LANGUAGE COGNITION AND MUSICAL EMOTIONS University of Birmingham

LANGUAGE, COGNITION, AND MUSICAL EMOTIONS University of Birmingham School of Computer Science 23 Nov 2010 Leonid Perlovsky Harvard University and the AF Research Lab http: //leonid-perlovsky. com/

OUTLINE • Mathematical difficulties – Computational Complexity • Dynamic Logic (DL) – Examples: recognition, situations – The Knowledge Instinct (KI) – Model of PSS • Future research – – The mind hierarchy, emotion of the beautiful Language and Cognition Musical emotions: cognitive function, origin Experimental tests: psychological, computer science

MATHEMATICAL DIFFICULTIES: COMPUTATIONAL COMPLEXITY • Perception, cognition, language involve evaluating large numbers of combinations – Pixels -> objects -> scenes -> abstract concepts – Sounds -> words -> phrases -> … • Combinatorial Complexity (CC) – A general problem (since the 1950 s) • Detection, recognition, tracking… language… • Pattern recognition, neural networks, rule systems… • Combinations of 100 elements are 100100 – This number > the size of the Universe • > all the events in the Universe during its entire life • Related to Gödelian limitations of logic

DYNAMIC LOGIC (DL) • DL unifies formal and fuzzy logic – A process-logic “from fuzzy to crisp” • Maximizes similarity between models and signals • Overcomes CC -> fast algorithms • Proven in neuroimaging experiments (Bar, 2006) – Initial representations-memories are vague-fuzzy – “close-eyes” experiment

EXAMPLE: RECOGNITION DL “from vague to crisp” a b c d e f g h Signal / Clutter ratio ~ 100 times improvement

ARISTOTLE VS. GÖDEL • Aristotle – Logic: a supreme way of argument – Forms: representations in the mind Ø Form-as-potentiality evolves into form-as-actuality Ø Potentialities are not logical -> logical actualities, (Dynamic Logic) – Language and thinking are closely linked • From Boole to Russell: formalization of logic – Logicians eliminated from logic uncertainty of language – Hilbert: formalize rules of mathematical proofs forever • Gödel (the 1930 s) – Logic is not consistent Ø Any statement can be proved true and false • Aristotle and Alexander the Great

objects EXAMPLE: Learning Situations DATA (SORTED) Situations (sorted)

objects SITUATIONS: DATA (RANDOM) Situations (random)

SITUATION LEARNING: ERRORS

THE MIND, KNOWLEDGE INSTINCT, AND DL • Mechanisms of the mind: – Instincts are like sensors measuring vital signals – Emotions are neural signals connecting instinctual and decisionmaking mechanisms – Concepts are mental representations, models – Behavior – Hierarchy • The knowledge instinct (KI) – – – Concept-models always have to be adapted KI increases similarity between models and the world DL is mathematically equivalent to KI KI-emotions: (dis)harmony between concepts and the world KI-emotions are aesthetic emotions Ø In every act of perception and cognition

DL and Perceptual Symbol System (PSS) • PSS – Mental representations are distributed bits and pieces of experiences – Simulations are the reenactment of perceptual experiences – Simulations reassemble perceptual experiences – Simulations are the mechanisms of perception and cognition, symbol-processes • DL – DL process “from vague-to-crisp” models PSS simulations – DL is a general mechanism of interacting bottom-up and topdown signals – Applicable to all levels of the cognitive processing hierarchy – How amodal signals emerge in PSS (final crisp state)

OUTLINE • Mathematical difficulties – Computational Complexity • Dynamic Logic (DL) – Examples: recognition, situations – The Knowledge Instinct (KI) – Model of PSS • Future research – – The mind hierarchy, emotion of the beautiful Language and Cognition Musical emotions: cognitive function, origin Experimental tests: psychological, computer science

HIERARCHY OF COGNITION … abstract ideas Top representations SURROUNDING LANGUAGE Purpose: unify entire life experience LANGUAGE aesthetic emotions are felt as beautiful Conceptual abstractand emotional contents are not differentiated words/phrases … … language descriptions Vague and unconscious of abstract representations thoughts situations Purpose: unify lower level phrases representations aesthetic emotions are conscious phrases for situations objects Vague and less conscious representations words aesthetic emotions could be conscious words for objects Sensory-motor signals Concrete conscious perceptions language Sensory-motor aesthetic emotions are below sounds language consciousness models

INTEGRATED LANGUAGE AND COGNITION • How language and cognition interact – Words and objects: zillions of combinations, how do we learn correct ones? – Each concept (mental representation) has linguistic and cognitive dual model Mm = { Mmcognitive, Mmlanguage }; – Language and cognition are fused at vague pre-conceptual level • before words and concepts are learned • Arbib: “language prewired brain” • Language and cognition mechanisms – – In a newborn mind, concept-models are dual vague blobs Language is learned “ready-made” from surrounding language Language models guide development of cognitive model (situations) Cognitive concepts are learned to match language models

INTEGRATED HIERARCHIES • High level cognition is only possible due to language COGNITION … abstract ideas situations LANGUAGE … abstract words/phrases SURROUNDING LANGUAGE … language descriptions of abstract thoughts phrases for situations objects words for objects Sensory-motor signals Sensory-motor language models language sounds

RECOGNITION • 2007 Gabor Award - The top engineering award from International Neural Network Society (INNS) • 2007 John L. Mc. Lucas Award - The top scientific award from the US Air Force • 2000 Best Paper Award, Zvezda • Elected to the Board of Governors of INNS • On Editorial Boards of 6 journals

BIBLE, NOBEL PRIZE, LANGUAGE • Why Adam was expelled from paradise ? – Did not want to think (cognitive models are vague) – Choose ready-made language rules (Maimonides, 13 th c. ) • Nobel Prize 2002, Kahneman (and Tversky) – Decisions are basically irrational – Not KI, but rules-heuristics (crisp language models) – KI-cortex (OFC) vs. rules-amygdala (DL & LP) • Language vs. irrational-rules – Language contains wealth of cultural knowledge, rules – Remember: opened eyes hide vague mental images – Similarly language hide vague abstract concepts

MUSIC IS A MYSTERY • Aristotle, Kant, Darwin - “must be ranked amongst the most mysterious (abilities) with which (man) is endowed” • Contemporary psychologists – Pinker, Justus, Hustler, Mc. Dermott, Houser, Trainor, Fitch, Huron, Cross, Mithen, Juslin, a series of reviews in Nature… – Dissanayake: “aberrance of modernity…” – Masataka: “Music is a human cultural universal that serves no obvious adaptive purpose, making its evolution a puzzle for evolutionary biologists” 16 -Sep-05 18

THEORETICAL HYPOTHESIS • Language – split concepts and emotions – accelerated differentiation of knowledge • Diverse knowledge induces contradictions in psyche - Cognitive dissonances - Need to connect language to cognition • Music restores unity of psyche - Along with diversity of knowledge • A number of contradictions is very large - Two pieces of knowledge contain a contradiction, cognitive dissonance - Combinations of knowledge pieces are very large, this is the reason for “infinity” of musical emotions • Music creates the diversity of emotions necessary to continue culture 16 -Sep-05 19

ORIGIN • Animals - Unity of psyche • Human voice separated into language & music - Language reduced emotionality and increased semantics - Violated unity of psyche - Diverse knowledge brings advantage if combined with unity of psyche - To restore unity, part of voice became more emotional, less semantic • Music - Evolved together with language to keep synthesis of differentiated psyche - “It is difficult to keep the scissor blades together” 16 -Sep-05 20

EVOLUTION • Undifferentiated proto language-music • Motherese • Earliest musical instruments 9, 000 years old (may be older) • Every major cultural-spiritual advantage required new music – Jaynes J. , The origin of consciousness, 1976 – Weiss P, Taruskin R. Music in the western world, 1984 – Cultural process is the combination of oppositions, differentiation and synthesis, music is needed to keep psyche together • Few examples – Contemporary consciousness (4 -5 BCE) Thales, Zechariah, Confucius – antiphony – Human emotions - Renaissance (13 -16) – tonality – The highest contradiction - Reformation (16) – Baroque, fugue, Bach – Rationality – Enlightenment (17 -18) – Classicism (Bach sons, Mozart) – Self – (19) – Romanticism (Beethoven, Chopin, Tchaikovsky) – Unconscious+objective – (20) all mixed-up, Modern, Postmodern – Pop songs – restore unity of everyday consciousness – Rap – a contemporary dithyramb, connects conscious and unconscious 16 -Sep-05 21

EXPERIMENTAL TESTS • There is no words for musical emotions, so how can we measure them? What to measure? • Psychologists – (1)Measure subjective emotional similarities (or distances) among musical phrases – Mathematically evaluate dimensionality of the emotional space – (2)Measure emotional similarities among musical phrases and cognitive dissonances – (3)Measure f. MRI or EEG signatures of musical phrases • Computer scientists – (1)Collect context (50 words) around emotional words on Google – Measure similarities (word overlap) between emotional words – Mathematically evaluate dimensionality of the emotional space – (2)Repeat for cognitive dissonances (look for word “choice”) 16 -Sep-05 22

PUBLICATIONS 380 publications 3 books OXFORD UNIVERSITY PRESS (2001; 3 rd printing) Neurodynamics of High Cognitive Functions with Prof. Kozma, Springer, 2007 Sapient Systems with Prof. Mayorga, Springer, 2007 2011: Dynamic Logic, Springer

FUTURE DIRECTIONS research, predictions and testing • Mathematical development KI in the hierarchy, combine with language and emotions Multi-agent simulations of cultural evolution • Psycholinguistic experiments Measure emotionality of various languages in labs • Music: theoretical and experimental Direct effect on emotions Concurrent evolution of music, consciousness, and cultures Multi-agent simulations of cultural evolution • Brain imaging Brain regions used by languages, music, in different cultures Neural mechanisms connecting language and cognition • Semantic Web and Cyberspace Adaptive ontologies Learn from human users, acquire cultural knowledge Enable culturally-sensitive communication Help us understand each other and ourselves Multi-agent simulations of cultural evolution • Improve human condition and understanding around the globe Develop predictive cultural models, integrate spiritual and material causes Identify language and music effects that can advance consciousness and reduce tensions 24

BACK-UP • Structure of the mind • Neural Modeling Fields • Dynamic logic • Neuro-imaging experimental confirmation • Beautiful and sublime • Cultural models 16 -Sep-05 25

STRUCTURE OF THE MIND • Concepts – Models of objects, their relations, and situations – Evolved to satisfy instincts • Instincts – Internal sensors (e. g. sugar level in blood) • Emotions – Neural signals connecting instincts and concepts • e. g. a hungry person sees food all around • Behavior – Models of goals (desires) and muscle-movement… • Hierarchy – Concept-models and behavior-models are organized in a “loose” hierarchy

NEURAL MODELING FIELDS from signals to concepts • Bottom-up signals – Pixels or samples (from sensor or retina) x(n), n = 1, …, N • Top-down concept-models Mm(Sm, n), parameters Sm, m = 1, …; – Models predict expected signals from objects • The knowledge instinct = maximize similarity between signals and models L = l ({x}) = l (x(n) | Mm) – MN items: all associations of pixels and models (=>CC) – New mathematical technique, DL, overcame this difficulty

DYNAMIC LOGIC (DL) non-combinatorial max of knowledge • Start with a set of signals and unknown objectmodels – any parameter values Sm – associate models with signals (vague) – (1) f(m|n) = r(m) l (n|m) / r(m') l (n|m') • Improve parameter estimation – (2) Sm = Sm + a f(m|n) [ ln l (n|m)/ Mm]*[ Mm/ Sm] • Continue iterations (1)-(2). Theorem: MF is a converging system (from vague to crisp) - similarity increases on each iteration - aesthetic emotion is positive during learning

DL AND BRAIN IMAGING • Neuro-imaging experiments proved that the brain works as predicted by dynamic logic • Bar et al (2006), Harvard University proved - Bottom-up signals (from eye retina) interact with top-down signals (from memory-models) - Initial top-down signals are vague - These interactions are unconscious • Barsalou et al (2006), Emory University proved - Distributed vague representations in the mind 16 -Sep-05 29

BEAUTIFUL AND SUBLIME • The highest aesthetic emotion, beautiful – improvement of the highest models (at the top of the hierarchy) – feel emotion of beautiful • Beautiful “reminds” us of our purposiveness – the “top” model unifies all our knowledge – vague – we perceive it as our purpose (“aimless purposiveness”) • Beauty is separate from sex – sex uses all our abilities, including beauty • Religiously sublime is related to behavior

MODELS OF CULTURAL EVOLUTION • Differentiation, D, synthesis, S, hierarchy, H d. D/dt = a D G(S); d. S/dt = -b. D + d. H H = H 0 + e*t G(S) = (S - S 0) exp(-(S-S 0) / S 1)

INTERACTING CULTURES • Two cultures – dynamic and traditional – slow exchange by D and S d. Dk/dt = ak Dk G(Sk) + xk. Dk d. Sk/dt = -bk. Dk + dk. Hk + yk. Sk Hk = H 0 k + ek*t

CULTURE AND LANGUAGE • Culture is transmitted through language • Examine mechanisms of – Language and cognition – Language and emotion • The following are hypotheses in need of verification

EVOLUTION OF CULTURES • The knowledge instinct - Two mechanisms: differentiation and synthesis • Differentiation - At every level of the hierarchy: more detailed concepts - Separates concepts from emotions • Synthesis - Connects concepts and emotions (knowledge and life) Ø Connects language and cognition Ø Created in the hierarchy: concepts acquire meaning at the next level • Evolutionary dynamics - Complex interaction of opposing mechanisms 16 -Sep-05 34

EMOTIONAL SAPIR-WHORF HYPOTHESIS • Language affects thinking and behavior – Bhartrihari 5 th CE (India), Humboldt 1836, Nietzsche 1876 – Benjamin Whorf and Edward Sapir in the 1930 s Ø E. g. , people better perceive colors, which have words in their language • Recent history – “Out of favor. ” Chomsky separated language and cognition – Recent resurgence of interest • We have to understand cultural differences – “European” thinking is not the only way • Emotional differences are no less important than semantical

EMOTIONS IN INTEGRATED HIERARCHIES • Look top-down: differentiation, more detailed concept-models, less emotions • Look bottom-up: synthesis, unifying general models, more emotions • At the top: meaning and purpose, emotions of the beautiful and sublime Meanings vague emotional COGNITION … SURROUNDING LANGUAGE … abstract ideas situations … abstract words/phrases language descriptions of abstract thoughts Objects crisp conceptual hierarchy: conceptual phrases for situations objects throughout Synthesis emotions words crisp low words for objects Sensory-motor signals Sensory-motor language models language sounds emotions

EMOTIONS IN LANGUAGES • Animals - Undifferentiated concepts-emotions-behaviors-vocalization - Vocal tract is controlled from ancient emotional limbic system • Human language evolution - Language evolved toward semantics and less emotions Still emotions are needed, otherwise, no synthesis, no meaning Two emotional centers: limbic (involuntary) and cortex (conscious) Emotionality: in voice sound (melody of speech) • Emotional differences among languages - All languages evolved toward less emotionality More semantic flexibility, but potential to lose meanings “Too fast” evolution => lose meaning “Too slow” evolution => culture stagnates Speed is determined by grammar, by inflections (Humboldt’s firmness) 16 -Sep-05 37

LANGUAGE EMOTIONS AND CULTURES • Conceptual content of culture: words, phrases Easily borrowed among cultures • Emotional content of culture In voice sound (melody of speech) Determined by grammar Cannot be borrowed among cultures • English language (Diff. > Synthesis) Weak connection between conceptual and emotional (since 15 c) Pragmatic, high culture, but may lead to crises (lost meaning) • Arabic language (Synthesis > Diff. ) Strong connection between conceptual and emotional Cultural immobility, but strong feel of identity and purpose 16 -Sep-05 38

DYNAMIC CULTURE Average synthesis, high differentiation; oscillating solution Knowledge accumulates; no stability

TRADITIONAL CULTURE High synthesis, low differentiation; stable solution Stagnation, stability increases

INTERACTING CULTURES 1) Early: Dynamic culture affects traditional culture, no reciprocity 2) Later: 2 dynamic cultures stabilize each other Knowledge accumulation + stability