Language and the brain Rajeev Raizada Dept of

Language and the brain Rajeev Raizada Dept. of Brain & Cognitive Sciences rajeev. raizada@gmail. com raizadalab. org

Language and the brain: why bother with brain stuff in the first place? Key language areas, and lesion deficits Lots of interactive brain areas • It’s not just a couple of areas on the left Interpreting brain activation: • Who cares which bit of the brain lights up? • We want brain imaging to tell us about linguistic information processing, or linguistic representations 2

Language areas in the brain Some brain areas are specialised for language • • Broca's area: speech production Wernicke's area: speech perception On the left side of the brain (in 95% of people) This is pretty much the only left-brain / right-brain saying that is actually true What does "specialised for language" actually mean? • • If you lose these areas, you lose language When you use language, you use those areas BUT: That does not mean that they only do language E. g. Broca's area may be involved in music perception

Broca's area: crucial for speech production Paul Broca (1861): patient "Tan” • Severe deficit in speech production: could only say “tan” • Good language comprehension Tan's brain: lesion (injury) in left frontal cortex

Auditory cortex and Wernicke's area Auditory cortex: all sounds pass into here • • Mostly specialised for low-level features, e. g. raw frequency Bilateral (on both left and right sides of the brain) Wernicke's area (Carl Wernicke, 1874) • • Patient with very poor speech comprehension Good speech production Lesion on left side, just behind auditory cortex Specialised for processing “higher level” sounds: speech

Auditory cortex and Wernicke's area From http: //www. physiology. wisc. edu/neuro 524/

Language areas in the brain From University of Washington's Digital Anatomist project

There's more to language in the brain than just Broca's and Wernicke's Friederici, A. D. (2012). The cortical language circuit: from auditory perception to sentence comprehension. Trends in cognitive sciences, 16(5), 262 -268. 8

The claim “language is on the left” is a total over-simplification Specht, K. (2013). Mapping a lateralization gradient within the ventral stream for auditory speech perception. Frontiers in human neuroscience, 7. 9

The claim “language is on the left” is a total over-simplification Peelle, J. E. (2012). The hemispheric lateralization of speech processing depends on what “speech” is: a hierarchical perspective. Frontiers in human neuroscience, 6. 10

Speech: Not what you might expect What makes speech sound the way it does? Sine-wave speech demo page: http: //www. lifesci. sussex. ac. uk/home/Chris_Da rwin/SWS/ How on earth can these weird whistles sound like spoken words? 11

Spectrograms and formants: Sound frequencies along time • • Formant = Peak in speech sound's frequency spectrum F 0 = fundamental freq, F 1 = 1 st peak, F 2 = 2 nd peak, etc. Mc. Gettigan, C. , & Scott, S. K. (2012). Cortical asymmetries in speech perception: what's wrong, what's right and what's left? . Trends in cognitive sciences, 16(5), 269 -276. 12

Spectrograms and formants: Sound frequencies along time • • Formant = Peak in speech sound's frequency spectrum F 0 = fundamental freq, F 1 = 1 st peak, F 2 = 2 nd peak, etc. Mc. Gettigan, C. , & Scott, S. K. (2012). Cortical asymmetries in speech perception: what's wrong, what's right and what's left? . Trends in cognitive sciences, 16(5), 269 -276. 13

Spectrograms and formants: Sound frequencies along time • • Formant = Peak in speech sound's frequency spectrum F 0 = fundamental freq, F 1 = 1 st peak, F 2 = 2 nd peak, etc. Mc. Gettigan, C. , & Scott, S. K. (2012). Cortical asymmetries in speech perception: what's wrong, what's right and what's left? . Trends in cognitive sciences, 16(5), 269 -276. 14

Place of articulation Mc. Gettigan, C. , & Scott, S. K. (2012). Cortical asymmetries in speech perception: what's wrong, what's right and what's left? . Trends in cognitive sciences, 16(5), 269 -276. 15

Some example formant transitions http: //www 2. psychology. uiowa. edu/faculty/mcmurray/speechglossary/ 16

Categorical perception: Not all acoustic differences make a difference as speech (Alvin Liberman & colleagues) • Stimuli spread evenly along the /ba/-/da/ continuum • Pure /ba/=1, pure /da/=10, perceptual boundary ~ 5 • Presented as pairs, constant 3 -step distance apart acoustically • 1 -4 and 7 -10 do not cross Raj boundary: perceived as same eev • 3 -6, 4 -7 do cross boundary, Raiz ada perceived as different SFN

Categorical perception and same / different judgments Acoustic difference within each pair is constant (1 -4, 4 -7, 9 -6 etc. ) Phonetic difference depends on whether category Raj eev boundary is crossed Raiz ada SFN

Spectrograms /ba/ /da/ Raj eev Raiz ada SFN

Vowels in formant space http: //www. phys. unsw. edu. au/jw/voice. html 20

Different languages carve up acoustic space differently: /ra/ and /la/ in English and Japanese speakers Raizada et al. , Cerebral Cortex (2010) 21

Formants and perceptual discriminability Raizada et al. , Cerebral Cortex (2010)

Some patterns are more separable than others, given a particular set of measurements Sumo wrestlers Basketball Students Weight Faculty players Height Wanted: • One single task, one set of stimuli • BUT: Some people can do task, other people cannot 23

Prediction: f. MRI patterns for /ra/ and /la/ are more separable in the brains of English speakers than in Japanese speakers English speakers: Can perceive /r/-l/ distinction f. MRI patterns are separable /ra/ /ra/ /la/ /ra/ /la/ Japanese speakers: Cannot perceive /r/-l/ distinction f. MRI patterns are not separable /la/ /ra/ /la/ /ra/ /la/ /ra/ /la/ 24

Will neural pattern separability match perceptual discriminability? Predicted pattern-separability, if it matches perception: English: F 3 -difference > F 2 -difference Japanese: F 3 -difference = F 2 -difference f. MRI pattern separability contrast: • F 3 -separability minus F 2 -separability Is this neural difference greater for English than Japanese? Key point: the classifier doesn't get told anything about people's behaviour, or about who is English or Japanese 25

Individual differences: Neural pattern separability predicts behavioural performance Raizada et al. , Cerebral Cortex (2010) Correlation after partialling out effects of group membership: r = 0. 389, p < 0. 05 2

Same speech sound, very different formant patterns http: //www 2. psychology. uiowa. edu/faculty/mcmurray/speechglossary/ 27

How do we pull out the “invariant” sound? An interesting (but misguided? ) theory: The motor theory of speech perception (Liberman et al. ) You don't actually pull out an invariant sound You somehow perceive the motor articulation which made the sound 28

An analogy: The skull theory of face recognition Google Images search for “Jon Stewart”

If motor theory is true, then motor-cortex damage should impair speech perception But that turns out not to be the case Stasenko, A. , Garcea, F. E. , & Mahon, B. Z. (2013). What happens to the motor theory of perception when the motor system is damaged? . Language and Cognition, 5(2 -3), 225 -238. Nonetheless, motor cortex is often active during language tasks. What is it doing? 30

Meaning 31

Embodied theory of meaning Words are represented in terms of bodily sense modalities: vision, hearing, movement, etc. • • • Barsalou, L. W. (2008). Grounded cognition. Annu. Rev. Psychol. , 59, 617 -645. Pulvermüller, F. (2013). How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics. Trends in cognitive sciences, 17(9), 458 -470. Binder, J. R. , & Desai, R. H. (2011). The neurobiology of semantic memory. Trends in cognitive sciences, 15(11), 527 -536. 32

Embodied theory of meaning • Binder, J. R. , & Desai, R. H. (2011). The neurobiology of semantic memory. Trends in cognitive sciences, 15(11), 527 -536. 33

Embodied theory of meaning • Pulvermüller, F. (2013). How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics. Trends in cognitive sciences, 17(9), 458 -470. 34

Example: somatotopic representation of motor words 35

Example: somatotopic representation of motor words Pulvermüller, F. , Trends in Cog Sci (2013). 36

Embodied-looking activation shows up even using corpus statistics “Gustatory cortex” for celery in Mitchell et al. (2008) Mouth / toungue areas 37

What about abstract words? Pulvermüller, F. , Trends in Cog Sci (2013). 38

Higher level “abstraction” areas? Pulvermüller, F. , Trends in Cog Sci (2013). 39

Decoding word meanings from the brain 40

Decoding as a test of whether we really understand anything about the encoding How does the brain represent the meanings of words? • Test: see if we can neurally decode the words 41

Video from CBS 60 Minutes https: //www. youtube. com/watch? v=8 jc 8 URRx. PIg 42

Semantic similarity: A quick demo to show that your brain cares about it You will be presented with a list of words. Try to remember as many as you can. sour candy chocolate heart cake pie tart soda honey nice tooth taste good bitter sugar Were any of the following words in that list? soft short sweet smooth Deese-Roediger-Mc. Dermott effect 43

People's attempts so far to neurally decode word-meanings Important contribution: Tom Mitchell et al. , Science, 2008 f. MRI data and semantic features publicly available at http: //www. cs. cmu. edu/~tom/science 2008 44

Modeling a continuous space Standard decoding: Neural responses for Stim A Neural responses for Stim B Present some test-set neural data: Q: Was it elicited by A or by B? Problem: What if it is a new stimulus, C? 45

Interpolating between stimuli, using a model of the stimulus space Pattern-information analysis: from stimulus decoding to computational-model testing. Kriegeskorte N. Neuroimage. 2011 May 15; 56(2): 411 -21. 46

Stimuli Snodgrass, J. G. , & Vanderwart, M. (1980). A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of experimental psychology: Human learning and memory, 6(2), 174. 47

The 60 words used as stimuli 48

The Mitchell study: word stimuli and semantic features Stimuli: concrete nouns • E. g. hammer, shirt, dog, celery (60 words in all) • 12 categories (tools, clothing, food, etc. ), each with 5 words Semantic features: action verbs • E. g. push, move, taste, see (25 semantic features in all) • Each noun has a 25 -element semantic feature vector of its co-occurrence freqs with the verbs, from Google textcorpus • hammer = 0. 13*break + 0. 93*touch + 0. 01*eat + … • celery = 0. 00*break + 0. 03*touch + 0. 84*eat + … 49

Example co-occurrence features Features for cat: say said says (0. 592), sees (0. 449), eat ate eats (0. 435), run ran runs (0. 303), hears heard (0. 208), opens opened (0. 175), smells smelled (0. 163), cleaned cleans (0. 146), moved moves (0. 088), listens listened (0. 075), touched touches (0. 075) … http: //www. cs. cmu. edu/~tom/science 2008/semantic. Feature. Vectors. html 50

Mitchell et al. (2008) What do the semantic features look like in the brain? 51

Predicting brain activation 52

The Mitchell study: training and testing the model Training (carried out separately in each subject) • Stage 1: represent each noun as weighted sum of semantic features • Stage 2: learn a linear mapping between those semantic features onto the word-elicited f. MRI patterns in each person's brain. Testing: can the model predict neural activation for untrained words? 53

The Mitchell study: decoding results “Leave two out” testing strategy • Remove each pair of words in turn from 60 -word set • Train the model on the remaining 58 words • Predict neural patterns for the two test-words • Match those predictions against the test-words' actual elicited activations Success rate for decoding left-out word-pairs: 77% • Chance-level is 50% Tried making a hand-crafted feature-set: 80. 9% 54

Summary The brain is astonishingly good at processing language • Nobody understands how it achieves this • But we do have some exciting leads Lots of brain areas, all representing multiple types of information, all communicating with each other • Not just Broca’s and Wernicke’s areas • Not just in the left hemisphere Challenges for neuroscience • What information processing tricks does the brain use? • What representations does it use, how does it use them? 55