PSY 369 Psycholinguistics Language Comprehension Word recognition speech

PSY 369: Psycholinguistics Language Comprehension Word recognition & speech recognition

Conceptualizer Thought Semantic Analysis Grammatical Encoding Lexicon Phonological Encoding Lexical Access Articulator Syntactic Analysis Word Recognition Letter/phoneme Recognition Comprehension Production Formulator

Lexical access n How do we retrieve the linguistic information from Long-term memory? n n n How is the information organized/stored? What factors are involved in retrieving information from the lexicon? Models of lexical access

Lexical access n How do we retrieve the linguistic information from Long-term memory? n n n How is the information organized/stored? What factors are involved in retrieving information from the lexicon? Models of lexical access

Lexical access Factors affecting lexical access Morphological structure n Role of prior context n Lexical ambiguity n Phonological structure n Concretness/abstractne n Grammatical class ss n Imageability n Frequency Some ofpriming these may reflect the structure of nn Semantic n n the lexicon Some may reflect the processes of access from the lexicon

Role of prior context n Swinney (1979): Cross Modal Priming Task: Listen to short paragraph. At some point during the paragraph a string of letters will appear on the screen. Decide if it is an English word or not. Say ‘yes’ or ‘no’ as quickly as you can.

Role of prior context n Swinney (1979) “Rumor had it that, for years, the government building has been plagued with problems. The man was not surprised bugsand in the corner of his when he found several spiders, roaches other room. ” ant Y Lexical decision: button press N

Role of prior context n Swinney (1979) “Rumor had it that, for years, the government building has been plagued with problems. The man was not surprised bugsand in the corner of his when he found several spiders, roaches other room. ” n Context related: Context inappropriate: Context unrelated: n 400 msec Lexical Decision task ant spy sew faster 700 msec faster No difference Results and conclusions n n Within 400 msecs of hearing "bugs", both ant and spy are primed After 700 msecs, only ant is primed

Lexical ambiguity n Hogaboam and Pefetti (1975) n n Words can have multiple interpretations The role of frequency of meaning n Task, is the last word ambiguous? n The jealous husband read the letter (dominant meaning) n The antique typewriter was missing a letter (subordinate meaning) n Results: Participants are faster on the second sentence. n The results may seem counterintuitive n n The task is the key, “is the final word ambiguous” In the first sentence, the meaning is dominant and the context strongly biases that meaning. So the second meaning may not be around, which in turn makes the it harder to make the ambiguity judgment in the first sentence

Lexical access n How do we retrieve the linguistic information from Long-term memory? n n n How is the information organized/stored? What factors are involved in retrieving information from the lexicon? Models of lexical access

Lexical access n How do we retrieve the linguistic information from Long-term memory? n n n How is the information organized/stored? What factors are involved in retrieving information from the lexicon? Models of lexical access Conceptualizer Thought Semantic Analysis Formulator Grammatical Encoding Lexicon Phonological Encoding Lexical Access Articulator Syntactic Analysis Word Recognition Letter/phoneme Recognition

(some) Models n Serial comparison models n n Search model (Forster, 1976, 1979, 1987, 1989) Parallel comparison models n n n of lexical access Logogen model (Morton, 1969) Cohort model (Marslen-Wilson, 1987, 1990) Connectionist models n Interactive Activation Model (Mc. Clelland Rumelhart, 1981) Important contrasting features: (1) interactive or autonomous, (2) serial or parallel, (3) cascaded or discrete

Search model (e. g. , Forster, 1976) n Access of the lexicon is considered autonomous, independent of other systems involved in processing language n n A complete perceptual representation of the perceived stimulus is constructed The representation is compared with representations in access files n Three access files: n n Orthographic Phonological Syntactic/semantic (for language production) Access files are organized in a series of bins (first syllable or letters) n n Position within the bins is organized by lexical frequency Access files have “pointers” to meaning information in semantic memory

Search model (e. g. , Forster, 1976) Visual input Pointers Decreasing frequency Entries in order of Access codes Auditory input /kat/ cat Mental lexicon mat cat mouse

Search model (e. g. , Forster, 1976) Search model (Forster, 1976, 1979, 1987, 1989, 1994) n n n Frequency effects n Bin organization Access codes Repetition priming effects n Temporary reordering of bins in response to recent encounter Semantic priming effects n Accounted for by cross referencing in the lexicon Context effects Mental lexicon n Search is considered to be autonomous, un affected by context (so context effects are “post-access”) Decreasing frequency n Entries in order of n Visual input Auditory input Pointers /kat/ cat mat cat mouse

Logogen model (Morton 1969) n The lexical entry for each word comes with a logogen n Logogens specify word’s attributes n n e. g. , semantic, orthographic, phonological Activated in two ways n n n cat By sensory input By contextual information Access (recognition) when reach threshold n Different thresholds depending on different factors n n e. g. , frequency Access makes information associated with word available dog cap

Logogen model (Morton 1969) Context system Auditory stimuli Visual stimuli Auditory analysis Visual analysis Semantic Attributes Logogen system Available Responses Output buffer Responses cat dog cap

Think of a logogen as being like a ‘strength-o-meter’ at a fairground When the bell rings, the logogen has ‘fired’

‘cat’ [kæt] • What makes the logogen fire? – seeing/hearing the word • What happens once the logogen has fired? – access to lexical entry!

‘cat’ [kæt] • So how does this help us to explain the frequency effect? – High frequency words have a lower threshold for firing –e. g. , cat vs. cot ‘cot’ [kot] Low freq takes longer

‘doctor’ [doktə] • Spreading activation from doctor lowers the threshold for nurse to fire – So nurse take less time to fire doctor ‘nurse’ [nə: s] Spreading activation network doctor nurse

Also goes by the name: Interactive Activation and Competition Model (IAC) Interactive Activation Model (IAM) Proposed to account for Word Superiority effect Mc. Clelland Rumelhart, (1981)

The Word-Superiority Effect (Reicher, 1969) + Until the participant hits some start key

The Word-Superiority Effect (Reicher, 1969) COURSE Presented briefly … say 25 ms

The Word-Superiority Effect (Reicher, 1969) U &&&&& A Mask presented with alternatives above and below the target letter … participants must pick one as the letter they believe was presented in that position.

The Word-Superiority Effect (Reicher, 1969) + + E KLANE E & T Letter only Say 60% + PLANE E &&&&& T Letter in Nonword Say 65% Letter in Word Say 80% Why is identification better when a letter is presented in a word?

Also goes by the name: Interactive Activation and Competition Model (IAC) Interactive Activation Model (IAM) Previous models posed a bottom-up flow of information (from features to letters to words). IAM also poses a topdown flow of information Nodes: • (visual) feature Mc. Clelland Rumelhart, (1981) • (positional) letter • word detectors • Inhibitory and excitatory connections between them.

Interactive Activation Model (IAM) n Inhibitory connections within levels n n If the first letter of a word is “a”, it isn’t “b” or “c” or … Inhibitory and excitatory connections between levels (bottom-up and top-down) n n If the first letter is “a” the word could be “apple” or “ant” or …. , but not “book” or “church” or…… If there is growing evidence that the word is “apple” that evidence confirms that the first letter is “a”, and not “b”…. . Mc. Clelland Rumelhart, (1981)

IAM & the word superiority effect n The model processes at the word and letter levels simultaneously n n Letters in words benefit from bottom-up and top-down activation But letters alone receive only bottom-up activation. Mc. Clelland Rumelhart, (1981)

Cohort model (Marslen-Wilson & Welch, 1978) n Specifically for auditory word recognition (covered in chapter 9 of textbook) n Speakers can recognize a word very rapidly n n Recognition point (uniqueness point) n n Usually within 200 -250 msec Point at which a word is unambiguously different from other words and can be recognized (strong emphasis on word onsets) Three stages of word recognition 1) activate a set of possible candidates 2) narrow the search to one candidate 3) integrate single candidate into semantic and syntactic context

Cohort model n Prior context: “I took the car for a …” /s/ /sp/ … soap spinach psychologist spin spit sun spank … spinach spin spit spank … time /spi/ spinach spin spit … /spin/ spin

Comparing the models n Each model can account for major findings (e. g. , frequency, semantic priming, context), but they do so in different ways. n n Search model is serial, bottom-up, and autonomous Logogen is parallel and interactive (information flows up and down) AIM is both bottom-up and top-down, uses facilitation and inhibition Cohort is bottom-up but parallel initially, but then interactive at a later stage

Different signals Visual word recognition Speech Perception Where are you going n n Some parallel input Orthography n n n Letters Clear delineation Difficult to learn n n Serial input Phonetics/Phonology n n n Acoustic features Usually no delineation “Easy” to learn

Different signals Visual word recognition Speech Perception Where are you going n n Some parallel input Orthography n n n Letters Clear delineation Difficult to learn n n Serial input Phonetics/Phonology n n n Acoustic features Usually no delineation “Easy” to learn

Speech perception n Articulatory phonetics n Production based n n Place and manner of articulation Acoustic phonetics n Based on the acoustic signal n Formants, transitions, co-articulation, etc.

Speech production to perception n Acoustic cues are extracted and stored in sensory memory and then mapped onto linguistic information n Air is pushed into the larynx across the vocal cords and into the mouth nose, different types of sounds are produced. n n The different qualities of the sounds are represented in formants The formants and other features are mapped onto phonemes

Acoustic features n Spectrogram n n Frequency n Time on the x-axis Frequency (pressure under which the air is pushed) on the y-axis Amplitude is represented by the darkness of the lines time

Acoustic features n Formants - bands of resonant frequencies n n Formant transitions - up or down movement of formants Steady states - flat formant patterns Bursts - sudden release of air Voice onset time (VOT) - when the voicing begins relative to the onset of the phoneme

Formants in a wide-band spectrogram <-- Formant transitions --> <-- F 3 <-- F 2 <-- F 1 n Formants - bands of resonant frequencies n n Formant transitions - up or down movement of formants Steady states - flat formant patterns

Formants in a wide-band spectrogram Burst --> <-- Formant transitions --> <-- F 1 n Bursts – sudden release of air <-- F 3 <-- F 2

Voice-Onset Time (VOT) bit 5 ms pit 40 ms

Categorical Perception n n Categorical Perception is the perception of different sensory phenomena as being qualitatively, or categorically, different. Liberman et al (1957) n Used the speech synthesizer to create a series of syllables panning categories /b/, /d/, /g/ (followed by /a/) n n n Was done by manipulating the F 2 formant Stimuli formed a physical continuum Result, people didn’t “hear” a continuum, instead classified them into three categories

Categorical Perception n Liberman et al (1957) 1. Set up a continuum of sounds between two categories /ba/ 1 - . . . 3 … /da/ 5 … 7

Categorical Perception n Liberman et al (1957) 2. Run an identification experiment Sharp phoneme boundary 100 n % /ba/ Our perception of phonemes is categorical rather than continuous. 0 1 . . . 3 … 5 … 7

Hard Problems in Speech Perception Wave form n Linearity (parallel transmission): Acoustic features often spread themselves out over other sounds n Where does show start and money end? n Demo's and info

Hard Problems in Speech Perception Wave form n Invariance: n One phoneme should have a one waveform n n But, the /i/ (‘ee’) in ‘money’ and ‘me’ are different There aren’t invariant cues for phonetic segments n Although the search continues n Demo's and info

Hard Problems in Speech Perception Wave form n Co-articulation: the influence of the articulation (pronunciation) of one phoneme on that of another phoneme. n Essentially, producing more than one speech sound at once n Demo's and info

Hard Problems in Speech Perception n Trading relations n Most phonetic distinctions have more than one acoustic cue as a result of the particular articulatory gesture that gives the distinction. n n slit–split – the /p/ relies on silence and rising formant, different mixtures of these can result in the same perception Perception must establish some "trade-off" between the different cues.

Hard Problems in Speech Perception n The Mc. Gurk effect: Mc. Gurk and Mac. Donald (1976) • Showed people a video where the audio and the video don’t match • Think “dubbed movie” n n Mc. Gurk effect 2 • visual /ga/ with auditory /ba/ often hear /da/ n Implications • phoneme perception is an active process • influenced by both audio and visual information

Motor theory of speech perception n A. Liberman (and others, initially proposed in late 50’s) n n Direct translation of acoustic speech into articulatorally defined categories Holds that speech perception and motor control involved linked (or the same) neural processes n Theory held that categorical perception was a direct reflection of articulatory organization n Categories with discrete gestures (e. g. , consonants) will be perceived categorically Categories with continuous gestures (e. g. , vowels) will be perceived continuously There is a speech perception module that operates independently of general auditory perception

Frontal slices showing differential activation elicited during lip and tongue movements (Left), syllable articulation including [p] and [t] (Center), and listening to syllables including [p] and [t] (Right) Pulvermüller F et al. PNAS 2006; 103: 7865 -7870 © 2006 by National Academy of Sciences

Motor theory of speech perception n Some problems for MT n n Categorical perception found in non-speech sounds (e. g. , music) Categorical perception for speech sounds in non-humans n Chinchillas can be trained to show categorical perception of /t/ and /d/ consonant-vowel syllables (Kuhl & Miller, 1975)

Other theories of speech perception n Direct Realist Theory (C. Fowler and others) n n n Similar to Motor theory, articulation representations are key, but here they are directly perceived Perceiving speech is part of a more general perception of gestures that involves the motor system General Auditory Approach (e. g. , Diehl, Massaro) n Do not invoke special mechanisms for speech perception, instead rely on more general mechanisms of audition and perception n For nice reviews see: n Diehl, Lotto, & Holt (2003) n Galantucci, Fowler, Turvey (2006)

Top-down effects on Speech Perception n n Phoneme restoration effect Sentence context effects

Phoneme restoration effect Listen to a sentence which contained a word from which a phoneme was deleted and replaced with another noise (e. g. , a cough) The state governors met with their respective legi*latures convening in the capital city. * /s/ deleted and replaced with a cough Click here for a demo and additional information

Phoneme restoration effect Typical results: Participants heard the word normally, despite the missing phoneme Usually failed to identify which phoneme was missing Interpretation We can use top-down knowledge to “fill in” the missing information

Phoneme restoration effect Further experiments (Warren and Warren, 1970): What if the missing phoneme was ambiguous The *eel was on the axle. The *eel was on the shoe. The *eel was on the orange. The *eel was on the table. Results: Participants heard the contextually appropriate word normally, despite the missing phoneme

Phoneme restoration effect n Possible loci of phoneme restoration effects n Perceptual loci of effect: n n Lexical or sentential context influences the way in which the word is initially perceived. Post-perceptual loci of effect: n Lexical or sentential context influences decisions about the nature of the missing phoneme information.

Beyond the segment Shillcock (1990): hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming) Hear: The scientist made a new discovery last year. NUDIST

Cross-modal priming Shillcock (1990): hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming) Hear: The scientist made a novel discovery last year. NUDIST

Cross-modal priming Shillcock (1990): hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming) Hear: The scientist made a novel discovery last year. The scientist made a new discovery last year. faster

Cross-modal priming Shillcock (1990): hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming) Hear: The scientist made a novel discovery last year. The scientist made a new discovery last year. faster NUDIST gets primed by segmentation error Although no conscious report of hearing “nudist”

Beyond the segment n Prosody and intonation n English: n Speech is divided into phrases. n Word stress is meaningful in English. n Stressed syllables are aligned in a fairly regular rhythm, while unstressed syllables take very little time. n Every phrase has a focus. n An extended flat or low-rising intonation at the end of a phrase can indicate that a speaker intends to continue to speak. n A falling intonation sounds more final.

Beyond the segment n Prosodic factors (supra segmentals) n Stress n n Rate n n Emphasis on syllables in sentences Speed of articulation Intonation n Use of pitch to signify different meanings across sentences

Beyond the segment n Stress effects n On meaning n n “black bird” versus “blackbird” Top-down effects on perception n Better anticipation of upcoming segments when syllable is stressed

Beyond the segment n Rate effects n How fast you speak has an impact on the speech sounds n n Faster talking - shorter vowels, shorter VOT Normalization n Taking speed and speaker information into account n Rate normalization n Speaker normalization