Auditoryvisual speech perception in young children language and

Auditory-visual speech perception in young children: language and agespecific factors Doğu Erdener @ 6 December 2007

Development of Speech Perception q First 6 months § phonetically-based, language-general q Beyond 1 st year phonologically-based, language-specific § Consonants 7 -11 months (Best, 1995) § Vowels 4 -6 months (Kuhl et al. , 1992) § Tones 6 -9 months (Burnham & Mattock, 2006) q A working heuristic (Burnham et al. , 2002) § Acoustic / Phonetic Stage (0 -6 months) § Phonological Stage (6 -12 months) § Semantic Stage (12 -24 months) § Orthographic Stage (6 -8 years)

Auditory-Visual Speech Perception (AVSP): Integration q Amodal (e. g. , Schwartz, et al. , 1997) Pre-categorical q Phonetic (e. g. , Burnham & Dodd, 2004) q Phonological / Postcategorical (e. g. , Massaro, 1998) q none/some/all of the above? Recall that auditory-only speech perception …. q in early infancy (till ~6 months) is phonetically based q beyond first year of life phonological AVSP integration research: We need differential and ontogentic data!

AVSP: Cross-Language (Differential) Studies q Cross-language phonemic differences § e. g. , Werker et al. (1992) Visual influence from /th/ in the Mc. Gurk stimulus [A]-/ba/ + [V]-/tha/ : • anglophones > francophones • more experienced francophones > less experienced francophones q Amount of visual speech influence across langauges: § American > Japanese (Sekiyama & Tohkura, 1993) § Japanese > Mandarin (Sekiyama, 1997) q Sekiyama studies Possible reasons: § Cross-cultural studies: Less eye/face contact in some cultures + Japanese Ss tend to look at the eye area whereas American Ss on the mouth area (Yuki et al. , 2007). § Linguistic 1: Tonal (e. g. Cantonese) and pitch-accent (e. g. , Japanese) information are not visually salient. § Linguistic 2: Fewer visually distinct consonant clusters and vowels in Japanese than in English.

Development of Auditory-Visual Speech Perception q Infancy § Infants match auditory and visual speech information (Burnham & Dodd, 2004, Rosenblum et al. , 1997) q Childhood § Mc. Gurk & Mac. Donald (1976) 3 to 8 years + adults § Massaro et al. (1986) 4 -6 years + adults q Adulthood § AV integration adults > children (Sekiyama & Burnham, in press; Massaro et al. , 1986) § Late adulthood older Ss > younger Ss on vısual speech information when auditory sensitivity is diminished (Cienkowski & Carney, 2002).

Focus: Developmental Data from Infants and Children q Why is there an age-related increase? q Infancy and childhood are marked with several novel language-relevant and language-specific challenges, such as § phonetic phonological § semantic vocabulary § reading

Sekiyama & Burnham (in press): an ontogenetic + differential study q Japanese and Australian English Speakers q Three child (6 -, 8 -, and 11 -year-olds) groups and a control group of adults. q Method § Mc. Gurk stimuli § DV: Visual Speech Index (VSI) Score based on the proportion of auditory-based responses § Identification Task: /ba/, /da/, or /ga/?

6 -yos AV+ minus A Augmentation 8 -yos A minus AVInterference Visual Speech Index (VSI) 11 -yos Adults [AV+] minus [AV-]

q Sekiyama and Burnham (in press) found: § Developmental increase in visual speech influence with age… • In English speakers but • Not in Japanese speakers q So…. . What factors modulate the development of auditory-visual speech perception (measured by VSI), especially between 6 and 8 years of age?

Experiment 1: Factors Investigated q Language Specific Speech Perception (LSSP) § Effect of phonological experience in native language: Native Speech Perception minus Non-native Speech Perception (N-NN) (Burnham, 2003). q Reading § Reading ability is positively related to LSSP in children (Burnham, 2003). § Evidence for reading-lipreading link: (de Gelder & Vroomen, 1998; Cavé et al. , 2007) q Articulation ability is positively related to visual speech influence § in children (Desjardins et al. , 1997), and § in adults with cerebral palsy (Siva et al. , 1995)

Experiment 1 Language-specific Speech Perception (LSSP) AVSP Articulation Regression Correlation Reading

Experiment 1: Method q Language Specific Speech Perception (LSSP) Test: (Native minus Non-native speech perception N-NN) 18 N [voiced vs. voiceless] + 18 NN [voiced vs. prevoiced] q Reading Test § WRAT-3 reading subtest: 15 letters + 42 words = 57 items q Articulation Test § Queensland Articulation Test (QAT) § Naming task: 64 pictures § Aus English consonants: initial, medial, & final positions q Auditory-Visual Speech Perception (AVSP) Test § Mc. Gurk paradigm: A & V combinations of /ba/, /da/, and /ga/ § 48 Mc. Gurk stimuli: (24 AV, 12 AO, and 12 VO), 3 consonants x 4 talkers (2 Eng & 2 Jap) x 2 congruence types x 2 background noise (Clear & noisy [+4 d. B]) versions § Visual Speech Index (VSI) AV (+) minus AV (-)

Experiment 1: Method q Participants n=96; Australian English 5 -, 6 -, 7 -, and 8 -year-olds n=48; Australian English adults N=144 q Procedure § Individual testing (~60 minutes) § Task orders counterbalanced § Clear & Noise (+4 d. B SN Ratio) § Each stimulus presented twice

Exp. 1 Results: AVSP Test q VSI scores: Linear increase with age (p <. 01) + Quadratic between 5 and 6 years (p <. 01). q VO scores: Linear increase with age (p <. 01). q AO scores: Linear increase with age (p <. 01).

Exp. 1 Results: LSSP, reading and articulation q LSSP scores: No age-based differences (p >. 01) but native > non-native (p<. 01) q Reading scores: Increase with age (p<. 01) q Articulation scores: Increase with age (p <. 01) + adults > children (p <. 01).

Exp. 1 Results: Regression Analyses (DV = VSI Scores) q Children Age + AO + VO + LSSP + Articulation + Reading VSI q Adults Age + AO + VO + LSSP +Articulation + Reading VSI

Exp. 1 Results: Correlation Coefficients

Experiment 1: Discussion q AVSP and lipreading (VO) increase with age § This increase is sharp at around 6 years of age, around the onset of reading instruction. q No age-based LSSP differences § native and non-native difference is largest at 6 years § Reading instruction a new challenge new sources of speech information used? q Regression Analyses § Children: lipreading + LSSP VSI § Adults : AO VSI q Correlation Analyses § Articulation is correlated with AO, lipreading (VO), and reading.

Exp. 1 Discussion Experiment 2 q Experiment 1: § LSSP & Lipreading AVSP Schooling and/or new language-specific challenges seem to increase visual speech influence. q Towards Experiment 2 (3 - & 4 -year-olds): § Status of LSSP AVSP link in pre-orthographic stages? § Are AVSP and basic Cognitive abilities related? § Are AVSP and Vocabulary Knowledge related?

Experiment 2: Method q Participants: N=48; 3 -year-olds & 4 -year-olds q AVSP – material, stimuli and procedure § AVSP: AX discrimination task § Three conditions AV(36), AO(12), VO(12) § AX-VSI (/1): based on different trials q LSSP – material, stimuli and procedure § 18 Native [pa-pha] & 18 Nonnative [ba-pa] trials. § AX-based category change paradigm. q Vocabulary Test – material, stimuli, and procedure § PPVT (Dunn & Dunn, 1997) § age-based word sets of 12 items § on average 5 -6 sets. q Cognitive Flexibility – material, stimuli and procedure § FIST (Jacques & Zelazo, 2001) § 15 items (shapes, sizes & colours)

Exp. 2 Results: AVSP Test q VSI-AX scores: 4 -year-olds > 3 -year-olds (p <. 01) q VO scores: 4 -year-olds ≈ 3 -year-olds (p =. 07) q AO scores: 4 -year-olds > 3 -year-olds (p <. 05)

Exp. 2 Results: LSSP, vocabulary and cognitive flexibility q LSSP scores: § 4 -year-olds > 3 -year-olds (p <. 01) § No stimulus language effect or contrast/age interaction (p >. 05) q Vocabulary scores: 4 -year-olds > 3 -year-olds (p <. 01) q FIST scores: 4 -year-olds > 3 -year-olds (p <. 01)

Exp. 2 Results: Regression Analyses q VSI scores as dependent variable Age + AO + VO + LSSP + Vocabulary + FIST VSI-AX

Exp. 2 Results: Correlation Coefficients

Experiment 2: Discussion q AVSP, LSSP, vocabulary and FIST improve with age § AVSP improves between 3 and 4 years. § Lipreading p=. 07 – same age-based trend, but small age difference? q Regression Analyses § AO & FIST VSI-AX § similar to adults in Exp. 1 (AO VSI) q Why does AO predict VSI-AX? § AVSP product of AO speech perception § Three and 4 years: object-word pairing is a challenge, but not as much as reading – so no need for extra info. § AVSP is determined by inherent abilities? e. g. , AO speech perception and cognitive function.

Regression Analyses: LSSP, age, language tests regressed onto AVSP 3, 4 years LSSP + VO + AO+ Cognitive + Vocab. AVSP is related only to auditory speech perception 5, 6, 7, 8 years LSSP + VO + Articulation + Reading AVSP is related to LSSP and lipreading Adults LSSP + VO + AO +Articulation + Reading AVSP is related only to auditory speech perception

General Discussion q Visual speech influence develops with age § AV: e. g. , Massaro et al. (1986) § VO: e. g. , Desjardins et al. (1997) (though p=. 07 in Exp. 2) q AVSP follows a phonetic phonological trend § AVSP is amodal and phonetic (Burnham & Dodd, 2004; Rosenblum et al. , 1997) § AVSP is phonological (Massaro, 1998) § But more importantly, AVSP seems to occur at all levels of language processing.

General Discussion q AVSP- Articulation Link § Vocabulary and articulation challenges in early childhood § Tests of AVSP, LSSP, and articulation for children (12 -24 months), with and without speech disorder (Dodd et al. , in press). q AVSP-Reading Link § Burnham (2003): reading LSSP § LSSP AVSP § So, integrative ability phonemic/phonological processing § This may be the case for English but not for other languages? q Around school age, language-relevant challenges emerge § Reading phoneme-grapheme mapping § LSSP intensifies (orthographic stage: Burnham et al. , 2002) § Children use all speech information available • Hyperarticulated speech styles (Lees & Burnham, 2005). e. g. , teacherese (Håkansson, 1987). • Cognitive skills activated + maturation

Past Current Future Articulation-lipreading link (Desjardins et al. , 1997) Burnham (2003) AVSP and reading link (Cave et al. , 2007; de Gelder & Vroomen, 1998)

Reading Auditory-only speech perception AVSP (integrative module) Speech Production (articulation) Lipreading

Some questions awaiting answer q Speech Production-Perception Link: § the perception-production link: a developmental perspective; e. g. , testing children with speech disorders (Dodd et al. , in press). q Visual Speech and Reading Link: Research on the link between study strong correlations between visual aspects of speech and reading § This study correlations between reading and AO, lipreading and articulation. § Cavé et al. (2007); de Gelder & Vroomen (1998). § Cross-language reading x AVSP studies.

Acknowledging……. q q q q Prof. Denis Burnham Prof. Barbara Dodd, Asc. Prof. Chris Davis, Prof. Kaoru Sekiyama MARCS Auditory Laboratories & MARCS Baby Lab. College of Arts, University of Western Sydney Australian Postgraduate Award Aprica Foundation, Osaka, Japan Participating schools, teachers, parents and kids in Sydney