Crossmodal integration Alignment of auditory and visual accent

Cross-modal integration: Alignment of auditory and visual accent structures in motion pictures Dr. Scott D. Lipscomb Institute for Music Research University of Texas at San Antonio August 16, 1999 SMPC Conference '99 1

Acknowledgments • University of California, Los Angeles – Dr. Roger A. Kendall & Dr. Ed Carterette • University of Texas at San Antonio – Institute for Music Research • Dr. Don Hodges, Director • Fellow Music Researchers August 16, 1999 SMPC Conference '99 2

Film Music Literature • past studies deal almost exclusively with the referential aspect of musical sound – “cognitive congruency” (Marshall & Cohen, 1988) – selected others: Tannenbaum (1956), Thayer & Levenson (1984) – special issue of Psychomusicology (vol. 13, 1994) • vs. accent structure alignment – i. e. , how often important events in the music coincide with important events in the visual scene August 16, 1999 SMPC Conference '99 3

Film Music Perception Paradigm (Lipscomb & Kendall, 1995) Aural Stimulus August 16, 1999 Visual Stimulus SMPC Conference '99 4

Alignment Conditions after Yeston (1975) Consonant Out-of-phase Dissonant August 16, 1999 SMPC Conference '99 5

Subject Responses … two VAME ratings • synchronization: “… how often important events in the music coincide with important events in the visual scene” • effectiveness: “… simply concerns [the observer’s] subjective evaluation of how well the two go together” August 16, 1999 SMPC Conference '99 6

Results of Previous Study (1997) presented at Penn State ASA Conference August 16, 1999 SMPC Conference '99 7

Problems with Dissonant Stimuli Solution Found August 16, 1999 SMPC Conference '99 8

Revised Stimulus Preparation • A-V alignment conditions were created using Media 100 software on a Macintosh G 3 computer • sound files were manipulated in Sonic Foundry’s Sound Forge 4. 0, using the Time Compress/Expand Sonic Foundry plug-in • completed A-V composites were recorded directly from Media 100 onto VHS tape August 16, 1999 SMPC Conference '99 9

Consonant Alignment Condition aligned as intended by the composer (visual & music only) August 16, 1999 SMPC Conference '99 10

Out-of-phase Alignment Condition alignment conditions based on Preliminary Study August 16, 1999 SMPC Conference '99 11

Dissonant Alignment Conditions • audio tracks for the Mc. Laren animations were “time expanded” by 115% • audio tracks for the “Obsession” excerpts were “time expanded” by 110% – since these excerpts were longer, the gradual misalignment could occur at a slower pace August 16, 1999 SMPC Conference '99 12

Subject Info & Method • N = 135 UTSA students taking music classes – musical training n (high = 17, mod = 31, low = 87) – gender n (female = 82, male = 53) – stimuli presented to groups of subjects • stimuli were presented to subjects using a Samsung VR 5855 video cassette recorder and a Pro. Scan PS 80690 80” big screen television – VAME ratings were provided on a continuous line response anchored by either “not synchronized - synchronized” or “ineffective - effective” not synchronized – each subject was assigned to one of three random stimulus presentation orders – response forms were generated so that order of VAME responses was also randomized into three versions August 16, 1999 SMPC Conference '99 13

Video excerpts August 16, 1999 SMPC Conference '99 14

Statistical Analysis - Between (SPSS’s “General Linear Model - Repeated Measures”) • repeated measures, fully-factorial ANOVA – three repeated measures: 2 VAME ratings x 6 AV combinations x 3 alignment conditions • Tests of Between-Subjects Effects – no significant between-groups variation • musical training: p = 0. 090; f(2, 129) = 2. 457 • gender: p = 0. 549; f(1, 129) = 0. 361 – interaction between AV combination & musical training is not statistically significant • p = 0. 622; f(2, 129) = 0. 477 August 16, 1999 SMPC Conference '99 15

Statistical Analysis - Within (SPSS’s “General Linear Model - Repeated Measures”) • Tests of Within-Subjects Effects – within-groups, ratings were significantly different depending upon the AV combinations • p <. 0005; f(5) = 20. 5 – within-groups, ratings were significantly different depending upon the alignment condition • p <. 0005; f(2) = 173. 036 • Within-Subjects Interaction Effects – alignment condition x VAME • p <. 0005; f(2) = 3. 906 – AV combination x alignment condition • p <. 0005; f(10) = 48. 720 August 16, 1999 SMPC Conference '99 16

Statistical Analysis - Within (SPSS’s “General Linear Model - Repeated Measures”) • Within-Subjects Effects - complex interactions – AV combination x VAME x musical training • p =. 006; f(10) = 2. 517 – AV combination x alignment condition x VAME • p <. 0005; f(10) = 3. 885 August 16, 1999 SMPC Conference '99 17

Experimental Results (r =. 997) August 16, 1999 SMPC Conference '99 18

Experimental Results (r =. 977) August 16, 1999 SMPC Conference '99 19

Experimental Results Subject Ratings Collapsed (n=3) Results of Previous Study August 16, 1999 SMPC Conference '99 20

Conclusions • accent structure alignment does, in fact, play an important role in subject ratings of both “synchronization” and “effectiveness” – ANOVA & Pearson correlation coefficient • based on the results of the present & previous investigations, there appears to be a dynamic relationship between the “association judgment” and “accent structure alignment” aspects of the model of Film Music Perception Revised Model August 16, 1999 SMPC Conference '99 21

Contact Info Dr. Scott D. Lipscomb UTSA Division of Music 6900 N. Loop 1604 West San Antonio, TX 78249 (210) 458 -4354 (210) 458 -4381 FAX lipscomb@utsa. edu http: //music. utsa. edu/~lipscomb August 16, 1999 SMPC Conference '99 22

Film Music Paradigm Aural Stimulus (revised) Visual Stimulus Perception Implicit Processes Audio-Visual Congruence Association Judgment Accent Structure Alignment No Yes AV combination perceived as Effective August 16, 1999 SMPC Conference '99 AV combination perceived as Ineffective 23