Music and Audio Computing I 216 306 A

Music and Audio Computing I 216 -306 A Prof. Marcelo M. Wanderley mwanderley@acm. org Weeks 12 and 13 Nov 29 01 - week 13 306 A - Music and Audio Computing

Interactive systems This week's topic: Gestural control of real-time multi-parametric systems. New digital musical instruments (where the computer generates sound) Interactive sound/video installations Extrapolation to other types of systems (not musical) Nov 29 01 - week 13 306 A - Music and Audio Computing 2

Interactive Systems Interactive systems becoming increasingly popular (installations, music and dance) Computers provide high computational power -> real-time processing Transducer technologies easily available Nov 29 01 - week 13 306 A - Music and Audio Computing 3

Terminology Interaction: mutual or reciprocal action or influence. v. Interact: act upon one another. Interactive System: A system that allows mutual action or influence between a user and a computer. Nov 29 01 - week 13 306 A - Music and Audio Computing 4

Terminology - Music applications Music - some examples: Interactive Musical Systems: Interactive computer music systems are those whose behavior changes in response to musical input. Interactive vs. Reactive systems Level of interaction Nov 29 01 - week 13 306 A - Music and Audio Computing 5

Instrumental Model A computer-based musical instrument model Nov 29 01 - week 13 306 A - Music and Audio Computing 6

A Technological Analysis Basic features from a technological point of view: Types of sensors - existing ones or new designs; Sensor characteristics Analysis/Conversion techniques for sensors outputs; Input Device Design Mapping strategies between control variables and sound generation; Sound synthesis methods. Nov 29 01 - week 13 306 A - Music and Audio Computing 7

A Technological Analysis Sensors and transducers: transducer (principle); sensor (device) Sensor characteristics: sensitivity, stability and repeatability Analog-digital conversion (usually MIDI, but not necessarily) Various analog-to-MIDI interfaces Inputs, outputs, speed, portability/size, price, etc. Nov 29 01 - week 13 306 A - Music and Audio Computing 8

A Technological Analysis Mapping of gesture parameters to sound parameters What is mapping? - Context dependent Mathematics: to assign to every element (of a set) an element of the same set or another set. Mapping Strategies? Simple (one-to-one) mappings? Neural networks? Complex strategies? Nov 29 01 - week 13 306 A - Music and Audio Computing 9

A Technological Analysis Sound Synthesis: Signal models: try to reproduce the sound (perceptual) characteristics or are based on abstract models (mathematics, etc. ) Physical models: try to reproduce the physical characteristics of the mechanisms that produced the sound. Nov 29 01 - week 13 306 A - Music and Audio Computing 10

Instrumental Interaction in Music Focus on computer-based musical instruments (digital musical instruments) Based or not on traditional instruments Expert interaction Manipulation (mostly) - tactile/force feedback Expressive intent Nov 29 01 - week 13 306 A - Music and Audio Computing 11

Instrumental Model Feedback Primary/passive feedback Visual, auditory and tactile-kinesthetic (tactual) Secondary/active feedback Auditory/force feedback Nov 29 01 - week 13 306 A - Music and Audio Computing 12

Performer Gestures Analysis of performer gestures in three levels: Effective gesture - necessary to produce the sound Accompanist gesture - associated body movements Figurative gesture - without direct correspondence to a physical movement Nov 29 01 - week 13 306 A - Music and Audio Computing 13

Gestural Acquisition In instrumental music: Direct acquisition: one captures the actual physical gesture or movement through the use of sensors Indirect acquisition: information obtained from the analysis of the instrument’s sound Nov 29 01 - week 13 306 A - Music and Audio Computing 14

Interfaces Analog - MIDI Different models (Sensorlab, i. Cube, Ato. MIC Pro, etc. ) Nov 29 01 - week 13 306 A - Music and Audio Computing 15

Gestural Controllers Instrument-like controllers: profit from highly developed (expert) gestures. Instrument-inspired controllers - Super. Polm Alternate controllers: design emphasis on capturing “natural” gestures - opposed to a learned gestural vocabulary - Hands Hybrid controllers Intermediate options! Extended instruments - hyperinstruments Nov 29 01 - week 13 306 A - Music and Audio Computing 16

Controllers Instrument-like controller: Saxophone MIDI Yamaha. WX 7 Nov 29 01 - week 13 306 A - Music and Audio Computing 17

Instrument-inspired controller Goto - Superpolm “Violin-inspired” controller that intends to capture part of the gestural vocabulary of a violin player in order to control other sounds (e. g. granular synthesis) Nov 29 01 - week 13 306 A - Music and Audio Computing 18

Alternate controllers Example of an alternate controller: Graphic drawing tablet - WACOM Nov 29 01 - week 13 306 A - Music and Audio Computing 19

Alternate Controllers Example: The Hands (M. Waisvisz STEIM) Nov 29 01 - week 13 306 A - Music and Audio Computing 20

Input Device Design Choice of input device for sound synthesis control Design principles: idiosyncratic (musical !? ) design methodology (engineering !? ) Nov 29 01 - week 13 306 A - Music and Audio Computing 21

Mapping Different points of view: Mapping is part of a composition - idiosyncratic Mapping is part of the instrument - affects instrument effectiveness Nov 29 01 - week 13 306 A - Music and Audio Computing 22

Mapping Electronic musical instruments Input device is independent from the sound synthesis device. No implicit mapping of one to the other Too often: one single control device corresponding to a single synthesis parameter. one-to-one mapping. Nov 29 01 - week 13 306 A - Music and Audio Computing 23

What do I mean by mapping ? An electronic musical instrument model: Mapping is the liaison or correspondence between control parameters (derived from performer actions) and synthesis parameters. Nov 29 01 - week 13 306 A - Music and Audio Computing 24

Mapping between controller outputs and synthesis inputs One-to-one; Convergent (many-to-one); Divergent (one-to-many); Any mixture of the above (many acoustic instruments) Nov 29 01 - week 13 306 A - Music and Audio Computing 25

Mapping Acoustic instruments: Sound generation device is inseparable from the human-control device. Yielding complex control relationships between human performers and their instruments Nov 29 01 - week 13 306 A - Music and Audio Computing 26

Simulation of traditional instrument performance Clarinet performance using a MIDI controller - Yamaha WX 7 and additive synthesis. Clarinet Reed behavior Gestures are not independent Embouchure is always 306 A - Music and Audio needed Nov 29 01 - week 13 Computing 27

Simulation of traditional instrument performance WX 7: independent outputs - fingering - breath pressure - lip pressure Clarinet reed behavior modeled in j. Max to simulate the interdependence of controller parameters. Nov 29 01 - week 13 306 A - Music and Audio Computing 28

Simulation of traditional instrument performance Simple and complex mappings - pedagogy Nov 29 01 - week 13 306 A - Music and Audio Computing 29

Simulation of the effect of performer movements Work with Philippe Depalle here at Mc. Gill! Clarinet movements may influence the sound recorded by a close microphone. Modeling of this effect -> Phasing! Real-time model in Max/MSP. Nov 29 01 - week 13 306 A - Music and Audio Computing 30

Simulation of the effect of performer movements Example of phasing (flanging) effect: http: //www. harmony-central. com/Effects/effects-explained. html Nov 29 01 - week 13 306 A - Music and Audio Computing 31

Simulation of the effect of performer movements Example of the real-time simulation Nov 29 01 - week 13 306 A - Music and Audio Computing 32

General Conclusions Different aspects of interactive systems Input from other fields - HCI, ergonomics, experimental psychology, etc. Need of a general corpus of knowledge: Trends in Gestural Control of Music ! ICMA/EMF working group/Gesture Research in Music Homepage Nov 29 01 - week 13 306 A - Music and Audio Computing 33

Web addresses • Gesture Research in Music homepage: http: //www. ircam. fr/gesture • Working Group ICMA/EMF www. kcng. org/wg/ Nov 29 01 - week 13 306 A - Music and Audio Computing