SENSORS MOBILE MUSIC Lalya Gaye Music controllers Interfaces

SENSORS & MOBILE MUSIC Lalya Gaye

Music controllers Interfaces * Body-based Human body as start for design: Expressive qualities of human movements The Hands, Waisvicz, STEIM, 1984

Music controllers Interaction * User movement - Choreographed body movement - Traditional instrumental gesture - Novel gestures Dark around the Edges, Winkler, 1997 Machover & Yoyo Ma, Hypercello, 1991 The Hands, Waisvicz, STEIM, 1984

Music controllers Interaction * User movement - Full-handed gesture Unfoldings, Interactive Inst. , 2003 - Empty-handed gesture Lady glove, Bongers & Sonami, 1991 Stranglophone, Sharon, ITP/NYU, 03

Music controllers Interfaces * Environment-based Interactive environments - Reactive floors - Digital realm: networked audio Global String, Tanaka & Toeplitz, 1998 Take advantage of the features of space Interactive environments: many people together, control of interaction parameters… Magic Carpet, MIT Medialab, 1996

Music controllers Interfaces * Wearables Musical jeans jacket (MIT Medialab, 1992) Tgarden (Fo. AM & sponge, ~2001) Expressive Footwear (MIT, 1997 -2000) ensemble (Kristina Andersen, ~2003) Intimate interfaces; Body movement and posture Theatrical vs. daily life dimensions

Music controllers Interfaces * Object-based Starting with existing instruments - augmented (hyperinstruments…) - digitalised (ex: piano synth) - interface used as controller (ex: MIDI keyboard) Use metaphor of object Taku Lippit, ITP/NYU, 2002 -03 Machover & Ma, Hypercello, MIT, 1991

Music controllers Interfaces * Object-based Repurposed everyday objects and materials: water, fabric, chemicals, vegetables … Daniel Skoglund, 8 Tunnel 2 Particles, Horio Kanta, 2003 MIDI Scrapyard Challenge, Brucker-Cohen & Moriwaki, 03 -04

Music controllers Interfaces * Object-based Take advantage of the material properties of objects f. e. x bendable, conducts electricity, etc Take into consideration human activities surrounding the objects: build upon it and / or break from it

Music controllers Output * Mechanical Guitarbot (Eric Singer et al. , LEMUR, 2003 -) * Electroacoustic Spherical speakers (Curtis Bahn) * Tactile output (haptics) Cutaneous Grooves (E. Gunther, MIT Medialab, 2001)

Sensors in Ubicomp technology * Computing where needed, not other way around. Invisible in use, in the fabric of everyday life, embodied interaction. Connection to place and moment of use. * Sensors: - in everyday environments (e. g. context-awareness) - on people (e. g. wearables) - on artefacts (Media cup - Tec. O) * Sensor fusion: combining different data and placements to gather context - sensor networks

Sensors in mobile music & locative audio * Combining NIME and Ubicomp type of sensors use * Urban settings + everyday: rich environment, familiar, unpredictable, dynamic, heterogeneous * Sensors on environments, users, objects * Interaction between: - user and objects - user and environment - user and user(s) + combinations and networks Possible uses, interactions, issues and implications of implementations?

Mobile music and locative audio Locative audio in public space * Space annotation: sensing proximity / location Hear&There (Rozier, MIT Medialab, 1999) Tejp / Audio tags (PLAY & FAL, 2003 -04)

Mobile music and locative audio Locative audio in public space * Radio pirates: sensing environmental factors Bit Radio (Bureau of Inverse Technology)

Mobile music and locative audio Mobile music * Mobile music sharing: sensing others Sound. Pryer (Mattias Östergren, Interactive Institute, 2001) Tun. A (Arianna Bassoli et al. , Medialab Europe, 2002) Push!Music (Håkansson et al. , Viktoria Institute, 2005)

Mobile music and locative audio Mobile music * Mobile music making Music making away from computer screen or performance setting: in the everyday Sensor technology + GPS -> situated music making Ad hoc & distributed networks throughout the city -> collaborative music making etc

Mobile music and locative audio Mobile music * Mobile music making: sensing user-environment interaction Sonic City (Gaye et al. , FAL & PLAY, 2002 -04) Sound Lens (Toshio Iwai, 200? )

Mobile music and locative audio Mobile music * Mobile music making: device as interface between user and space Sound Mapping (Iain Mott et al. , Reverberant, 1998)

Mobile music and locative audio Mobile music * Mobile music making: sensing user-user + user-device interaction Cos. Tune (Nishimoto, ATR, 2001) Sound Lens (Toshio Iwai, 200? ) Malleable Mobile Music (Atau Tanaka, Sony CSL, 2004)

Mobile music and locative audio Sound Walks: mapping audio world to physical paths * Sound-art installations Electric walks (Christina Kubisch) Drift (Teri Rueb) * Walking through digital space Seven Mile Boots (Beloff et al. , 2003 -04)

Mobile and locative sound Wearable audio Personal instrument (Krzysztof Wodiczko, 1969)

Mobile and locative sound Output Headphones vs Boombox vs Using everyday objects Soundbug. TM speakers & piezos Flower Speakers (LET’S corporation, Japan, 2004)

Mobile and locative sound Output Wearables Nomadic Radio (Nitin Shawney, MIT Medialab, 1998) Sonic Fabric (Alice Santaro, 2002)

Demo DIY music controller * System set-up Tracking & other sensors Micro-controllers MIDI protocol Interactive softwares

DIY music controller * Components - sensors: potentiometer + switch / light + proximity sensors - micro-controller: Basic. X-24 - protocol: MIDI - software: Pd

Tracking & other sensors * Contact-based tracking Isometric • Pressure, switches, etc Movement sensing • Rotation: pots, goniometers, joysticks • Linear movement: sliders, tension sensors, pads, tablets • Bending Ref: “Human Movement Tracking Technology”, Mulder, A. Technical Report, NSERC Hand Centered Studies of Human Movement project. Burnaby, B. C. , Canada: Simon Fraser University.

Tracking & other sensors * Contact-based tracking Inside-in • Emitter + receiver on subject body-centred • Workspace in principle unlimited • ex: flex sensors, biometric sensors… Inside-out • Sensor on subject + external emitter • Workspace limited if source artificial, unlimited if source natural • ex: accelerometers, gyroscope, compass… Ref: “Human Movement Tracking Technology”, Mulder, A. Technical Report, NSERC Hand Centered Studies of Human Movement project. Burnaby, B. C. , Canada: Simon Fraser University.

Tracking & other sensors * Contactless tracking Outside-in • • External sensor + emitter on subject Least obtrusive Workspace limited ex: video tracking + markers Indirect acquisition • Deduction from audio output • Latency Ref: “Human Movement Tracking Technology”, Mulder, A. Technical Report, NSERC Hand Centered Studies of Human Movement project. Burnaby, B. C. , Canada: Simon Fraser University.

Tracking & other sensors * Other sensors Objects • More or less same as human tracking sensors Environment • Light, sound, temperature, humidity, electricity, magnetism… Digital information • ex: activity on internet Ref: “Human Movement Tracking Technology”, Mulder, A. Technical Report, NSERC Hand Centered Studies of Human Movement project. Burnaby, B. C. , Canada: Simon Fraser University.

* Micro-controllers • Collecting sensor data and sending them to processor (e. g. PC) as serial data (e. g. MIDI signal) • Can also be used to trigger actuators (f. ex: LED) Common micro-controllers • • • Basic. X-24 Basic Stamp II PIC

* MIDI protocol • MIDI=Musical Instrument Digital Interface • Standardised serial communications protocol between synthesizers and other digital music devices • Controllers / receivers • Midi command = status byte + 2 data bytes – action (note on, note off, pitch bend, control change ) – pitch – velocity (how loud)

* Interactive music softwares Common softwares • MAX/MSP • Pd (Pure Data)… • Using MIDI signals as control data…

* Reading sensor values with BX-24 • connect sensor to ADC pins • power supply them with the BX’s 5 V DC output power (! BX power = 9 V) • add ”Serial. Port” module for communicating with serial port • write routine for reading voltage on pins • download program to EPPROM

Option Explicit Dim volt. In As Byte Dim switch As Byte Public Sub Main() volt. In = 1 switch = 1 Do 'potentiometer volt. In = c. Byte(get. ADC(16)) 'switch = Get. Pin(17) Debug. Print "volt. In: "; c. Str(Volt. In) Debug. Print "switch: "; c. Str(switch) Call Sleep(0. 05) Loop End Sub

* Sending values as MIDI signal - convert data into MIDI scale (0 -127) - create buffer - adapt baud rate to MIDI speed - write subroutine loop for sending MIDI - MIDI command 144 (note on) + 128 (note off) - or on + ”velocity” used as ID + ”pitch” used as sensor value - download on EPPROM - sending out serial data via MIDI adapter circuit and MIDI-USB adapter Ref: Physical Computing, Tom Igoe. http: //www. tigoe. net/pcomp/code/archives/bx-24/000249. shtml

Option Explicit Dim Input. Buffer(1 To 12) As Byte Dim Output. Buffer(1 To 10) As Byte Dim midi. Cmd As Byte Dim vel As Byte Dim midi. Task. Var(1 To 50) As Byte Dim volt. In As Byte Dim switch As Byte Ref: Physical Computing, Tom Igoe. http: //www. tigoe. net/pcomp/code/archives/bx-24/000249. shtml

Public Sub Main() volt. In = 1 switch = 1 Call open. Queue(Inputbuffer, 12) Call open. Queue(Outputbuffer, 10) Call Open. Com(1, 9600, Input. Buffer, Outputbuffer) Register. ubrr = 14 midi. Cmd = 144 Call. Task "midi. Task", midi. Task. Var Do 'potentiometer volt. In = c. Byte(c. Sng(get. ADC(16)) * 127. 0 / 1023. 0) 'switch = Get. Pin(17) Call Sleep(0. 05) Loop End Sub Ref: Physical Computing, Tom Igoe. http: //www. tigoe. net/pcomp/code/archives/bx-24/000249. shtml

Sub midi. Task () Do vel=1 Call put. Queue(Output. Buffer, midi. Cmd, 1) Call put. Queue(Output. Buffer, volt. In, 1) Call put. Queue(Output. Buffer, vel, 1) vel=2 Call put. Queue(Output. Buffer, midi. Cmd, 1) Call put. Queue(Output. Buffer, switch, 1) Call put. Queue(Output. Buffer, vel, 1) Call Sleep(0. 05) Loop End Sub Ref: Physical Computing, Tom Igoe. http: //www. tigoe. net/pcomp/code/archives/bx-24/000249. shtml

* Sending values as MIDI signal Ref: Physical Computing, Tom Igoe. http: //www. tigoe. net/pcomp/midi. shtml

* Receiving MIDI data in Pd • C: …/pd/bin • pd –midiindev 1 • route data according to ID (”vel”) • use ”pitch” as control values

Discussion * Mobile music application using sensors: Possible uses, interactions, issues and implications of implementations? * Props: sensor platform, soundbug, tell me * Focus: - sensor positioning - physical interaction and relation between sound, body and place - combining data

Links DIY links • BX-24: http: //www. basicx. com • Pd: http: //www. crca. ucsd. edu/~msp/software. html • More micro-controllers etc: ITP Physical computing http: //tigoe. net/pcomp/index. shtml Book Physical Computing – Dan Sullivan & Tom Igoe • On i. Paq: Linux + PDa (by Gunther Geiger): http: //gige. xdv. org/pda/

Links Sensors & Mobile Music Links New Interfaces for Musical Expression: http: //www. nime. org Mobile Music & Locative Audio: http: //www. netzwissenschaft. de/mob. htm http: //www. viktoria. se/~lalya/tamabi 05/ Ubiquitous Computing: http: //www. ubicomp. org/