Digital media I Audio Glenn Bresnahan glennbu edu

Digital media I: Audio Glenn Bresnahan glenn@bu. edu Robert Putnam putnam@bu. edu 1 BPC: Art and Computation – Fall 2006

Outline l Part I (Glenn) – What is sound? – How do we hear? l Part II (Robert) – Qualities of sound – Sound reproduction • analog v. digital – Sound in VR BPC: Art and Computation – Fall 2006 2

Waves revisited BPC: Art and Computation – Fall 2006 3

Waves – (non)artistic rendering BPC: Art and Computation – Fall 2006 4

Wave properties l How might we describe waves? BPC: Art and Computation – Fall 2006 5

Wave properties l How might we describe waves? – Height – Time between waves – Speed of the wave BPC: Art and Computation – Fall 2006 6

Shoals and tides BPC: Art and Computation – Fall 2006 7

Tide tables BPC: Art and Computation – Fall 2006 8

Cause of tides l Gravity from moon and sun 1 day New moon 27. 3 days (29. 5 days) Full moon 365 days BPC: Art and Computation – Fall 2006 9

Phases of the moon Moon phases: New moon Waxing crescent First quarter Waxing gibbous Full moon Waning gibbous Last quarter New moon BPC: Art and Computation – Fall 2006 10

Moon phases BPC: Art and Computation – Fall 2006 11

Sunrise and sunset BPC: Art and Computation – Fall 2006 12

Sunrise and sunset solstice BPC: Art and Computation – Fall 2006 13

Waves – sine waves l Sine wave is the fundamental wave BPC: Art and Computation – Fall 2006 14

Waves – properties Wavelength (distance) Amplitude BPC: Art and Computation – Fall 2006 15

Waves in motion – properties Period (time for one cycle) Time 1 Frequency cycles per time interval 2 BPC: Art and Computation – Fall 2006 16

What is sound? l Examples BPC: Art and Computation – Fall 2006 17

What is sound – vibration Striking an object will cause it to vibrate l The vibration is a sine wave l Objects have a natural vibration frequency l – Resonance frequency – Frequency depends on type of material, thickness, length/size, tension – May have multiple vibrating frequencies The pitch depends on the frequency l Loudness (amplitude) depends on size of the object l BPC: Art and Computation – Fall 2006 18

What is sound – vibrations moves air Wave Energy (pluck) Air pressure level string vibration BPC: Art and Computation – Fall 2006 19

Properties of sound Pitch is perception of frequency l Frequency is measured in cycles per second (cps) l – Hertz (Hz) = cycles per second – The A above middle C is 440 Hz. – Humans hear appox. 20 -20, 000 Hz l Sound travels at approx. 1100 feet/second – Speed depends on pressure and temperature – Approx. 750 miles/hour – Approx. 1 mile every 4. 8 seconds l Perceived loudness depends on pressure level – Sound pressure is measured in (micro)pascals (20 u. Pa) – Loudness is usually expressed in decibels (d. B) BPC: Art and Computation – Fall 2006 20

Real Waves BPC: Art and Computation – Fall 2006 21

Properties of sound l Real sounds are far more complex than simple sine waves – Objects produce vibrations at multiple frequencies – Sound waves interact with other objects • Waves bounce (reflect) off surface – Reverberation/echo • Wave are absorbed by materials – Sound waves interact with each other BPC: Art and Computation – Fall 2006 22

Combinations of waves BPC: Art and Computation – Fall 2006 23

Properties of sound – real sounds BPC: Art and Computation – Fall 2006 24

Electrification of sound l Microphones – Convert pressure levels into electrical signals (voltages) l Guitar pickups – Converts string vibration to voltages • The pickup contains a magnet and a coil • The vibrating metal strings alter the magnetic field and induce a voltage in the coil l Loud speakers convert an electrical signal back into air pressure BPC: Art and Computation – Fall 2006 25

How do we hear? l Sound waves move through the air from the sound source to the ear BPC: Art and Computation – Fall 2006 26

Anatomy of the ear BPC: Art and Computation – Fall 2006 27

Anatomy of the ear - outer l Divided into three principal sections – Outer ear – Middle ear – Inner ear l Outer ear – External ear, aka pinna – Ear canal – Outer ear funnels the ear have to the eardrum BPC: Art and Computation – Fall 2006 28

Anatomy of the ear - middle l Middle ear – Eardrum – Set of 3 ear bones • the 3 bones are rigid – Act as a mechanical amplifier – The 3 rd bone, stapes, induces a vibration into the inner ear, i. e. the cochlea BPC: Art and Computation – Fall 2006 29

Anatomy of the ear - inner l Inner ear / cochlea l Where the real work is done l Cochlea is a spiral tube and filled with fluid l Stapes causes a wave to pass through the fluid BPC: Art and Computation – Fall 2006 30

Anatomy of the ear - inner Cochlea is a spiral tube lined with hair cells on a membrane (~15 K HCs) l Hairs vary in length and thickness along the tube l Hairs resonate at different frequencies l – High freq on near end, low at far end BPC: Art and Computation – Fall 2006 31

Anatomy of the ear - inner BPC: Art and Computation – Fall 2006 32

Anatomy of the ear - inner Hair cells are connected to the auditory nerve cells l The vibrations excite the nerve cells and cause them to fire (electrical signal) l A series of nerve cells pass the signal to brain l BPC: Art and Computation – Fall 2006 33

Binaural hearing - why two ears? l Two ears, so we can identify locations of sounds – Time difference – Intensity difference – Sound color difference (caused by movement of sound around head and shoulders) BPC: Art and Computation – Fall 2006 34

Sound localization – pinna Sound waves interact with the asymmetric Pinna l The effect on the sound varies with the direction l Up/down, back/front waves result in different sounds entering ear canal back l front BPC: Art and Computation – Fall 2006 35

Digital media I: part II l Other qualities of sound: pitch, timbre, “noise”, envelope l Sound reproduction: analog v. digital l Sound in VR BPC: Art and Computation – Fall 2006 36

What is pitch? l l l Our perception of the highness or lowness of a tone. Closely related to frequency When frequency doubles, pitch rises by an “octave” Examples But, what happens when there’s more than one frequency in a sound? BPC: Art and Computation – Fall 2006 37

Review: modes of vibration of a string Fundamental [e. g. , 110 Hz] 2 nd harmonic [e. g. , 220 Hz] 3 rd harmonic [e. g. , 330 Hz] Examples BPC: Art and Computation – Fall 2006 38

Timbre Sound color, or “timbre” is a quality of sound that derives from the particular combination of frequencies (a. k. a. , “harmonics” or “partials”) in a tone. l Two sounds can contain the same harmonics but sound very different because their individual harmonics are of different amplitudes. l Examples l BPC: Art and Computation – Fall 2006 39

Timbre, continued l Easy to demonstrate timbre with human voice l Hum. l Slowly open mouth. l Hear how the sound color changes from “dark” to “bright” l Example BPC: Art and Computation – Fall 2006 40

Timbre, continued. l Timbre changes as a wind instrument is played louder or softer. l Example BPC: Art and Computation – Fall 2006 41

Unpitched sounds l Can use human voice to demonstrate another distinction: pitched versus unpitched sounds l Make “s” sound l No identifiable “pitch” l Related to concept of “noise” l Examples BPC: Art and Computation – Fall 2006 42

Examples l Pitched sounds – – l Birdsong Flutes Stringed instruments Etc. Examples of unpitched sounds – Certain percussion instruments (cymbals, ratchets, etc. ) – Wind, rain, footsteps in snow l Listen now. What do you hear? Frequencies, amplitudes. Pitched, unpitched. External versus internal sources. BPC: Art and Computation – Fall 2006 43

Time variation of sounds l Most naturally occurring sounds are not static; i. e. , they vary over time – Amplitude – Pitch – Timbre l Examples BPC: Art and Computation – Fall 2006 44

Sound recording technologies l Analog l Digital BPC: Art and Computation – Fall 2006 45

Analog recording l Analog: “device or system that represents changing values as continuously variable physical quantities. ” l Example: clock with hour, minute and second hands l Question: what values are changing when we hear sound? BPC: Art and Computation – Fall 2006 46

Analog recording technologies l Phonautograph BPC: Art and Computation – Fall 2006 47

Analog recording technologies l Mechanical: Gramophone, LP record, etc. BPC: Art and Computation – Fall 2006 48

Analog recording technologies l Magnetic: Wire, tape recorder. BPC: Art and Computation – Fall 2006 49

Analog recording technologies l Optical: movie soundtrack. BPC: Art and Computation – Fall 2006 50

Analog sound reproduction l Amplification l Loudspeaker demo BPC: Art and Computation – Fall 2006 51

Digital recording l Digital: “device or system that represents changing values as discontinuous, or ‘discrete, ’ values. ” l Example: clock with number readout. BPC: Art and Computation – Fall 2006 52

Digital recording l With digital recording, we do not store a continuous record of the rise and fall of air pressure. We make measurements of the air pressure (or the voltage produced by a microphone) thousands of times per second and store these measurements as numbers. BPC: Art and Computation – Fall 2006 53

Digital recording l Analog: continuous waveform l Digital: discrete values BPC: Art and Computation – Fall 2006 54

Some buzzwords l ADC: analog-to-digital converter l DAC: digital-to-analog converter l Sampling rate: samples/second l Word size: how much storage for each sample l Quantization: [see next slide] BPC: Art and Computation – Fall 2006 55

Quantization l Selecting sample value from finite set of numbers. l 16 bits = 65, 536 choices l 20 bits = 1, 048, 576 choices Source: http: //advisor. matrasi-tls. fr/digital_sampling_index. html BPC: Art and Computation – Fall 2006 56

CD audio l 44, 100 samples per second. l 16 -bit samples (65536 different possible values) l Frequency range: 0 -22 KHz. BPC: Art and Computation – Fall 2006 57

Some benefits of digital audio l Easy to edit (visual interface) l No noise with additional generations l Flexible signal processing (no special hardware) l Examples (reverb, pitch shift, noise reduction, etc. ) BPC: Art and Computation – Fall 2006 58

An aside: MP 3 audio l CD takes up a lot of space: 3 minute song = 44100*2*2*3*60 = 31752000 B. l MP 3 compression results in a factor of 5 -10 savings in storage, but lower fidelity (e. g. , noisier). BPC: Art and Computation – Fall 2006 59

Another aside: MIDI l Musical Instrument Digital Interface l A communications scheme for computers, synthesizers, sequencers, etc. l Suited for popular music l Stores Note-On, Note-Off, Velocity, etc. (i. e. , not waveforms) l Example BPC: Art and Computation – Fall 2006 60

DAFFIE audio l Soundserver – Plays sound files associated with objects – Mixes many simultaneous sounds – Internet telephony support BPC: Art and Computation – Fall 2006 61

DAFFIE localization All sounds are assigned a location in virtual space. l Typically, sounds are associated with visible objects, but “ambient” sound (e. g. , wind, nature) is supported too. l Direction and distance are indicated by variations in loudness among the loudspeakers – done automatically by soundserver. l BPC: Art and Computation – Fall 2006 62

Uses of sound in VR l Communication (via telephony) l Sound effects l Music selections l Ambient audio l Live audio (via telephony) l Previous projects have involved controlling synthesizers or musical instruments remotely. BPC: Art and Computation – Fall 2006 63

Recording sounds for DAFFIE l Field recording versus studio recording l Record, record l Remember that sounds are combined “in real time” by the soundserver (so no need to put everything in a single file). l Stereo is good. BPC: Art and Computation – Fall 2006 64

Recording sounds for DAFFIE, part II. l Remember that you can change sounds in various ways – Change pitch/tempo – Use filters to change spectrum – Cut and paste BPC: Art and Computation – Fall 2006 65

DAFFIE demo l Demonstrate – Proximity triggering – (Variable) Distance attenuation – Sound localization BPC: Art and Computation – Fall 2006 66