Toward Synthesized Environments A Survey of Analysis and

  • Slides: 28
Download presentation
Toward Synthesized Environments: A Survey of Analysis and Synthesis Methods for Sound Designers and

Toward Synthesized Environments: A Survey of Analysis and Synthesis Methods for Sound Designers and Composers Ananya Misra, Perry Cook Princeton University

Motivation • Multitude of sounds in the sonic landscape • Multitude of algorithms •

Motivation • Multitude of sounds in the sonic landscape • Multitude of algorithms • Better knowledge of a variety of algorithms => empowerment to create rich sound scenes

Some Related Surveys • Smith. “Viewpoints on the History of Digital Synthesis. ” ICMC

Some Related Surveys • Smith. “Viewpoints on the History of Digital Synthesis. ” ICMC 1991. • Pope. “A Taxonomy of Computer Music. ” Contemporary Music Review 1996. • Tolonen et al. “Evaluation of Modern Sound Synthesis Methods. ” Tech. Rep. 1998. • Vercoe et al. “Structured Audio: Creation, Transmission, and Rendering of Parametric Sound Representations. ” Proc. IEEE 1998. • Widmer et al. “Sound and Music Computing: Research Trends and Some Key Issues. ” JNMR 2007. This one: from the perspective of creating complex environmental sound scenes or compositions

Overview • Abstract synthesis algorithms • Synthesis from “scratch” • Synthesis from existing sounds

Overview • Abstract synthesis algorithms • Synthesis from “scratch” • Synthesis from existing sounds – Concatenative techniques – Additive synthesis – Subtractive synthesis and other techniques • Analysis not for synthesis Disclaimer: No taxonomy is clean or 1 -1 from method to class

Abstract synthesis algorithms

Abstract synthesis algorithms

Oscillators • From analog days Sine wave Triangle wave Sawtooth wave Alarm clock Examples

Oscillators • From analog days Sine wave Triangle wave Sawtooth wave Alarm clock Examples by Chuc. K

Frequency Modulation • Modulation of one oscillator’s phase by another’s output Basic example Doorbells

Frequency Modulation • Modulation of one oscillator’s phase by another’s output Basic example Doorbells Examples by Chuc. K

More • Circle maps as nonlinear oscillators (Essl, ICMC 2006) • Errant sound synthesis:

More • Circle maps as nonlinear oscillators (Essl, ICMC 2006) • Errant sound synthesis: “the potential of any algorithm cast into the audio range” (Collins, ICMC 2008) • Auditory display and sonification

Synthesis from scratch Synthesis from physical or perceptual models, without the raw material of

Synthesis from scratch Synthesis from physical or perceptual models, without the raw material of existing audio samples

Physical models High-level parametric control over synthesized sound • Plucked string model, waveguides and

Physical models High-level parametric control over synthesized sound • Plucked string model, waveguides and more • Reed and bowstring models, singing voice synthesis, percussive sounds • Synthesis Tool. Kit => Pe. RColate (Max/MSP), Chuc. K, Super. Collider • Real-world contact / motion sounds Modal synthesis Gait modeling Cook, 2002

Perceptual models Give desired perceptual characteristics • For speech and singing (Cook, CMJ 1996):

Perceptual models Give desired perceptual characteristics • For speech and singing (Cook, CMJ 1996): – Formant synthesizers – Formant wave functions (FOFs) (Rodet, CMJ 1984) • General: Feature-based synthesis (Hoffman, ICMC 2006) =>

Synthesis from existing sounds Creation of sound from existing sound, synthesis by analysis

Synthesis from existing sounds Creation of sound from existing sound, synthesis by analysis

Concatenative techniques • Rearrangement of samples in the time domain • Wavetable synthesis •

Concatenative techniques • Rearrangement of samples in the time domain • Wavetable synthesis • Concatenative synthesis (Schwarz, JNMR 2006): – – Source sound segmented into units Target sound Set of unit descriptors Unit selection algorithm

Concatenative techniques • Applications: singing, instruments, audio mosaicing • Granular synthesis: concatenating usually short

Concatenative techniques • Applications: singing, instruments, audio mosaicing • Granular synthesis: concatenating usually short “sound grains” (Truax, 1990)

Concatenative techniques: granular synthesis • Formant wave functions, when using arbitrary sound samples •

Concatenative techniques: granular synthesis • Formant wave functions, when using arbitrary sound samples • Dictionary-based methods with time-localized waveforms (Sturm, ICMC 2008) => analytical counterpart • TAPESTREA: granular synthesis by parametrically looping transformed events

Concatenative techniques for sound textures • Splitting soundscapes into syllable-like segments (Hoskinson, ICMC 2001)

Concatenative techniques for sound textures • Splitting soundscapes into syllable-like segments (Hoskinson, ICMC 2001) • Soundscape generation from database of annotated sound files (Birchfield, ICMC 2005) • Fast sound texture synthesis using overlapadd (Fröjd, ICMC 2007) From Fröjd&Horner, 2007; algorithm offered in TAPESTREA

Additive synthesis Spectral analysis, addition of resulting signals • Channel vocoder: bank of bandpass

Additive synthesis Spectral analysis, addition of resulting signals • Channel vocoder: bank of bandpass filters • Phase vocoder: FFT to get phase as well as magnitude of each frequency band • Pitch and time transformations (offered in TAPESTREA) • Cross-synthesis

Additive synthesis: sinusoidal modeling • Speech signals can be modeled using a few sinusoids

Additive synthesis: sinusoidal modeling • Speech signals can be modeled using a few sinusoids (Mc. Aulay 1986, Quatieri 1986) • Spectral modeling synthesis (Serra, 1989): sines + noise • Lemur/Loris, CLAM, SMS, SPEAR, Audio. Sculpt • Used to extract and transform some environmental sounds in TAPESTREA Transformed windchimes Baby chorus / cacophony

Sines + Transients + Noise Decompose into transients (brief, noisy events) as well as

Sines + Transients + Noise Decompose into transients (brief, noisy events) as well as sines and noise • Transient / onset detection techniques: – Time-domain envelope following (in TAPESTREA) – Comparing energy envelopes of original and residual noise signals (Levine, 1998) – Comparing energy in short and long signal segments (Verma, 1998) (in TAPESTREA) – Frequency-domain techniques

Subtractive synthesis and linear predictive coding • Filtering a signal to shape it by

Subtractive synthesis and linear predictive coding • Filtering a signal to shape it by subtracting unwanted components • LPC: Source-filter model where next = linear combination of previous samples (Atal, 1970) • Musical composition (Lansky, 1989) • Sound texture synthesis (Athineos, 2003; Zhu, 2004) Noise-excited LPC Time-frequency LPC; Athineos&Ellis, 2003

Other tools for sound textures • Wavelet-tree learning (Dubnov, 2002) (offered in TAPESTREA) •

Other tools for sound textures • Wavelet-tree learning (Dubnov, 2002) (offered in TAPESTREA) • Parametrically modeling and transforming stochastic components (Miner, 2002) • Inferring statistical distributions of events (Zhu, 2003)

Analysis not for synthesis Content-based analysis methods not necessarily designed for synthesis. May provide

Analysis not for synthesis Content-based analysis methods not necessarily designed for synthesis. May provide information to guide synthesis algorithms.

Analyze to… • Represent an audio signal in structurally or perceptually meaningful ways •

Analyze to… • Represent an audio signal in structurally or perceptually meaningful ways • Understand use a collection of sounds on a global level

Representing a signal • Source separation: Computational auditory scene analysis (Ellis, 1992; Melih, 2000)

Representing a signal • Source separation: Computational auditory scene analysis (Ellis, 1992; Melih, 2000) – Automatic and manual grouping of partials offered in TAPESTREA • Source separation: Multiple fundamental frequency estimation (Klapuri, 2004) • Blind source separation (Hoffman, 2009) • Music transcription

Understanding a collection • Comparison actions: content-based classification, search, recommendation, … • Automatic timbre

Understanding a collection • Comparison actions: content-based classification, search, recommendation, … • Automatic timbre recognition • Music information retrieval tools, e. g. MARSYAS

Conclusions

Conclusions

Advantages of parametric synthesis algorithms • Many algorithms may contribute to one simple piece

Advantages of parametric synthesis algorithms • Many algorithms may contribute to one simple piece • Many available tools: programming languages, specialized libraries, graphical software • Compression • Ability to re-render over and over with changes, interactively or in real-time • Rich palette of techniques for composers

Taxonomy of methods by sounds Sound/Goal Methods Abstract FM, non-linear oscillators, feature-based synthesis, wavetables,

Taxonomy of methods by sounds Sound/Goal Methods Abstract FM, non-linear oscillators, feature-based synthesis, wavetables, granular synthesis Acoustic instruments Wavetables, waveguides / physical models, granular synthesis, additive synthesis Contact sounds Physical models Cross-synthesis LPC, vocoders Pitch/time transformations LPC, vocoders, granular synthesis, additive synthesis Pitched sounds Additive synthesis, granular synthesis, FM, oscillators Singing voice FM, formant synthesis, FOFs, granular synthesis, additive synthesis Speech Formant synthesis, FOFs, granular synthesis, vocoders, additive synthesis, LPC Textures and soundscapes Granular synthesis, LPC, stochastic and wavelet-based methods Transients Onset detection, physical models, granular synthesis, sines+transients+noise models