MPEG 4 Structured Audio Algorithmic Sound for the

MPEG 4 Structured Audio: Algorithmic Sound for the Internet and Beyond John Lazzaro John Wawrzynek Sep 1, 1999 CS Division University of California at Berkeley www. cs. berkeley. edu/~johnw

MPEG 4 Structured Audio Outline: u. Motivation for structured audio u. Introduction to MP 4 -SA u. Example encoding u. C translator u. Physical Instrument Modeling u. Hardware Architectures u. Future directions

Digital Audio Basics amp 16 -bit samples v mono: 705. 6 kbps u u u Cell-phone network: 5 -10 kbps dialup modems: 50 kpbs x. DSL: 128 to 1000 kbps Traditional Compression: encoder time 44. 1 k. Hz sample rate decoder v How well does this work? u True u Lossless: 2. 5 X reduction ²Shorten, T. Robinson (Cambridge University) “Perceptually Lossless” : 10 X-20 X reduction ²MP 3, Dolby AC 3, …

The Kolmogorov alternative: Write a computer program that generates the desired audio stream. Transmit the computer program. To decode, execute the program. Similar to Postscript! v MPEG-4 Structured Audio (MP 4 -SA) uses this approach. v Final draft standard: Nov 15, 1998. v Eric Schierer, Editor (MIT Media Lab). u http: //sound. media. mit. edu/~eds/mpeg 4/

MP 4 -SA Encoding v may be a creative act: writing a program. u directly (emacs), or u indirectly (GUI, webpage) Ü In this case, MP 4 -SA is a lossless compressor. v may be automatic -- given a sound, an encoder writes a program that generates the sound. u Automatic encoding is a hard problem in the general case. MP 4 -SA Decoders v are interpreters or compilers.

Key Application: Music Production ¯ Modern Music Production is Computer based. u Musicians enter performances into computers as control information, not audio waveforms. u Digital synthesizers, effects, and mixes create the final audio, under engineer/producer control. MP 4 -SA Maps to Modern Music Production “The Program” synthesis algorithms effects “boxes” mixers Musical performance Mix-down control information Network “The Decoder” sound rendering Premium on low-bandwidth

Key Application: Music Production ¯ Modern Music Production is Computer based. u Musicians enter performances into computers as control information, not audio waveforms. u Digital synthesizers, effects, and mixes create the final audio, under engineer/producer control. MP 4 -SA Maps to Modern Music Production “The Program” synthesis algorithms effects “boxes” mixers Musical performance Mix-down control information Standard Framework “The Decoder” sound rendering File System Ideal format for collaborative productions, remixes, . . .

MPEG 4 Structured Audio: v A binary file format that encodes: The programming language SAOL (say: sail). The musical score language SASL. Legacy support for MIDI. Audio sample data. v Result is normative: an MP 4 -SA file will sound identical on all compliant decoders. èDifferent from MIDI files.

MPEG 4 Standard MPEG 4 video audio Natural coding AAC T/F CELP Parametric system Synthetic coding SA TTS Structured Audio: One “component” in the MPEG audio standard. ISO/IEC 14496 -3 sec 5

MPEG 4 Standard MPEG 4 video audio Natural coding AAC T/F CELP Parametric system Synthetic coding SA TTS Advanced Audio Coding: successor to MP 3, delivers highest quality audio, and highest bit-rate.

MPEG 4 Standard MPEG 4 video audio Natural coding AAC T/F CELP Parametric system Synthetic coding SA TTS Time-Frequency Coding: Meant for a moderate bit/sec range, with moderate quality.

MPEG 4 Standard MPEG 4 video audio Natural coding AAC T/F CELP Parametric system Synthetic coding SA Code Excited Linear Prediction: Low bit rate coder, works best as a speech coder. TTS

MPEG 4 Standard MPEG 4 video audio Natural coding AAC T/F CELP Parametric system Synthetic coding SA Parametric coders: Very-low bit rate coder, works best as as a speech coder. TTS

MPEG 4 Standard MPEG 4 video audio Natural coding AAC T/F CELP Parametric system Synthetic coding SA TTS Text-to-Speech: Takes phonetic and prosadic control information, produces syntesized speech.

MPEG 4 Standard MPEG 4 video audio Natural coding AAC T/F CELP Parametric system Synthetic coding SA TTS “System” level includes mechanisms for composing and synchronizing audio (& video) components.

Why SAOL and MP 4 -SA? Why not Java? v Musical performance have temporal structure that changes over several timescales: Sample-by-sample 10’s of usec Amplitude & timbre envelopes: 10’s of msec Note-by-note: 100’s of msec v Writing sound generation code in a conventional language results in code dominated by time-scale management. u Hard to maintain, hard to optimize.

Time management is built into SAOL. v A SAOL program executes by moving a simulated clock forward in time, performing calculations along the way in a synchronous fashion. v Work is scheduled to happen: u at the a-rate (the audio sample rate) u at the k-rate (envelope control rate) u at the i-rate (rate for new notes) v Language variables are typed as a/k/i-rate. v A language statement is scheduled based on the rate of the variables it contains.

SAOL, SASL, and Scheduling: v Sound creation in MP 4 -SA can be compared to a musician playing notes on an instrument. v A SAOL subprogram (called an instr or instrument) serves as the instrument. v SASL commands (called score lines) act to play notes on SAOL instruments. v Many instances of a SAOL instr can be active at one time, making sounds corresponding to notes launched by different score lines in a SASL file.

Single Note Execution Trace SAOL Instruments. . . Contains all the instructions for playing a note: -- Code that runs at note launch. (once per i-pass) -- Code that models timbre evolution at the k-rate. (once per kpass) -- Code to generate audio samples at the a-rate. (once per a-pass) Executing a Note … (k-rate: 4 k. Hz, a-rate: 40 k. Hz) time(us) pass 0 0 0 25 50. . . 225 250 275 300. . . 475 i-pass k-pass a-pass 500 525. . . k-pass a-pass a-pass

An example: v SAOL instrument tone, that plays a gated sine wave. (SAOL code in next slide. ) v This SASL file plays melody on tone: 0. 5 When 1. 5 2. 5 instance is launched 3 3. 25 3. 5 4 5 tone tone end How long instrument runs 0. 75 0. 25 0. 5 52 64 63 59 61 63 64 0. 25 0. 225 0. 25 Instance parameters (note number, loudness)

SAOL code for tone instr tone (note, loudness) { ivar a; // sets osc f ksig env; // env output asig x, y; asig init; // osc state a = 2*sin(3. 141597*cpsmidi(note)/s_rate); env = kline(0, 0. 1, 0. 5, dur-0. 2, 0. 5, 0. 1, 0); if (init == 0) // first a-pass only { x = loudness; init = 1; } } x = x - a*y; y = y + a*x; // the FLOPS happen in // these 3 statements output(y*env); // creates audio output // end of instr tone

SAOL code for tone instr tone (note, loudness) { ivar a; // sets osc f ksig env; // env output asig x, y; asig init; // osc state i-rate a = 2*sin(3. 141597*cpsmidi(note)/s_rate); env = kline(0, 0. 1, 0. 5, dur-0. 2, 0. 5, 0. 1, 0); if (init == 0) // first a-pass only { x = loudness; init = 1; } } x = x - a*y; y = y + a*x; // the FLOPS happen in // these 3 statements output(y*env); // creates audio output // end of instr tone

SAOL code for tone instr tone (note, loudness) { ivar a; // sets osc f ksig env; // env output asig x, y; asig init; // osc state k-rate a = 2*sin(3. 141597*cpsmidi(note)/s_rate); env = kline(0, 0. 1, 0. 5, dur-0. 2, 0. 5, 0. 1, 0); if (init == 0) // first a-pass only { x = loudness; init = 1; } } x = x - a*y; y = y + a*x; // the FLOPS happen in // these 3 statements output(y*env); // creates audio output // end of instr tone

SAOL code for tone instr tone (note, loudness) { ivar a; // sets osc f ksig env; // env output asig x, y; asig init; // osc state a-rate a = 2*sin(3. 141597*cpsmidi(note)/s_rate); env = kline(0, 0. 1, 0. 5, dur-0. 2, 0. 5, 0. 1, 0); if (init == 0) // first a-pass only { x = loudness; init = 1; } } x = x - a*y; y = y + a*x; // the FLOPS happen in // these 3 statements output(y*env); // creates audio output // end of instr tone

SAOL: Unique Features v Rate semantics: u i/k/a-rate execution v Vector arithmetic: u ex: A=B+C for i=1, n A[i]=B[i]+C[i] v All floating-point arithmetic. v Extensive build-in audio function library: usignal generators, table operators, pitch converters, filters, fft, sample rate conversion, effects, . . .

SAOL: Unique Features v Instrument communication through bus structures: B A bus C D v Dynamic instrument creation and control. v Scheduler and language support for MIDI and SASL scores.

Sfront - a SAOL-to-C translator v Converts MP 4 -SA files to a C program, that when executed, produces audio. foo. mp 4 sfront sa. c v Handles SAOL, SASL, MIDI, uncompressed samples. SAOL SASL MIDI Uncompressed samples u Runs sfront foo. mp 4 sa. c on UNIX, Win 98/NT. u Licensed under the GNU public license (GPL). u www. cs. berkeley. edu/~lazzaro/sa

Sfront Benchmarks Sfront version 0. 36 Machine: 450 Mhz Pentium III, 128 MB, gcc version egcs-2. 91. 66, -O 3 optimizer Audio sample rate: 44. 1 k. Hz for all examples MP 3 compression ratio = 11

Sfront Performance Summary: v Rendering (file decoding): u Current performance: a benchmark suite of moderately complex MP 4 -SA streams computes in a time equivalent to the audio it generates, on a 400 Mhz Ultrasparc & 450 Mhz Pentium. v Real-time interaction: u with a MIDI keyboard with acceptable latency (~20 ms) and microphone input.

Interesting Issues: v MP 4 -SA puts emphasis on sound synthesis methods that can be described in a small amount of space. J Physical Modeling L Sampling Natural Instruments good bad v If models are chosen carefully, compression ratios of 100 to 10, 000 are possible. v Physical Modeling is relatively immature, but holds much promise.

Struck/Plucked Instrument Model Examples: struck bars, bells, drums, plucked strings attack section linear modes (resonances) M 1 Aluminum Bar Sounds M 2 M 3 striker single strike output multiple strikes amplitude frequency Mn Digital resonator: Yn = Yn-1 + Yn-2 + Xn Parameters: striker characteristics, resonator constants

Blown Instrument Model Examples: pipes, flutes, etc. jet x excitation non-linear element y linear element (resonant modes) Blown Pipe Sounds tube amplitude y brass pipe x overblown frequency Parameters: shape of non-linear function, resonator constants

Physical Modeling Summary v Models instrument not sound. v Advantages over traditional synthesis techniques (FM, sample-based): u Compact descriptions. u Physical parameterization leads to: ²more intuitive control ²lower control bandwidth u State accurate simulation leads to: ²efficiency in re-excitation ²emulation of otherwise missing effects u Ultimately - more realistic sounds.

Physical Modeling Summary (cont. ) v Disadvantages: u potential for high computational complexity v Approaches: u PDE (partial differential equation) approach would be nice, but probably not practical. u ODE (ordinary differential equation, lumped circuit models) practical and very general. Capture essential physics. u Wave-guide filters provide a more efficient alternative in some cases.

Interesting Issues (cont. ): v MP 4 -SA specifies that a decoder produces audio that “sounds identical” to computing the program accurately. v A new role for psychophysics: Instead of using psychophysics to squeeze bits out of a sound representation, MP 4 -SA decoders will use psychophysics to squeeze FLOPS out of sound computations. v Leverage spectral and temporal masking.

Interesting Issues (cont. ): v MP 4 -SA can be used in a way similar to traditional compression except that the compression method can be ad hoc: u Frame-work for experimentation in encoding. u Hope for automatic encoding, if done in a voice specific way: ² vocals ² guitar ² sax ² and other hard-to-synthesize sounds.

Running SAOL on Conventional Architectures v Lessons Learned from SAOL development: u Temporal typing of variables has the nice side effect of marking the inner loops. ²Typically, a-rate = 10 X to 100 X k-rate u A-rate code optimization : moving subexpressions into k-rate or i-rate. u SAOL semantics support a static heap. ²No recursion, all variables sp floats, no pointers. . . simplifies optimization. u Other researchers (Giorgio Zoia - ETH) focusing on blocking all a-passes for an instance, reducing overhead. v Processors with SIMD FP support (Intel SSE, AMD 3 DNow!) will be a good match.

Fixed-Function Hardware for SAOL Accelerators v Unlike MPEG-2 chips, DVD chips, etc. , its not clear how MP 4 -SA can be accelerated by rolling an ASIC. u Since every MP 4 -SA file is a new algorithm. v Common opcodes can be hardwired and the general characteristics of typical MP 4 -SA files could be leveraged to specialize a conventional processor design. u But the language is only six months old; execution frequencies are not known. v Reconfigurable computing architectures might hold promise (however, MP 4 -SA is all floating point).

Directions / Research Opportunities v Compiler optimizations for: SAOL and other languages with rate semantics u high-performance SIMD architectures u runtime code specialization u v Runtime scheduling under limited compute resources. v SAOL programming environments. v Physical modeling. v Automatic encoding.