Stealing Sound The Use of Generative Music in

Stealing Sound: The Use of Generative Music in the Next Thief Paul Weir Audio Director, Composer & Sound Designer, Eidos-Montreal paul. weir@eidosmontreal. com

- The benefits of generative music - Place generative music within a theoretical framework - Issues regarding current game music practice - Demonstrate GRAMPS - Best practice for similar systems

Hitchcock in his 1944 film 'Lifeboat' was not keen on having music, where would the music come from in the middle of an ocean? The composer David Raksin's response was "Go back and ask him where the camera comes from and I'll tell him where the music comes from!"* On many game projects it feels like we haven't reached even that level of analysis for the audio. *http: //filmsound. org/articles/horrorsound. htm

In 2007 The Sound Agency played a generative ambient composition in Glasgow Airport. * The music was played one day on, one day off and sales recorded for two months. Dramatic impact on sales, between 5 -10% Music clearly alters behaviour, we must use all the tools available. * http: //www. thesoundagency. com/

As with previous Thief titles, use of audio is integral to our game Trying to build on the excellent work of Eric Brosius Audio treated VERY seriously in the team (refreshing) Deeply embedded in the game design, for example each environment has its audio mapped for key moments, dynamic and dramatic range, music cues, before it's developed.

I run Earcom Ltd (www. earcom. net), worked on over 50 titles as composer, sound designer and voice director. Unusual set up as I'm the Audio Director, composer and sound designer on the title but not a typical outsource model, I'm very closely integrated into the team. Daily webcam conferences, more like an off-site office. Supported by two excellent on-site sound integrators and a programmer.

Kotaku article – The Year I Gained The Courage To Ignore Videogame Music* “Sometimes I would listen to the game, but lower the volume and reach for my i. Phone if I'd failed at a challenge. If I was going to have to re-play part of a game, I reasoned, I didn't have to re-listen to it as well. I could listen to something else, even just for 30 second, 30 minutes or however long I was repeating something I'd seen and heard but not correctly done before. ” *http: //kotaku. com/5730637/the-year-i-gained-the-courage-to-ignore-video-game-music

May not agree with the article (about 70% of the comments disagreed) Neither should we dismiss the argument out of hand As composers, we need to pro-actively find ways to engage our audience, creating a great composition is not the end point Player's reaction affected by repetition, the mix, cue points, relationship with the sound effects and voice, dynamic range, etc.

Several issues facing game music, here are four of particular interest. . .

#1 Repetition Habituation - something that's there all the time starts to disappear. Oliver Sacks in Musicophilia (2007) talks about a man who becomes deaf to his wife A question of balance, too much may do more harm than good

“There is then, after repeated hearing, an optimum point of appreciation after which the law of diminishing aesthetic returns sets in. The attainment of [this point] may be accelerated or delayed, but in the long run it is psychologically inevitable. ” - Russell, P Getz, The Effects of Repetition on Listening Response, Journal of Research in Music Education, Vol 14, Issue 3, 1966

When only using linear music, compromises have to be made, as George Sanger has written: "A normal intelligent game developer will budget for about an hour of music in his game, more or less…That hour is destined to be stretched over, let’s say, a 40 -hour entertainment experience. That’s the normal, intelligent way to do things, and it’s pretty stinking ridiculous. "* *George Sanger, First Things First, Second, and Third: It's About Repetition in Game Audio, 2003

Ways to stave off the inevitable: - Live stem mixing - Branching structures - Stings - Red Dead Redemption great music, but all had to be written in A minor and 130 bpm

#2 Time & Structure Question over the mapping of music onto game structure and their relationship with time. - Music most linear of all the arts - 'Temporal friction' “. . . music makes us vividly aware of the duration and succession of events and our sense of change and continuity. ” - The New Grove Dictionary of Music

Interesting parallel with that other traditionally linear medium, story telling: “There's a conflict between interactivity and storytelling. Most people imagine there's a spectrum between conventional written stories on one side and total interactivity on the other. But I believe that what you really have are two safe havens separated by a pit of hell that can absorb endless amounts of time, skill, and resources. ” - Walter Freiteg quoted in Charles Platt's article 'Interactive Entertainment: Who writes it? Who reads it? Who needs it? ', Wired Magazine 1995

#3 Music & Sound Design There's rarely an interplay between the music and sound design - We can cross-fertilise (Dead Space) - Procedural audio - Too often 100% music and 100% sfx, film has a more defined filtering process - Shopping list syndrome

- Limited dynamic and dramatic space, use it wisely - Game music at its worst is like an annoying child - Right ingredients but don't always have the cook (audio director)

#4 Stylistic Range Film music can reflect an inner psychological world Literal and non-literal music Game music tends towards the literal & reinforcing what we already know (the Steiner model)

“Whereas composers like Max Steiner. . . used an intellectual approach to scoring -- where, for example, the villain is announced with stereotypical 'villain music'. . . Herrmann's was a passionate response: music that commented on the interior psychology of a character in a scene. ”

Stylistically the John Williams style score still dominates (ban the anvil) Big budget games very rarely experiment with musical styles Film is also largely derivative but there ARE notable exceptions, usually driven by a strong director Five examples from recent films, can you name them?

Why do games tend to be musically conservative? - A hard sell to do something even a little different - Needs close relationship with the composer from early on - Design by committee, not by auteur (some notable game exceptions) - The role of the audio director is essential

Summary: - Too much repetition - The musical structure conflicts with the game structure - Effects and music exist in ignorance of each other - Conservative music styles and use of music

Generative, procedural, algorithmic, interactive, emergent music. . what the hell does it all mean? Algorithmic tends towards having an intelligent musical 'brain' - Noatikl (Koan) - Max. MSP / Pd - Impromptu The more sophisticated the algorithm, the more musically restrictive the outcome Keith Jarret Impromptu example (http: //homepage. mac. com/digego/study_in_keith. mov)

Randomised music far from new, arguably the norm pre-19 th Century - 'Mozart's Dice' - John Cage's chance music - Charles Ive's aleatoric music - Generative music needs a mechanical system to work – clockwork music - Need to balance the amount of change over time

“. . . music that is maximally surprising would not, in fact, be maximally interesting or “good. ” Perhaps the most distinctive aspect. . . is the sense of freedom engendered by the music. This sense does not arise just from surprising listeners, but from doing so in appropriate (and moderate) ways that involve manipulating expectation through the transformation of both large and fine grain structure. ” - Rivers of Consciousness: The Nonlinear Dynamics of Free Jazz, Borgo & Goguen, 2004

Advantages of a 'live' music system: - Mapping of game stimuli and states into the music system (but avoid 'Mickey Mousing') - Can react, reinforce or lead the narrative - Transitional music states become part of the composition - Any element of the music can be changed in real-time, such as the mix, individual sounds, scale, tempo

GRAMPS Generative Reactive Adaptive Music Playback System Generating Real-time Audio Motivating Players System Great Rocking Audio Melting Players Speakers Great Red-haired Amazonian Maiden Putty Stove

What is GRAMPS? - A randomised file player with basic logic - GRAMPS Edit is the GUI version - Sits on top of Wise going through their sound effects system (doesn't use their interactive music system) - Liberal use of RTPCs - Built from a long history of generative systems

Master control panel - Preview interpolation time between moods - Set tempo and volume per mood - Reserve game objects (number of Wise voices) - Set scale (new scales can be defined in the XML) - Set sound to be triggered on piece start

Channel Panel

Each channel panel has the following features: - Set Wise event (can represent any number of sounds) - How frequently the event is likely to play (value is in beats) - How many notes can be triggered simultaneously - If more than one note triggered, spread can force them apart - Set octave, if both boxes are 0 then don't transpose, if both boxes the same value other than 0 then transpose, if both values are different, snap to the scale. - Randomise volume, pan and LPF - Chord forces notes to snap to chords (listed in the XML) - Stagger adds a random delay when the piece is first started

Interpolation First slider represents the point at which the interpolation starts Second slider is how long it takes for the interpolation to go from one mood to another Interpolation can be controlled by game state, environment position, trigger event, clock, etc.

Advantages: - Encourages top down composition and therefore different musical ideas than when working with a sequencer - Mood system gets round issue that generative systems are good at micro change, but not macro - Inputs are open and can be wired to a large number of states - Can change any element of the mix at any point - Works well when combined with linear pre-rendered music

Disadvantages: - Can be challenging to get clarity in the mix - As it's real-time, uses valuable voices (game objects in Wise) - Time consuming to test the mix by nature of it being random - Generative systems tend to peak, so useful to have a limiter on the music bus - Important to mix musical styles and include more rhythmic prerendered or thematic material for dramatic contrast

Best practice: - Extra emphasis on timbre and the relationship between sounds - As with sound effects, best to normalise all sounds - Go as granular as possible - Take care that the music and sound effects work together in the mix and don't make the generative music too dense - Traditional sequencers are useful for prototyping ideas

Summary: - Generative music an alternate approach but not the only one, however it can help alleviate some serious issues facing game music - It's a creative solution, not a technical one, there's no magic bullet - Relationship between sfx and music even more important, in Thief all the sound effects are entirely original - Exploit the advantages of having a live music playback system but player need not be aware