Modern Game Development Oliver Franzke Javier Blazquez Lucas

Modern Game Development Oliver Franzke & Javier Blazquez Lucas. Arts ofranzke@lucasarts. com Stanford University February 6 th, 2009 jblazquez@lucasarts. com

Agenda • Introduction – Facts • ‘Next-gen’ graphics – Challenges – Realistic Rendering – Asset pipeline • Multithreaded games – Parallel game loop approaches

Introduction • Disclaimer This talk is not specifically about Lucas. Arts! It is an attempt to present typical challenges games companies have to deal with as well as common processes and techniques used in the industry. Please be aware that every company is unique and will therefore handle things differently.

Introduction • Modern Game Development Facts – Timeframe: 2 – 4 years – Team: 100+ (peak) – Budget: 15+ million $ – Number of source code lines: ~1 million – Code build times: up to 30 minutes – Size of assets: 100+ GB – Units sold: 2 – 3 million [VGChartz]

Introduction • Team structure Engineering Lead Project Lead Art Design Lead Gameplay AI Core Graphics Tools Network Concept Character Environment Animation Effects System Level Narrative Technical Artist Production Producer Product Manager Sound/Music Quality Assurance HR PR Finance

Introduction • Timeline (3 year production) – Concept phase (1 – 2 month) • Develop ideas and prototypes – Pre-production (18 month) • Prove all game-play features • Prepare engine and tools for production – Production (12 month) • Create final code and content – Finaling (4 - 5 month) • Remove bugs and polish – Support • Customer support, patch development and updates (DLC)

‘Next-gen’ graphics • Introduction • Realistic rendering • Data pipeline Image courtesy of Crytek Gmb. H

Challenges • Consumer expectations – High resolution models – Detailed textures – Huge environments • Rendering speed – HD resolution: up to 1920 x 1080 – At least 30 Hz (goal is 60 Hz)

Challenges • Platform diversity – Next-gen consoles (Xbox 360, Playstation 3) – Handhelds (Nintendo DS, PSP, i. Phone) – High-end PCs • Assets, assets… – Increased development complexity – Specialized code and art for every platform

Comparison • TFU character complexity – Main character (Apprentice) • ~15. 000 triangles • 3 x 1024² textures – Other characters: 7 k – 10 k • Comparison – Quake 3: 800 – 900 (1999, id Software) – Half Life 2: 4 k – 8 k (2004, Valve) [Stirling 07] – Gears of War: 10 k – 15 k (2006, Epic) [Stirling 07]

Comparison • TFU character complexity – Main character (Apprentice) • ~15. 000 triangles • 3 x 1024² textures – Other characters: 7 k – 10 k • Comparison – Wolfenstein 3 D: 2 (1992, id Software) – Quake 3: 800 – 900 (1999, id Software) – Half Life 2: 4 k – 8 k (2004, Valve) [Stirling 07] – Gears of War: 10 k – 15 k (2006, Epic) [Stirling 07]

Comparison • TFU level complexity – 500 k triangles per frame in opaque pass – Memory limitation – Next sector is streamed in • Comparison [Stirling 07] – Virtua Fighter 5: 100 k – 300 k (2006, Sega) – Lost Planet: ~500 k (2007, Capcom)

Realistic Rendering • Goals – Imitation of light transport (kind of) – Create convincing images – Increased immersion • Solution – Solve the rendering equation – High quality approximation

Realistic Rendering • Rendering Equation [Kajiya 86] [Wikipedia 1]

Realistic Rendering • How hard can it be to solve? – Recursive integral equation – Spatial, temporal and spectral domain • Very hard!

Realistic Rendering • Constraints / Typical Simplifications – Must be controllable by artists!!! – Limited changes in the lighting condition: – BRDF is time independent: – Spatial relationships are static

Realistic Rendering • Solution – Distinguish between static / dynamic lighting and geometry – Pre-calculate static lighting information offline + No expensive calculations during run-time + Quick access - Low frequency lighting only - Memory intensive – Add dynamic lighting during run-time - No global lighting effects (local model) + Simplified lighting model

Realistic Rendering • Recap: Lightmaps

Realistic Rendering • ‘Radiosity Normal Mapping’ [Mc. Taggart 04] [Mitchell 06] – Source™ Engine shading model – Developed by Valve for Half Life 2 Image courtesy of Valve

Realistic Rendering • ‘Radiosity Normal Mapping’ – Use of basis function projection for lighting – Basis defined in tangent space y Tangent space: z x

Realistic Rendering • ‘Radiosity Normal Mapping’ (Offline) – Project lighting into basis – Assumptions • Surface is smooth • BRDF is view independent (diffuse)

Realistic Rendering • ‘Radiosity Normal Mapping’ (Runtime) – Project perturbed normal into basis – Reconstruct incident lighting Normal map #1: Normal map #2:

Realistic Rendering • Lightmaps vs. ‘Radiosity Normal Mapping’

Realistic Rendering • ‘Radiosity Normal Mapping’ + Low resolution sampling (Vertex, Lightmaps) + Easy to composite with dynamic lights + Works with other bump techniques - Only low-frequency diffuse lighting captured - More memory required (basis matrix vs. color)

Realistic Rendering • XNA demo is available (including source)!

Realistic Rendering • ‘Ambient Cube’ [Mc. Taggart 04] [Mitchell 06] – Directional ambient lighting for objects – Simplified irradiance volume [Greger 98] +y -x -z +z +x -y

Realistic Rendering • ‘Ambient Cube’ (Example) Ambient cube #2 Ambient cube #1

Realistic Rendering • ‘Ambient Cube’ (Example) Image courtesy of Valve

Asset Pipeline • ‘Next-gen’ challenge: Amount of assets • Asset-to-data transformation – Pre-calculation (lighting, PVS, nav mesh…) – Adaption for console hardware – Packing – Compression • Transformations depend on the game – Unique pipeline for every game

Asset Pipeline • Problem: Very time consuming • Solution: Build farm – Distribute asset transformation – Collection of dedicated machines – Server-client architecture • Pros and Cons + Speed-up + Incremental builds (fragment caching) + Scalable – Previewing assets can be complicated – Extra development cost

Conclusion • Complexity is increasing • Basis function projection used to improve image quality – HL 2 2 basis, Spherical Harmonics, Wavelet • Amount of data is a big problem • Looking ahead – Sub-surface scattering – Facial animation techniques – GPU used for general purpose computations

Multithreading • Why a multithreaded game is important – Frame time on a typical game is very short • 16 ms or 33 ms max for everything • Can’t drop frames – Very noticeable, both visually and audibly – Might disrupt the player’s actions • Increasing amount of computation per frame – More precise physics – Realistic enemy behavior – Larger and more complex worlds

Multithreading • One thread is no longer enough – The console CPU’s single-threaded performance is not very good. – You would be wasting ~80% of the CPU processing capacity. – Your game wouldn’t live to expectations. – Future trend is to add more cores, not to have faster ones.

CPU Comparison Clock Frequency # of cores / HW threads Cache Memory Bandwidth SIMD Registers Out-of-order Execution Intel Core i 7 PS 3 PPU PS 3 SPU Xbox 360 3. 2 GHz 4/8 1/2 6 3/6 256 KB L 2 / 8 MB L 3 512 KB L 2 N/A (256 KB LS) 256 KB L 2 / 2 MB L 3 25. 6 GB/s 22. 4 GB/s 8 x 128 -bit SSE 4. 2 32 x 128 -bit Alti. Vec 128 x 128 -bit VMX 128 Yes No No out-of-order execution kills single-threaded performance

Multithreading Challenges • What makes parallelizing games difficult: – Many different kinds of processes: • Graphics, Physics, Audio • AI, Gameplay – Lots of shared state: • World geometry, game objects, navigation graphs, player state, etc. – Game loop has traditionally been very sequential in nature.

Typical Game Loop Process Input Update Player Gameplay AI Animation Audio Physics Render

Parallelizing the Game Loop • Several approaches: – Coarse-grained multithreading • Long processes on their own thread – Task-based multithreading • Short jobs scheduled dynamically • Well-defined dependencies – Hybrid

Coarse-grained Approach • Move certain systems to their own thread – Good candidates: Rendering, Physics, Audio • Provide sync points with the main thread – Retrieve computed data – Update state – Kick off next frame’s work • Relatively straightforward to set up

Render frame 1 Sync Render frame 0 Input Player Logic AI Frame 1 Anim Audio Input Sync Physics Main Render Coarse-grained Loop Physics for frame 2 Player Frame 2 Physics for frame 3

Coarse-grained Approach • Problems: – Main thread modifying state mid-frame • Defer state-changing operations – Use a command queue for the system – Main thread expecting results mid-frame • Return last frame’s data – Use a cache / double-buffering scheme – Expose a deferred query API • Might introduce off-by-one-frame issues

Coarse-grained Approach • Problems (cont. ): – Worker thread calling back into game • E. g. collision response • Can happen at any time – Queue callbacks for later processing – Or make systems thread-safe – Uneven hardware thread usage • Difficult to schedule work properly • E. g. audio thread usually idle – Not the best suited for PS 3

Task-based Approach • Code is partitioned into small tasks – Tasks operate on streams of regular data • Well-defined inputs and outputs • Limited access to other resources – Tasks can be scheduled independently • Ordering is defined by data dependencies

Task-based Approach • Custom scheduler for running tasks – Build / traverse dependency graph – Schedule independent tasks in parallel • Opportunity for additional simulations – Cloth – Fluids • Best suited for PS 3 / Xbox 360 architecture

Task-based Approach Picture from [Turner 07]

Task-based Approach • Problems: – Typical game code is very difficult to partition in tasks • Lots of dependencies, very order-sensitive • Object-oriented designs do not help – Many small objects – Required data is in disparate memory locations – Not very suitable for stream processing • Focus is usually on maintainability, not performance

Conclusions • Game must be parallelized as much as possible – Including systems that were traditionally sequential – Usually requires hand-tailored solutions – OOP sometimes gets in the way

Q&A

References (1) • [Stirling 07] ‘Yes, but how many polygons? ’, Rick Stirling, Blog, 2007, http: //www. rsart. co. uk/2007/08/27/yes-but-how -many-polygons/ • [VGChartz] http: //www. vgchartz. com/ • [Wikipedia 1] http: //en. wikipedia. org/wiki/Rendering_equation • [Kajiya 86] James Kajiya, “The rendering equation”, SIGGRAPH, 1986, Pages 143 – 150

References (2) • [Mitchell 06] Jason Mitchell, Gary Mc. Taggart and Chris Green, “Shading in Valve’s Source™ Engine”, SIGGRAPH, 2006, Advanced Real-Time Rendering in 3 D Graphics and Games Course • [Mc. Taggart 04] Gary Mc. Taggart, “Half Life 2 / Valve Source™ Shading”, Game Developer Conference, 2004 • [Greger 98] Gene Greger, Peter Shirley, Philip M. Hubbard and Donald P. Greenberg, “The Irradiance Volume, ” IEEE Computer Graphics & Applications, 1998, Pages 32 -43

References (3) • [Acton 07] Mike Acton, “SPU Shaders Introduction”, Insomniac R&D • [Acton 08] Mike Acton, “More on SPU Shaders”, Insomniac R&D • [Turner 07] Randall Turner, “Saints Row Scheduler”, GDC 2007 • [Valenzuela 08] Joe Valenzuela, “SPUs for gameplay”, Insomniac R&D

Additional References • [Ramamoorthi 01] Ravi Ramamoorthi and Pat Hanarahan, An efficient representation for irradiance environment maps, SIGGRAPH 2001 • [Sloan 02] Peter-Pike Sloan, Jan Kautz and John Snyder, Precomputed Radiance Transfer for Real -Time Rendering in Dynamic, Low-Frequency Lighting Environments, SIGGRAPH 2002 • [Green 03] Robin Green, Spherical Harmonic Lighting: The gritty Details, 2003