Tools for Investigating Graphics System Performance Matthew Fisher

Tools for Investigating Graphics System Performance Matthew Fisher Steve Pronovost

Goal A video game runs slowly, skips frames, has high latency, etc. and the developers want to fix this problem n The problem is almost always a cascade of bottlenecks at the application, CPU, and GPU levels that is very challenging to investigate locally n We want tools that lets programmers solve these problems faster n

Approaches n Profiling – Rig the game events with logging or use an automatic profiler n PIX (for Windows and Xbox 360) – All calls by the game to the graphics API are logged n GPUView – OS logs all CPU, graphics kernel and graphics driver events

Profiling Manual profiling requires a significant amount of development effort n Polling-based automatic profiling can work reasonably well for CPU applications but doesn’t capture graphics or memory transfer events well n Percentage-based statistics (“you spent 45% of the time in function X”) can sometimes be useful and sometimes extremely misleading n

PIX Released by Microsoft as part of the Direct. X SDK n Multiple modes for investigating performance targeted at game developers n – Interactive mode – Frame logging – Frame capture and playback

PIX – Interactive Mode Various counters stream by as the game runs n You can change the counters, hope is to find that the observed problem correlates with one of the counters n

PIX – Interactive Mode

Commonly Used Counter Types Number, type, and size of draw primitive calls n Number of texture, vertex/index buffer locks, and what memory pool was locked n Object creation and destruction events n Allocated system and video memory n Frame latency, seconds per frame n Page faults n

PIX – Frame Capture Mode

PIX – Debug Pixel

Questions PIX is good at Are object locks causing the frame skipping problem users are experiencing? n Are we allocating too many resources we don’t use? n What are the API calls that are taking the longest time to execute? n Why was this pixel in the sky green? n

GPUView

Windows Display Driver Model The XP Display Driver Model required applications to cede control of the graphics infrastructure and was largely designed assuming a single 3 D application would be running n The Vista Display Driver Model added standard scheduling principles forcing applications to share control of graphics memory and compute resources n

GPUView The graphics model switch induced a variety of constraints on graphics applications and forced highly optimized graphics drivers to be restructured n Many games were running more slowly on Vista than they did on XP (~5% - 30% slower) n GPUView was designed to help investigate these problems and see what stage was causing the speed difference n

Event Tracing n The GPUView logger enables logging of a vast set of events in the OS, such as – All calls to the Windows graphics kernel • All resource creation, lock, destruction, etc. events • All command buffer submissions – Context switches (w/ stack trace and reason) – Kernel mode enter/exits (w/ stack trace) n World of Warcraft generates approximately 1 GB every 3 seconds

GPUView Without Any Graphics

Windows Display Driver Model Applications build up local command buffers n When these command buffers get big enough they are submitted to the application’s local graphics queue for processing n The graphics scheduler selects which application should be running on which graphics card and submits work to the corresponding hardware queue n

One Second of a Game

Setup

Multiple Applications Fighting

Simple Problems

Relatively Normal Execution

GPU Starvation

GPU Idle

Sleepy App

Huge Render Times (GPU Bound)

GPU and CPU Starvation

Answering Questions

Why Did Our Thread Context Switch?

Does Surface Allocation Cause Frame Stuttering?

Thoughts Surprisingly, the overhead of GPUView logging is pretty minimal and the traces often reflect the underlying problem well n The biggest advantage of GPUView over PIX is that PIX can’t tell you crucial things like when the GPU is blocked on the CPU n GPUView is excellent for telling you what part of the application needs optimization n

Driver Perspective n Provides a lot of detail to let display driver writers and the Direct. X graphics kernel diagnose problems with task submission, the command buffer submission threads, GPU preemption, video skipping, etc.