Variability in Architectural Simulations of Multithreaded Workloads Alaa

Variability in Architectural Simulations of Multi-threaded Workloads Alaa R. Alameldeen and David A. Wood University of Wisconsin-Madison {alaa, david}@cs. wisc. edu http: //www. cs. wisc. edu/multifacet/ 10/14/2002

Motivation Ø Experimental scientists use statistics Ø Computer architects in simulation experiments don’t! Ø Why ignore statistics? l Simulations are deterministic This HPCA 2003 can lead to wrong conclusions! Alaa Alameldeen and David Wood 2

Workload Variability OLTP HPCA 2003 Alaa Alameldeen and David Wood 3

Workload Variability Slower memory is better! OLTP HPCA 2003 Alaa Alameldeen and David Wood 4

What Went Wrong? Ø Many possible executions for each configuration Ø Why? Different timing effects l l l OS scheduling decisions Different orders of lock acquisition Different transaction mixes Ø This is magnified by short simulations Variability HPCA 2003 can lead to wrong conclusions Alaa Alameldeen and David Wood 5

Overview Ø Variability is a real phenomenon for multithreaded workloads l Runs from same initial state can be different Ø Variability l Simulations are short Ø Our l is a challenge for simulations solution accounts for variability Multiple runs, statistical techniques HPCA 2003 Alaa Alameldeen and David Wood 6

Outline Ø Motivation and Overview Ø Variability in Real Systems l Time and Space Variability Ø Variability in Simulations Ø Accounting for Variability Ø Conclusions HPCA 2003 Alaa Alameldeen and David Wood 7

What is Variability? Ø Differences between multiple estimates of a workload’s performance Ø Time Variability: l Performance changes during different phases of a single run Ø Space l Variability: Runs starting from the same state follow different execution paths HPCA 2003 Alaa Alameldeen and David Wood 8

Time Variability in Real Systems One-second intervals OLTP HPCA 2003 Alaa Alameldeen and David Wood 9

Time Variability Example (Cont’d) Ø How l is this handled in real experiments? Solution: Run your experiment long enough! One-minute intervals OLTP HPCA 2003 Alaa Alameldeen and David Wood 10

Space Variability in Real Systems One-second averages 5 runs OLTP HPCA 2003 Alaa Alameldeen and David Wood 11

Space Variability Example (Cont’d) Ø How l is this handled in real experiments? Same Solution: Run your experiment long enough! One-minute averages 5 runs 16 -day simulation OLTP HPCA 2003 Alaa Alameldeen and David Wood 12

Outline Ø Motivation and Overview Ø Variability in Real Systems Ø Variability in Simulations l l l Simulation Infrastructure Injecting Randomness The Wrong Conclusion Ratio Ø Accounting for Variability Ø Conclusions HPCA 2003 Alaa Alameldeen and David Wood 13

Simulation Infrastructure Ø Workloads l Ø Ø Target System: E 10000 -like 16 -node system Full System Simulation l l l Ø Two scientific and five commercial benchmarks Virtutech Simics running Solaris 8 on SPARC V 9 A blocking processor model (Simics) An Oo. O processor model (TFSim – Mauer et al. , SIGMETRICS’ 02) Memory system simulator l MOSI invalidation-based broadcast coherence protocol (Martin et al. , HPCA-02) HPCA 2003 Alaa Alameldeen and David Wood 14

Simulating Space Variability? Ø Simulations are deterministic Ø Variability cannot be ignored for multithreaded applications l l One execution may not be representative Execution paths affect simulation conclusions We HPCA 2003 need to obtain a space of results Alaa Alameldeen and David Wood 15

Injecting Randomness Ø We introduce artificial random perturbations in each simulation run Ø For each memory access, latency in nanoseconds becomes Latency + r (r = -2, -1, 0, 1, 2 nanoseconds, uniform dist. ) Ø Roughly models contention due to DMA traffic Ø Other methods are possible HPCA 2003 Alaa Alameldeen and David Wood 16

Simulated Space Variability 20 runs ~10 hrs sim. Space HPCA 2003 variability exists in our benchmarks Alaa Alameldeen and David Wood 17

Quantifying Variability: The Wrong Conclusion Ratio (WCR) 20 runs 50 Xacts OLTP WCR (16, 32) = 18% Ø WCR (16, 64) = 7. 5% Ø WCR (32, 64) = 26% Ø HPCA 2003 Alaa Alameldeen and David Wood 18

Outline Ø Motivation and Overview Ø Variability in Real Systems Ø Variability in Simulations Ø Accounting for Variability Ø Conclusions HPCA 2003 Alaa Alameldeen and David Wood 19

Confidence Intervals Ø Definition: l Range of values expected to include population parameter (e. g. mean) Ø Confidence l Probability that true mean lies inside confidence interval Ø For l Probability: the same confidence probability: Sample Size ↑ → Confidence Interval ↓ HPCA 2003 Alaa Alameldeen and David Wood 20

Accounting for Space Variability OLTP HPCA 2003 Alaa Alameldeen and David Wood 21

Accounting for Space Variability OLTP Simple solution: Estimate #runs such that confidence intervals do not overlap Ø Tests of hypotheses can be used (paper) Ø HPCA 2003 Alaa Alameldeen and David Wood 22

Conclusions Ø Short runs of multi-threaded workloads exhibit variability Ø Variability can lead to wrong simulation conclusions Ø Our Solution: l l l Injecting randomness Multiple runs Apply statistical techniques HPCA 2003 Alaa Alameldeen and David Wood 23

Backup Slides HPCA 2003 Alaa Alameldeen and David Wood 24

Effects of OS Scheduling HPCA 2003 Alaa Alameldeen and David Wood 25

WCR Definition Ø Percentage of comparison simulation experiments that reach a wrong conclusion Ø The correct conclusion is the relationship between averages of the two populations Ø WCR can be used to estimate the wrong conclusion probability for single experiments HPCA 2003 Alaa Alameldeen and David Wood 26

Confidence Intervals - Equations Ø The confidence interval for the mean of a normally distributed infinite population: Ø Sample Size needed to limit mean relative error to r: HPCA 2003 Alaa Alameldeen and David Wood 27

Hypothesis Testing Ø Tests whethere is no difference between two population means l Hypothesis: μ 32 = μ 64 tests whether the two means of the 32 and 64 ROB configurations are different Ø Hypothesis is tested using sample means and variances Ø If hypothesis rejected Our conclusion is significant HPCA 2003 Alaa Alameldeen and David Wood 28

Accounting for Time Variability Ø Is time variability caused by the same effects that cause space variability? l Use Analysis of Variance (ANOVA) Ø If time variability is caused by different effects, we need to obtain a time sample l Observations obtained from different starting points HPCA 2003 Alaa Alameldeen and David Wood 29

Multi-threaded Workloads and Simulation Ø Multi-threaded l l workloads are important Workloads for commercial servers New architectures support multi-threading Ø Performance metrics are different from traditional benchmarks l l Throughput-oriented (transactions) IPC is not appropriate (idle time!) Simulation Challenge: Comparing systems running multi-threaded applications HPCA 2003 Alaa Alameldeen and David Wood 30

Simulation of Multi-threaded Workloads Ø Simulation l is slow! We cannot simulate the whole workload Ø Solution: l l l Run for a fixed number of transactions Measure the per-transaction runtime (cycles per transaction) Use to compare different systems HPCA 2003 Alaa Alameldeen and David Wood 31