Benchmarking Benchmarks Project 3 cs 533 Introduction Benchmarks
Benchmarking Benchmarks Project 3 cs 533
Introduction • Benchmarks are uniform way of comparing • • performance on different computer systems However, if benchmark is not setup properly, it will not adequately reflect actual application (workload) performance You are to test the effectiveness of a benchmark of your choosing with reflecting the application workload of your creation
Overview • Select benchmark • Install benchmark • Run benchmark • Create application-level benchmark • Report results
Select Benchmark • “Well-known” – described peer-reviewed forum – published results for computer systems for comparison • Possibilities: – – – – HINT SPEC CPU 2000 Linpack BYTEMark Quake 3 Bonnie …
Install Benchmark • Download • Read documentation • Configure • Run validation tests (as appropriate) • Figure out how to run it • Figure output generated
Run Benchmark • Run benchmark – Appropriate flags, number of iterations • At least two different systems – Systems should be selected so as to get “different” results – Ex: CPU speeds, memory sizes, architectures …
Create Application Level Benchmark • Create your own workload • Design so can compare with benchmark • Examples: – Compile large software system – Run large simulation – Perform numerous database transactions
Report Results (1 of 2) • Background on well-known benchmark – – who, where, when invented where published what the intent of the benchmark is measures of performance and some published performance results. • Descriptions of the systems you run – – CPU type and speed memory size cache levels and sizes operating system
Report Results (2 of 2) • Details on your application level benchmark – – Type of workload Length of run Number of iterations Measure of performance • Results and analysis depicting results clearly – tables and graphs – appropriate stats and presentation techniques • Conclusions on the meaning of your data – data that supports your conclusions – subjective opinions you may have
- Slides: 9