HOW TO DO BAD ARCHITECTURE EVALUATION with apologies

HOW TO DO BAD ARCHITECTURE EVALUATION (with apologies to Dave Patterson) David A. Wood University of Wisconsin-Madison

Only use simulation • Simulation is the only effective evaluation technique • Conferences will take your papers without validation and without stating most of your assumptions • Never do detailed hardware design • If you can write it in C++, then it must be implementable • Never measure real hardware and software • That requires running billions of instructions, not the few million that really matter • Never use analytic models • Who needs insight when you can get exact cycle counts

Simulation Infrastructure • Option 1: Write your own simulator from scratch • You are the expert; no one else could do it as well • Everyone will be impressed that you did as much as an entire open source community • That way no one will ever know what assumptions you make • No one can ever reproduce your results, so they can’t be wrong • Why waste time developing software that others can use? • Option 2: Use an existing simulator as a black box • Don’t waste time understanding it, someone already did that • Of course the authors made the same simplifying assumptions that are appropriate for your study • Keep any extensions/improvements to yourself; open-source projects only work if someone else does the work

Never validate • Don’t waste time verifying your simulation results • Whoever wrote the simulator must have already done that • Simulators never have bugs • Any difference between a simulator’s results and an analytic model MUST be because the model is wrong

Benchmarks • Use yesterday’s benchmarks to design tomorrow’s computers • They’ve worked for 20 years, why change now? • Why develop new benchmarks? That wastes time that can be better spent re-analyzing old ones.

Principles to NOT do bad evaluation • Use all methods as appropriate • Computer evaluation methods are not one size fits all • More of the community’s activity should benefit the community • Expand the use of open source • Treat computer evaluation as a science • More reproducible results • Drive standardization and validation • Simulation interfaces and validated modules (e. g. , CPUs) • Integrate virtual machines and simulators • Get the best of emulation/measurement and simulation