Lecture 2 Technology Trends and Performance Evaluation Performance
- Slides: 28
Lecture 2: Technology Trends and Performance Evaluation Performance definition, benchmark, summarizing performance, Amdahl’s law, and CPI
Major Faces in Today’s Market u Desktop computers u Servers u Embedded computers – Office, personal computing, multimedia, etc. – Cost-performance trade-off – Providing larger-scale and more reliable file and computing service – Designed for performance, availability, and scalability – Networking switches, printer, palm, cell phone, etc. – Real-time performance requirements – Lost cost and low power
Technology Trends Implementation technologies change dramatically – – Integrated circuit logic technology Semiconductor DRAM Magnetic disk technology Network technology Software compatibility: software is more expensive than hardware
Cost, Price, and Their Trends u Cost and price may determine if a computer product will be successful in markets u In many cases cost is the single important factor in design considerations – Add a new feature or not? – Trade performance with cost and price Especially true for desktop and embedded market
Processor Performance Trends
Processor Price Trend
DRAM Price Trend
What Does Performance Mean? u Response time – A simulation program finishes in 5 minutes u Throughput – A web server serves 5 million request per second u Other metrics – MIPS (million instruction per second) – MFLOPS – Clock frequency
Execution Time u Processor design is concerned with processor consumed by program execution. Shorter execution time=> – Shorter response time – Higher throughput u Execution time = #inst×CPI×Cycletime – What affects #inst, CPI, and cycle time? – Almost all designs can be interpreted u Any other metrics is meaningful only if consistent with execution time
Performance of Computers Performance is defined for a program and a machine. How to compare computers? Need benchmark programs: – Real applications: scientific programs, compilers, text-processing software, image processing – Modified applications: providing portability and focus – Kernels: good to isolate performance of individual features u Lmbench: measure latency and bandwidth of memory, file system, networking, etc. – Toy benchmarks – Synthetic benchmarks: matching average execution profile
Performance Comparison “X is n times faster than Y”: n: speedup if we are considering an enhancement, optimization, etc. u What does “improving” mean? u – Improve performance: decrease execution time, increase throughput – Improve execution time: decrease execution time – Degrade performance: the reverse of the above; brings negative speedup
Benchmark Suite u Benchmark suite is a collection of benchmarks with a variety of applications – Alleviating weakness of a single benchmark – More representative for computer designers to evaluate their design – Benchmarks test both computer and compilers, and OS in many cases u u u Desktop benchmarks: CPU, memory, and graphics performance Sever benchmarks: throughput-oriented, I/O and OS intensive Embedded benchmarks: measuring the ability to meet deadline and save power
Summarizing Performance Given the performance of a set of programs, how to evaluate the performance of machines? A B C P 1 (secs) 1 10 20 P 2 (secs) 1000 100 20 Total (secs) 1001 110 40 u Which computer is the “best” one?
Arithmetic Mean u Total execution time / (number of programs) – Simple and intuitive – Representative if the user run the programs an equal number of times
Weighted Arithmetic Mean u Give (different) weights to different programs – Considering the frequencies of programs in the workload
Geometric Means u u Based on relative performance to a reference machine Relative performance is consistent with different reference machines – If C is 2 x faster than B (using B as the reference), B is 2 x faster than A (A as the reference), then C is 4 x faster than A (A as the reference)
Harmonic Mean u Given speedups s 1, s 2, …, s_n, the average speedup by harmonic mean is n / (1/s 1 + 1/s 2 + … + 1/s_n) Why not arithmetic mean?
Amdahl’s Law We know about performance: defining, measuring, and summarizing How to maximize performance gains from the beginning in our design? u Principle: Make the Common Case Fast!
Amdahl’s Law u Predict overall speedup from “local speedup” by an enhancement, provided the frequency to use the enhancement is know. – “Local speedup” is related to design and optimization objectives, like to double CPU frequency, to reduce cache latency by half
Amdahl’s Law
Equation Based on Instruction Types
Make Design Choice Using CPU Time Equation Assume we need to improve the performance of a graphics engine: FP FPSQR Other Frequency 25% 2% 75% CPI 4. 0 20 1. 33 Alternative 1: CPIFPSQR 20 2 Alternative 2: CPIFP 4 2. 5 Which one is better? Calculate speedups.
Amdahl’s Law Choice one: Speed up FP Square root by 10 x Choice two: Speed up all FP instruction by 1. 6 x 20% time are used by FP Square root, 50% for all FP inst Which choice is better? Implication: Optimizing for the common case first
SPEC CPU Benchmark u SPEC: Standard Performance Evaluation Corporation u CPU-intensive benchmark for evaluating processor performance of workstation u Four generations: SPEC 89, SPEC 92, SPEC 95, and SPEC 2000 u Two types of programs: INT and FP u Emphasizing memory system performance in SPEC 2000
SPEC CPU 2000 Profiling Dynamic instruction mix Instruction Load int Store int Load fp Store fp Add All fp inst Cond br. All ctrl inst Int avg 26% 10% 19% 12% 16% FP avg 15% 2% 15% 7% 23% 41% 4% 4%
Other SPEC Benchmarks u SPECviewperf and SPEapc: 3 D graphics performance u SPEC JVM 98: performance of clientside Java virtual machine u SPEC JBB 2000: Server-cline Java application u SPEC WEB 99: evaluating WWW servers u SPEC HPC 96: parallel and distributed computing
Server Benchmarks u SPEC CPU 2000, WBB 99, SFS 97 u TPC Measuring the ability of a system to handle transactions – TPC-C: online transaction processing (OLTP) benchmark (for bank systems) – TPC-H: ad hoc decision make support – TPC-R: decision make support with standard queries – TPC-W: simulating business-oriented transactional web server
Embedded Benchmark u EEMBC (Embedded Microprocessor Benchmark Consortium) benchmarks – Based on kernel performance – Five classes: automotive/industrial, consumer networking, office automation, and telecommunications Embedded benchmarks are not mature
- 01:640:244 lecture notes - lecture 15: plat, idah, farad
- Hardware technology trends
- Ethical and social issues in the digital firm
- Progress and performance measurement and evaluation
- Evaluation in progress
- Measurement and evaluation in human performance 5e download
- Measurement and evaluation in human performance
- Performance management lecture
- Wbb99
- Evaluation of technology learning
- Computer architecture performance evaluation methods
- Portfolio performance evaluation problems
- Sales force performance management
- Are criteria for evaluating performance of channel members
- Evaluation of computer system
- Oscd usmc
- Direct support professional performance evaluation
- Performance evaluation for decentralized operations
- Evaluating purchasing performance
- Citibank performance evaluation case study
- Housekeeping items meeting
- Unit 6 principles of management
- Stronge leader effectiveness performance evaluation model
- Performance evaluation using variances from standard costs
- Library staff performance evaluation
- Channel performance evaluation
- Performance evaluation model
- Chra europe region employment
- Unm pep