ICOM 6115 Computer Systems Performance Measurement and Evaluation
- Slides: 27
ICOM 6115: Computer Systems Performance Measurement and Evaluation August 11, 2006
Question p Describe a performance study you have done n p Work or School or … Describe a performance study you have recently read about n n n Research paper Newspaper article Scientific journal
Outline Objectives p The Art p Common Mistakes p Systematic Approach p (next)
Objectives (1) p Select appropriate evaluation techniques, performance metrics and workloads for a system. n n Techniques: measurement, simulation, analytic modeling Metrics: criteria to study performance p n ex: response time Workloads: requests by users/applications to the system
Objectives (2) p Conduct performance measurements correctly n Need two tools: load generator and monitor System Software Can it be observed? Can the actions be observed?
Objectives (3) p Use proper statistical techniques to compare several alternatives n One run of workload often not sufficient p n Many non-deterministic computer events that effect performance Comparing average of several runs may also not lead to correct results p Especially if variance is high Exec time Run
Objectives (4) p Design measurement and simulation experiments to provide the most information with the least effort. n p Often many factors that affect performance. Separate out the effects that individually matter. How many experiments are needed? How can the performance of each factor be estimated?
Objectives (5) p Perform simulations correctly n n Select correct language, seeds for random numbers, length of simulation run, and analysis Before all of that, may need to validate simulator
Outline Objectives p The Art p Common Mistakes p Systematic Approach p (done) (next)
The Art of Performance Evaluation p Evaluation cannot be produced mechanically n n p p Requires intimate knowledge of system Careful selection of methodology, workload, tools No one correct answer as two performance analysts may choose different metrics or workloads Like art, there are techniques to learn n n how to use them when to apply them
Example: Comparing Two Systems p Two systems, two workloads, measure transactions per second System A B Workload 1 20 10 Workload 2 10 20 They are equally good! p … but is A better than B? p Average 15 15
The Ratio Game p Take system B as the base System A B Workload 1 2 1 Workload 2 0. 5 1 A is better! p … but is B better than A? p Average 1. 25 1
Outline Objectives p The Art p Common Mistakes p Systematic Approach p (done) (next)
Common Mistakes (1) p Undefined Goals n n n p Biased Goals n n p There is no such thing as a general model Describe goals and then design experiments (Don’t shoot and then draw target) Don’t show YOUR system better than HERS (Performance analysis is like a jury) Unrepresentative Workload n n Should be representative of how system will work “in the wild” Ex: large and small packets? Don’t test with only large or only small
Common Mistakes (2) p Wrong Evaluation Technique n n p Inappropriate Level of Detail n n p Use most appropriate: model, simulation, measurement (Don’t have a hammer and see everything as a nail) Can have too much! Ex: modeling disk Can have too little! Ex: analytic model for congested router No Sensitivity Analysis n n Analysis is evidence and not fact Need to determine how sensitive results are to settings
Common Mistakes (3) p Improper Presentation of Results n p It is not the number of graphs, but the number of graphs that help make decisions Omitting Assumptions and Limitations n n Ex: may assume most traffic TCP, whereas some links may have significant UDP traffic May lead to applying results where assumptions do not hold
Outline Objectives p The Art p Common Mistakes p Systematic Approach p (done) (next)
A Systematic Approach 1. 2. 3. 4. 5. 6. 7. 8. 9. State goals and define boundaries Select performance metrics List system and workload parameters Select factors and values Select evaluation techniques Select workload Design experiments Analyze and interpret the data Present the results. Repeat.
State Goals and Define Boundaries p Just “measuring performance” or “seeing how it works” is too broad n p Ex: goal is to decide which ISP provides better throughput Definition of system may depend upon goals n n Ex: if measuring CPU instruction speed, system may include CPU + cache Ex: if measuring response time, system may include CPU + memory + … + OS + user workload
Select Metrics Criteria to compare performance p In general, related to speed, accuracy and/or availability of system services p Ex: network performance p n n n p Speed: throughput and delay Accuracy: error rate Availability: data packets sent do arrive Ex: processor performance n Speed: time to execute instructions
List Parameters List all parameters that affect performance p System parameters (hardware and software) p n p Workload parameters n p Ex: CPU type, OS type, … Ex: Number of users, type of requests List may not be initially complete, so have working list and let grow as progress
Select Factors to Study p Divide parameters into those that are to be studied and those that are not n n p Ex: may vary CPU type but fix OS type Ex: may fix packet size but vary number of connections Select appropriate levels for each factor n n n Want typical and ones with potentially high impact For workload often smaller (1/2 or 1/10 th) and larger (2 x or 10 x) range Start small or number can quickly overcome available resources!
Select Evaluation Technique Depends upon time, resources and desired level of accuracy p Analytic modeling p n p Simulation n p Medium effort, medium accuracy Measurement n p Quick, less accurate Typical most effort, most accurate Note, above are all typical but can be reversed in some cases!
Select Workload Set of service requests to system p Depends upon measurement technique p n n n p Analytic model may have probability of various requests Simulation may have trace of requests from real system Measurement may have scripts impose transactions Should be representative of real life
Design Experiments Want to maximize results with minimal effort p Phase 1: p n n p Many factors, few levels See which factors matter Phase 2: n n Few factors, more levels See where the range of impact for the factors is
Analyze and Interpret Data Compare alternatives p Take into account variability of results p n p Statistical techniques Interpret results. n n The analysis does not provide a conclusion Different analysts may come to different conclusions
Present Results Make it easily understood p Graphs p Disseminate (entire methodology!) p "The job of a scientist is not merely to see: it is to see, understand, and communicate. Leave out any of these phases, and you're not doing science. If you don't see, but you do understand communicate, you're a prophet, not a scientist. If you don't understand, but you do see and communicate, you're a reporter, not a scientist. If you don't communicate, but you do see and understand, you're a mystic, not a scientist. "
- Progress and performance measurement and evaluation
- Evaluation in progress
- Measurement and evaluation in human performance 5e download
- Vertical jump
- Yaesu ftm-400xdr vs icom 5100
- Icom dstar
- Icom
- Icom model project management
- Network performance measurement in computer networks
- Network performance measurement in computer networks
- Network performance measurement in computer networks
- Computer architecture performance evaluation methods
- The assessment of computer performance is called
- Concept map of measurement assessment and evaluation
- Prognostic test
- Strategic measurement and evaluation
- Definition of measurement in education
- Measurement and evaluation for health educators
- Approaches to measuring performance
- Defining performance and choosing a measurement approach
- Performance measurement in decentralized organizations
- Performance measurement in decentralized organizations
- Network performance measurement tools
- Nist sp 800-55
- Comparative performance measurement system
- Network performance measurement tools
- Quantum performance measurement
- Portfolio performance evaluation problems