Performance Testing vs Performance Monitoring Jan Warren Enbridge

Performance Testing vs. Performance Monitoring Jan Warren Enbridge April 2009

What we shall talk about • Some of the many terms……. . • What the two disciplines mean • Comparison and contrast • Where they overlap (or do they? ) • Throw in a little ITIL • Our tools • Nirvana in each world • In a joint ideal world……. . • Conclusion • Questions

Top of the food chain • • • Performance Management System Measurement Management System Assurance Anything else?

Performance Testing • AKA Quality Assurance (or Quality Control) • Performance testing: Testing conducted to evaluate the compliance of a system or component with specified performance requirements. [IEEE] • Performance testing, unit testing, system testing, integration testing, functional testing, regression testing, load testing, stress testing, volume testing

Performance Monitoring • AKA Quantity Assurance? • Monitoring of the End User Experience (EUE), monitoring of system infrastructure (servers, network, switches, …. . )

Compare and Contrast Testing • • • Performance and accuracy Pre-production Controlled Load (forced/simulated) Agent based Agentless Network Server Capacity definition SLA baselining Recreate production issues Reproducible measurement under controlled conditions Monitoring • • • Performance and availability Production Normal load Agent based Agentless Network Server Capacity planning SLA monitoring Diagnose production issues Live real time snapshot of “in service” performance

Where do they overlap (or do they)? • • • NOT the environment The Medium (server, network, etc. ) NOT the role of application folks NOT in terms of load NOT in their role i. e. Testing vs. Monitoring Possibly the tools

ITIL • Performance Testing – Release Management • ICT Infrastructure Management – Deployment • Performance Monitoring – Service Delivery • Service Management – Service Level Management

Our tools Testing • HP – – – Quality Center Quick Test Pro Load. Runner Site. Scope Diagnostics (for SAP) Monitoring • Compuware – Server. Vantage – Client. Vantage Agentless – Vantage. View • Whats. Up Gold • PRTG • Scrutinizer

Testing Nirvana • True quality assurance—nothing goes into production until completely tested and confirmed acceptable

Monitoring Nirvana • Indicators warn us ahead of an issue and it is already resolved before it surfaces • Failing that, the issue surfaces, the proper folks are alerted and the issue is being worked on before the calls to the helpdesk start

In an ideal world……. . Tests and infrastructure set up for performance testing could be ported to production for performance monitoring…………….

Conclusions Do I have any? • ITIL keeps them separate • Enbridge keeps them separate • Currently, although the groups talk, nothing created in the pre-prod world can be ported to the production world • Customer expectation management vs. Customer experience management • Both groups often get forgotten during development

Questions?