Basics of Performance Testing Abu Bakr Siddiq Topics

Basics of Performance Testing Abu Bakr Siddiq Topics to be discussed q Introduction q Performance Factors q Methodology q Test Process q Tools q Conclusion

Introduction q Why Performance Test? q What is Performance Test? q How Performance Test? q When to Performance Test? q Performance Vocabulary

Why, What, How & When Performance Test q Why Performance test the application? To improve the software speed, scalability, stability and confidence of the system under different loads for desired performance. Ex: USA Shopping in Dec. Indian Railway Reservation. q What is Performance testing? Testing for performance factors against the acceptable or suggestible configuration is performance testing. q How Performance test the application? Using a Tool which follows Performance Methodology & Performance Process q When Performance test the application? Available Performance Requirements, Stable Build, Resource Availability, Defined Performance Plan and methodology, Set entry and Exit Criteria.

Performance Testing Vocabulary q Capacity Testing: Determining the server’s failure point is called Capacity Testing. q Component Testing: Testing the architectural component of the application like, servers, databases, networks, firewalls, and storage devices. q Endurance Testing: Testing for performance characteristics with work load models anticipated during production. q Load Testing: Subjecting the system to a statistically representatives(load) is called Load Testing. q Smoke Testing: A initial run to see the system performance under normal Load. q Spike Testing: Testing for performance characteristics with work load models during production which repeatedly increase beyond anticipated. q Stress Testing: Evaluating the application beyond peak load conditions. q Validation Testing: Testing against the expectations that have been presumed for the system q Volume Testing: Subjecting the system to variable amount of data and testing for performance is Volume Testing. Through put The different types of performance testing Stress Testing Saturation point Load Testing User Load

Performance Factors Throughput q Response Time Capacity Planning Response Time Benchmarking Latency q Tuning q Benchmarking q Capacity Planning Tuning

Throughput, Response Time & Latency Throughput : The number of requests/business transactions processed by the system in a specified time duration. N 1 Internet Client O 1 A 1 N 4 N 2 A 2 DB Server Web Server A 3 Network Latency = N 1 + N 2 + N 3 + N 4 Product Latency = A 1 + A 2 + A 3 Actual Response Time = Network Latency + Product Latency = Actual Response Time + O 1 N 3

Tuning, Benchmarking & Capacity Planning q Tuning : Procedure by which the system performance is enhanced by setting different values to the parameters (variables) of the product, OS and other components. Ex: “Search” q Benchmarking: Comparing the throughput and response time of the system with that of competitive products. Ex: Comparing Open office with that of MS- Office. q Capacity Planning: The exercise to find out what resources and configurations are needed. Ex: suggesting the ideal software, hardware and other components for the system to the customer.

Methodology q Collecting Requirements q Writing Test Cases q Automating Test Cases q Executing Test Cases q Analyzing Results q Tuning q Benchmarking q Capacity Planning • Collecting Requirements • Authoring Test Cases • Automating Test Cases • Executing Test Cases • Analyzing Results • Tuning • Benchmarking • Capacity Planning

Collecting Performance Requirements types: q Generic q Specific Performance Requirements Characteristics: q Testable q Clear q Specific Sources for gathering Requirements: q Performance compared to previous release of the same product q Performance compared to the competitive product(s) q Performance compared to absolute numbers derived from actual need q Performance numbers derived from architecture and design Graceful Performance Degradation

Authoring Test Cases Performance Test Cases should have the following details: q List of Operations or business transactions to be tested q Steps for executing those operations/transactions q List of Product, OS parameters that impact the Performance Testing and their values q Loading Pattern q Resource and their configuration (network, hardware, software configurations q The expected results q The product version/competitive products to be compared with the related information such as their corresponding fields

Automating Performance Test Cases Performance Testing naturally lends itself to automation, reasons : q Performance testing is repetitive q Performance test cases cannot be effective without automation q Results of Performance testing need to be accurate, manually calculating may introduce inaccuracy q Performance Testing takes into account too many permutations and combinations hence if done manually will be difficult q Extensive Analysis of performance results and failures needs to be taken into account, is very difficult to do things manually

Executing Performance Test Cases The following aspects need to be considered while executing Performance Tests: q Start and End time of Execution q Log and trace/audit files of the product and OS. (for future debugging and repeatability) q Utilization of resources ( CPU, memory, disk, network utilization and so on) on periodic basis q Configuration of all Environmental factors (Hardware, software and other components) q The performance factors listed in the test case documentation at regular intervals

Analyzing Performance Test results are concluded as follows: q Whether performance of the product is consistent when tests are executed multiple times q What performance can be expected for what type of configuration q What parameters impact and how they can be derived for better performance q What is the effect of scenarios involving a mix of performance factors q What is the effect of product technologies such as caching q What Loads of performance numbers are acceptable q What is the optimum throughput and response time of the product for set of factors q What performance requirements are met how the performance looks compared to previous version or expectations

Performance Tuning Two ways to get optimum mileage q Tuning the Product parameters q Tuning the Operating system parameters Product parameters: q Repeat the Performance tests for different values of each parameters that impact q A Particular Parameter change values is changed, it needs changes in other q Repeat the tests for default values of all parameters( factory settings tests) q Repeat the performance tests for low and high values of each parameter and combinations Operating system parameters: q Files system related parameters q Disk Management parameters q Memory Management parameters q Processor Management parameters q Network management parameters

Performance Benchmarking & Capacity Planning The steps involved in Performance Benchmarking are the as follows: q Identify the transactions/ scenarios & the test configuration q Comparing the performance of different product q Tuning the parameters of the products bring compared fairly to deliver the best performance q Publishing the results of performance benchmarking Capacity Planning is identifying the right configuration, which is of 3 types: q Minimum required configuration q Typical configuration q Special configuration

Test Process q Resource Requirements q Test Lab Setup Resource Requirement Test Lab Setup q Responsibilities q Setting up Traces & Audits Responsibilities q Entry & Exit Criteria Setting up Traces & Audits

Performance Test Process Obtain Measurable, testable requirements Design Test Cases Evaluate Entry Criteria Evaluate Exit Criteria Create a Performance Test Plan Automate Test Cases Perform and Analyze performance test cases

Resource Requirements: All the resources are specifically needed , hence shall be planned and obtained. Resources are to be exclusively dedicated to the current system without interchanging the roles and responsibilities often. Test – Lab Setup: Test Lab with all required equipment is to be setup prior to execution. Test lab has to be configured cautiously as a single mistake can lead to running the tests again. Responsibilities: Performance defects may cause changes to architecture, design and code. Team facing customer communicates requirements for performance. Multiple teams are involved in Performance testing the system. Hence a matrix which describes the responsibilities of the team is part of the test plan.

Setting up Product traces, Audits: Performance test results need to be associated with traces and audit trails to analyze the results. Audits and traces are to be planned in advance or else may start impacting the performance results. Entry and Exit Criteria: Performance tests require a stable product due to its complexity and accuracy that is needed. Changes to the product mean tests are to be repeated. Hence Performance tests starts after the product meets a set criteria.

Tools for Performance Testing q Commercial Tools Load Runner – HP QA Partner - Compuware Silk Test - Segue MS Stress Tool – Microsoft Ø Ø q Open Source Tools Ø Web Load Ø JMeter Ø Open STA q Challenges q Conclusion Commercial Open Source Load Runner Web Load Silk Test JMeter MS Stress Open STA

Thank you