Metrics in Automation testing theory and practice Igor

Metrics in Automation testing theory and practice Igor Cernopolc 15 th May 2014

Why Metrics? TQM Customer Focus Process Improvement Human Side Of Quality Metrics, Models, Measurements and Analyses Testing Metrics help organization to obtain the information it needs to continue improve its testing process, reduce errors and increase the quality of deliverables, thus achieving the desired Goal.

What is a that? Metric is a standard unit of measurement that quantifies results. It is obtained using measurement based techniques which are applied to processes, products and services as applicable.

Metrics categories Coverage • parameters for measuring: • testing scope • success of testing Progress • parameters that help identify test progress to be matched against success criteria • metrics are collected iteratively over time Quality • measures of testing product/process quality • usability, performance, scalability, overall customer satisfaction, and defects reported are a few examples

Lifecycle Analysis • Identify metrics to use • Define metrics identified Acceptance • Explain the need of metric to stakeholder and testing team • Educate the team to prepare and collect the metrics Evaluation • Capture the data • Verify the data • Calculate metrics Reporting • Develop the report with effective conclusion • Distribute report to stakeholders • Obtain feedback

What are automated testing metrics? Automated testing metrics are metrics used to measure the performance (past, present, and future) of the implemented automated testing process and related efforts and artifacts.

Is my metric a good one? A good automated testing metric has the following characteristics: • • • It is objective It is measurable It is meaningful Data for it is easily gathered It can help identify areas of improvement It is simple.

Metrics in automation testing • Automatable Percentage • Test Automation coverage • Automation test design productivity • Automation test execution productivity • Effort savings through test automation • ROI of Test Automation

Automatable Percentage 100 92 85 90 82 80 70 60 50 50 69 67 C 6 C 7 48 35 40 30 20 10 0 C 1 C 2 C 3 C 4 Automatable % C 5 Goal C 8

Test Automation Coverage 100 85 90 80 92 90 82 80 69 70 60 50 50 48 45 35 40 67 67 45 40 32 30 30 20 10 0 C 1 C 2 No of TC's C 3 C 4 C 5 Automated TC's C 6 C 7 % Automated C 8

Automation test design productivity 160 135 140 120 100 100 85 80 60 60 50 40 15 20 0 day 1 day 2 day 3 day 4 day 5 % of TC's automated day 6 Goal day 7 day 8

Automation test execution productivity 160 140 135 120 100 100 85 80 60 60 50 40 20 15 0 day 1 day 2 day 3 % of TC's automated day 4 day 5 Goal day 6 day 7 Lower control limit day 8

Effort savings through test automation 50 45 40 35 30 25 25 20 15 18 10 5 0 3 8 C 1 C 2 17 10 5 C 3 Manual execution time C 4 C 5 C 6 Automation execution time 2 C 7 Saved C 8

ROI of Test Automation 30 25 20 20 20 18 15 15 16 15 18 17 16 13 10 8 5 8 7 7 6 3 0 Q 1 Q 2 Cost of Automation Q 3 Q 4 Cost Savings from Automation Q 5 Q 6 Net Cost Savings Q 7 Q 8

Let’s share How we all automated, with and without metrics

Few more things • Reliability vs Validity • Ratio, Proportion, Rate, Percentage, Absolute values • Don't measure for the sake of measure • Track and measure on an ongoing basis

Q&A are Welcomed

In one word Before measuring anything,

In one word Before measuring anything, set goals

Thank you Igor Cernopolc igor. cernopolc@gmail. com