Using Metrics to Reduce Cost of Rework Dwight

Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton

Opening Question “What is the job of testing? ” 2

Potential Response “The job of testing is to find defects. ” 3

Myth Exposed Myth: The Job of Testing Is to Find Defects 1 1 The Seven Principles of Lean Software Development by Mary Poppendieck 4

“Quality is everyone’s responsibility” Myth: The Job of Testing Is to Find Defects “The job of tests, and the people that develop and run tests, is to prevent defects, not find them. A quality assurance organization should champion processes that build quality into the code from the start rather than test quality in later. ” 1 1 The Seven Principles of Lean Software Development by Mary Poppendieck 5

At a recent software seminar “The common attributes of successful software projects are l Heroes l Chaos l Re-work l Untracked OT” 6

Today’s Discussion l Case Study – 4 projects l 2 Definitions l Testing Metrics (what to collect & analyze) l Metrics from first 2 projects l Process Improvement Steps l Success Story – Projects 3 & 4 l Some trends & conclusions 7

Project Background Legacy mature system (mainframe) l Industry regulatory enhancements l Time-boxed: 3 to 4 months l Waterfall SDLC First 2 Projects l Project 1 – Dec 09 – 1 X l Project 2 – June 10 – 2. 3 X l 8

Definition: Re-work – AD time to fix and testing time to report and re-test System Testing and Production defects 9

Definition: Static Testing - Testing of a component or system at specification or implementation level without execution of that software, e. g. , reviews or static code analysis. 10

Static vs. Dynamic Testing From System Testing with an Attitude, p. 14, by Nathan Petschenik 11

Testing Defect Analysis l When was the defect introduced (SDLC phase) l When was the defect discovered (testing phase) 12

Points of Origin for Defects 2 1. 2. 3. 4. 5. Requirements Design Code User documents Bad fixes (new defects due to repairs of older defects) 2 Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies by Capers Jones 13

Defect Discovery Phases 1. Static Testing l l Document walk-throughs & reviews Pre-testing Test Plan review All before Build is complete System Testing 3. Production 2. 14

Tracking the Cost of Re-work Defect Fix Time 2. Re-test Time 1. COR (Cost of Re-work) – calculated by multiplying time x rate – cost per defect 15

Testing Rework Workflow Re-work time during Test Execution is tracked as 2 Components 1. 2. Time to Fix a Defect Time to Re-test Defect Fixes 16

Project 1 - Defect Fix Cost 17

Issues from first 2 projects 1. High % of project = Cost of Rework 18

COR = 23 to 29% Project 19

Issues from first 2 projects High % of project = Cost of Rework 2. High % of Requirements & Design (R&D) defects found later 1. 20

20 -30% R&D Defects found later 21

Issues from first 2 projects High % of project = Cost of Rework 2. High % of Requirements & Design (R&D) defects found later 3. DDP (Defect Discovery %) – slightly better than average 1. 22

Defect Discovery Percentage (DDP) DDP = % of Total Defects discovered before implementation Provides a metric to evaluate the overall effectiveness of testing. 90% - Good 95% - Very Good 23

DDP = average + 24

Process Improvement Steps 1. 2. 3. 4. 5. 6. Developed framework in Quality Center to track COR Presentations to business partners & executives Lively discussion during presentations AD appointed a new AD Lead AD improved their internal code review More emphasis on and participation during reviews, walk-throughs – Static Testing 25

Process Improvement Steps More collaboration between AD and testing teams 8. Increased emphasis on unit testing 9. Risk-based testing approach 10. Improved regression work 7. 11. Staff increased in domain knowledge 26

Projects 3 & 4 – Process Improved Project 1 – Dec 09 – 1 x 2. Project 2 – June 10 – 2. 3 x 3. Project 3 – May 11 – 3. 4 x 4. Project 4 – June 11 – 2. 3 x 1. 27

Static Testing emphasized 28

System Testing Defects 29

% R&D Defects found later 30

DDP (Defect Discovery %) 31

COR as % Project Cost 32

Defect Discovery Rate DDR = % of Total Defects discovered during an SDLC or Test Phase Static Testing System Testing Transition Goal - Trying to find out if we are finding defects earlier. l l l 33

DDR Trends --------Coding--------- As DDR for Static Testing increases, DDR for System Testing and Production Defects decreases 34

When is Unit Testing complete? “A necessary criterion for completion of developer testing is that the developers believe that the software is ready to be delivered to the users on the day it is delivered to the system-test team. ” From System Testing with an Attitude, p. 33, by Nathan Petschenik 35

Today’s Discussion l Defect metrics – powerful story l Static Testing works l It is a collaborative effort l The positive effect ripples into Unit Testing, System Testing, UAT, and implementation l No shortcuts here – influence your project to do it right 36

Some Final Thoughts Training needed for our staff (testing, AD, business, PM community) - how to be more effective in document reviews and walk-throughs. Perspectives l Tester – static testing > a paradise (a new maturity level) l 37

Questions & Answers · ? ? ? 38