Debashish Barua Oct 17 th 2016 PACIFIC NW

Debashish Barua Oct 17 th, 2016 PACIFIC NW Defect prediction model for estimating project schedule SOFTWARE QUALITY CONFERENCE

Bugs are everywhere. . PACIFIC NW SOFTWARE QUALITY CONFERENCE The Systems Sciences Institute at IBM has reported that “the cost to fix an error found after product release was four to five times as much as one uncovered during design, and up to 100 times more than one identified in the maintenance phase. ” Photo Credit: Shutterstock Source: http: //blog. celerity. com/the-true-cost-of-a-software-bug 2

Problem Statement PACIFIC NW SOFTWARE QUALITY CONFERENCE 1. How to estimate the number of defects prior to a product’s launch? 2. How do we define project milestone schedule? v What if Alpha, Beta, and Production Release are planned too early / late? v How does the number of open defects impact the program schedule? v Are there any existing tools which can be used for scheduling project milestones? Ultimate Goal is to improve the Quality! 3

Our Proposal PACIFIC NW SOFTWARE QUALITY CONFERENCE Ø Build a Defect Prediction Model which will help us in: v Scheduling project milestones. v Identifying and evaluating the Risk in existing Program schedule. v Monitoring in-process quality of products. v Taking preventive actions on unforeseen Defects / Issues. v Meeting time to the market need with improved quality. 4

PACIFIC NW SOFTWARE QUALITY CONFERENCE Photo Credit: Shutterstock How did we build a prediction model?

Pre-requisites – What do we need? PACIFIC NW SOFTWARE QUALITY CONFERENCE v Historical Defect Data. Previous project data is precious! v Track of all Major milestones – Alpha, Beta, PRQ (Product Release Qual) v The date at which defects were logged (Submitted date). v All defects in a project including Closed, Open, Rejected, etc. . We used Defect Tracking Systems – Jira, Bugzilla, etc. Jira source: www. atlassian. com Bugzilla source: https: //www. bugzilla. org/ 6

Major Project Milestones PACIFIC NW SOFTWARE QUALITY CONFERENCE Alpha: v Validation maybe pending. Expect ~50% of the total defects by this mark. Beta: v Feature Complete. Expect 60 -70% of the total defects by this mark. PRQ: v Final qualified release product. Expect 70 -80% of the total defects before shipping. 7

Photo Credit: Shutterstock SOFTWARE PACIFIC NW Let’s plot some data. . QUALITY CONFERENCE

PACIFIC NW SOFTWARE QUALITY Plotting the historical defect data Week # 1 2 3 4 5 6 7 8 9 10 … 30 31 32 … 50 51 52 … 89 90 WW 201338 201339 201340 201341 201342 201343 201344 201345 201346 201347 … 201415 201416 201417 … 201435 201436 201437 … 201522 201523 Cumulative Defects 1 1 1 3 3 4 5 5 … 292 335 432 … 2005 2100 2153 … 3484 3486 Do we see any pattern“ CONFERENCE S”? ! 9

Photo Credit: Shutterstock SOFTWARE PACIFIC NW What’s next? What to do with “S”? Build a mathematical model out of it. . QUALITY CONFERENCE

Mathematical Model for an “S” curve PACIFIC NW SOFTWARE QUALITY CONFERENCE where, e = the natural logarithm base (also known as Euler's number) Source: https: //en. wikipedia. org/wiki/Sigmoid_function Logistic function: where, • x 0 = the x-value of the sigmoid’s midpoint • L = the curve's maximum value • k = the steepness of the curve Source: https: //en. wikipedia. org/wiki/Logistic_function 11

Photo Credit: Shutterstock SOFTWARE PACIFIC NW Let’s apply it to our defect trends QUALITY CONFERENCE

What happens when we apply the Logistic Model to our Defect trends. . PACIFIC NW SOFTWARE QUALITY CONFERENCE where, • Inflection Point = the Work Week at which the Cumulative Defects is 50% of the total Defects • Asymptote = the total Cumulative Defects, and • Growth Rate = the rate at which the Defects grow with respect to Work Week *We are applying the Logistic 3 P model (Logistic regression) using the JMP® Pro 11. 1. 1 (64 -bit) software to calculate the parameters. JMP source: http: //www. jmp. com/support/help/Fit_Curve_Options. shtml 13

Example of “not-so-perfect” data Week # WW Cumulative Defects 1 201345 1 2 201346 1 3 201347 1 4 201348 1 5 201349 3 6 201350 4 7 201351 4 8 201352 4 9 201353 4 10 201402 4 … 30 … 201422 … 101 31 201423 115 32 201424 135 … 50 … 201442 … 501 51 201443 516 52 201444 565 … 86 … 201526 … 1689 87 201527 1690 PACIFIC NW SOFTWARE QUALITY CONFERENCE Growth Rate Inflection Point Asymptote 59 0. 1 1856 It works! 14

Photo Credit: Shutterstock PACIFIC NW for prediction? SOFTWARE QUALITY CONFERENCE

Defect Prediction Model Steps 1. Generate Reference Parameters from past Projects 2. Discuss Parameter vs. Dependencies Coding Speed / Capability Dev. methodology Verification Speed / Capabiity D Time of Max. Code generation Defect Experience Time of Max. Verification Activity # resources coding # resources verifying Max. resources distribution Milestone/ Release Activities Camps date Max. resources distribution # of features e. LOC Architecture type Project culture Defect Density Verification technology Regular Re-assessment 5. PRQ Window Discussion Adjusting Project Activities Calculate Future Defect Accumulation and PRQ window 4. Estimate Delta to Future Project 3. 0. 07 120 40000 where Unknown Defects is the Estimated Defects after the final Release (PRQ) 16

reference parameters from past projects PACIFIC NW SOFTWARE QUALITY CONFERENCE Photo Credit: Shutterstock

Using JMP – Version JMP® Pro 11. 1. 1 (64 -bit) 1. 6. 2. 5. 3. 4. 18

Generated parameters for Projects A, B, and C Project Milestones / Parameters Defect Growth Rate Inflection Point Asymptote Alpha Beta PRQ Project A 0. 046 159 30831 WW 2311 WW 3511 WW 0112 Project B 0. 04 128 49897 WW 1512 WW 4012 WW 0513 Project C 0. 044 143 20462 WW 4813 WW 1014 WW 2516 PACIFIC NW SOFTWARE QUALITY CONFERENCE Growth Rates are similar! 19

Photo Credit: Shutterstock PACIFIC NW ers vs depende ncies SOFTWARE QUALITY CONFERENCE

PACIFIC NW What dependencies to discuss? SOFTWARE QUALITY CONFERENCE v Architectural Complexity v More or Less features v Number of Programmers v Development Methodology, Project culture v Development / Verification technology v Project Timelines Photo Credit: Shutterstock 21

Photo Credit: Shutterstock SOFTWARE PACIFIC NW Step 3: Delta discussion QUALITY CONFERENCE

Parameter generation from Past Projects Project Name Project A Project B Project C Project D Growth Rate 0. 18 0. 15 0. 04 0. 05 Inflection Point 25 37 78 78 Asymptote Alpha Beta PRQ 3037 4147 260 309 WW 0215 WW 1015 WW 4813 WW 4214 WW 1915 WW 1415 WW 1014 WW 0815 WW 2615 WW 2215 WW 2614 WW 2615 PACIFIC NW SOFTWARE QUALITY CONFERENCE v Project A, B, C, and D are roughly 1 year long. We note the following differences: 1. The Growth Rate varies from 0. 04 to 0. 18 2. The Inflection Point ranges from 25 to 78 3. A huge difference in Asymptote from 260 to 4147 defects Based on this information, for a completely new project, will you be able to predict the parameters? 23

Photo Credit: Shutterstock SOFTWARE PACIFIC NW answer is QUALITY CONFERENCE

Let me provide you with more details about the projects. . Project Name Project A Project B Project C Project D Growth Rate 0. 18 0. 15 0. 04 0. 05 Inflection Point 25 37 78 78 Asymptote Alpha Beta PRQ 3037 4147 260 309 WW 0215 WW 1015 WW 4813 WW 4214 WW 1915 WW 1415 WW 1014 WW 0815 WW 2615 WW 2215 WW 2614 WW 2615 PACIFIC NW SOFTWARE QUALITY CONFERENCE v Indeed. A and B are similar v B was derived from A with extra features. v Same goes for C and D. The parameters now make sense! 25

Project Milestones / Parameters Defect Growth Rate Inflection Point Asymptote Alpha Beta PRQ Project A 0. 046 159 30831 WW 2311 WW 3511 WW 0112 Project B 0. 04 128 49897 WW 1512 WW 4012 WW 0513 Project C 0. 044 143 20462 WW 4813 WW 1014 WW 2516 PACIFIC NW SOFTWARE Estimate Delta to future project QUALITY Estimated Project D 0. 05 130 12282 WW 5115 WW 1016 WW 2616 CONFERENCE Actual Defects . . . Actual defects following the Predicted trend 26

Photo Credit: Shutterstock PACIFIC NW accumulation and prq window SOFTWARE QUALITY CONFERENCE

Future unknown defects after PRQ PACIFIC NW SOFTWARE QUALITY CONFERENCE ~8000 Unknown Defects Looks bad! ~8000 defects after final release! 28

Photo Credit: Shutterstock SOFTWARE PACIFIC NW Discussion Adjusting Project Activities QUALITY CONFERENCE

How to mitigate Escape Risk PACIFIC NW SOFTWARE QUALITY CONFERENCE Old PRQ ~4000 Unknown Defects ~8000 Unknown Defects Now it looks much better! 30

Results PACIFIC NW SOFTWARE QUALITY CONFERENCE 31

Key Takeaways PACIFIC NW SOFTWARE QUALITY CONFERENCE 1. Eliminating the uncertainty in declaring project milestones. 2. Achieving in process quality monitoring of current projects. 3. Suggesting corrective actions during an on-going project. 4. Delivering projects on schedule and with utmost quality. 32

Future work PACIFIC NW SOFTWARE QUALITY CONFERENCE 1. Use Machine Learning on all previous projects to find patterns. 2. Figure out other dependencies which correlate with the parameters. 33

PACIFIC NW SOFTWARE QUALITY Contact Info CONFERENCE Debashish Barua Intel Corporation, Santa Clara email: debashish. barua@intel. com Linked. In: https: //www. linkedin. com/in/debashish-barua-15 a 9649 b Facebook: https: //www. facebook. com/debashishbarua Thank you! 34