Mistakes Errors and Defects 2262021 Mistakes Errors Defects

Mistakes, Errors and Defects 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 1

Basic Concepts o o You are building a project. The project is made up of “phases”. n o o o You complete a phase of the project and then move onto the next phase In the Lab. , some people would consider each “task” as a “phase” Some people would call each part of a task a phase The key element is – you decide a “phase” for your project, and then you decide n n 2/26/2021 whether you have completed that “phase” and can move onto the next stage, or whether you have to keep working to make this phase work Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 2

Mistakes in a project with 3 tasks Task 1 o 5 mistakes are made o 3 mistakes are discovered before moving onto Task 2 o 4 mistakes are made o 3 mistakes are discovered before moving onto Task 3 o 1 mistake made in Task 1 is discovered and fixed Task 3 o 4 mistakes are made o 1 mistake from Task 1 is discovered PRODUCT RELEASE 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 3

Questions to ask o o o How many mistakes made during Task 3, will the programmer find before releasing the project to the customer? How likely is it that the programmer will, during Task 3, find the mistake left behind from Task 2? How many mistakes can be expected to be left behind after the product has been released? 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 4

Errors and Defect o Mistake – anything you do wrong o Error – are mistakes you make in “ this phase” of the project and FIND before moving on to the “next phase” of the project. o Defects – are mistakes you make in “ this phase” of the project and DON’T FIND before moving on to the “next phase” of the project o There is no mention of constitutes an error or a defect is – it is anything you do wrong! Errors and defects can both be syntax errors of logical errors. The difference is one of time -- “when did you find the mistake” 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 5

Mistakes in a project with 3 tasks Task 1 o 5 mistakes are made o 3 mistakes are discovered before moving onto Task 2 o 4 mistakes are made o 3 mistakes are discovered before moving onto Task 3 o o 1 mistake made in Task 1 is discovered and fixed 3 ERRORS 2 DEFECTS 3 ERRORS 1 DEFECT RECOVERY Task 3 o 4 mistakes are made o 1 mistake from Task 1 is discovered ? ERRORS ? DEFECTS 1 DEFECT RECOVERY PRODUCT RELEASE HOW MANY PROBLEMS? 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 6

Error free with defects Defect free with errors? Task 1 o 5 mistakes are made o 3 mistakes are discovered before moving onto Task 2 o 4 mistakes are made o 3 mistakes are discovered before moving onto Task 3 o o 1 mistake made in Task 1 is discovered and fixed Task 3 o 4 mistakes are made o 1 mistake from Task 1 is discovered or 3 ERRORS 2 DEFECTS 3 ERRORS 1 DEFECT RECOVERY ? ERRORS ? DEFECTS 1 DEFECT RECOVERY SHOULD PRODUCT BE RELEASED BASED ON THIS PROGRAMMERS SKILLS OR SHOULD THE PROGRAMMER KEEP WORKING ON TASK 3 PRODUCT RELEASE 2/26/2021 HOW MANY PROBLEMS? Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 7

Mistake Recovery Efficiency How to calculate Mistake Recovery Efficiency Errors found for a = ------------------given project phase Mistakes made (Errors + Defects) Can’t really be calculated for THIS project till the project is finished (Must have past history available) Task 1 o 5 mistakes are made o 3 mistakes are discovered before moving onto Task 2 o Mistake Recovery Efficiency 3 ERRORS 2 DEFECTS = MRE = 3 / 5 = 60% recovery Task 2 o 4 mistakes are made o 3 mistakes are discovered before moving onto Task 3 o o 3 ERRORS 1 DEFECT 1 mistake made in Task 1 is discovered and fixed 1 DEFECT RECOVERY Mistake Recovery Efficiency = MRE 3 / 4 = 75% recovery Average MRE = (60 + 75) / 2 = 67. 5% (2/3) 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 8

Defect Recovery Efficiency How to calculate Defect Recovery Efficiency for a given project phase = Defects found ------------------Defects at this point in project Can’t really be calculated for THIS project till the project is finished (Must have past history available) Task 1 o 5 mistakes are made o 3 mistakes are discovered before moving onto Task 2 o Defect Recovery Efficiency 3 ERRORS 2 DEFECTS = Has no meaning at this stage of the project Task 2 o 4 mistakes are made o 3 mistakes are discovered before moving onto Task 3 o o 3 ERRORS 1 DEFECT 1 mistake made in Task 1 is discovered and fixed 1 DEFECT RECOVERY Defect Recovery Efficiency = Found 1 from Task 1 / 2 made in Task 1 = 50% recovery Average DRE = 50% 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 9

Problems with final customer code How many? – based on MRE Task 1 o 5 mistakes are made o 3 mistakes are discovered before moving onto Task 2 o 4 mistakes are made o 3 mistakes are discovered before moving onto Task 3 o 1 mistake made in Task 1 is discovered and fixed 3 ERRORS 1 DEFECT RECOVERY Task 3 o 4 mistakes are made o 1 mistake from Task 1 is discovered ? ERRORS ? DEFECTS 1 DEFECT RECOVERY PRODUCT RELEASE HOW MANY PROBLEMS? 3 ERRORS This programmer averages MRE = 67. 5% in each phase – therefore expect that of the 4 mistakes made in task 3 – 4 * 0. 675 of those will be found 1 defect left in customer code from Task 3 2/26/2021 Mistakes, Errors, Defects , 10 Copyright M. Smith, ECE, University of Calgary, Canada

Problems with final customer code How many? – based on DRE Task 1 o 5 mistakes are made o 3 mistakes are discovered before moving onto Task 2 o 4 mistakes are made o 3 mistakes are discovered before moving onto Task 3 o 1 mistake made in Task 1 is discovered and fixed 3 ERRORS 1 DEFECT RECOVERY Task 3 o 4 mistakes are made o 1 mistake from Task 1 is discovered ? ERRORS ? DEFECTS 1 DEFECT RECOVERY PRODUCT RELEASE HOW MANY PROBLEMS? 3 ERRORS This programmer averages DRE = 50% in each phase – Start of Task 3 – there is one defect from Task 1 and 1 from Task 2 left undiscovered Based on this programmers track record – only 50% of defects uncovered. Therefore one of the two defects will not be uncovered. Mistakes, in Task 3 2/26/2021 Errors, Defects , 11 Copyright M. Smith, ECE, University of Calgary, Canada

Analysis – Manager’s point of view o Manager’s point of view – this programmer will leave behind 2 defects in this product based on past performance 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 12

Analysis – Programmer’s point of view o On average I make 1 defect (that I don’t find until after moving onto the next stage) for every 3 errors made (that I find before moving onto the next stage ERRORS FOUND TASK 1 4 TASK 2 3 TASK 3 4 EXPECTED # DEFECTS 1 1 1 Based on past programming style, programmer expects to have 3 defects in code – but has only found 2 Based on past-performance the programmer should spend a little more time checking the code looking for one more probable error How long to you keep looking? – good question – no clear answer 2/26/2021 Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 13

If you are going to do this sort of metrics seriously o The programmer needs to know their own MRE and DRE. n o The programmer needs to know need to know whether the tasks were of equal difficulty n o A more difficult task can be expected to cause more than the average number of defects Is this a safety critical task n o Must collect “metrics” from past experiences Might spend more time looking for possible defects (Perhaps write more tests (TLD)) Time estimation – how long will the product take to develop? n n n 2/26/2021 How big is the product? Learn estimation skills What is your efficiency (lines of debugged code / hour)? Errors / 1000 lines of code? Defects / 1000 lines of code? How long does it take you to fix a “typical” error or defect? When fixing errors and defects – how many new mistakes do you introduce for each one you fix? (Industry about 1 in 3) What are your most common mistakes? And what are you doing to fix them in time for final exam and post-lab 3 quiz Mistakes, Errors, Defects , Copyright M. Smith, ECE, University of Calgary, Canada 14

Example Post. Lab 3 or Final question on Errors / Defects / Mistakes A student is doing 5 tasks – all roughly the same size and difficulty. You know that the students MRE = 60% and DRE = 20% Task 1 Task 2 Task 3 Task 4 Task 5 10 errors discovered 5 errors discovered 10 errors discovered 12 errors discovered 10 errors discovered 2 defects from Task 1 found 1 defects from Task 1 + 2 defects from Task 2 + 2 defects from Task 3 found 2 defects from Task 2 + 2 defects from Task 3 + 1 defect from Task 4 Explain both qualitatively (reasonability argument) and quantitatively (number argument) the following issues 1) 2) 3) The programmer moved onto Task 3 from Task 2 at the correct time – true or false The programmer should stop working on Task 5 and release the product Why is impossible in Task 5 to discover a defect from Task 5 This programmer has been using test first development (test driven development). 4) Explain why you might think that this student does not know how to write TDD tests (some people are good at testing and some are not) 5) 6) 7) 8) 9) Looking back at your previous classes – make an estimate of your MRE and DER rates Make a comment on what you feel is your MRE and DER rates now Has TDD helped you or hindered you? Some people feel that writing tests before coding takes longer that writing tests after the coding. In theory – since you need the same tests – this is not true. But in practice it often is true – explain the discrepancy What is the difference between a unit test and a system test? Using one of your labs as an example give details of a unit test and a 2/26/2021 Mistakes, Errors, Defects , 15 system test? Copyright M. Smith, ECE, University of Calgary, Canada