Amadori Course Delivering an Effective Test Process Effective

Amadori Course: Delivering an Effective Test Process Effective Reporting of Test Execution Progress 1

Amadori Course: Delivering an Effective Test Process Introduction • In this session we will look in detail at two different methods of reporting test progress • Which can be used individually or in combination • Unless we report testing progress effectively the value of even the best test execution effort is considerably reduced 2

What does EFFECTIVE mean in this context? • Progress MUST be measured in terms of requirements coverage • Recording progress in terms of tests executed tells you nothing • Except how hard the test team are working • And even this can be “gamed” Amadori Course: Delivering an Effective Test Process 3

Amadori Course: Delivering an Effective Test Process 1. testing have executed 907 tests over the last 5 days. 602 tests have yet to be executed 2. Of the 90 requirements tested 32 have been fully tested and 36 partially tested Question Which summary adds most value to stakeholders And Why? 4

2. Of the 90 requirements tested 32 have been fully tested and 36 partially tested Answer Is the better summary as it measures BREADTH of test coverage 1. testing have executed 907 tests over the last 5 days. This measures only test effort and not progress Amadori Course: Delivering an Effective Test Process 5

Word of Warning If you haven’t been through a structured requirements extraction and mapping process it is very difficult to accurately report requirements coverage Not something you can addon at the last minute Amadori Course: Delivering an Effective Test Process 6

Amadori Course: Delivering an Effective Test Process 1. Mapping against an existing requirements map Methods of tracking requirements coverage 2. Using a Boston Matrix to provide a high level indication of test coverage Two approaches can be combined to provide both summary and detailed metrics to stakeholders 7

Amadori Course: Delivering an Effective Test Process Mapping test progress against requirements • Simple when tests relate to a single requirement • Not so simple when there is a many to many relationship between tests and requirements • Let’s start with a simple example…. 8

A simple example Requirements Tests A 1 -10 B 11 -15 C 16 -95 D 96 97 -100 Amadori Course: Delivering an Effective Test Process 9

Requirements A Tests 1 -10 B 11 -15 C 16 -95 D 96 97 -100 Requirement Tests A 10 Passed 6 B 16 12 C 70 50 D 4 4 Failed 4 10 Not Run 0 % Executed 100. 00% 4 75. 00% 10 85. 71% 0 100. 00% % Passed Pass Rate Comments 60. 00% Initial testing complete but defects have not yet been fixed 75. 00% 100. 00% Testing execution well advanced, no issues found so far 71. 43% 83. 33% Testing execution well advanced, some issues found 100. 00% Testing complete, no issues found • % executed and % passed measure scope of testing • Pass rate measures quality of the code tested to date Amadori Course: Delivering an Effective Test Process 10

Something More Complex… Tests Requirements A 1 -10 B 11 -15 C 16 -95 D 96 97 -100 Amadori Course: Delivering an Effective Test Process 11

Two Stage Process • Allocate test results against each requirement Tests 1 -10 11 -15 16 -95 96 97 -100 Totals Tests 10 15 70 1 4 100 Passed 6 10 50 Failed 4 Not Run 0 5 20 0 0 25 1 3 8 1 67 A Y B Y C Y Y y D Y Y y y Y y • Work out test metrics as before Requirement A B C D Tests Passed 14 16 99 29 Failed 7 10 67 17 Not Run 7 1 7 7 % Executed 100. 00% 5 68. 75% 25 74. 75% 5 82. 76% % Passed Pass Rate 50. 00% 62. 50% 67. 68% 58. 62% 50. 00% 90. 91% 90. 54% 70. 83% Amadori Course: Delivering an Effective Test Process Comments 12

What else can these numbers tell us? • Can help prioritise testing effort Tests 1 -10 11 -15 16 -95 96 97 -100 Totals Tests 10 15 70 1 4 100 Passed 6 10 50 1 67 Failed 4 1 3 8 Not Run 0 5 20 0 0 25 A Y B Y C Y Y y D Y Y y y Y y • Ensures that requirements aren’t forgotten about during the heat of battle…. Amadori Course: Delivering an Effective Test Process 13

But not all requirements are equal…. . • Two ways to take this into account when reporting progress…. • Add a weighting factor so more important requirements count for more • Or produce different versions of the report • 1. all requirements • 2. Only business critical requirements Amadori Course: Delivering an Effective Test Process 14

Issues with this approach • This approach provides excellent detail about how well or badly individual requirements are being delivered BUT… • Requires the reader to have a good understanding of how testing works • Can be difficult to see the bigger picture • This is where a Boston Matrix can help Amadori Course: Delivering an Effective Test Process 15

What is a Boston Matrix ? 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering an Effective Test Process 16

Understanding the 4 quadrants is the Key. . • Quadrant A – High Pass Rate, Low Coverage • Looks good at first sight but be careful…. . • Quadrant B – High Pass Rate, High Coverage • This is where you want to be • Quadrant C – Low Pass Rate, Low Coverage • Bad news, unless testing has just started…. . • D – Low Pass Rate, High Coverage • Lots of issues but at least we know what they are…. . Amadori Course: Delivering an Effective Test Process 17

Boston Matrix Heat Map 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management 18

Alternative Views 100 % B C D Pass Rate A 100% A B Pass Rate 0% C 0% 100% Test Coverage D Test Coverage 100% Amadori Course: Delivering an Effective Test Process 19

How is it populated? • Divide testing scope into a number of areas • Try where possible to reflect the way the project is seen by stakeholders…. • For each area work out • Test Coverage - % of requirements tested to date • Pass Rate (tests passed/tests failed) Amadori Course: Delivering an Effective Test Process 20

A Worked Example Area Trade Entry Trade Confirmations Market & Credit Risk Analytics Reporting Compliance Financial Accounts Pass Rate 92% 43% 12% 67% 71% 78% Coverage 64% 12% 46% 80% 20% 90% 40% Where on the Matrix does each area go? Amadori Course: Delivering an Effective Test Process 21

Trade Entry: Pass Rate 92%, Coverage 64% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering an Effective Test Process 22

Confirmations: Pass Rate 92%, Coverage 12% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering an Effective Test Process 23

Risk: Pass Rate 43%, Coverage 46% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering an Effective Test Process 24

Analytics: Pass Rate 12%, Coverage 80% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management 25

Reporting: Pass Rate 67%, Coverage 20% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering an Effective Test Process 26

Compliance: Pass Rate 71%, Coverage 90% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering an Effective Test Process 27

Accounts: Pass Rate 78%, Coverage 40% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management 28

Questions? • What if a requirement is only partially tested? • Establish a rule of thumb and stick to it…. . • Either • If a requirement is more than 50% tested it counts as complete • Requirements are only covered when 100% of tests have been executed • Not all areas are equally important…. • Then highlight the important areas in some way to make them stand out…. Amadori Course: Delivering an Effective Test Process 29

Highlight the key areas of delivery 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering an Effective Test Process 30

Exercise • You have a meeting with the project manager coming up • Using the Boston Matrix • What would you tell him about the project’s overall status? • Which areas give you most concern? • What are the key activities in each area moving forward? • Where would you prioritise available test resource? Amadori Course: Delivering an Effective Test Process 31