Managing the Requirements Elicitation Process Amadori Courses Delivering

Managing the Requirements Elicitation Process Amadori Courses: Delivering Effective Test Management 1

• I much prefer the term elicitation to the term extraction Requirements ELICITATION? • Reminds everyone that this should be a collaborative process with benefits for all concerned • Not down to testing to beat requirements out of the stakeholders • Stakeholders and test should work together to ensure that the requirements of the project are complete and unambiguous Amadori Courses: Delivering an Effective Test Process 2

3 This is ELICITATION Amadori Courses: Delivering an Effective Test Process

4 And this is EXTRACTION Amadori Courses: Delivering an Effective Test Process

I know which I prefer Amadori Courses: Delivering an Effective Test Process 5

Introduction • This session will provide guidelines which allow Test Managers to effectively • Identify and manage sources of requirements • Address issues with requirements which are • Unavailable • Vague • Contradictory • Untestable • Subject to constant change • And explain the best ways of communicating progress and issues in this with Senior Stakeholders Amadori Courses: Delivering an Effective Test Process 6

Why are Requirements so important to the test process? If no-one can tell us where we are going… How can test confirm whether we have arrived? Amadori Courses: Delivering an Effective Test Process 7

What I am not going to do…. . • Is spend the next hour explaining how to derive testable requirements from source documentation • I assume you already know how to do this. . • Will spend the time instead looking at ways in which the overall process can be better managed • And progress and issues reported on Amadori Courses: Delivering an Effective Test Process 8

What the effective test manager needs to do here. . Get the Big Picture clear Assess where requirements are coming from in each area Build Preparedness Matrix Assign Priorities Work through requirements in priority order Identify and Manage Issues with Requirements Provide Regular Metrics showing completeness Move on to script writing…. Amadori Courses: Delivering an Effective Test Process 9

• Begin by dividing the delivery into its various components • Work out • Criticality • Complexity Get the Big Picture Clear • Of each component and use this to assign a weight to each area • This is the % of the time available you should aim to spend in each area • Areas with higher weights should take priority • However should try and have at least some high level test requirements in all areas Amadori Courses: Delivering an Effective Test Process 10

Get the Big Picture Clear • Making the Effort to understand where all of the individual pieces fit will pay huge dividends in the longer run Amadori Courses: Delivering an Effective Test Process 11

Example • Begin by dividing the delivery into its various components • Work out • Criticality • Complexity • Of each component and use this to assign a weight to each area • This is the % of the time available you should aim to spend in each area • Areas with higher weights should take priority • However should try and have at least some high level test requirements in all areas Amadori Courses: Delivering an Effective Test Process 12

System X Manual Updates Audit Calcs. Feed from Vendor 1 Usability & Performance Feed from Vendor 2 Feed to Regulator Daily Reports Monthly Reports Amadori Course End of Year Reports 13

Classify each area by complexity and criticality • Values don’t have to be exact, just a way of differentiating one area from another Amadori Course Element Feed from Vendor 1 Feed from Vendor 2 Manual Updates Calcs. Audit Usability & Performance Daily Reports Monthly Reports End of Year Reports Feed to Regulator Criticality High Medium Low High Complexity Low Medium Low High Low Low Low 14

Apply a Weighting • Values don’t have to be exact, just a way of differentiating one area from another • Greater complexity and/or complexity should normally mean that greater weight should be paid to it Amadori Course Element Feed from Vendor 1 Feed from Vendor 2 Manual Updates Calcs. Audit Usability & Performance Daily Reports Monthly Reports End of Year Reports Feed to Regulator Criticality High Medium Low High Complexity % Weighting Low 15 Medium 5 Low 10 High 25 Low 10 Low 5 Low 10 15

Allocate Resource Accordingly • I’ve assumed 10 man-days are available for requirements analysis in total to make the sums easier… • Becomes easier to spot if your team are spending too long on a particular area • Will use these weightings to assess requirements coverage also Amadori Course Element Feed from Vendor 1 Feed from Vendor 2 Manual Updates Calcs. Audit Usability & Performance Daily Reports Monthly Reports End of Year Reports Feed to Regulator Criticality High Medium Low High Man. Complexity % Weighting Days Low 15 Medium 5 Low 10 High 25 Low 10 Low 5 Low 10 16 1. 5 0. 5 1 2. 5 1 0. 5 1

Assess where requirements are coming from in each area Identify all the possible sources where information about each area may come from Amadori Courses: Delivering an Effective Test Process Include informal sources such as meetings and conversations with end users as well as formal specifications and business process diagrams 17

A matrix like this one may help Element Criticality Complexity Formal Spec Meeting Worked Minutes Examples Feed from Vendor 1 High Low x Feed from Vendor 2 High Medium x Manual Updates High Low x Calcs. High x Audit Medium Low Discussions with Stakeholders x x Names/Location of sources Owner x x x Usability & Performance Low Daily Reports High Low Monthly Reports Low x End of Year Reports High Low x Amadori Courses: Delivering an Effective Test Process Existing Business Processes x x x 18

Formal vs Informal Sources • Require Slightly different treatment Amadori Courses: Delivering an Effective Test Process 19

Formal Sources • Formal Requirements • Ensure that each testable requirement you extract includes the section/page reference it relates back to • That way if the document changes it is easier to work out which requirements are affected by the changes and which are not Amadori Courses: Delivering an Effective Test Process 20

Informal Sources • Informal Requirements • You should formally document your understanding of the requirement • This will form the basis of testing in this area • And makes it harder for other people to claim they said X and not Y later in the project • Always give the other parties involved the chance to review these requirements • But make clear that no feedback is tacit acceptance of what you have written and not a get out of jail card…… Amadori Courses: Delivering an Effective Test Process 21

If any source of requirements on your matrix is not available or is currently undergoing revision make this clear Availability Also make clear that if a source of requirements is not made available for testers to analyse NO testing of that source will be carried out

Stages in the Requirements Elicitation Process 23

This should contain • All requirement sources grouped by area Build Preparedness Matrix Amadori Courses: Delivering an Effective Test Process And measure 1. are they available 2. have they been analysed 3. % testable 4. % missing 5. % with issues 6. Actual and Potential requirements in each area 7. impact on overall preparedness in terms of requirements coverage 24

Actual Requirements Actual and Potential Requirements Testable requirements extracted from a source Potential requirements Additional requirements expected once all issues with requirements sources resolved

Why try and measure Potential Requirements? Questions How do we measure Potential Requirements? Amadori Courses: Delivering an Effective Test Process 26

Why try and measure Potential Requirements ? Because if we don’t and report only on we risk • 1. testable requirements we have extracted • 2. requirements with known issues • Seriously understating the amount of functionality required • Giving the impression that our test coverage is much more complete than is When we report progress to stakeholders Amadori Courses: Delivering an Effective Test Process 27

For Example • Returning to the project we referenced earlier • This is how requirements coverage might usually be reported Element Reqs with Issues Testable Reqs Feed from Vendor 1 12 2 Feed from Vendor 2 7 3 Manual Updates 7 1 3 2 5 1 39 10 Calcs. Audit Usability & Performance Daily Reports Monthly Reports End of Year Reports Feed to Regulator Amadori Courses: Delivering an Effective Test Process 28

For Example • Which at first sight looks quite healthy until you realise that we have no requirements AT ALL in certain key areas Element Reqs with Issues Testable Reqs Feed from Vendor 1 12 2 Feed from Vendor 2 7 3 Manual Updates 7 1 3 2 5 1 39 10 Calcs. Audit Usability & Performance Daily Reports Monthly Reports End of Year Reports Feed to Regulator Amadori Courses: Delivering an Effective Test Process 29

For Example • Wouldn’t it be better to include an allowance for the work we know will have to be carried out in these areas? • But how by definition do you place a value on something whose scope is currently unknown • One thing is certain however, we don’t know whethere will be actually 1, 100 or 1, 000 requirements in each of these areas…. • But we can be certain there will not be 0 requirements • So any value we include will be more accurate than what we show currently… Amadori Courses: Delivering an Effective Test Process 30

So How do we measure Potential Requirements? • There a couple of obvious solutions here which use the data values we assigned to each area earlier Element Criticality Complexity Feed from Vendor 1 High Low Feed from Vendor 2 High Medium Manual Updates High Low 10 Calcs. High 25 Audit Medium Low 10 Usability & Performance Low 5 Daily Reports High Low 10 Monthly Reports Low 5 End of Year Reports High Low 5 Feed to Regulator High Low 10 Amadori Courses: Delivering an Effective Test Process % Weighting 15 5 31

Method 1 • • Assign a set level of requirements to each source based on its complexity and criticality The actual values can be anything you want as they will be updated as soon as more accurate information is available Criticality Complexity Low 3 Low Medium 6 Low High 15 Medium Low 7 Medium 15 Medium High 12 High Low 12 High Medium 30 High 50 Amadori Courses: Delivering an Effective Test Process Requirements 32

Method 1 • Which Gives the following set of Numbers Element Criticality Complexity Feed from Vendor 1 High Low 12 Feed from Vendor 2 High Medium 30 Manual Updates High Low 12 Calcs. High 50 Audit Medium Low 7 Usability & Performance Low 3 Daily Reports High Low 12 Monthly Reports Low 3 End of Year Reports High Low 12 Feed to Regulator High Low 12 Total 153 Amadori Courses: Delivering an Effective Test Process reqs. 33

Method 2 • Estimate an overall number of requirements for the project as a whole and assign according Element % Weighting Feed from Vendor 1 15 Feed from Vendor 2 5 Manual Updates 10 Calcs. 25 Audit 10 Usability & Performance 5 Daily Reports 10 Monthly Reports 5 End of Year Reports 5 Feed to Regulator 10 Amadori Courses: Delivering an Effective Test Process 34

Element Feed from Vendor 1 Method 2 • Estimate an overall number of requirements for the project as a whole and assign according • So if we assume there will be exactly 200 requirements overall % Weighting Reqs 15 30 Feed from Vendor 2 5 10 Manual Updates 10 20 Calcs. 25 50 Audit 10 20 Usability & Performance 5 10 Daily Reports 10 20 Monthly Reports 5 10 End of Year Reports 5 10 Feed to Regulator 10 20 Amadori Courses: Delivering an Effective Test Process 35

Important !!!!! • Sometimes People can get all hung up on the accuracy or otherwise of their initial estimates Don’t!!! • Neither method is better or more accurate than the other… • They are just a quick and easy way to get you started • As you work through the detail of the various sources you can use this info to refine your original estimates Amadori Courses: Delivering an Effective Test Process 36

37 Example – Method 2 • Our initial estimate tells us that • But when we work through the document we discover only 27 separate requirements… • We could refine our model to take account of this but it would be a waste of effort • Only worth doing when your initial guesses turn out to be wildly under or over the mark Element Feed from Vendor 1 % Weighting Reqs 15 30 • If for example you only found 6 requirements this might suggest that there will be only be 40 requirements in total and it might then be worth recalculating your estimate Amadori Courses: Delivering an Effective Test Process

How do we use this data Element Testable Reqs with Issues Feed from Vendor 1 12 2 Feed from Vendor 2 7 3 Manual Updates 7 1 Usability & Performance 3 2 Daily Reports 5 1 39 10 Calcs. Audit • If we apply our estimates to the example shown below, we get a much more meaningful measure of preparedness Monthly Reports End of Year Reports Feed to Regulator Amadori Courses: Delivering an Effective Test Process 38

Method 1 Element Analysis Complete Feed from Vendor 1 potential reqs. Total Reqs coverage % Testable Reqs with Issues Y 12 2 14 85. 71% Feed from Vendor 2 Y 7 3 10 70. 00% Manual Updates Y 7 1 8 87. 50% Calcs. 50 0 0 50 0. 00% Audit 7 0 0 7 0. 00% 3 2 5 60. 00% Usability & Performance Y Daily Reports Y 5 1 6 83. 33% Monthly Reports 3 0 0 3 0. 00% End of Year Reports 12 0 0 12 0. 00% 5 1 6 83. 33% 39 10 121 32. 23% Feed to Regulator Overall Y 72 Amadori Courses: Delivering an Effective Test Process 39

Method 2 Element Analysis Complete potential reqs. Testable Reqs 0 12 Feed from Vendor 1 Y Feed from Vendor 2 Y 0 Manual Updates Y 0 Calcs. Audit Reqs with Issues Total Reqs coverage % 2 14 85. 71% 7 3 10 70. 00% 7 1 8 87. 50% 50 0 0 50 0. 00% 20 0 0 20 0. 00% Usability & Performance Y 0 3 2 5 60. 00% Daily Reports Y 0 5 1 6 83. 33% Monthly Reports 10 0 0 10 0. 00% End of Year Reports 10 0 0 10 0. 00% 0 5 1 6 83. 33% 90 39 10 139 28. 06% Feed to Regulator Y Amadori Courses: Delivering an Effective Test Process 40

41 You may initially encounter some resistance to the whole notion of “potential requirements” May be viewed as a waste of time and energy It gives a more accurate measure of the state of play as we have shown above You need to explain that Helps reveal areas the extent of areas where requirements are unclear or entirely absent Help identify key areas on which project as a whole needs to focus Amadori Courses: Delivering an Effective Test Process Using these methods in practice

Using these methods in practice • You should also ask the question • “If Testing do not clearly understand what is required in these areas what are the development team coding against? ” • Most defects stem not from poor coding but from unclear or invalid requirements • Or worst of all developers trying to help by filling the gaps themselves…. • Identifying and addressing these issues before coding starts will save a lot of time and money in the longer run Amadori Courses: Delivering an Effective Test Process 42

Dealing with Changing Requirements • You must agree a baseline against which initial test preparation will take place • Otherwise you are promising a potentially infinite amount of work and rework with only finite time and resources • You should agree the baseline with all parts of the projects particularly development so that their code references exactly the same set of requirements as your tests • Changes to this baseline should then be treated as Feature Requests and planned and resourced outside the initial testing phase Amadori Courses: Delivering an Effective Test Process 43

Next Steps • The purpose of extracting testable scripts is to allow a full set of testcases to be written against them • In the real world there may not always be time to write a full set of tests against every requirement • So you should classify each requirement according to its business criticality to allow maximum test coverage to be achieved with the time and resource available Amadori Courses: Delivering an Effective Test Process 44

45 Exercise • Review the requirements coverage metrics in the handout I am passing round now • What should be your 3 highest priorities in the limited time remaining to you and your team? • And Why? • What is expected level of risk to the project posed by any gaps in requirements coverage this leaves • How might you mitigate this risk? Amadori Courses: Delivering an Effective Test Process

Conclusion • This session has suggested some simple processes and techniques to assist Test Managers keep control of a project’s scope • It has demonstrated the best way to handle areas where requirements are incomplete and missing • Suggested a format by which useful data about requirements coverage can be communicated to stakeholders • And emphasised the point of establishing a baseline in terms of the requirements against which testing of a particular delivery will be carried out • In the next session we will look at ways of effectively managing test execution and defect management Amadori Courses: Delivering an Effective Test Process 46