Tests prioritization Tor Stlhane How to prioritize There

Tests prioritization Tor Stålhane

How to prioritize There are several ways to prioritize tests. We will however, focus on risk based prioritization. This lecture will cover • An introduction to risk assessment • How to use risk as a prioritization mechanism for tests • A small example

Risk-Based Testing Risk-based testing is not a new concept. Most companies that develop or buy software think risk, albeit often in an unstructured, undocumented manner. Only companies with a tradition for using risk analysis will use systematic methods for handling risk through analysis and testing.

Risk and stakeholders Risk has only meaning as a relationship between a system and its environment. Thus, what is a risk and how important it is, will vary from stakeholder to stakeholder. While the probability of an event is a system characteristic, the consequences will vary. Thus, we need to identify all stakeholders before we start to discuss and analyze risk.

Stakeholders - 1 We have two main groups of stakeholders, each with their own concerns if a risk is allowed to become a problem, e. g. : • Customers: lose money, either directly – e. g. an expensive failure – or indirectly – losing business. • The company that develops the system: lose business – e. g. lose marked shares.

Stakeholders - 2 All stakeholders must be involved in the risk assessment process. They will have different areas of expertise and experience. Thus, the methods we use must be simple to: • Use – no long training period needed. • Understand – people don’t have confidence in something they don’t understand.

Risk identification We start with the system’s use cases. Prepare the use case diagram for the function to be analyzed. Each participant should familiarize himself with the use case diagram. Start with a warm-up exercise, for instance going through results from previous risk identification processes. The warm-up exercise will help us to clear up misunderstandings and agree on a common process

Use case – high level example Review Treatment Plan Review Drug Data Doctor Review Documents Review Diagnoses Order Tests Lab Send Test Results

Use case – detailed example Update existing schedule Create new schedule Re(schedule) train conflicting schedules Control center operator <<extends>> Change the schedule of a train <<extends>> Change the schedule of a track maintenance activity

Risk identification from use cases For each function we need to consider: • How can this function fail? Identifies failure modes and what part of the code will contribute to this failure mode. • What is the probability that there are defects in this part of the code? • Which consequences can this failure have for the stakeholders? • Document results in a consequence table. • Assess the severity of each failure mode.

Consequence Table Subsystem: Function What the user wants to achieve Failure mode description How the function may fail Consequences Code involved System parts involved. How likely is it that they will cause the failure mode User Cust. Dev.

Risk assessment - 1 Even though risk assessment is a subjective activity it is not about throwing out any number that you like. To be useful, a risk assessment must be • Based on relevant experience. • Anchored in real world data. • The result of a documented and agreed-upon process. 12

Risk assessment - 2 Risk assessment uses the participants’ experience and knowledge to answer questions such as • Can this really happen; e. g. has it happened before? • Can we describe a possible cause consequence chain for the event? • How bad can it get? • How often has this happened in the past? 13

How to make an assessment We will look at the two main methods for risk assessment: • Qualitative risk assessment based on – The probability / consequence matrix – The GALE (Globally At Least Equivalent) method • Quantitative risk assessment based on the CORAS model

Qualitative assessment We can assess consequences, probabilities and benefits qualitatively in two ways. We can use: • Categories – e. g. High, Medium and Low • Numbers – e. g. values from 1 to 10. Note that this does not make the assessment quantitative – it is just another way to document the assessments. 15

Categories – 1 When using categories, it is important to give a short description for what each category implies. E. g. it is not enough to say “High consequences”. We must relate it to something already known, e. g. • Project size • Company turn-over • Company profit 16

Categories – 2 Two simple examples: • Consequences: we will use the category “High” if the consequence will gravely endanger the profitability of the project. • Probability: we will use the category “Low” if the event can occur but only in extreme cases. 17

Consequences and probability - 1 Consequences Probability H M L H H M L M L L 18

Consequences and probability - 2 The multiplication table is used to rank the risks. It can not tell us how large they are. In the general case, we should only use resources on risks that are above a certain, predefined level. 19

The GALE method is a method for making decision about introducing or not introducing a change in e. g. a process or construction. We will, however, only use the scoring scheme for risk assessment. The scoring scheme focuses on deviations from current average. This is reasonable, given that it is mainly concerned with comparing status quo to a new situation. 20

The GALE risk index is computed based on our assessment of an incident’s • Frequency score – how often will the event occur. The event here is a defect that has not been removed. • Probability score – what is the probability that the event will cause a problem • Severity score – how serious is the problem. The risk index I = FE + PE + S

Frequency score for event Frequency class Occurrences per project FE Very frequent 200 Every project 6 Frequent 100 Every few projects 5 Probable 40 Every 10 th project 4 Occasional 10 Every 100 th project 3 Remote 1 A few times in the company’s lifetime 2 Improbable 0. 2 One or two times during the company’s lifetime 1 Incredible 0. 01 Once in the company’s lifetime 0 22

Probability score for problem Classification Interpretation PE Probable It is probable that this event, if it occurs, will cause a problem 3 Occasional The event, if it occurs, will occasionally cause a problem 2 Remote There is a remote chance that this event, if it occurs, will cause a problem 1 Improbable It is improbable that this event, if it occurs, will cause a problem 0 23

Severity score for event Severity class Severe Average Minor Interpretation The portion of occurring problems that have serious consequences is much larger than average The portion of occurring problems that have serious consequences is similar to our average The portion of occurring problems that have serious consequences is much lower than average S 2 1 0 24

The CORAS model – 1 CORAS was developed as a framework for assessment of security risks. What should concern us here, however, is how they related the qualitative risk categories, not to absolute values, but to the company’s turnover. 25

The CORAS model – 2 Quantitative risks and opportunities give us real values. The usefulness of this is, however, limited since it is • Difficult to find real values for all risks. • Not obvious how we can compare qualitative and quantitative risks. When we use the CORAS tables it is important to remember that developers, customers and users will have different values – e. g. different company incomes. 26

The CORAS consequence table Consequence values Category Measured related to income Measured loss due to impact on business Insignificant Minor Moderate Major Catastrophic 0. 0 – 0. 1% 0. 1 – 1. 0% 1 – 5% 5 – 10% 10 – 100% Lost profits Reduce the resources of one or more departments Loss of a couple of customers Close down departments or business sectors No impact on business. Minor delays Out of business 27

The CORAS frequency table - 1 As we will see on the next slide, CORAS allows us to interpret frequency in two ways: • The number of unwanted incidents per year – e. g. the number of times a function will fail. • The failing portion of demands – e. g. number of service demands to a system. 28

The CORAS frequency table - 2 Frequency values Category Rare Unlikely Possible Likely Almost certain Number of unwanted incidents per Year 1/100 – 1/50 - 1 1 - 12 > 12 Number of unwanted incidents per demand 1/1000 (1/500) 1/50 (1/25) 1/1 Each five times the system is used Each tenth time the system is used Every second time the system is used Interpretation of number of demands Unwanted incident never occurs Each thousand time the system is used 29

An alternative consequence table We have been introducing risk based testing in a Norwegian company that develop software that is used in health care. This company have made a table including: • Service receiver – the patient • Service provider – the hospital • Developing company – e. g. the software developers

Consequence Levels Consequence level High – 3 Medium – 2 Low – 1 Service receiver Service provider One person killed or several persons seriously injured Several persons killed or many seriously injured Bad press Lose customer One person seriously injured User looks bad to his superior(s) or large sums lost Dissatisfied customers or users Minor irritant, small amount of money lost or wasted - Minor irritant Developing company

Earlier test experiences – Worry list Testing is a sampling process. Thus, if we find a lot of defects in a component or sub-system, we should conclude that this component has many defects. The conclusion will not necessarily be the same if we conclude based on an inspection.

The Worry List Consequences High (3) Number of errors registered High 10 Medium 3– 9 Low 0 -2 Medium (2) Low (1)

Testing and resources We will always have limited resources for testing. This is made worse by the fact that testing often starts late in the project We make best use of the available resources by trying to minimize the total system-related risk.

Risk-Based Testing – 1 Risk-based testing has the following steps • Identify how the product interact with its environment. This is necessary to understand failure consequences • Identify and rank risks – probability and consequence. If this is a white box or grey box test we should also identify possible causesevent chains to understand the failure mechanisms.

Risk-Based Testing – 2 • Define tests that can be used to ensure that the code defect probability is low. – Black box test – apply e. g. random testing or domain testing. – White box or grey box test – make sure all identified cause-event chains are exercised. • Run the tests – highest risk first

ATM example – 1

ATM example – 2 Subsystem: ATM transaction Function Failure mode description Consequences Code involved User Cust Dev. Withdrawal Withdraw more than allowed Wrong amount registered Wrong account W-1, M-1 Acc-2 L H L M H H H Deposit Wrong amount registered Wrong account M-1, V-1 Acc-2 H H H Inquiry Wrong account Wrong value returned Acc-2, V-1, M-1 M M L M M M

ATM example – 3 Frequency class Occurrences per project FE Very Frequent 200 Every project Frequent 100 Every few projects 5 Probable 40 Every 10 th project 4 Occasional 10 Every 100 th project 3 Remote 1 A few times in the company’s lifetime 2 Improbable 0. 2 One or two times during The company’s lifetime 1 Incredible 0. 01 Once in the company’s lifetime 0 Function Deposit – wrong amount registered Inquiry – wrong account 6 Classification Interpretation PE Probable It is probable that this event, if it occurs, will cause a problem 3 Occasional The event, if it occurs, will occasionally cause a problem 2 Remote There is a remote chance that this event, if it occurs, will cause a problem 1 It is improbable that this event, if it occurs, will cause a problem 0 Improbable Components S FE PE I M-1, V-1 2 4 3 9 Acc-2, V-1 1 3 2 6