Riskbased regression testing in a telecommunication system node

Risk-based regression testing in a telecommunication system node Master’s thesis presentation 14. 8. 2007 Author: Mika Lehto Supervisor: Professor Heikki Hämmäinen Instructor: M. Sc. Pekka Hämäläinen

Agenda § § Background and objectives Risk-based testing Regression testing Case Study – – – Ericsson’s Media Gateway for Mobile Networks (M-MGw) M-MGw Node Functional Verification Risk model Risk analysis and prioritization Scenarios for the case study Results § Conclusions § Future research Ericsson Confidential 2 2007 -07 -20

Background § § § The costs of regression testing comprise 20 -30 % of the whole testing costs on average in software development → regression testing is often the target of cost reduction activities in order to achieve shorter time-to-market and lower cost structure of the product The efficiency of regression testing is usually attempted to improve by using regression testing methods such as regression test selection, test suite reduction, and test case prioritization One approach that has been studied is called risk-based regression testing but only a little information about the feasibility of risk-based regression testing is currently available Ericsson Confidential 3 2007 -07 -20

Objectives § The primary objective of this thesis was to study whether it would have been beneficial to use risk-based regression testing in an old M-MGw Node Functional Verfication project § This thesis also analyzed the differences in test case selections between previously used and risk-based test case selections methods § The aim of this thesis was also to demonstrate how risk -based testing approach can be utilized in regression testing Ericsson Confidential 4 2007 -07 -20

Risk-based testing § The fundamental difference between traditional testing and risk-based testing is that risk-based testing brings formal risk assessment methods (risk identification, risk analysis, and risk prioritization) in testing process § The aim of risk-based testing is to focus the test effort on the most critical areas of the system § Risk-based testing approach can be used for instance to: – Select the most important test objects and test cases – Evaluate the current level of system quality and risk – Provide input for the decision to continue or stop testing Ericsson Confidential 5 2007 -07 -20

Regression testing § “Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements” (IEEE Software glossary) § Regression testing is typically used during iterative software development and to support application maintenance § The biggest challenge in regression testing is the test case selection § In risk-based regression testing, formal risk analysis methods are used to select the most risky test cases for execution Ericsson Confidential 6 2007 -07 -20

Ericsson’s Media Gateway for Mobile Networks (M-MGw) § § § M-MGw operates in the connectivity layer of the core network and its fundamental purpose is to connect the core network to other networks such as UTRAN and PSTN M-MGw also supports media conversion, bearer control and payload processing M-MGw is remotely controlled by the MSC or SGSN Servers which operate in the network control layer. Ericsson Confidential 7 2007 -07 -20

M-MGw Node Functional Verification § § § Testing in the NFV is based on requirements (e. g. Specifications and call flows) NFV is a testing phase in which the M-MGw software is first time tested in a real node environment Test environment is established of one M-MGw node and all the external interfaces to this node are simulated Ericsson Confidential 8 2007 -07 -20

Risk model § The case study was based on a qualitative risk model and a risk was measured as a risk exposure: Risk Exposure = Probability x Consequence § Risk items were categorized into three classes according to the probability and consequence of a risk: tolerable, undesirable, and intolerable Ericsson Confidential 9 2007 -07 -20

Risk analysis and prioritization § Risk analysis was performed both for the test objects and test cases using systems experts – § 2 systems experts participated into the risk analysis of test objects and 8 systems experts analyzed the risks related to test cases Test objects and test cases were prioritized according to risk exposure – In a case of test case prioritization, also other prioritization criteria (consequence, old test case priority) were needed because several test cases had the same risk exposure Ericsson Confidential 10 2007 -07 -20

Scenarios § § § The analysis was based on 5 scenarios in which the amount of executed test cases was changed In scenario 1, as many test cases were selected for execution per test object as in the original case but the test cases were selected based on risk In other scenarios, the test effort was focused on the most risky test objects by defining how many percent of the test cases per test object were executed – – Scenario 2: Intolerable 25%, Undesirable 15%, Tolerable 10% Scenario 3: Intolerable 20%, Undesirable 10%, Tolerable 5% Scenario 4: Intolerable 15%, Undesirable 10%, Tolerable 0% Scenario 5: Intolerable 10%, Undesirable 5%, Tolerable 0% Ericsson Confidential 11 2007 -07 -20

Results – used man hours § § § Significant savings in used man hours would have been achieved in scenarios 3 -5 In the original case, the used man hours were distributed equally to all test objects In risk-based approach (i. e. scenarios 2 -5), approximately 55 % of man hours were consumed to 20 % of test objects Ericsson Confidential 12 2007 -07 -20

Results – risk level § § § Risk level defines how many percent of the total risk is still uncovered Risk level would have been 10 percentage units lower in the original case if the test cases had been selected based on risk In the scenarios 3 -4, risk level is at the same level than in the original case even though much less man hours were consumed Ericsson Confidential 13 2007 -07 -20

Results – detected faults § § § Only 40 % of the faults were found in scenario 1 in which the test cases were selected based on risk but the test effort was not concentrated on the most risky test objects By concentrating also the test effort to the most risky test objects, 40 -60 % of the faults were found Deeper analysis to these poor fault-detection rates revealed that the risk analysis of test cases of TO_04 had failed and caused faults to slip through testing Ericsson Confidential 14 2007 -07 -20

Results – test case selections § § Differencies in test case selections between previously used and risk-based approach was analyzed by determining test case coverage which defines how many percent of test cases executed in the original case was also executed in designed scenarios In addition, the average risk exposure per test case was determined to each scenario Ericsson Confidential 15 2007 -07 -20

Conclusions § Risk-based regression testing would not have brought cost savings in used man hours because the costs of the faults that slipped through testing would have eliminated all the gained cost savings § Testing effort was not concentrated on the most risky test object in the original case § It can not be said that testers would have utilized riskbased testing approach when selecting regression test cases in the original case Ericsson Confidential 16 2007 -07 -20

Future Research § A possible pilot project for risk-based testing in order to get more empirical experience how risk-based testing works in practice § The concept and usage of risk level should be investigated more because risk level could be possible used as an indicator to tell when it is time to stop testing § To develop a framework for assessing the costs of the faults quantitatively Ericsson Confidential 17 2007 -07 -20

Ericsson Confidential 18 2007 -07 -20