RISKBASED TESTING Liza Katz liza katzgmail com 054

RISK-BASED TESTING Liza Katz liza. katz@gmail. com 054 -4977802

RISK-BASED TESTING: SETTING THE COURSE For any complex system, an infinite number of tests exist … … but we can’t forever test! So, out of the infinite cloud of possible tests, testers must select a finite number of tests, and we need to be very selective and smart about our test coverage

"Quality is never an accident; it is always the result of intelligent effort. “ John Ruskin (1819 -1900) Leading English art critic of the Victorian era “Software testing is the art of trying to increase confidence in a product by pointing out everything that is wrong with it. ”

WHAT IS SOFTWARE QUALITY The degree to which a system, component , or process meets specified requirements. The degree to which a system, component , or process meets customer or user needs or expectations. (IEEE definition)

THE CHALLENGES Time constrains Resource Constrains Quality Requirements Risk Factors: � New technology � System complexity � Lack of knowledge or experience � Etc…

WHAT IS RISK-BASED TESTING Risk-based testing (RBT) prioritizes the features and functions to be tested based on the risk they represent, a function of their importance and likelihood or impact of failure Quality Risk – the possibility that product or system might fail to deliver one or more of the key quality attributes, endangering our ability to achieve the quality outcomes we want

RISK BASED TESTING (RBT)PROCESS • Stakeholders Objectives • Risk Factors • Functional break up • Impact Analysis • Probability of Failure • Risk Categorization • Weighted Averages Identification Analysis Risk Exposure RBT Governance Test Planning Testing Policy • Progressive Monitoring • Readiness Calculation • Reporting • Effort Distribution • Test Case Prioritization • Test Scope Coverage • Testing Techniques

TYPES OF RISKS Business or Operational � High use of a subsystem, function or feature � Criticality of a subsystem, function or feature, including the cost of failure Technical � Geographic distribution of development team � Complexity of a subsystem or function External � Sponsor or executive preference � Regulatory requirements

RISK IDENTIFICATION The activity of identifying risk answers these questions: � Is there risk to this function or activity? � How can it be classified? Risk identification involves collecting information from stakeholders about the project and classifying it to determine the amount of potential risk in the test phase and in production (in the future).

RISK IDENTIFICATION - EXAMPLE Capability Can it perform the required functions? Reliability Will it work well and resist failure in all required situations? Usability How easy is it for a real user to use the product? Performance How speedy and responsive is it? Installability How easily can it be installed onto its target platform? Compatibility How well does it work with external components & configurations? Supportability How economical will it be to provide support to users of the product? Testability How effectively can the product be tested? Maintainability How economical will it be to build, fix or enhance the product? Portability How economical will it be to port or reuse the technology elsewhere?

RISK ANALYSIS Analysing risks means determining the effect or impact of potential risks. Risk analysis involves asking questions such as: � Is this a risk or not? � How serious is the risk? � What are the consequences? � What is the likelihood of this risk happening? Decisions are made based on the risk being analysed. The decision(s) may be to mitigate, manage or ignore.

RISK EXPOSURE Very often is it impossible to quantify this accurately, but the use of high-medium-low (1 -2 -3) may be good enough to rank the individual functions. By combining the consequence and the probability (from risk identification above) it should be possible to rank the individual functions of a system. The ranking could be done based on “experience” or by empirical calculations.

RBT GOVERNANCE In the test phase it is important to monitor: Number of test planned vs. executed vs. completed Number of defects found, number of defects per function, classification of defects, number of hours testing per defect, number of hours in fixing per defect etc. Readiness and completion calculations The metrics can be reported graphically and trend analysis applied.

CASE STUDY

SYSTEM UNDER TEST (SYSTEM TEST) provisioning inventory performance assurance OSS layer element management network & service monitoring service activation Network Management System under test virtual network abstraction network

TEST DIMENSIONS AND VARIABLES

SCOPE Functional System Test Black box testing of features and flows. Non-functional System Test Scale and performance, installation, migration, and documentation.

STAKEHOLDERS Product management Geographically distributed between Israel and US System architecture Development Utilizing Agile development methodology Geographically distributed between Israel, US and India Project Management Executive management Customers

THE CHALLENGE Transition from sustaining mode to development mode with large amount of open defects and not fully tested functionality Large number of developers new to the code Utilization of a new development methodology– Agile. Functional test moved to development responsibility. Very short release cycles – 3 -4 months 3 to 4 releases per year Geographically distributed project team – Israel, US, India Scale requirements

THE STRATEGY Risk Analysis was performed to identify the most critical areas. The System Test Strategy document was rewritten to reflect Risk-Based approach and performed Risk Analysis, including list of Test and Development teams test responsibilities. The System Test Process and Organisation was improved. This included defining the test process by preparing procedures, planning the test, controlling the process, progress tracking, and defining roles and responsibilities Automated Testing used extensively.

RISK-BASED APPROACH Risk Type Risk Identification Examples of correlation with system components Business or Operational Strategic - anything that has special importance to the business. CE support, NCCM, Activation Scripts, Scale, profiling, fault management, … Popular - anything that will be used a lot. Installation, migration, topology discovery, pathtrace… Critical - anything whose failure could cause substantial damage Installation, migration, fault management, … New - anything that has no history in the product. Management Reachability, TFS (Trap Forwarding Service) GUI, Win 7, … Changed - anything that has been changed or improved Fault management, … Impacted – anything that can be affected by new or changed items Severity Aspects, … Buggy - anything known to have a lot of problems VLAN Discovery, correlation, … Recent failure - anything with a recent history of failure Severity Aspects, … Technical

Analysis Example 160 l 2 vpn-discovery 122 6% P 1 140 logical-inv 91 5% VNE P 1 120 fault-mgmt-gw 89 4% P 1, P 2 100 nccm-neim 61 3% P 2 tech-ipran 56 3% P 1 tech-core 50 3% P 2 migration 50 3% P 1, P 2 Component physical-inv vne-fw topology fault-mgmt-gw logical-inv gateway-alarms gw-fw gateway-general iox-vne-drivers alarms-parsing topo-discovery database gui-fw Bug Count Bug % Test Priority 44 9% VNE P 1 40 8% P 2 26 5% P 1, P 2 23 5% P 1, P 2 22 5% VNE P 1 22 5% P 1, P 2 20 4% P 1, P 2 19 4% VNE P 1 15 3% VNE P 1 14 3% P 1, P 2 12 3% P 1, P 2 10 2% P 1, P 2 5% 67 4% 4% 4% 3% 20 61 3% 56 3% 4% 50 50 3% 2% 3% 3% 0 1% 0% lul t-m inv gm t-g w vn efw nc cm -c a gu i-f w nc cm ev -g cui di sc ov er nc cm y -n ei m te ch -ip ra n te ch -c or e m ig ra tio n P 1 78 CFDs’ S 1 -S 3 – 3. 6. 3 -3. 7. 1 50 45 40 35 30 25 20 15 10 5 0 44 1/1/2009 – 10/20/2010 40 26 9% 23 22 22 8% 5% 5% 20 4% 19 4% 15 3% 14 12 10 3% 3% 2% t-m y gm t-g w lo gi ga c al te w ay inv -a la rm s ga gw te w -fw ay -g io en xer vn al edr al iv ar er m s sp to ar po si ng -d is co ve ry da ta ba se gu i-f w 3% 81 fa 67 og evc-discovery 40 po l P 1, P 2 5% to 4% 83 6% fa ul 78 60 er y nccm-gui 89 gi ca P 1, P 2 fw 4% -fw 81 7% di gui-fw 89 80 gw P 1, P 2 n- 4% vp 83 l-i nv nccm-ca l 2 P 1, P 2 ic a 4% 91 ys 89 7% 122 6% ph vne-fw 8% 134 ov P 1 lo 7% e- 134 IFDs’ S 1 -S 3– 3. 7. 1 Test Priority vn gw-fw Bug Count Bug % sc Component 10% 9% 8% 7% 6% 5% 4% 3% 2% 1% 0%

Components Risk Matrix

PROGRESS TRACKING System Test Overall - Planned vs. Actual

DEFECT TRACKING

BENEFITS OF RISK-BASED TESTING - SUMMARY Running the tests in risk order gives the highest likelihood of discovering defects in severity order. Allocating test effort based on risk is the efficient way to minimize the residual quality risk upon release. Measuring test results based on risk will allow organization to make smart release decisions. If schedule requires, dropping tests in reverse risk order reduces the test execution period with the least possible increase in quality risk. With risk-based testing, you can show management that you strive to make the best use of the resources they invest.

FINALLY… If two vital factors need to be chosen to make risk-based testing work, it would be … experience - Over a period of time, any product line or technology will reveal its pattern of characteristic problems. Learn from that. teamwork - Do whatever you can to invite different people with different points of view into the risk analysis process.

THANK YOU FOR LISTENING