Software Risk Assessment based on UML models Requirementsbased

Software Risk Assessment based on UML models Requirements-based and Performance-based Risk Assessment By: Kalaivani Appukkutty, MS thesis, WVU This work is funded in part by grants to West Virginia University NASA Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program (SARP) managed through the NASA Independent Verification and Validation (IV&V) Facility in Fairmont, West Virginia.

Overview Ø Introduction Ø Research Contribution Ø Requirements-based Risk Analysis Ø Performance-based Risk Analysis Ø Conclusions and Future work

Introduction • Risk is the possibility of suffering loss. – Software risk is defined as a combination of two factors: probability of failure of software and the severity of the failure • Risk Assessment is the process of identifying the potential risk and their impact – It identifies potentially troublesome software components that require careful development and allocation of more testing efforts • It can be performed at various phases of life software cycle.

Research Objectives Requirements-based Risk Assessment Ø ü ü ü To identify the requirements that are at high risk early in the software life cycle To assess requirements-based risk from UML models To develop a methodology that can be automated Performance-based Risk Assessment Ø ü ü To assess performance based risk taking into account the severity of performance failures To develop a methodology based on the UML profile for performance, schedulability and time.

Requirements-based Risk • Software risk is the product of : • • The probability of malfunctioning (failure) of the software The consequence of malfunctioning (severity) • The probability of failure is proportional to the complexity of the system. • Requirements-based risk is assessed for a requirement in a specific failure mode

Requirements-based Risk • Requirements are mapped to UML use case and scenarios • A failure mode of a scenario refers to the way in which a scenario fails to achieve its requirement • The risk factor of a particular scenario (a), in a particular failure mode (b) is: Rfab = Normalized Dynamic Complexityab X Severityab

Calculating Complexity • We estimate complexity as: Complexity = CC * Msg – CC : Mc. Cabe’s cyclomatic complexity of a scenario for each failure mode – Msg: The number of messages exchanged between the components in the sequence diagram to the point where the failure mode occurs in the scenario According to Fenton et al, “The product of cyclomatic complexity (CC) and sig. FF was shown to be a good predictor of fault proneness. . . The combined metrics appear to be better than both sig. FF and Cyclomatic complexity on their own and also better than the size metric”

Requirement Risk Matrix Failure Modes Requirements Scenarios R 1 S 1 FM 2 … FMn Rf S 2 R 2 S 3 … Rm Risk factor of scenario S 1 in Failure mode FM 2

Requirement-based Risk Assessment Methodology STEP 1: Map the requirements to UML sequence diagrams For each Sequence diagram STEP 2: Construct the control flow graph of the scenario from the sequence diagram STEP 3: Identify the different failure modes on the sequence diagram and the control flow graph For each failure mode STEP 4: Assess the severity of the failure mode using Function Failure Analysis (FFA) STEP 5: Determine the cyclomatic complexity of the subcontrol flow graph STEP 6: For the failure mode, measure the number of messages exchanged between the components in the sequence diagram. STEP 7: Calculate the complexity of the scenario for the failure mode as: Cyclomatic complexity * Number of messages STEP 8: Calculate the Risk factor of the scenario for the failure mode as: Complexity * Severity End For each failure mode End for each sequence diagram

Cardiac Pacemaker system A cardiac pacemaker is an implanted device that assists cardiac functions by pacing the heart, when the heart pulse fails.

Requirement R 2 of Pacemaker : Mapped to Atrial-Ventricular Inhibit (AVI) Scenario Atrial-Ventricular Inhibit Scenario: Failure Modes and Msg Values marked Control Flow Graph in the sequence diagram

Calculating Risk Complexity Failure Modes Cyclomatic Complexity (CC) Number of Messages (Msg) Complexity (CC * Msg) Normalized Complexity FM 1 3 4 12 0. 111 FM 2 6 7 42 0. 389 FM 3 6 9 54 0. 500 Severity • Severity is assessed using Function Failure Analysis (FFA) • The severity factor corresponding to the failure of requirement R 2 in failure modes FM 1, FM 2 and FM 3 is 0. 95 (catastrophic category) Risk Failure Modes Requirements Scenarios FM 1 FM 2 FM 3 R 2 AVI 0. 11 0. 37 0. 475

Requirement Risk Matrix (Cardiac Pacemaker) Requirements Scenarios Failure Modes FM 1 FM 2 FM 3 R 1 AVI 0. 11 0. 37 0. 475 R 2 AAI 0. 151 0. 283 0. 566 R 3 Programming FM 4 FM 5 0. 017 0. 233

Requirement Risk Chart (Cardiac Pacemaker)

Performance-based Risk Analysis • Considers the non-functional performance requirements of the system, which can sometimes be more important than the functional-requirements • Highlights the performance critical components and scenarios at an architectural-level • Uses the software and the system execution models which are well defined ways of modeling the performance aspects of a system

Performance-based Risk Analysis Methodology • For each scenario in each use case and for each scenario is that use case – STEP 1 - Assign demand vector to each action/interaction in sequence diagram and build a Software Execution Model for that scenario – STEP 2 - Add hardware platform characteristics on the deployment diagram and conduct stand-alone analysis – STEP 3 - Devise the workload parameters; build a System Execution Model and conduct contention-based analysis and estimate probability of performance failure – STEP 4 - Conduct severity analysis and estimate severity of performance failure for the scenario – STEP 5 - Estimate the performance risk of the scenario; Identify high-risk components

Step 1 • • Each “action” in the sequence diagram such as number of cpu instructions, amount of disk access and the data passed from one component to another A software execution model is built which considers the maximum service demand path of the scenario execution

Step 2 • • • The annotated Deployment Diagram is used for identifying the hardware platform characteristics on which the software system is deployed The system execution model is built by mapping the software execution model to the hardware component service times A standalone analysis is conducted by considering total service time of the scenario against the response time objective of that scenario

Step 3 • Contention-based analysis is performed by for objective workload and response time of the scenario is the total demand of the scenario and the is the maximum demand in that scenario Z 1: Failure probability (Z 1) = 0 Z 2: Failure probability (Z 2) = (UB-OBJ) / (UB-LB) Z 3: Failure probability (Z 3) = 1 UB stands for Upper Bound, LB for Lower Bound and OBJ stands for Performance Objective

Step 4 and 5 Step 4 Severity analysis is conducted to obtain the severity factor of the performance failures of the scenarios. We use a framework for conducting severity analysis of the performance failures of the scenario. The framework uses – Function Failure Analysis(FFA) – Failure Mode Effect Analysis(FMEA) – Fault Tree Analysis(FFT) Step 5 • Identification of performance-critical components - we first estimate the overall residence time of each component in a given scenario and then normalize it with the response time of the scenario • A high-risk component is said to be the one with a higher normalized service time in a scenario or across many scenarios

E-commerce case study The methodology is illustrated with a typical e-commerce case study

Sequence Diagram Annotations and Execution Graph • The sequence diagram shows the Place. Requisition scenario and the various performance annotations

Demand Vectors • The sequence diagram is converted to a software execution graph and the execution path with the longest service demand is considered for estimating the PLACE-REQ SPLIT RA 4 RS 1 CA 4 SPLIT CA 5 DOA 1 CI 3 OS 1

Service Time and Demand Vectors • The deployment diagram is annotated with hardware characteristics • The completion time of the Place Requisition scenario is equal to 0. 1326 seconds

Estimating Failure Probability The performance objectives are : • A response time objective of 1. 5 seconds • A workload of 15 customers The results are: • UB=2. 0295 sec • LB=1. 35 sec and OBJ=1. 5 sec • Probability of performance failure = 0. 7792 • • The severity assigned for this scenario is catastrophic (0. 95) The risk factor of this scenario is 0. 74024

Performance-based Risk: E-commerce case study

Residence Time of Components The components with taller bars(high-risk) and colored red(high-severity) are the ones that are more performance-critical and require further investigation and testing

Conclusion and Future Work • The requirements based risk assessment methodology presented, fills the gap between completely domainexpert dependant methods that are applied at the requirement analysis stage and formal analytical methods that do not assess risk at the requirements level • The performance-based risk assessment methodology gives a systematic approach to use UML profile for performance to assess risk based on failure probability and severity

Conclusion and Future Work • Future work will focus on developing tool support to assist the analyst with the steps of the methodology. • We also intend to apply the methodologies and supporting tools on large scale applications.

Publications • K. Appukkutty , Hany H. Ammar, K. Goseva-Popstojanova, “Software Requirement Risk Assessment Using UML”, Accepted in The 3 rd ACS/IEEE International Conference on Computer Systems and Applications, Jan 2005. • K. Appukkutty , Hany H. Ammar, K. Goseva-Popstojanova, “Early Risk Assessment of Software Systems”, Accepted in The 3 rd International Conference on Computer Science, Software Engineering, Information Technology, e-Business, and Applications, Dec 2004. • V. Cortellessa, K. Goseva-Popstojanova, K. Appukkutty, A. Guedem, A. Hassan, R. Elnaggar, W. Abdelmoez, and H. H. Ammar “Performance-based Risk Analysis of UML Models” submitted to IEEE Transactions on Software Engineering