IVV Facility FY 2002 Initiative IVV of UML

IV&V Facility FY 2002 Initiative IV&V of UML Less risk, sooner. A Catch Phrase by Coach Menzies WVU UI: Architectural-level Risk Assessment Hany Ammar, Katerina Goseva-Popstojanova, V. Cortelessa, Ajith Guedem, Diaa Eldin Nassar, Walid Abdel. Moez, Ahmad Hassan, and Rania Elnaggar LANE Department of Computer Science and Electrical Engineering West Virginia University Ali Mili, Bo Yu College of Computing Science New Jersey Institute of Technology 1

Outline IV&V Facility • • • Objectives What we can do Why UML & NASA Project Overview Architecture-Based Risk Analysis The Risk Assessment Methodology Performance – based risk Accomplishments Future Work Publications 2

Objectives IV&V Facility Before bad software After bad software The ARIANE 5 explosion • Automated techniques V&V of dynamic specifications – Performance and timing analysis – Fault-injection based analysis, • Less risk, sooner – Risk assessment • Technologies: – UML – Architectures – Risk assessment methodology • Benefits: – Find & rank critical • use cases, scenarios, • components, connectors 3

What We Can Do IV&V Facility Not contributing Minor Major Critical Catastrophic • Identify and rank critical components based on risk factors and severity classes • How? - details follow 4

Why UML IV&V Facility • Unified modeling language – Rational software – The three amigos: Booch Rumbaugh, Jacobson. • International standard in system specification An international standard In system specification 5

UML & NASA IV&V Facility • Increasing use at NASA • Informal (very) survey – – Google search: “rational rose nasa” 10, 000 hits 3 definite projects, just in first ten • We use a case study based on the UML specs of a component of the International Space Station 6

Project Overview IV&V Facility FY 01 • Developed of an automated simulation environment for UML dynamic specification, suggested an observer component to detect errors • Conducted performance and timing analysis of the NASA case study FY 02 • Develop a fault injection methodology Define a fault-model for components at the specification level • Develop a methodology for architecture-based risk analysis Determine critical use case List Determine critical component/connector list (based on recent paper by Yacoub & Ammar on IEEE Trans. on Software Engineering, June 02) FY 03 • Develop a methodology for performance-based/reliability-based risk assessment • Validation of the risk analysis methodology on several NASA projects 7

Architecture-Based Risk Analysis IV&V Facility • Develop architecture-based approach for risk assessment – Overall system/subsystem – Different use cases – Key scenarios associated with use cases • Heavily used scenarios • Scenarios that are used infrequently but perform critical functions • Develop components and connectors risk factors – Define components risk factors as Normalized dynamic complexity * Severity – Estimate dynamic complexity measure based UML sequence diagrams and state charts – Estimate severity measure based FEMA and hazard analysis – Consistent with the NASA definition of risk Probability of an undesired event * Consequences if that event should occur – Define connectors risk factor as Normalized dynamic coupling * Severity 8

Risk Assessment Methodology IV&V Facility • • For each use case – For each scenario • For each component – Measure dynamic complexity – Assign severity based on FMEA and hazard analysis – Calculate risk factor • For each connector – Measure dynamic coupling – Assign severity based on FEMA and hazard analysis – Calculate risk factor • Construct Markov model • Calculate scenario level risk factor • Determine critical component/connector list – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 9

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – – – • For each connector – – – • • • Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor Measure dynamic coupling Assign severity based on FEMA and hazard analysis Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 10

IV&V Facility NASA CASE STUDY Use Case Diagram 11

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – – – • For each connector – – – • • • Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor Measure dynamic coupling Assign severity based on FEMA and hazard analysis Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 12

Both Pumps Retry- scenario IV&V Facility 13

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – Measure dynamic complexity – Assign severity based on FMEA and hazard analysis – Calculate risk factor • For each connector – – – • • • Measure dynamic coupling Assign severity based on FEMA and hazard analysis Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list • – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk • • Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 14

Component Dynamic Complexity IV&V Facility Dynamic Complexity of a component Oi in scenario x is defined as : Where CCx(oi ) =ti – ci + 2 Is the cyclomatic complexity of a component Oi in scenario x Cx(oi ) : A finite set of states for a component Oi for a scenario x , ci is the cardinality of this set. Tx(oi ) : A finite set of transitions from one state to another for Oi component in a scenario x, ti is the cardinality of this set. Ox : is the set of components collaborating during the execution of a scenario 15

Component Severity (FEMA) IV&V Facility 16

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – – – Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor • For each connector – Measure dynamic coupling – Assign severity based on FEMA and hazard analysis – Calculate risk factor • • • Construct Markov model Calculate scenario level risk factor Determine critical component/connector list • – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk • • Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 17

Connector Dynamic Coupling IV&V Facility Dynamic coupling for connector between component Oi and component Oj is the percentage of the number of messages sent from Oi to OJ with respect to the total number of messages exchanged during the scenario x MTx(Oi , Oj) : is the set of messages sent from component Oi to component Oj during the execution of scenario x MTx : is the set of total messages exchanged between all components during the execution of scenario x. 18

Connector Dynamic Coupling IV&V Facility • Dynamic Coupling for Connector between component C 1 and component C 2 is EOC(C 1, C 2) =2/5=0. 4 C 2 C 1 C 3 M 1 M 2 M 3 M 4 M 5 19

Connector Severity (FEMA) IV&V Facility 20

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – – – • Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor For each connector – – – Measure dynamic coupling Assign severity based on FEMA and hazard analysis Calculate risk factor • Construct Markov model • Calculate scenario level risk factor • Determine critical component/connector list • – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk • • Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 21

The Markov Chain Model for Both Pumps Retry- scenario IV&V Facility FAILURE STATES OF VARIOUS SEVERITIES PFMC_M T S RPCM_MT Minor Major PFMC_LT FRITCS Critical SCITCS RPCM_LT T Catastrophic 22

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – – – • For each connector – – – • • • Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor Measure dynamic coupling Assign severity based on FEMA and hazard analysis Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list • • • Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 23

IV&V Facility Distribution of risk factors of each scenario over the severity classes 24

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – – – • For each connector – – – • • • Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor Measure dynamic coupling Assign severity based on FEMA and hazard analysis Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list • Calculate system level risk • • Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope 25

Overall System risk distribution over the severity classes IV&V Facility MINOR MARGINAL CRTICAL CATASTROPHIC 0. 3014 0. 0103 0. 2192 0. 2879 The overall system risk factor is: 0. 8189 26

Sensitivity analysis of components IV&V Facility 27

Risk Assessment Methodology IV&V Facility • For each use case – For each scenario • For each component – – – • For each connector – – – • • • Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor Measure dynamic coupling Assign severity based on FEMA and hazard analysis Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list • – Calculate use case level risk factors – Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system scope 28

IV&V Facility Determine Critical Component/Connector List Not contributing Minor Major Critical Catastrophic 29

Performance – based risk IV&V Facility • Performance failure is the inability of the system to meet its performance objective(s) • Define components performance-based risk as Normalized component demand factor * Severity X 1 X 2 X 3 T 11 T 21 D 21 T 22 D 22 T 31 T 23 D 23 is demand for resource (e. g, CPU, disk, etc. ) in state Tij (state j of component i ) Scaling vector scales the resource demands accordingly to the corresponding service times of the resources D 11 = D 12 T 32 D 31 D 32 30

Performance – based risk IV&V Facility • Total demand of component xi in a scenario Sk is • Overall demand of a scenario Sk is where m is total number of components and l total number of states for a given component in a given scenario • Normalized demand factor of component xi in Scenario Sk DFi = ( . SCT) / ( . SCT) 31

Accomplishments IV&V Facility • Developed analytical techniques and a methodology for Architecture-Based Risk Analysis • A lightweight approach based on static analysis of dynamic specifications is developed and automated • A tool will be presented in the Tools session • Applied the methodology and tool to the NASA case study 32

Future Work IV&V Facility • The main thrust of our future work will be in the development of a cohesive methodology for performance- based /reliability- based risk assessment • Compare risk factors based on other Complexity and coupling metrics obtained from static analysis of UML dynamic specs. – COSMIC-Full Function Point measurement maybe a good complexity predictor. – COCOMO II’s effort prediction may be another good complexity predictor • Validation of methodology using several NASA case studies 33

Publications IV&V Facility 1. 2. 3. 4. 5. 6. 7. Sherif M. Yacoub, Hany H. Ammar , “A Methodology for Architecture-Level Reliability Risk Analysis, ” IEEE Transactions on Software Engineering, June 2002, pp. 529 -547 H. H. Ammar, T. Nikzadeh, and J. B. Dugan "Risk Assessment of Software Systems Specifications, " IEEE Transactions on Reliability, September 2001 Hany H. Ammar, Sherif M. Yacoub, Alaa Ibrahim, “A Fault Model for Fault Injection Analysis of Dynamic UML Specifications, ” International Symposium on software Reliability Engineering, IEEE Computer Society, November 2001 Rania M. Elnaggar, Vittorio Cortellessa, Hany Ammar, “A UML-based Architectural Model for Timing and Performance Analyses of GSM Radio Subsystem” , 5 th World Multi. Conference on Systems, Cybernetics and Informatics, July. 2001, Received Best Paper Award Ahmed Hassan, Walid M. Abdelmoez, Rania M. Elnaggar, Hany H. Ammar, “An Approach to Measure the Quality of Software Designs from UML Specifications, ” 5 th World Multi. Conference on Systems, Cybernetics and Informatics and the 7 th international conference on information systems, analysis and synthesis ISAS July. 2001. Hany H. Ammar, Vittorio Cortellessa, Alaa Ibrahim “Modeling Resources in a UML-based Simulative Environment”, ACS/IEEE International Conference on Computer Systems and Applications (AICCSA'2001), Beirut, Lebanon, 26 -29 June 2001 A. Ibrahim, Sherif M. Yacoub, Hany H. Ammar, “Architectural-Level Risk Analysis for UML Dynamic Specifications, ” Proceedings of the 9 th International Conference on Software Quality Management (SQM 2001), Loughborough University, England, April 18 -20, 2001, pp. 179 -190 34