Introduction to Risk Management And Software Architecture Risk

ﺑﺴﻢ ﺍﻟﻠﻪ ﺍﻟﺮﺣﻤﻦ ﺍﻟﺮﺣﻴﻢ ﻭﺍﻟﺼﻼﺓ ﻭﺍﻟﺴﻼﻡ ﻋﻠﻰ ﺭﺳﻮﻝ ﺍﻟﻠﻪ ، ﺍﻟﺤﻤﺪ ﻟﻠﻪ Introduction to Risk Management And Software Architecture Risk Assessment Hany H. Ammar LANE Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, West Virginia, USA, and Faculty of Computers and Information, Cairo University, Cairo, Egypt 1

OUTLINE • Risk Management • Software Architecture Risk Assessment – Maintainability-based risk • Conclusions • Next Steps SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 2

Risk Management SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 3

SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 4

For NASA Programs • RISK MANAGEMENT: An organized, systematic decision-making process that efficiently identifies risks, assesses or analyzes risks, and effectively reduces or eliminates risks to achieving program goals. • RISK: A Program “Risk” is any circumstance or situation that poses a threat to: crew or vehicle safety, Program controlled cost; Program controlled schedule; or major mission objectives, and for which an acceptable resolution is deemed unlikely without a focused management effort SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 5

Risk Management Cycle Identify: Identify that a risk exits and give it a meaningful name. Analyze: Determine the severity of the risk according to the risk matrix. If the risk is negligible (low to medium severity, low likelihood of occurrence), stop here. However, if the risk could cause damage to the system or the system's users, continue. Plan: Decide how to combat the risk based on the risk's severity and likelihood of occurrence. Mitigate: Follow the plan formulated in the previous phase as closely as possible to combat the risk. If this approach does not work, return to the previous phase and make a new plan. If the plan does work, continue analyzing the risk to determine whether it has been reduced to an acceptable severity level. Track: Once the risk has been mitigated to an acceptable severity level, the risk should be tracked to ensure the continued control of the risk. If at any time the risk seems to resurface, the risk management cycle should begin again, starting with the analysis phase. SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 6

SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 7

Risk Definition • According to NASA Software Safety Technical Standard, risk is defined as: “exposure to the chance of injury or loss. It is a function of the possible frequency of occurrence of the undesired event, of the potential severity of resulting consequences, and of the uncertainties associated with the frequency and severity”. • For software intensive systems, a risk is a combination of a likelihood of occurrence of an abnormal event or failure and the potential consequences or severity of that event or failure to a system's operators, users, or environment SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 8

Risk Matrix Likelihood of Occurrence Severity Probable Occasional Catastrophic High Critical High. Medium Marginal Negligible Remote High. Medium Improbable Medium. Low High. Mediu m Medium. Low Low SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 9

SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 10

NASA IV&V Facility NPD 2820. 1 C for Software IV&V Policy states: "Task the IV&V Facility in Fairmont, West Virginia to manage the performance of all IV&V for software identified per the established criteria, and for any other safety critical software (as defined in NASA-STD-8719. 13)" SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 11

IV&V Function • Software Independent Verification & Validation (IV&V) is a systems engineering process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the software life cycle. • Software IV&V is adapted to the characteristics of the project. Different projects require different level of IV&V SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 12

IV&V Lifecycle Activities System Preliminary Requirements Design Review Initial IVVP Signed Baseline IVVP Signed Concept Phase 2. 0 Critical Design Review S/W FQT System Test - IV&V provides support and reports for Project milestones - Technical Analysis Reports document major phases - IVVP is updated to match changes in Project Requirements Phase 3. 0 Design Phase 4. 0 Implementation Phase 5. 0 IV&V Phase Independent Support 1. 0 Test Phase 6. 0 Mission System Readiness Retirement Launch Review IV&V Final Provides Report Co. FR Operations & Maintenance Phase 7. 0 Note: numbers correspond to IV&V WBS • Life-cycle IV&V is designed to mesh with the Project schedule and provide timely inputs to mitigate risk • Dialog between the IV&V Facility and the Project must begin before SRR • For most Projects, IV&V ends (and the Final Report is delivered) on or about MRR. Some Projects have extended S/W development post -launch or major upgrades/maintenance (e. g. Shuttle, MER) SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 13

Software Project Resolution is commonly categorized into three resolution types: 1. Successful Projects – Completed and operational, and: • On Schedule • On Cost • With all originally specified features and functions 2. Challenged Projects – Completed and operational, but: • Behind Schedule------- Project Risk • Over Cost-------- Project Risk • With fewer features and functions than originally specified ------ Product Risk 3. Failed Projects: – Cancelled before completion or never implemented SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 14

Software CHAOS The Standish Group has examined 30, 000 Software Projects in the US since 1994. This "CHAOS" research has revealed a decided improvement in IT project management with the implementation of standards and practices such as IV&V. This improvement correlates with the rise in project success depicted in the chart below: Project Resolution History (1994 -2000) The Standish Group International, Inc. : Extreme CHAOS (2001) - The 2001 update to the CHAOS report. http: //www. standishgroup. com/sample_research/PDFpages/extreme_chaos. pdf SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 15

Error Detection/Correction Early error detection and correction are vital. The cost to correct software errors multiplies during the software development lifecycle. Early error detection and correction reduce costs and save time. Direct Return on Investment of Software Independent Verification and Validation: Methodology and Initial Case Studies, James B. Dabney and Gary Barber, Assurance Technology Symposium, 5 June 2003. SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 16

IV&V Characteristics • Includes Risk Identification and Mitigation Techniques • Provides Independent Evaluation / Assessment of: – Are we building the product right? = Verification – Are we building the right product? = Validation • Requires Technical, Managerial and Financial Independence • Makes a value added contribution, everyone shares the same mission success objective – For NASA Management - Provides Mission Assurance – For Project Management - Provides Unbiased Source of Help • Helps deliver – Risk Identification and Mitigation – Increased Quality and Safety – Improved Timeliness and Reliability – Reduced Rework Cost NPD 8730. 4: Requires NASA programs and projects that contain mission or safety critical software to document decisions concerning the use of IV&V. SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 17

OUTLINE • Risk Management • Software Architecture Risk Assessment – Reliability-based risk – Performance-based risk – Maintainability-based risk – Component Ranking • Conclusions • Next Steps SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 18

Software Architecture Risk Assessment This work is funded in part by grants to West Virginia University Research Corp. from the NSF (ITR) Program, and from the NASA Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program (SARP) managed through the NASA Independent Verification and Validation (IV&V) Facility, Fairmont SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 19

Project Overview • Risk Assessment of software architecture components, usage What keeps satellites working 24/7 ? scenarios, and requirements • Risk definition is based on * Frequency of abnormal events * Severity or consequences of events • • • Reliability-based risk, Performance-based risk Requirement–based risk Severity Analysis Maintainability-based risk SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 20

Software Architecture Risk Assessment • An architecture-based approach for risk assessment Componentsconnectors, requirements, and scenario risk, • Define several types of risk factors Reliability-based Risk [IEEE Trans. on Rel 2001, on SE, 2002, 2003] Probability of failure * Severity or Consequences of this failure Maintainability-based Risk [RAMS 06, ICSM 05, ICSM 04] Probability of performing maintenance task * Cost of performing this task • The losses caused by low system maintainability can be: – High cost of maintenance effort – Loss of the system by aging Performance-based Risk [IEEE Trans. SE, Jan. 2005] Probability of missing timing or performance requirements * Severity or Consequences SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 21

Importance / Benefits Components Risk Factor Components • Identify the high risk components of the system in terms of Reliability/Maintainability/Performance SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 22

Software Architecture Risk Assessment Methodology Requirements Model System Architecture Model Software Architecture Risk Assessment Reliabilitybased Risk Analysis Maintainabilitybased Risk Analysis Components Risk Factors Performancebased Risk Analysis Components Ranking SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 23

OUTLINE • Risk Management • Software Architectures • Software Architecture Risk Assessment – Maintainability-based risk • Conclusions • Next Steps SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 24

Maintainability-based Risk [ICSM 05] Abdel. Moez, Goseva, Ammar, Mili, Fuhrman, “Architectural level Maintainability Based Risk Assessment” [RAMS 06] Abdel. Moez, Goseva, Ammar, ” Methodology for Maintainability Based Risk Assessment”, Jan 2006. SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 25

Importance / Benefits Maintainability-based Risk • According to Pigoski, 60%-80% of the system budget is spent on maintenance l Enhancements (perfective/ adaptive maintenance) account for 78%83% of the maintainer effort SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 26

Importance / Benefits Maintainability-based Risk • Unisys holds the NASA contract to maintain and support 14 million lines of ground software for the space shuttle • There were 3, 800 requirement changes made to the software after the loss of Challenger. These changes resulted in 900 software releases, of which 30 applied to the mission-control center with 3 of • Reference: these being major upgrades IEEE Software, Vol. 6, No. 1, pp. 116 -119 http: //hebb. cis. uoguelph. ca/~dave/343/Lectures/introduction. html SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 27

Importance / Benefits Trade off Analysis for Perfective Maintenance Risk Factor Components Risk Components Patterns Maintainability risk for perfective maintenance (open source case study Borg) SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 28

Software Architecture Risk Assessment Methodology: Maintainability-based Risk Requirments maturity Index / change / error reports (1) Estimate components Initial Change Probability (ICP) SW Architecture (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) CP=[cpi/j] SC=[sci/j] ICP=[icpi] (4) Estimate component risk factor SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 29

Maintenance change propagation Incoming maintenance Outgoing maintenance SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 30

Estimating Change Propagation C 1 Change in Provided Service V 11 =1 V 12 V 13 C 2. . V 13 Required Services =0 • Change Propagation Probabilities matrix CP=[cpij ] cpij is the probability that a change in Ci due to corrective/ perfective maintenance requires a change in Cj while maintaining the overall function of a system S cpij = P([Cj] [Cj'] | [Ci] [Ci'] ^ [S] = [S'] ) · cpij is estimated by cpij = SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 31

Estimating Size of Change C 1 V 1 Change in Provided Service C 2 V 11 V 12 V 13 M 1 M 2 M 3 … M 7 Receiving Component methods · Size of change SC=[scij ] scij is defined as the ratio between the number of affected methods of the receiving component caused by the changes in the interface elements of the providing components and the total number of methods in the receiving component · scij is estimated by scij = SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 32

CM 1 Maintainability-Based Risk in Adaptive Maintenance Context SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 33

Case Study: NASA CM 1 UML Model Structure Diagram • The UML-RT Model of CM 1 was Developed by WVU students (Nathan, Tom and Rajesh, Summer 2004) based on the CM 1 software design specification SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 34

Change Propagation Probabilities for CM 1 • The Change Propagation probabilities CP is estimated using the CM 1 UML model • The Change Propagation probabilities CP can be automatically estimated from UML -RT models, or java source SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 35

Size of Change for CM 1 • The Size of Change metrics SC is estimated using the CM 1 UML model • The Size of Change metrics SC Probabilities CP can be automatically estimated from UML-RT models, or Java source SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 36

Software Architecture Risk Assessment Methodology: Maintainability-based Risk Requirments maturity Index / change / error reports (1) Estimate components Initial Change Probability (ICP) SW Architecture (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) CP=[cpi/j] SC=[sci/j] ICP=[icpi] (4) Estimate component risk factor SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 37

Maintainability-based Risk For corrective maintenance (case study CM 1) ICP is estimated using error reports data SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 38

Prioritizing Corrective Maintenance Tasks for CM 1 Components Severity Level BIT CCM DCI DCX DPA EDAC ICUI 1553 SCUI SSI TIS TMALI Minor Cat. Minor Major Critical Cat Major Critical SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 39

Maintainability-based Risk Maintainability risk for Adaptive maintenance (case study CM 1) ICP is estimated using change reports data SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 40

Tool Support SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 41

Technology Readiness Level The Software Architecture Risk Assessment Tool Support SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 42

Conclusions • Risk Management is vital to the success of projects and products • A risk Assessment process is needed • Software Architecture is a major determinant of software quality • Software Architecture can be used to manage project and product risks • Development of a methodology and a process for software architecture risk assessment • Continued development of a software architecture risk assessment tool to support the methodology SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 44

Papers Published 1. 2. 3. 4. 5. 6. 7. 8. 9. Vittorio Cortellessa, Katerina Goseva-Popstojanova, Kalaivani Appukkutty, Ajith R. Guedem, Ahmed Hassan, Rania Elnaggar, Walid Abdelmoez, Hany H. Ammar, “Model-Based Performance Risk Analysis, IEEE Transactions on Software Engineering, January 2005, (Vol. 31, No. 1), pp. 3 -20. Katerina Goseva-Popstojanova, Ahmed Hassan, Ajith Guedem, Walid Abdelmoez, Diaa Eldin M. Nassar, Hany Ammar, Ali Mili, "Architectural-Level Risk Analysis Using UML", IEEE Transactions on Software Engineering, October 2003 (Vol. 29, No. 10), pp. 946 -960. S. Yacoub, H. Ammar, “A Methodology for Architectural-Level Reliability Risk Analysis, ” IEEE Transactions on Software Engineering, Vol. 28, No. 6, June 2002. W. Abdel. Moez, K. Goseva-Popstojanova, H. H. Ammar, ” Methodology for Maintainability-Based Risk Assessment”, Proc. of the 52 nd Annual Reliability & Maintainability Symposium (RAMS 2006), Newport Beach, Ca. , January 23 -26, 2006. Israr P. Shaik , W. Abdelmoez, R. Gunnalan, M. Shereshevsky, A. Zeid, H. H. Ammar, A. Mili, C. Fuhrman, “Change Propagation for Assessing Design Quality of Software Architectures”, Proc. of 5 th IEEE/IFIP Working Conference on Software Architecture (WICSA), Pittsburgh, Pa. , USA, November 6 -9, 2005. Abdel. Moez, W. , I. Shaik, R. Gunnalan, M. Shereshevsky, K. Goseva-Popstojanova, H. H. Ammar, A. Mili, C. Fuhrman, “Architectural level Maintainability Based Risk Assessment”, Proc. of poster papers in IEEE International Conference on Software Maintenance (ICSM 2005), Budapest, Hungary, September 25 -30, 2005. W. Abdelmoez, D. M. Nassar, M. Shereshevsky, N. Gradetsky, R. Gunnalanm and H. H. Ammar, Bo Yu, and Ali Mili "Error Propagation in Software Architectures". In Proceedings of the 10 th International Symposium on Software Metrics (METRICS'04), September 11 - 17, 2004 , IEEE Comp. Soc. , pp 384 -393 Abdelmoez, W. , M. Shereshevsky, R. Gunnalan, H. H. Ammar, Bo Yu, S. Bogazzi, M. Korkmaz, A. Mili, "Software Architectures Change Propagation Tool (SACPT), ” 20 th IEEE International Conference on Software Maintenance (ICSM'04) September 11 - 14, 2004 , Chicago, Illinois, IEEE Comp. Soc. , pp 517 A. Hassan, K. Goseva-Popstojanova, and H. Ammar, “UML Based Severity Analysis Methodology”, Proceedings of the 2005 Annual Reliability and Maintainability Symposium (RAMS 2005), Alexandria, VA, January 2005. SW Architecture Risk Assessment Keynote Presentation My. SEC’ 06 45