Software Quality Safety Assessment Using Bayesian Belief Networks

Software Quality & Safety Assessment Using Bayesian Belief Networks Joanne Bechta Dugan Susan Donohue, Ganesh Pai University of Virginia Department of Electrical and Computer Engineering University of Virginia 1

Problems Under Consideration • GETR: How does one decide that a software system is “good enough to release”? • SWQ-BBN: Can I combine process assessment and product assessment metrics to predict quality/reliability of a software system? Department of Electrical and Computer Engineering University of Virginia 2

Approach: Bayesian Belief Networks (BBN) • We use BBN models as the basis of both projects • BBN models effectively allow the combination of quantitative and qualitative assessment (that is, measures and expert judgment) in the same model Department of Electrical and Computer Engineering University of Virginia 3

GETR Approach (with S. Donohue) • For the GETR (Good Enough to Release) project, we are developing a BBN model of the decision process – What evidence is used, and how is it weighed – Determining conditional probabilities from expert opinion (to get probability parameters for the model) • GETR is building a mathematical framework based on BBN to understand facilitate the decision making process Department of Electrical and Computer Engineering University of Virginia 4

GETR Decision How can we investigate and document the decision process that is used to go from. . . to… Is the system good enough to release? Test Results Personal and Team CMM Formal Methods Quality Assurance Requirements Prototype Review Performance Risk Assessment Code Inspection Engineering Judgment I have an acceptable level of belief that the system will operate as specified. for a computer-based system Department of Electrical and Computer Engineering University of Virginia 5

Department of Electrical and Computer Engineering University of Virginia 6

Quantifying Judgment for BBN Department of Electrical and Computer Engineering University of Virginia 7

Quantifying Judgment for BBN (QJ BBN) Conditional probabilities (NPT entries) are generated as a function of the contribution of evidence to support a premise. For example, • Acceptable results from testing supports the conclusion that verification is acceptable. • Unacceptable documentation supports the premise that the artifact quality is unacceptable. • Evidence can overlap, be disjoint or synergistic. • Proofs of coherence of functions used in QJ methodology help assure rational decisions. • Importance and sensitivity analysis can help guide decision makers in seeking new evidence. • BN model provides a record of evidence analysis. Department of Electrical and Computer Engineering University of Virginia 8

Application to NASA: Seal of Approval Process (SOAP) for PRA tools Is the tool “fit for use”? Is it “good enough” to share? Is the tool appropriate for use in a given domain? What evidence is available for review? How much influence does certain evidence have on the approval process? Department of Electrical and Computer Engineering University of Virginia 9

Department of Electrical and Computer Engineering University of Virginia 10

SWQ BBN Approach (with G. Pai) • For the SWQ BBN project, we are developing techniques to build a BN to model the software development process and the products (artifacts) – BBN model represents causally related phases and activities within the phases. – Measurements or expert opinion can be used to determine probability parameters for the model. • Model can be used to assess the process/product with respect to reliability (defect density) or other quality attribute Department of Electrical and Computer Engineering University of Virginia 11

Department of Electrical and Computer Engineering University of Virginia 12

Candidate BBN for design phase Department of Electrical and Computer Engineering University of Virginia 13

Hypothetical illustrative example • Hypothetical priors • Model result – Medium defect content – Actual values dependent on the mapping between node states and range values Posterior • E. g. Vlow, Low, Medium, High, Vhigh 0 -20, 20 -40, 40 -60, 60 -80, 80 -100 – Model results Defect content would lie in 40 – 60 range Evidence Department of Electrical and Computer Engineering University of Virginia 14

• Feedback to the designer greater value – Network itself can provide feedback – Propagation of evidence • In this case: knowledge of high specification quality, observation of high defect content – Change in distribution indicates potential problem area Department of Electrical and Computer Engineering University of Virginia 15

Application to IV&V (joint work with Titan (Khalid Lateef)) Use IVV process for use case analysis, construct BBN from process model • Relevant process parameters and inputs represent parent nodes • Child nodes of BBN represent features desired from the requirements specification Department of Electrical and Computer Engineering University of Virginia 16

Department of Electrical and Computer Engineering University of Virginia 17

Example analysis • Probabilities reflect either measurement or analysts’ beliefs • The state ‘true’ is less than 95% Not mature enough. Department of Electrical and Computer Engineering University of Virginia 18

Technology Readiness Level Department of Electrical and Computer Engineering University of Virginia 19

Data / Case Study Availability GETR case study domain: “lightweight” V&V for inhouse developed analytical tools being considered for release to other centers or research groups. Identified case studies – RAP (JPL), SIAT (IV&V), and MATT (IV&V) SWQ BBN case study domain: Case study of system development, including artifacts & defect data. OO or ODC would be great Working with Khalid Lateef to develop case study for OO requirements analysis Department of Electrical and Computer Engineering University of Virginia 20

Barriers to Research or Application Case studies Department of Electrical and Computer Engineering University of Virginia 21