IVV Facility RealWorld Software Reliability Assessment WVU UI7

IV&V Facility Real-World Software Reliability Assessment (WVU UI#7: Sensitivity of Software Reliability to Operational Profile Errors: Architecture-Based Approach) PI: Katerina Goseva – Popstojanova Students: Sunil Kamavaram & Olaolu Adekunle Lane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, WV katerina@csee. wvu. edu

What we are doing? IV&V Facility Anyone can see a fire What we need are smoke detectors But what about the sensitivity and accuracy of the alarms ? 2

Problem statement & Our goal IV&V Facility § Traditional view: Point estimate of software reliability computed from the model using point estimates of input parameters § Problem: Estimation of a trustworthy operational profile is difficult § IV&V information on operational profiles - limited, may be inaccurate § Single operational profile could not be sufficient to describe the use by different users § Software systems evolve - operational profile may change § Our goal: Reliability “sensitometer” that enables us to answer the question “How parameters uncertainty propagates into overall application reliability? ” § Develop an architecture-based methodology for uncertainty analysis of software reliability & apply it on case studies 3

What we can do? IV&V Facility Entropy as a measure of uncertainty Execution rates & uncertainty of components Reliability frequency chart & distribution fitting Certainty bands (percentiles) § Benefits to IV&V § Software reliability assessment throughout the life cycle (keeping track of the software evolution) § Allocation of testing efforts § Software certification 4

IV&V Facility Architecture - based methodology for uncertainty analysis Uninformed Approach Intended Approach Informed Approach (maximum entropy) (historical data, UML) (component traces) Growth models R 1 1 p 12 R 2 2 Non-failed executions p 23 1 -p 12 3 R 3 1 -p 23 1 E Fault injection Uncertainty analysis 5

Methods for uncertainty analysis IV&V Facility Uncertainty analysis Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis 6

Choice of the method IV&V Facility § Choose the method using the following criteria § § Data requirements & ability to collect data Reliability measures Accuracy of the solution Scalability with respect to the number of components § Our goal: fill the table Method Data requirements Reliability measures Accuracy of the solution Scalability 7

IV&V Facility Construction of the software architecture model §Structural phase – establishment of static software architecture 1 p 12 § Software specifications § § Architectural design Parser-based or lexically based tools (SIAT tool - Titan Systems Corporation) §Statistical phase – estimation of the relative frequencies of component interactions, that is, transition probabilities 2 1 -p 12 p 23 3 1 -p 23 1 E Uniform distribution – maximum entropy approach § Historical data § Software specification (e. g. UML use case & sequence diagrams) § Component traces from profiles or test coverage tools (Testing tool for JSC AERCam project - Dr. Yann-Hang Lee, ASU) § 8

European Space Agency case study IV&V Facility Component traces obtained during testing were used for constructing software architecture & estimating transition probabilities § Almost 10. 000 lines of C code § The program has been extensively used after the last fault removal without failures; this gold version is used as an oracle Informed Approach (component traces) R 1 1 p 12 R 2 2 p 23 1 -p 12 3 R 3 1 E 1 -p 23 Two faulty versions were obtained reinserting the real faults discovered during the integration testing and operational usage Fault Injection (real faults) 9

Parameter estimation IV&V Facility § Two versions § Version A: faulty components 1&2, fault-free component 3 § Version B: faulty components 2, fault-free components 1&3 § Transition probabilities where is the number of times control was transferred from component i to component j, and Version p 12 p 23 A 0. 5933 0. 7704 B 0. 7364 0. 6866 § Component reliability where is the number of failures and is the number of executions of component i in N randomly generated test cases Version R 1 R 2 R 3 A 0. 8428 0. 8346 1 B 1 0. 8346 1 10

IV&V Facility Construction of the architecture – based software reliability model 1 p 12 R 1 2 1 -R 2 p 23 R 2 (1 -p 12)R 1 3 1 -R 1 F (1 -p 23)R 2 1 -R 3 E 1 C 11

IV&V Facility Traditional View: Point estimates of software reliability § Actual reliability of the software where F is the number of system failures in N randomly generated test cases § Estimated reliability from the model § Results Version Actual reliability Estimated reliability Error A 0. 7393 0. 7601 2. 81% B 0. 8782 0% 12

Methods for uncertainty analysis IV&V Facility Uncertainty analysis Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis 13

Sensitivity of software reliability to variations in operational profile IV&V Facility Version A reliability Rmax = 0. 8414 Rmin = 0. 7048 Version B reliability Rmax = 0. 9983 Rmin = 0. 8363 14

Methods for uncertainty analysis IV&V Facility Uncertainty analysis Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis 15

Uncertainty study based on entropy IV&V Facility § Entropy quantifies the uncertainty present in a stochastic source where represents the usage distribution and the transition probabilities § Higher entropy implies an exponentially greater number of statistically typical paths § Maximum entropy – all transitions that are exit arcs from each state are equiprobable 16

Uncertainty of the operational profile IV&V Facility Hmax = 0. 5514 max==0. 0404 HH min Hmin = 0. 0404 § Operational profile A (H=0. 4707) is more uncertain than operational profile B (H=0. 4604) § Software systems that have uniform operational profile are more uncertain and thus would require more testing 17

Uncertainty of software reliability IV&V Facility Operational profile Version A uncertainty Version A reliability Version B uncertainty Version B reliability § Considering software failure behavior increases the uncertainty for both versions compared to the uncertainty due to operational profile § Version B, which is more reliable, is less uncertain than version A 18

IV&V Facility Uncertainty of components for the operational profile § Uncertainty of component i is estimated using the conditional entropy Version A Version B § Uncertainty of component i will be higher if it transfers the control to more components and the transition probabilities are equiprobable 19

IV&V Facility Uncertainty of components for the software reliability model Version A Version B § Uncertainty of component 1 version B remains the same because § For all other components uncertainty increases due to § Components that have higher expected execution rate, higher component uncertainty, and moderate reliability should be allocated more testing effort 20

Methods for uncertainty analysis IV&V Facility Uncertainty analysis Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis 21

IV&V Facility Uncertainty study based on the method of moments § Method of moments involves the following steps 1. Obtain the expression for the system reliability using the architecture-based software reliability model 2. Expand the expression for system reliability using Taylor series 3. Determine the moments of the components reliabilities 4. Estimate the mean and the variance of the system reliability using the parameter moments and Taylor series coefficients 22

First order Taylor series IV&V Facility § First order Taylor series expansion where is the mean component reliability, and § Mean reliability is § Variance of the reliability is where is the variance of the component reliability 23

Second order Taylor series IV&V Facility § Second order Taylor series expansion § Mean reliability is § Variance of the reliability is 24

Method of moments for the case study IV&V Facility First order Taylor series Second order Taylor series 0. 7601 0. 0825 Variance 0. 0068 Mean reliability 0. 8782 0. 0589 0. 0035 Mean reliability Version A Standard deviation Version B Standard deviation Variance § Second order approximation does not improve accuracy significantly § Version B is more reliable with less variance of the reliability 25

Methods for uncertainty analysis IV&V Facility Uncertainty analysis Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis 26

IV&V Facility § Uncertainty study based on Monte Carlo simulation involves the following steps 1. Obtain the expression for the system reliability using the architecture-based software reliability model 2. Assign probability distributions to the transition probabilities and components reliabilities 3. Sample the distributions 4. Compute the reliability of the system using the sampled values 5. Repeat steps 3&4 until the desired number of values of system reliability has been generated 6. Calculate the moments, frequency chart and percentiles for the system reliability, do the distribution fitting 27

IV&V Facility Variation of the operational profile: Frequency chart and distribution fitting Mean 0. 7600 Standard deviation (Spread of the distribution) 0. 0210 Variance (Spread of the distribution) 0. 0004 Skewness (Lean of the distribution) 0. 2072 Kurtosis (Peakedness of the distribution) 2. 6047 28

Variation of the operational profile: Percentiles IV&V Facility 95% 75% § 95% certainty band shows the range of values in which reliability has 95% chance of falling 29

Convergence of the mean IV&V Facility Mean reliability =0. 7600 The estimation of the mean reliability converges after around 3000 iterations 30

IV&V Facility Variation of the operational profile: Sensitivity measured by contribution to variance § Reliability is more sensitive to p 1 E; the variance is positive § Reliability is also sensitive to p 12; the variance is negative 31

Variation of the operational profile and component reliabilities: Frequency charts IV&V Facility Version A Version B Mean 0. 7589 0. 8780 Standard deviation (Spread of the distribution) 0. 0860 0. 0660 Variance (Spread of the distribution) 0. 0074 0. 0044 Coefficient of variation (Relative measure of spread) 0. 1493 0. 0752 -0. 5190 -0. 9646 3. 1367 4. 2254 Skewness (Lean of the distribution) Kurtosis (Peakedness of the distribution) 32

IV&V Facility Variation of the operational profile and component reliabilities: Distribution fitting & percentiles Version A Version B 33

Making a choice IV&V Facility Data requirements Reliability measures Accuracy of the solution Sensitivity Point estimates Sensitivity of the point estimate Exact analytical solution Large systems Entropy Point estimates NA Exact analytical solution Large systems Method of moments Moments of the parameters Moments § Approximate solution: accuracy may be increased by higher order Taylor series Small to medium systems Monte Carlo simulation § Distribution functions of the parameters § Generation of random numbers § Distribution § Approximate solution: Large systems accuracy may be increased by increasing the sample size § Sampling errors may be involved in case of long tail distributions Method § Moments Scalability 34

Accomplishments IV&V Facility § Architecture-based methodology for uncertainty analysis of software reliability was developed § Four different methods already developed § These methods were illustrated on the European Space Agency software 35

Future work IV&V Facility § Develop other methods for uncertainty analysis § Complete “Make a choice” table § Apply & validate all methods using NASA case studies § SIAT tool - Titan Systems Corporation § Testing tool for JSC AERCam project - Dr. Yann-Hang Lee, ASU 36