Four MegaChallenges Facing The Commercial Software Quality Movement

  • Slides: 54
Download presentation
Four Mega-Challenges Facing The Commercial “Software Quality Movement”: Definition, Composition, Certification, and Commercialization and

Four Mega-Challenges Facing The Commercial “Software Quality Movement”: Definition, Composition, Certification, and Commercialization and Return. On-Investment Jeffrey M. Voas Chief Scientist

Definitional Problem

Definitional Problem

Ask the CIO Q: Are you interested in software quality? A: What do you

Ask the CIO Q: Are you interested in software quality? A: What do you mean by that? Q: Are you interested in quality software? A: Yes.

What is Software Quality? Subjective term that produces confusion among most software engineering professionals

What is Software Quality? Subjective term that produces confusion among most software engineering professionals

IEEE • “Totality of features of a software product that bears on its ability

IEEE • “Totality of features of a software product that bears on its ability to satisfy given needs. ” [Source: IEEE-STD -729] • “Composite characteristics of software that determine the degree to which the software in use will meet the expectations of the customer. ” [Source: IEEE-STD-729]

Three High-Level Attributes Quality Software Reliable/ Accurate (integrity) Secure/ private Timeliness

Three High-Level Attributes Quality Software Reliable/ Accurate (integrity) Secure/ private Timeliness

High-Level Attributes Quality Software Reliable/ Accurate (integrity) Secure/ private Timeliness Problem: Intuitive, but not

High-Level Attributes Quality Software Reliable/ Accurate (integrity) Secure/ private Timeliness Problem: Intuitive, but not formal

Lower-Level Attributes Quality Software Reliable/ accurate reliability fault tolerance testability Functional attributes Secure/ private

Lower-Level Attributes Quality Software Reliable/ accurate reliability fault tolerance testability Functional attributes Secure/ private Timeliness privacy security availability confidentiality intrusion tolerance performance fault tolerance Non-functional attributes (“ilities”)

Position Statement Software Quality (or Quality Software) must be viewed/defined as some combination of:

Position Statement Software Quality (or Quality Software) must be viewed/defined as some combination of: (1) the degree to which the functional requirements are met, as well as, (2) the degree to which the non-functional requirements are met. reliability fault tolerance testability Functional attributes privacy security availability confidentiality + intrusion tolerance performance fault tolerance Non-functional attributes (“ilities”)

Position Statement Software Quality then is some combination of the following functional and non-functional

Position Statement Software Quality then is some combination of the following functional and non-functional attributes: Reliability [R], Performance [P], Safety [Sa] Fault Tolerance [F], Security [Se], Availability [A] Testability [T], and Maintainability [M]

However …. Software Quality can also be viewed as some combination of the previous

However …. Software Quality can also be viewed as some combination of the previous attributes PLUS: Scalability, Usability, Sustainability, Survivability, Interoperability, Extensibility, Reusability, Readability, etc.

Eight in an Equation? Q = a. R + b. P + c. F

Eight in an Equation? Q = a. R + b. P + c. F + d. Sa + e. Se + f. A + g. T + h. M where a, b, c, d, e, f, g, and h are units of quantitative or qualitative measures of a particular attribute.

Key Problems • The equation cannot be linear, since the units of measure for

Key Problems • The equation cannot be linear, since the units of measure for each attribute cannot be standardized (the apples and oranges problem). • d. Sa = Q – (a. R + b. P + c. F + e. Se + f. A + g. T + h. M) • Most “ilities” are not quantifiably measurable. • Reliability, Availability, and Performance are measurable (via testing).

For Example …Maintainability • Size, defect density, amount of testing, T, R, cohesion, coupling,

For Example …Maintainability • Size, defect density, amount of testing, T, R, cohesion, coupling, documentation, complexity, depth of inheritance, number of objects, testing infrastructure, mean-time-to-repair, experience of maintenance personnel as well as their domain knowledge, existence of impact analysis tools, etc. , all impact M. • Q: So how can you assign a single numerical score for M?

Security • The level of security of an information system is a function of

Security • The level of security of an information system is a function of the partially unknown threat space, that changes by the minute. • Q: So how can you assign a single numerical score for Se? • A: You can assess, for a bounded set of anticipated threats, how the system will respond to those, e. g. , 100 known threats, 50 mitigated. • A: Or you could measure the percentage of patches that are installed based on the number that need to be, and then test to make sure those installed work. Such information could also be used to give security “a score” but once again, that is only a score based on known threats and available patches.

The “Culprit” Phenomenon It is more difficult to directly measure the quality of software

The “Culprit” Phenomenon It is more difficult to directly measure the quality of software than to achieve quality.

So Where Does That Leave Us? Without a Numerical Quality Equation, But With a

So Where Does That Leave Us? Without a Numerical Quality Equation, But With a Way to Discuss the Attributes of Quality Software, and Therefore With a Means for Industry to Define Quality Goals

Composition Problem

Composition Problem

Two Software Components x y

Two Software Components x y

With Attributes x x has the following properties: (a. R, b. P, c. F,

With Attributes x x has the following properties: (a. R, b. P, c. F, d. Sa, e. Se, f. A, g. T, h. M) y has the following properties: (i. R, j. P, k. F, l. Sa, m. Se, n. A, o. T, p. M) y

What Have You Got? x y Then f(x y) will inherit some level of

What Have You Got? x y Then f(x y) will inherit some level of Quality from the individual components. Is that level of quality an integer? Probability? An n-tuple of values? Color coded (green red yellow)? Key Point: The Composite Quality must represent something from which predictions of future behavior can be made.

Key Problems • It is hard enough to know, with any precise accuracy, what

Key Problems • It is hard enough to know, with any precise accuracy, what the composite reliability score will be as a result of the a and i values (let alone for the non-functional attributes). • But an even greater challenge exists here. For example, the security mechanisms in component y could thwart the performance that is built into component x. • Attributes are only reasonable to talk about within the context of a system, i. e. , it is not reasonable to talk about them and attempt to measure them as standalone component properties. Their eventual target environments must weighed into their individual assessments.

Environment Operational environment Reliable/ accurate reliability fault tolerance Quality Secure/ private Timeliness privacy security

Environment Operational environment Reliable/ accurate reliability fault tolerance Quality Secure/ private Timeliness privacy security availability confidentiality testability intrusion tolerance performance fault tolerance Non-functional attributes (“ilities”)

So Where Does That Leave Us? In Search of a Calculus or Calculi for

So Where Does That Leave Us? In Search of a Calculus or Calculi for Predicting How a Composite System Will Behave in the Future in a Specific Environment

Product Certification and Software Engineering Standards To Aide the Composition Problem Standardized Parts?

Product Certification and Software Engineering Standards To Aide the Composition Problem Standardized Parts?

What is a Standard? Ideally, it is a line in the sand from which

What is a Standard? Ideally, it is a line in the sand from which a certificate of compliance can be written.

Pros Any bar or hurdle is better than no bar or hurdle

Pros Any bar or hurdle is better than no bar or hurdle

Cons Possibly the developers would have done more to improve quality but now feel

Cons Possibly the developers would have done more to improve quality but now feel they have a license to do less.

Premise for SW Product Certification Commercially built software should be tagged with some guarantee

Premise for SW Product Certification Commercially built software should be tagged with some guarantee (or at least a “warm fuzzy”) as to how good the software should be. Problem: Software Of Unknown Pedigree (SOUP) Goal of Product Certification: SO(Known)P

Three Schools of Thought All SE standards incorporate one or more of these perspectives

Three Schools of Thought All SE standards incorporate one or more of these perspectives Processes People Products

1. Process: Clean Pipes, Dirty Water? Certifying that you know how to do things

1. Process: Clean Pipes, Dirty Water? Certifying that you know how to do things correctly does not mean that you do them correctly!

2. People The IEEE Computer Society has developed a program to certify software engineering

2. People The IEEE Computer Society has developed a program to certify software engineering professionals. This program provides formal recognition of professionals who have successfully achieved a level of proficiency commonly accepted and valued by the industry.

3. Product: The Software Itself 0% confidence 100% confidence Spectrum of possibilities as to

3. Product: The Software Itself 0% confidence 100% confidence Spectrum of possibilities as to what a certificate proclaiming that some “quantified” level of quality has been built in could state --- it could say anything in the range between “Nothing” (e. g. , “here is a piece of software”, etc. ) to “This software will always work perfectly under all conditions” (i. e. , a 100% guarantee of perfection).

But Problems Exist With Standards – Vague: Develop software that only does "good" things

But Problems Exist With Standards – Vague: Develop software that only does "good" things • Common sense "dos" and "don'ts" - Very watered done by voting time – Disclaimers by publishing organizations • Profitable to organization that publishes them – – – Used only if mandated Return-on-investment is unknown Thwart intellectual creativity • "Protectionist" legislation – Paperwork • 2167 A: ~400 English words per Ada code statement – "Old news" before being ratified – Relating one to another is very hard • Hundreds in existence – Cannot be easily tested for compliance • Mis-certifications are possible

– Different interpretations – Lack of fairness during certification judgment – So much legacy

– Different interpretations – Lack of fairness during certification judgment – So much legacy code exists that complies with no standards and therefore get excluded in heterogeneous systems, making it’s impact to the system unknown.

Example of “Standards” Confusion ØSuppose you have the following logical expression: (A and B)

Example of “Standards” Confusion ØSuppose you have the following logical expression: (A and B) or (B and C) or (A and C) where A, B, and C are Boolean variables ØTo meet verification requirements for Level A software in RTCA DO 178 -B, you need to know the number of conditions in this statement Condition: A Boolean expression containing no Boolean operations How many conditions are there? 3, 4, 6, or 9 [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C. M. Holloway, Presented at the 26 th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

The FAA Says … Distribution of Responses from 39 FAA Certification Authorities 41. 0

The FAA Says … Distribution of Responses from 39 FAA Certification Authorities 41. 0 35. 9 17. 9 5. 1 [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C. M. Holloway, Presented at the 26 th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

And the Answer is … 6 [Source: “Challenges in Software Aspects of Aerospace Systems”,

And the Answer is … 6 [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C. M. Holloway, Presented at the 26 th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

Explanation (A and B) or (B and C) or (A and C) has 6

Explanation (A and B) or (B and C) or (A and C) has 6 conditions ØThe full definition for condition is not contained in the glossary entry for that term ØPart of the definition is given in the entry for decision Decision: A Boolean expression composed of conditions and zero or more Boolean operators. A decision without a Boolean operator is a condition. If a condition appears more than once in a decision, each occurrence is a distinct condition. [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C. M. Holloway, Presented at the 26 th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

So Where Does That Leave Us? In Need of More Precise, Less Vague, and

So Where Does That Leave Us? In Need of More Precise, Less Vague, and Repeatable Processes, for Grading The Quality of Software

Commercialization and ROI Issues

Commercialization and ROI Issues

Commercialization Issues • Proven technology? (empirical vs. anecdotal) • Prototypes? Are they Maintainable/Extensible or

Commercialization Issues • Proven technology? (empirical vs. anecdotal) • Prototypes? Are they Maintainable/Extensible or Trashware? • Scalable? Theoretical or Practical? Maturity? • Automated? Is it a solution or standalone? • What languages/architectures does it support? Fad/Lifetime? • Difficult to learn? Ease of use? Time-to-market enabler or disabler? • Client base: commercial or government? Number of site? • Evolutionary (leap frog-able) vs. Revolutionary?

 • Compatible with existing technology (e. g. , Microsoft)? • Point of Origination:

• Compatible with existing technology (e. g. , Microsoft)? • Point of Origination: University? Small business? , Large Corporation? , Government? • Competing foreign technologies? • Process, People, and Product - oriented? • Which attribute(s) does it address? Measurement or design? If design, how much of that attribute can it offer? • SOUP or SOKP? • Does this technology self-certify Quality after use?

Final Thoughts

Final Thoughts

1. Quality is a Recipe As with food, ingredients of different types (liquids, powders,

1. Quality is a Recipe As with food, ingredients of different types (liquids, powders, vegetables, meats, etc. ) can all be mixed together. What food tastes like is a function of the ingredients and their proportions. Quality Software can be viewed/defined in a similar manner. And certain ingredients overpower others.

2. Standards Beat Chaos • Software engineering standards are completely necessary despite their limitations.

2. Standards Beat Chaos • Software engineering standards are completely necessary despite their limitations. They usually are a “good rule of thumb”, but not an absolute process for achieving perfection. • Virtually any standard beats development chaos • What is Missing in SE Standards? • Not technology! • How to Implement, • How to gain regulatory approval given uncertain knowledge as to how judgment will be rendered, • Fairness in the certification processes, • And ROI (most are anecdotes, not statistical studies)

3. Only Product Certification Can Address The Composition Problem • Until the non-functional attributes

3. Only Product Certification Can Address The Composition Problem • Until the non-functional attributes of software components can be graded, and assumptions about the target environments of those components can be nailed down (HUGE PROBLEM), a priori certification of the quality of any software component is suspect. • Research is needed into how to compose both functional and nonfunctional attributes.

4. Attributes Need to Be Pre. Defined • Requirements should prescribe at some level

4. Attributes Need to Be Pre. Defined • Requirements should prescribe at some level of granularity as to what the weights are for various “ilities”, as well as how much of each “ility” is desired. • But HOW? • Ignoring the non-functional attributes is not an option for high assurance and trustworthy systems! Make an attempt to discuss them with the client even if quantification is not possible. Just get the issue on the table!

5. Weighting is Important w 1 R w 5 Se w 2 P w

5. Weighting is Important w 1 R w 5 Se w 2 P w 6 A w 3 F w 7 T w 4 Sa w 8 M in order to not over-design any attribute into the system. For example, for an e-commerce application, w 4 would probably equal 0. 0 and w 7 would also be less than something like w 2

6. Tradeoffs How much will you spend for increased reliability knowing that doing so

6. Tradeoffs How much will you spend for increased reliability knowing that doing so will take needed, financial resources away from security or performance or …?

 • Security vs. Performance • Fault tolerance vs. Testability • Fault tolerance vs.

• Security vs. Performance • Fault tolerance vs. Testability • Fault tolerance vs. Performance • etc.

Counterintuitive Realities • 100% safety and 0% reliability • 100% reliability and 0% safety

Counterintuitive Realities • 100% safety and 0% reliability • 100% reliability and 0% safety • 0% functionality/reliability and 100% security • 100% availability and 0% reliability • 100% availability and 0% performance • 0% performance and 100% safety

7. The Software Quality Movement … …. is alive and well. There are still

7. The Software Quality Movement … …. is alive and well. There are still many interesting and fascinating software engineering research challenges to pursue that not only will benefit NASA, but industry at-large.

21351 Ridgetop Circle Dulles, VA 20166 USA www. cigital. com Jeffrey Voas phone: 703.

21351 Ridgetop Circle Dulles, VA 20166 USA www. cigital. com Jeffrey Voas phone: 703. 404. 9293 e-mail: voas@cigital. com