Verification Validation IS 301 Software Engineering Lecture 30

Topics Ø Verification and validation planning Ø Software inspections Ø Automated static analysis Ø

Verification and Validation ØAssuring that software system meets user's needs 3 Note content copyright

Objectives Ø To introduce software verification and validation and to discuss distinction between them

Verification vs Validation Ø Verification: “Are we building the product right? ” q software

V & V Process Ø Whole life-cycle process q. V & V at each

Static and Dynamic Verification Ø Software inspections q. Analysis of static system representation q.

Static and Dynamic V&V 8 Note content copyright © 2004 Ian Sommerville. NU-specific content

Program Testing Ø Can reveal q. Presence of errors q. But NOT their absence

Types of Testing Ø Defect testing q. Tests designed to discover system defects q.

V & V Goals Ø Establish confidence that software is fit for purpose Ø

V & V Confidence Ø Depends on system’s purpose, user expectations and marketing environment

Testing and Debugging Ø Defect testing and debugging are distinct processes Ø Verification and

Debugging Process 14 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright ©

V & V Planning Ø Careful planning required to get most out of testing

V-Model of Development 16 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright

Structure of Software Test Plan Ø Testing process Ø Requirements traceability Ø Tested items

Software Inspections Ø People examine representation of system to find anomalies and defects q.

Inspection Success Ø In testing, one defect may mask another so several executions are

Inspections and Testing Ø Inspections and testing are complementary and not opposing verification techniques

Program Inspections Ø Formalized approach to document reviews Ø Intended explicitly for defect DETECTION

Inspection Pre-Conditions Ø Precise specification must be available Ø Team members must be familiar

Inspection Process 23 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright ©

Inspection Procedure Ø System overview presented to inspection team Ø Code and associated documents

Inspection Teams Ø Made up of at least 4 members q. Author of code

Inspection Checklists Ø Checklist of common errors should be used to drive inspection Ø

Inspection Checks (1) Data Faults Ø Are all program variables initialized before their values

Inspection Checks (2) Control Faults Ø For each conditional statement, is the condition correct?

Inspection Checks (3) Ø Input/Output Faults q. Are all input variables used? q. Are

Inspection Checks (4) Ø Storage Management Faults q. If a linked structure is modified,

Inspection Rate Estimating cost of inspection for 500 lines of code: Ø 500 statements/hour

Automated Static Analysis Ø Static analyzers = software for source text processing q. Parse

Static Analysis Checks 33 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright

Stages of Static Analysis (1) Ø Control flow analysis q. Checks for loops with

Stages of Static Analysis (2) Ø Information flow analysis q. Identifies dependencies of output

$138% more lint_ex. c #include <stdio. h> printarray (Anarray) int Anarray; { printf(“%d”, Anarray);$

Use of Static Analysis Ø Particularly valuable for language such as C q. Weak

Cleanroom Software Development Ø Name derived from 'Cleanroom' process in semiconductor fabrication Ø Philosophy

Cleanroom Process 39 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright ©

Cleanroom Process Characteristics Ø Formal specification using state transition model Ø Incremental development Ø

Incremental Development 41 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright ©

Formal Specification and Inspections Ø State-based model = system specification Ø Inspection process checks

Cleanroom Process Teams Ø Specification team q. Develops and maintains system specification Ø Development

Cleanroom Process Evaluation Ø Results in IBM impressive q. Few discovered faults in delivered

Key Points (1) Ø Verification and validation are not same thing. Verification shows conformance

Key points (2) Ø Program inspections are very effective in discovering errors Ø Program

Homework Ø Required by Wed 17 Nov 2004: q. For 32 points ü 22.

DISCUSSION 48 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004

Slides: 48

Download presentation

Verification & Validation IS 301 – Software Engineering Lecture #30 – 2004 -11 -10 M. E. Kabay, Ph. D, CISSP Assoc. Prof. Information Assurance Division of Business & Management, Norwich University mailto: mkabay@norwich. edu V: 802. 479. 7937 1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Topics Ø Verification and validation planning Ø Software inspections Ø Automated static analysis Ø Cleanroom software development 2 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Objectives Ø To introduce software verification and validation and to discuss distinction between them Ø To describe program inspection process and its role in V & V Ø To explain static analysis as verification technique Ø To describe Cleanroom software development process 4 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Verification vs Validation Ø Verification: “Are we building the product right? ” q software should conform to its specification Ø Validation: “Are we building the right product? ” q software should do what user really requires 5 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

V & V Process Ø Whole life-cycle process q. V & V at each stage in software process Ø Two principal objectives q. Discover defects in system q. Assess whether system is usable in operational situation 6 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Static and Dynamic Verification Ø Software inspections q. Analysis of static system representation q. Static verification q. May be supplemented by tool-based document and code analysis Ø Software testing q. Exercising and observing product behavior (dynamic verification) q. System executed with test data and its operational behavior observed 7 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Program Testing Ø Can reveal q. Presence of errors q. But NOT their absence Ø Successful test q. Discovers one or more errors Ø Only validation technique for non-functional requirements Ø Should be used in conjunction with static verification to provide full V&V coverage 9 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Types of Testing Ø Defect testing q. Tests designed to discover system defects q. Successful defect test reveals presence of defects in system Ø Statistical testing q. Tests designed to reflect frequency of user inputs q. Used for reliability estimation 10 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

V & V Goals Ø Establish confidence that software is fit for purpose Ø Does NOT mean completely free of defects Ø Good enough for its intended use Ø Type of use will determine degree of confidence needed 11 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

V & V Confidence Ø Depends on system’s purpose, user expectations and marketing environment q. Software function üLevel of confidence depends on how critical software is to organization q. User expectations üUsers may have low expectations of Sound certain kinds of software familiar q. Marketing environment ? üGetting product to market early may be more important than finding defects in program 12 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Testing and Debugging Ø Defect testing and debugging are distinct processes Ø Verification and validation concerned with establishing existence of defects in program Ø Debugging concerned with locating and repairing these errors Ø Debugging involves formulating hypotheses about program behavior then testing these hypotheses to find system error(s) 13 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

V & V Planning Ø Careful planning required to get most out of testing and inspection processes q. Start early in development process Ø Identify balance between q. Static verification and q. Testing Ø Test planning is about q. Defining standards for testing process rather than q. Describing product tests 15 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Structure of Software Test Plan Ø Testing process Ø Requirements traceability Ø Tested items Ø Testing schedule Ø Test recording procedures Ø Hardware and software requirements Ø Constraints 17 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Software Inspections Ø People examine representation of system to find anomalies and defects q. May be applied to any representation of system üRequirements, design, test data. . . Ø Do not require execution of system q. Used before implementation Ø Very effective technique for discovering errors 18 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Success Ø In testing, one defect may mask another so several executions are required Ø Reuse domain and programming knowledge so reviewers are likely to have seen types of error that commonly arise Ø Thus many different defects may be discovered in single inspection 19 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspections and Testing Ø Inspections and testing are complementary and not opposing verification techniques Ø Both should be used during V & V process Ø Inspections can check q. Conformance with specification q. But not conformance with customer’s real requirements Ø Inspections cannot check non-functional characteristics q. Performance, usability, etc. 20 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Program Inspections Ø Formalized approach to document reviews Ø Intended explicitly for defect DETECTION (not correction) Ø Defects may be q. Logical errors, q. Anomalies in code that might indicate erroneous condition ü(E. g. uninitialized variable) q. Or non-compliance with standards 21 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Pre-Conditions Ø Precise specification must be available Ø Team members must be familiar with organization standards Ø Syntactically correct code must be available Ø Error checklist should be prepared Ø Management must accept that inspection will increase costs early in software process Ø Management must not use inspections for staff evaluations q. I. e. , finding errors does not necessarily mean programmer is BAD 22 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Procedure Ø System overview presented to inspection team Ø Code and associated documents distributed to inspection team in advance Ø Inspection takes place and discovered errors noted Ø Modifications made to repair discovered errors Ø Re-inspection may or may not be required 24 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Teams Ø Made up of at least 4 members q. Author of code being inspected q. Inspector who finds errors, omissions and inconsistencies q. Reader who reads code to team q. Moderator who chairs meeting and notes discovered errors Ø Other roles: q. Scribe q. Chief moderator 25 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Checklists Ø Checklist of common errors should be used to drive inspection Ø Error checklist is programming language dependent Ø ‘Weaker’ type checking needs larger checklist Ø Examples: q. Initialization q. Constant naming q. Loop termination q. Array bounds, etc. 26 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Checks (1) Data Faults Ø Are all program variables initialized before their values are used? Ø Have all constants been named? Ø Should the lower bound of arrays be 0, 1 or something else? Ø Should the upper bound of arrays be equal to the size of the array or (size – 1)? Ø If character strings are used, is a delimiter explicitly assigned? 27 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Checks (2) Control Faults Ø For each conditional statement, is the condition correct? Ø Is each loop certain to terminate? Ø Are compound statements correctly bracketed? Ø In case statements, are all possible cases accounted for? 28 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Checks (3) Ø Input/Output Faults q. Are all input variables used? q. Are all output variables assigned a value before the are output? Ø Interface Faults q. Do all function and procedure calls have the correct number of parameters? q. Do formal and actual parameter types match? q. Are the parameters in the right order? q. If components access shared memory (globals), do they have the same model of the shared memory structure? 29 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Checks (4) Ø Storage Management Faults q. If a linked structure is modified, have all links been correctly reassigned? q. If dynamic storage is used, has space been allocated correctly? q. Is space explicitly deallocated after it is no longer required? Ø Exception Management Faults q. Have all possible error conditions been taken into account? 30 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Inspection Rate Estimating cost of inspection for 500 lines of code: Ø 500 statements/hour during overview = 1 hr person x 4 people = 4 person-hours Ø 125 source statements/hour during individual preparation = 4 hours x 4 people = 16 person-hours Ø 90 -125 statements/hour can be inspected in meeting with 4 people in team = ~5 hours x 4 people = ~20 person-hours Ø Inspection is therefore expensive process q Inspecting 500 lines thus takes ~ 4 + 16 + 20 = ~40 person-hours q Estimate programmer salary $80 K/2 K hr ~$40/hr q Multiply by 2 for extended costs = $80/hr q Therefore costs of 40 person-hours effort = ~ $3, 200 31 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Automated Static Analysis Ø Static analyzers = software for source text processing q. Parse program text q. Try to discover potentially erroneous conditions q. Report to V & V team Ø Effective aid to inspections q. Supplement to but not replacement for inspections 32 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Stages of Static Analysis (1) Ø Control flow analysis q. Checks for loops with multiple exit or entry points, finds unreachable code, etc. Ø Data use analysis q. Detects uninitialized variables, variables written twice without intervening assignment, variables which are declared but never used, etc. Ø Interface analysis q. Checks consistency of routine and procedure declarations and their use 34 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Stages of Static Analysis (2) Ø Information flow analysis q. Identifies dependencies of output variables. Does not detect anomalies itself but highlights information for code inspection or review Ø Path analysis q. Identifies paths through program and sets out statements executed in that path. Again, potentially useful in review process Ø Both these stages generate vast amounts of information. Must be used with care. 35 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

$138% more lint_ex. c #include <stdio. h> printarray (Anarray) int Anarray; { printf(“%d”, Anarray);$

138% more lint_ex. c #include <stdio. h> printarray (Anarray) int Anarray; { printf(“%d”, Anarray); } main () { int Anarray[5]; int i; char c; printarray (Anarray, i, c); printarray (Anarray) ; } LINT Static Analysis 139% cc lint_ex. c 140% lint_ex. c(10): warning: c may be used before set lint_ex. c(10): warning: i may be used before set printarray: variable # of args. lint_ex. c(4) : : lint_ex. c(10) printarray, arg. 1 used inconsistently lint_ex. c(4) : : lint_ex. c(11) printf returns value which is always ignored 36 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Use of Static Analysis Ø Particularly valuable for language such as C q. Weak typing q. Hence many errors undetected by compiler Ø Less cost-effective for languages like Java q. Strong type checking q. Can therefore detect many errors during compilation 37 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Cleanroom Software Development Ø Name derived from 'Cleanroom' process in semiconductor fabrication Ø Philosophy is defect avoidance rather than defect removal Ø Software development process based on: q. Incremental development q. Formal specification q. Static verification using correctness arguments q. Statistical testing to determine program reliability 38 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Cleanroom Process Characteristics Ø Formal specification using state transition model Ø Incremental development Ø Structured programming q. Limited control and abstraction constructs are used Ø Static verification using rigorous inspections Ø Statistical testing of system (covered in Ch. 21)(but not in this course) 40 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Formal Specification and Inspections Ø State-based model = system specification Ø Inspection process checks program against this model Ø Programming approach defined so that correspondence between model and system is clear Ø Mathematical arguments (not proofs) are used to increase confidence in inspection process 42 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Cleanroom Process Teams Ø Specification team q. Develops and maintains system specification Ø Development team q. Develops and verifies software. q. Software NOT executed or even compiled during this process Ø Certification team q. Develops set of statistical tests to exercise software after development q. Reliability growth models used to determine when reliability acceptable 43 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Cleanroom Process Evaluation Ø Results in IBM impressive q. Few discovered faults in delivered systems Ø Independent assessment: q. Process no more expensive than other approaches q. Fewer errors than in ‘traditional’ development process Ø Not clear how this approach can be transferred to environment with q. Less skilled or q. Less highly motivated engineers 44 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Key Points (1) Ø Verification and validation are not same thing. Verification shows conformance with specification; validation shows that program meets customer’s needs Ø Test plans should be drawn up to guide testing process. Ø Static verification techniques involve examination and analysis of program for error detection 45 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Key points (2) Ø Program inspections are very effective in discovering errors Ø Program code in inspections is checked by small team to locate software faults Ø Static analysis tools can discover program anomalies which may be indication of faults in code Ø Cleanroom development process depends on incremental development, static verification and statistical testing 46 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Homework Ø Required by Wed 17 Nov 2004: q. For 32 points ü 22. 1 – 22. 5 (@4) ü 22. 9 & 22. 10 (@6) (think hard) Ø Optional by Mon 29 Nov 2004 q. For up to 25 extra points, write detailed answers for any or all of ü 22. 6, 22. 7 (@10) ü 22. 8 (@5) 47 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.