Cleanroom Software Engineering Kris Read Paul Werbicki Introduction

Cleanroom Software Engineering Kris Read Paul Werbicki

Introduction • Name borrowed from hardware cleanrooms • Software that exhibits zero defect rate • Uses mathematical and statistical models to prevent and detect defects 9/25/2020 K. Read & P. Werbicki 2

What is Cleanroom? • • 9/25/2020 Incremental development cycle Mathematically based design specifications Statistical verification and certification Software re-engineering K. Read & P. Werbicki 3

Why Cleanroom? • • 9/25/2020 Very high quality Higher productivity Reduced life cycle costs Higher return on investment K. Read & P. Werbicki 4

When is Cleanroom used? • Suitable for critical applications • Used when – defects can cause loss of life – defects can cause critical financial loss – software cannot be updated at a later time 9/25/2020 K. Read & P. Werbicki 5

It’s all about no errors • “Errorless Software Development” • If no errors enter development, no errors need to be tested or debugged • Elimination of testing and debugging means faster product development • Introduce no errors in the first place 9/25/2020 K. Read & P. Werbicki 6

9/25/2020 Figure. Cleanroom Process Flow (Linger & Trammell, 1996) K. Read & P. Werbicki 7

CUSTOMER ANALYSIS QUALITY INTERACTION & SPECIFICATION EVALUATION QUALITY EVALUATION REENGINEERING USAGE ANALYSIS & DESIGN REQUIREMENTS & MODELING & SPECIFICATION VERIFICATION SPECIFICATION TEST PLANNING FUNCTION USAGE STATISTICAL SPECIFICATION CERTIFICATION TESTING INCREMENT DEVELOPMENT 9/25/2020 K. Read & P. Werbicki PLANNING 8

Incremental Development Cycle • • • 9/25/2020 Early and continual quality assessment Increased user feedback Facilitates process improvements Permits requirements changes Avoids integration risk K. Read & P. Werbicki 9

Mathematically Based Design Specifications • All design and code are verified as mathematically correct • Cleanroom achieves this through – Increment Planning – Box Structure Specification and Design 9/25/2020 K. Read & P. Werbicki 10

Running Example • Phone Number entry component • GUI based • Allows users to enter phone numbers in various common formats 9/25/2020 K. Read & P. Werbicki 11

Increment Planning • Based on Referential Transparency • Pipeline of user-function software increments • Early and continual quality assessment and user feedback • Avoids risks inherent in late component integration • Permits systematic management of requirement changes over the development cycle 9/25/2020 K. Read & P. Werbicki 12

Box Structure Specification and Design • A way of specifying software designs • Semi-formal program/component verification • Three box structure stages – Black box – State box – Clear box 9/25/2020 K. Read & P. Werbicki 13

Black Box Structure • Program or component is modeled as a mathematical function • Defines the system boundaries • Defines all stimuli and responses 9/25/2020 K. Read & P. Werbicki 14

9/25/2020 K. Read & P. Werbicki Figure. Example Black Box Diagram 15

State Box Structure • Maps stimulus and response to old and new state transitions • Defines state data and initial functions • Defines transition functions • Used to verify Black Box Structure 9/25/2020 K. Read & P. Werbicki 16

9/25/2020 K. Read & P. Werbicki Figure. Example State Box Diagram 17

Clear Box Structure • Used to design controls and operations • Shows use of new and reused Black Boxes • Used to verify State Box Structure 9/25/2020 K. Read & P. Werbicki 18

9/25/2020 K. Read & P. Werbicki Figure. Example Clear Box Diagram 19

9/25/2020 K. Read & P. Werbicki Figure. Refinement and Verification (Bohner, 2001) 20

Repeated Verification • After each box stage the design is refined • There is a review by all team members • Refinement continues until there is code 9/25/2020 K. Read & P. Werbicki 21

Integrating Box Structure with Z • Design and code are verified as mathematically correct • Cleanroom uses a semi formal design specification already • Focus is on correct design • Code is not written, compiled or executed until final stage of testing 9/25/2020 K. Read & P. Werbicki 22

Mathematically Based Design Specifications • Catch errors before they enter the code • No unit testing • Statistical quality testing forms the only real test of the software • Software results can be certified 9/25/2020 K. Read & P. Werbicki 23

Statistical Quality • Cleanroom uses statistical quality control • Encompasses development and testing • Focuses on detecting the statistically significant defects 9/25/2020 K. Read & P. Werbicki 24

Statistical Usage Testing • Software has an infinite number of paths • It is impossible to test all paths • A large enough random sample can represent the entire system 9/25/2020 K. Read & P. Werbicki 25

Advantages of this Approach • • • 9/25/2020 Mathematically sound measurement Conducted as scientific experiments Proven in hardware engineering Tests for fitness of use Highly efficient K. Read & P. Werbicki 26

The Testing Procedure • • • 9/25/2020 Pre-test Correctness Verification Create a Usage Model Generate random test cases Execute statistical tests Execute additional tests if needed K. Read & P. Werbicki 27

Correctness Verification • Function-theoretic static code analysis • Verifies correctness of code compared to specifications • Defined conditions code must meet • Mental/verbal proofs performed during reviews • Based on control structures 9/25/2020 K. Read & P. Werbicki 28

Correctness Example • User Requirement: All characters shall be numeric digits for the phone number to be considered valid. • Functional Specification: A method called check. Numeric will return true if the character is a digit, false otherwise. 9/25/2020 K. Read & P. Werbicki 29

Correctness Example function boolean check. Numeric (Char C) { boolean x; if ( ! C. is. Number() ) x = true; else x = false; return x; } 9/25/2020 K. Read & P. Werbicki 30

Correctness Example • When C. is. Number() is true… • Does x = true make the function true? • When C. is. Number() is false… • Does x = false make the function false? • YES + YES = Passed Correctness 9/25/2020 K. Read & P. Werbicki 31

Correctness Pros and Cons • • 9/25/2020 Highly effective in practice Requires good specifications Can be done incrementally Code re-use may require re-engineering K. Read & P. Werbicki 32

Usage Models • • 9/25/2020 Define all possible scenarios of use Includes probabilities for each scenario Multiple models can be built Generate test cases automatically K. Read & P. Werbicki 33

Usage Specification • Identify and classify users, scenarios, and environments • Establish hierarchical usage breakdown and probability distribution for software • Obtain agreement with the customer on the basis for software certification. 9/25/2020 K. Read & P. Werbicki 34

Usage Specification Format • Not defined explicitly by Cleanroom • Often represented as a Markov chain – A directed graph – Nodes are states of use – Arcs are stimuli that cause transitions 9/25/2020 K. Read & P. Werbicki 35

Usage Model Example • Assume only two user types: USA and Japan • Assume everyone is running the program in an identical environment 9/25/2020 K. Read & P. Werbicki 36

USA 30% STARTING STATE EMPTY PART DONE 10% FIRST INPUT SECOND INPUT EMPTY PART 15% EMPTY PART . . . LAST INPUT 9/25/2020 DONE 9% EMPTY PART K. Read & P. Werbicki DONE 37

Test Execution • • 9/25/2020 Tests are randomly chosen Tests are generated from the models Controlled tests are executed Result validated against project goals K. Read & P. Werbicki 38

Conclusions • Survey says: – A wonderful advance in software development – Downright weird approach – Maybe a little of both • Theorists love Cleanroom • Some parts are reasonable, some are not • Similar resistance to formal languages 9/25/2020 K. Read & P. Werbicki 39

Conclusions • Mathematically based software development is difficult to do • Main-stream software isn’t written this way • Specific to certain types of software • Still fairly unproven as a software development process 9/25/2020 K. Read & P. Werbicki 40

References Bohner, A. (2001). Software Engineering CS 5704. http: //www. nvc. cs. vt. edu/~bohner/cs 5704/CS 5704 -Class 14. pdf Carnige Mellon Software Engineering Institute (2000). Cleanroom Software Engineering – Suftware Technology Review. http: //www. sei. cmu. edu/activities/str/descriptions/cleanroom_body. html Cleansoft – Cleanroom Software Engineering Inc. http: //www. cleansoft. com/ Hendrix, S. (2000). CSCI 3308 – Spring 2001. http: //www. cs. colorado. edu/~hendrixs/classes/lecture_18. pdf Linger, R. and Trammel, C. (1996). Cleanroom Software Engineering Reference Model Version 1. 0. http: //www. sei. cmu. edu/pub/documents/96. reports/pdf/tr 022. 96. pdf Linger, R. (1993). Cleanroom Software Engineering for Zero Defect Software. http: //www. bsn. usf. edu/departments/isds/faculty/hevner/ism 6124/Linger 1. pdf Wolack, C. (2001). Taking The Art Out Of Software Development – An In-Depth Review of Cleanroom Software Engineering. http: //www. scisstudyguides. addr. com/papers/cwdiss 725 paper 1. htm Oshana, R, and Linger, R. (1999). Capability Maturity Model Software Development Using Cleanroom Software Engineering Principles - Results of an Industry Project http: //www. computer. org/proceedings/hicss/00017/00017042. PDF Pressmen and Associates (2000). Cleanroom Engineering Resources. http: //www. rspa. com/spi/cleanroom. html Stavely, A. (2000). Integrating Z and Cleanroom. http: //archive. larc. nasa. gov/shemesh/Lfm 2000/Proc/cpp 13. pdf THANK YOU! 9/25/2020 K. Read & P. Werbicki 41