INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Experience Report: Error Distribution in Safety-Critical Software & Software Risk Analysis Based on Unit Tests Stephan Ramberger (stephan. ramberger@arcs. ac. at) Thomas Gruber (thomas. gruber@arcs. ac. at) ARC Seibersdorf research Gmb. H WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 1

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Error Cost – Example Ariane 5 Flight 501 § § § § § June 4, 1996: Maiden flight of Ariane 5 37 secods after ignition horizontal speed = 32. 768, 0 internal units Exception in „idle“ subsystem by 16 bit – integer conversion (overflow) Subsystem software had been tested only for the smaller Ariane 4 model. Redundant computer had the same problem 72 ms earlier – switched off Main controller interprets diagnostic data as trajectory data Unreasonable control commands to the engines Purposeful blowing-up after 40 Seconds at 3. 700 m altitude Damage: § € 130. 000 launch cost § € 435. 000 cluster satellites § € 310. 000 subsequent improvements http: //uprhmate 01. upr. clu. edu/~pnm/notas 4061/ariane 5. htm WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 3

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Testing: Cost and Benefit Total cost safety-critical software commercial software test effort WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 4

INFORMATION TECHNOLOGIES Term Definitions SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY § Test ►Testing is the process of exercising or evaluating a system or a system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results. [IEEE 729 -1983 "IEEE Standard Glossary of SW Engineering Terminology"] § Verification ►Verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled. Have we built the system correctly ? [ISO 8402: 1994] § Validation ►Validation means confirmation by examination and provision of objective evidence thatsystem the particular for a specific Does the meet requirements our intended use are fulfilled. [ECSS-P-001 A, Rev. 1] expectation ? WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 5

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Test & Verification requirements + design docs document verification SUT testclass test case software under test static analysis V&V report test results dynamic test WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 6

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Classification of Tests black box test white box test Acceptance tests ? Integration test Unit test re-test 1 st test ? ? bugfix WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber ? ? System test existing code regression test ? 7

INFORMATION TECHNOLOGIES Unit Test with IPL Cantata++ test object C++ classes SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY coverage, white & black box test cases C++ frames with test commands main precompile ctp make instrumented (compile code cpp & link) C++ C++ WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber Cantata+ + wrapper classes C++ C/C++ libraries Cantata++ libraries C++ reports C++ 8

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Test Metrics § Residual error rate § Value e. g. 20% § Ascending over time (Yet many errors occurring? ) § Convergence (Do we discover less errors by time? ) § Test coverage (dynamic) § „Our“ test tool Cantata++ implements 11 different coverage metrics § Ideal: 100%, often <100% due to „dead code“, „defensive programming“. . . § Complexity (static) § Statements per unit (static) § LOCs per unit (static) WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 9

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Test Coverage – Examples Statement Decision ? ? 1 2 WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber Path coverage ? 4 test cases 10

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Cyclomatic Complexity c=e–n+2 c = cyclomatic complexity e = number of edges n = number of nodes Cyclomatic complexity 1 -10 11 -20 21 -50 >50 WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber risk simple, low-risk program more complex, medium risk complex, high risk unstable program, very high risk 11

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Experience Report WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 12

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Error Types § Documentati on Errors § Other § Coding Errors § Coverage Errors WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 13

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Distribution of Error Types WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 14

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Documentation Errors WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 15

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Coding Errors WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 16

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Incomplete Coverage WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 17

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY From Errors to Complexity WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 18

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Defining Risk Areas WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 19

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Categorization Example WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 20

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Errors by Cyclomatic Complexity WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 21

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Mean Error Count WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 22

INFORMATION TECHNOLOGIES SAFETY AND QUALITY THROUGH INFORMATION TECHNOLOGY Conclusion § detailed design § code review § thorough verification WSRS Ulm – 20 Sept. 2004 St. Ramberger / Th. Gruber 24