Software Testing and Reliability Testing RealTime Systems Aditya

Software Testing and Reliability Testing Real-Time Systems Aditya P. Mathur Purdue University May 19 -23, 2003 @ Guidant Corporation Minneapolis/St Paul, MN Graduate Assistants: Ramkumar Natarajan Baskar Sridharan Last update: April 17, 2003 Functional testing

Learning objectives n Issues in testing real-time systems n Methodology for testing real-time systems n Tools for testing real-time systems Software Testing and Reliability Aditya P. Mathur 2002 2

References n Testing Embedded Systems, Do You have the Gu. Ts for it? , Vincent Encontre, Rational June 2001, http: //www. rational. com/products/whitepapers/final_encontre_testing. jsp n n n Code. Test Vision for Embedded Software, Applied Microsystems Corporation, http: //www. amc. com/products/codetest/. G-Cover Object Code Analyzer, Green Hills Software Inc. , http: //www. ghs. com/products/safety_critical/gcover. html DO-178 B, http: //www. rtca. org and http: //www. windriver. com/products/html/do 178 b. html Software Testing and Reliability Aditya P. Mathur 2002 3

The RTCA DO-178 B Guidelines n n n The FAA’s DO-178 B offers guidelines for the development of airborne systems equipment software. The guidelines in DO-178 B impose constraints on the software development process so that the resulting system is safe. Most RTOS tool vendors have accepted the guidelines in DO 178 B and begun to offer tool support. Software Testing and Reliability Aditya P. Mathur 2002 4

Levels of Criticality-A, B n n Level A: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a catastrophic failure condition for the aircraft. Level B: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a hazardous/severemajor failure condition for the aircraft. Guidant products: Level A or B? Software Testing and Reliability Aditya P. Mathur 2002 5

Levels of Criticality-C, D, E n n n Level C: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a major failure condition for the aircraft. Level D: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function resulting in a minor failure condition for the aircraft. Level E: Software with anomalous behavior, as shown by the system safety assessment process, would cause or contribute to a failure of system function with no effect on aircraft operational capability or pilot workload. Software Testing and Reliability Aditya P. Mathur 2002 6

Guidelines Related to Testing n Test Procedures are correct n Results are correct n Compliance with Software Verification Plan n Test coverage of High-Level requirements n Test coverage of Low-Level requirements n Test coverage of object code n Test coverage - Statement, Decision, and Modified condition/decision n Test coverage - Data coupling and control coupling Software Testing and Reliability Aditya P. Mathur 2002 7

Issues in Testing Real-Time Systems n n Separation of development and execution platforms Variety of execution platforms (combinations of hardware and OS, e. g. Z 80/QNX, Tornado/Power. PC) n Tight resources and timing constraints n Emerging quality and certification standards Software Testing and Reliability Aditya P. Mathur 2002 8

Test Methodology: Unit Testing n Identify the module to be tested. This module becomes Mu. T. n Prepare Mu. T for testing. n n Isolate the Mu. T from its environment. Make Mu. T an independently executable module by adding a stub and a test driver. Test harness Test Driver Generate tests Software Testing and Reliability Aditya P. Mathur 2002 Mu. T Module under test stub Replaces the rest of the application 9

Issues in Unit Testing n How to generate tests? n What to observe inside Mu. T? n n Returned parameters Internal state such as values of global variables, coverage, control flow for verification against a UML sequence diagram, etc. How to observe? n Use coverage tools or debuggers n Insert probes When to stop testing? Software Testing and Reliability Aditya P. Mathur 2002 10

Test Methodology: Integration Testing n Combine multiple Mu. T into a larger software component. n Build a test harness. n Look for the correctness of interactions amongst the various Mu. Ts in the component. Again, UML sequence diagrams can be useful in validating the interactions against design. Software Testing and Reliability Aditya P. Mathur 2002 11

Test Methodology: System (Unit) Testing n The Mu. T is now a complete software application. n Test is executed under the RTOS. n n Communication amongst individual modules might be via the RTOS. Observation is done using probes. The data so collected is analyzed on the host platform. The test driver could now be a simulator. The system needs to be brought to a desired initial state and then generate and send to the system a sequence of test messages. Usually one does grey-box testing at this stage but more could be done. Software Testing and Reliability Aditya P. Mathur 2002 12

Test Methodology: System Integration Testing n n Integrate other software components of the end-application. These might be off-the-shelf components. Test is integrated system Software Testing and Reliability Aditya P. Mathur 2002 13

Test Methodology: System Validation Testing n For overall embedded system, perform the following: n n n Check if the system meets the end-user functional requirements. Perform non-functional testing. This includes load testing, robustness testing, and performance testing. Check for conformance with any inter-operability requirements. Software Testing and Reliability Aditya P. Mathur 2002 14

Run-time Analysis Source Compiler Assembler/Linker Executable GUI Analysis tools RTA server Target platform Wind. River and Agilent offer real-time trace tools that work with Agilent’s logic analyzers. Software Testing and Reliability Aditya P. Mathur 2002 15

Applicability of Test Techniques to Real-Time Systems n n n All techniques for test data generation and coverage analysis covered in this course are applicable when testing real-time applications. Due to the embedded nature of such applications, one generally needs to use special tools that operate in a clientserver mode; the server running on the host machine and the client on the target environment. The client provides data gathered during the execution of the embedded application and the server analyses and presents it to the tester. Software Testing and Reliability Aditya P. Mathur 2002 16

Tools for Testing Real Time Systems n n Rational: Test Real. Time: code coverage; execution sequence, unit testing; integration testing, checking assertions for C++ classes; testing C threads, tasks, processes; evaluatin the UML model; requirement to test traceability; plug-in for mathworks Simulink. Applied Microsystems: Code. Test: A suite of several tools; performance measurement; tracing of execution history; finds memory leaks; code coverage measurement; supports QNX. Green Hills Software: G-Cover: Coverage analysis at the object code level; indirect source code coverage information. Wind. River: RTA: Detects memory leaks; identifies hot spots; per task profiling. Software Testing and Reliability Aditya P. Mathur 2002 17

Summary n Issues in testing real-time systems n Test tool architecture n Test methodology Software Testing and Reliability Aditya P. Mathur 2002 18