2014 REU Program at ECU Software Testing Foundations

2014 REU Program at ECU Software Testing - Foundations, Tools, and Applications Lecture 1 May 27, 2014 Introduction to Software Testing Dr. Sergiy Vilkomir 1/42

Dr. Sergiy Vilkomir Instructor’s office: S & T Building, C-111 Email: vilkomirs@ecu. edu http: //www. cs. ecu. edu/reu/index. html http: //core. ecu. edu/vilkomirs http: //core. ecu. edu/STRG/ 2/42

Software life cycle • • concept requirements design implementation (coding) testing installation operation and maintenance Is testing important ? 3/42

Software Errors Software errors can be found and eliminated during software testing “Software bugs, or errors, are so prevalent and so detrimental that they cost the U. S. economy an estimated $59. 5 billion annually, or about 0. 6 percent of the gross domestic product, according to a newly released study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST)”. (NIST News, June 42, 2002 ) 4/42

Software Errors Buggy Software May Have Crashed Mars Polar Lander (March 2000) • The software problem crashed the Mars Polar Lander into the Red Planet’s frozen ground • The most likely cause of the lander’s failure, investigators decided, was that a spurious sensor signal associated with the craft’s legs falsely indicated that the craft had touched down when in fact it was some 130 -feet (40 meters) above the surface. • This caused the descent engines to shut down prematurely and the lander to free fall out of the martian sky. • The error was traced to a single bad line of software code. 5/42

Causes of software defects Software Bugs with Fatal Consequences • 1985 - 1986 few patients were killed in a hospital in USA because the medicine dose was calculated wrongly (software bugs in Therac-25 treatment machine lead to radiation overdoses, approximately 100 times the intended dose). 6/42

7/42

Causes of software defects • In 1996 the Ariane 5 rocket of the European space agency was destroyed because they used the software from Ariane 4 rocket (was self-destroyed 40 seconds after takeoff; more than $370 million was lost ). 8/42

Causes of software defects Software Bugs with Fatal Consequences • In 1962 NASA lost their 80 Million Dollar "Mariner 1" because of a missing "hyphen" in the program code. 9/42

Causes of software defects Software Bugs with Fatal Consequences During the Gulf War in February, 1991, a Patriot missile system operating in Dhahran, Saudi Arabia, failed to track and intercept an incoming Scud. The Iraqi missile impacted into an army barracks, killing 42 U. S. soldiers and injuring another 98. The cause of the missile system failing to defend against the incoming Scud was traced back to a bug in Patriot’s radar and tracking software. The bug occurs in the calculation of the next location of the incoming target. That modified software reached Dhahran the day after the attack. 10/42

Computers and Software Errors • On Tuesday, June 3, 1980, at 1: 26 a. m. , the display system at the command post of the Strategic Air Command (SAC) near Omaha, Nebraska, indicated that two submarine-launched ballistic missiles (SLBMs) were headed toward the United States. • The SAC duty controller directed all alert crews to move to their B-52 bombers and to start their engines • Land-based missile crews were put on a higher state of alert, and battle-control aircraft prepared for flight • Fortunately, there were a number of factors which made those involved in the assessment doubt that an actual attack was underway • The cause of these incidents was eventually traced to the failure of a single integrated circuit chip in a computer which was part of a communication system 11/42

Causes of software defects Software Bugs with Fatal Consequences • Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983. • The software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so. • Disaster was averted when a Soviet commander, based on what he said was a '. . . funny feeling in my gut', decided the apparent missile attack was a false alarm. The filtering software code was rewritten. 12/42

Software Errors Investigation found that the satellite in question had picked up the sun's reflection off the cloud tops and somehow interpreted that as a missile launch. 13/42

CNN Money - August 9, 2012 Computer glitch that set fire to $440 million of Knight Capital Group's funds In less than an hour, Knight Capital's computers executed a series of automatic orders that were supposed to be spread out over a period of days. Millions of shares changed hands. The resulting loss, which was nearly four times the company's 2011 profit, crippled the firm and brought it to the edge of bankruptcy. 14/42

Top Software Failures Of 2011 (from http: //www. businesscomputingworld. co. uk) 1. Financial services giant fined $25 million for hiding software glitch that cost investors $217 million 2. Computer system bugs cause Asian banking facilities’ downtime 3. Cash machine bug benefits customers by giving them extra money 4. 22 people wrongly arrested in Australia due to failures in new NZ $54. 5 million courts computer system 5. 42, 420 cars recalled after airbag-related software glitch 15/42

Top Software Failures Of 2011 (from http: //www. businesscomputingworld. co. uk) Conclusions • Software failures are costing companies and consumers large amounts of money • The worst software failures have damaged reputations, impacted negatively on financials and caused stress to users. • Each of the top 2011 software failure examples could easily have been avoided through an effective quality management and proper software testing 16/42

• Systems where failure means: death, injury, destruction, chaos, severe financial loss, etc. • Such systems often are safety-critical systems or mission -critical systems Student project 17/54

18/54

• • Nuclear power plant systems Military systems Space systems Avionic software Medical software Large telecommunication systems. . . 19/54

Software testing • Software errors can be found during software testing • Testing can help but it is not for free • The cost of software testing could be around 40% of the overall costs of software development 20/42

Terminology Validation Verification Functional testing Regression testing Error Unit testing Failure Fault Software testing Oracle Input domain V&V 21/42

Terminology • Unfortunately, SE terminology in textbooks is often contradictory • Two different textbooks could provide two different definitions for the same term • What to do in this situation? Solution: use terminology from standards 22/42

IEEE (Institute of Electrical and Electronics Engineers) standard IEEE Std 610. 12 1990 IEEE Standard Glossary of Software Engineering Terminology 23/42

New standard • IEEE Std 610. 12 1990 has not been updated for many years • Some terms are obsolete • Instead: ISO/IEC/IEEE 24765: 2010 “Systems and software engineering – Vocabulary” 24/42

New International Software Testing Standard “Software and systems engineering — Software testing” • ISO/IEC/IEEE 29119 -1: Concepts & Definitions (published September 2013) • ISO/IEC/IEEE 29119 -2: Test Processes (published September 2013) • ISO/IEC/IEEE 29119 -3: Test Documentation (published September 2013) • ISO/IEC/IEEE 29119 -4: Test Techniques (draft) • ISO/IEC/IEEE 29119 -5: Keyword Driven Testing (draft) 25/42

From IEEE Std 610. 12 1990 testing - the process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component Contrast with: Static Analysis, Inspection 26/42

From IEEE Std 610. 12 1990 • Static Analysis - the process of evaluating a system or component based on its form, structure, content, or documentation. • Inspection - a static analysis technique that relies on visual examination of development products to detect errors, violations of development standards, and other problems. 27/42

From IEEE Std 610. 12 1990 • Fault - an incorrect step, process, or data definition in a computer program. Note: In common usage, the terms "error" and "bug" are used to express this meaning. • Failure - the inability of a system or component to perform its required functions within specified performance requirements. The result of the fault. 28/42

From IEEE Std 610. 12 1990 • test case - a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement. 29/42

• Input domain – the set of all possible inputs to a program • test oracle - a source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), a user manual, or an individual’s specialized knowledge, but should not be the code 30/42

Y= a X + b – it is very easy to determine expected results ? 31/42

Goal of software testing Increasing our confidence on the quality and reliability of the program Test case that did not find an error - successful Test case that discovered a new error - successful Increasing the quality and reliability of the program after removing the error 32

Goal of software testing Dual Goal • To find undiscovered errors AND • to get confidence on the quality and reliability of the program 33

Lifecycle phase in which testing takes place 34 © Aditya P. Mathur 2005

From IEEE Std 610. 12 1990 • black-box testing -Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions. Syn: . functional testing Contrast with: structural testing. • white-box testing -Testing that takes into account the internal mechanism of a system or component. Types include branch testing, path testing, statement testing. Syn: structural testing, glass-box testing Contrast with: functional testing 35

Is testing always done in the same way? Types of software Program A Program B Program C Program D Personal software Company’s internal software Commercial software Safety-critical software Consequences of defects nothing serious internal problems financial losses, damaged reputation loss of human life 36/42

• Testing is done differently in different contexts. – Type of software – Recourses (constraints): human recourses, financial recourses, time – Regulatory requirements • Conclusion: testing is context dependent 37/42

Causes of software defects Human mistake (error) Defect (bug, fault) in software Failure during software execution 38/42

How much testing is enough? Exhaustive testing is impossible Testing everything (all combinations of inputs and preconditions) is not feasible Example: • The exam has 30 questions and four possible answers for each question. • What is a number of all possible combinations of answers? • If we can test 1000 combinations per second, how long will we test all combinations? (An hour? A day? A month? ) Total number of combinations: 4 4 4 …. 4 = 430 1. 000 30 times 1 year 42. 000 sec. We need 42. 000 years for testing 39/42

How much testing is enough? Not possible to say in general Depends on each specific situation • Pressures on a project include time and budget • Risk assessment to decide how much testing to do. Important: • prioritization (do the most important tests first) • setting criteria when to stop testing • Take into account technical and business risk • Take into account project constraints. 40/42

Testing principles Testing shows presence of defects Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness “Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of ? 41/42

Testing principles Testing shows presence of defects “Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of Dijkstra Edsger Dijkstra (1930 – 2002) was a Dutch computer scientist. Winner of the 1972 Turing Award; creator of "structured programming. " 42/42