CS 4723 Software Validation and Quality Assurance Lecture

CS 4723 Software Validation and Quality Assurance Lecture 01 Introduction

Course Instructor Ø Name: Dr. Xiaoyin Wang (Sean) Ø Office: NPB 3. 310 Ø Email: xiaoyin. wang@utsa. edu Ø Experiences Ø Ø Ø Got my Ph. D from Peking University, China Did my postdoc in UC Berkeley Worked for Microsoft (. net project), and Ensighta (a startup company at Berkeley with 7 -8 people) 2

Course Meetings, Web Pages, etc. Ø Course Meetings: TR 11: 30 am – 12: 45 pm MH 3. 02. 18 Ø Office Hours: TR 10 am – 11: 30 am Ø 3 Course Web Page: http: //xywang. 100871. net/CS 4723. htm

Course Textbooks Ø Just reference books l l l 4 Software Testing and Quality Assurance: Theory and Practice, by Kshirasagar Naik and Priyadarshi Tripathy Software Testing: Principles and Practice, by Srinivasan Desikan and opalaswamy Ramesh Introduction to Software Testing, by Paul Ammann and Jeff Offutt

Course Topics 5 Ø Software Testing Ø Software Debugging Ø Software Verification Ø Anti-pattern Checking and Refactoring Ø Bug Management Ø Research Progress: Ø Automatic Oracles Ø Automatic Program Repair

Grading Scheme 6 Ø Exams: 20% * 2 = 40% Ø Mid-term project: 20% Ø Final project: 20% Ø Assignment: 10% Ø Course participation and presentations: 10%

Exams Ø Ø Time: Ø Oct 2 nd: 11: 30 am to 12: 45 pm Ø Dec 4 th: 11: 30 pm to 12: 45 pm Location: Ø Ø 7 MH 3. 02. 18 Content Ø Selected important topics in the course Ø We will have a review class before exams

Project I: Unit Testing Ø Ø Write unit tests for a class in existing open source software Steps Ø Ø Ø Write one or more unit tests for a list of public methods to achieve preset test coverage Deliverable Ø 8 Understand the source code The testing code with comments

Project II: Android UI Testing Ø UI script testing for an Android App Ø Steps Ø Ø Explore the Android App Ø Write Testing Reports Ø Write Scripts for Regression Testing Deliverable Ø 9 A testing report and the testing scripts

Now, let’s go to the real lecture … 10

Why software quality matters? Ø Prevalent usage Ø Billions of PCs and mobile phones Ø Ø 11 Working, Chatting, Gaming, … About 1 million lines of code on a new car, MIT Tech Review 2013 Ø Some software on almost all major or small appliances Ø ATMs Ø Counters at shopping centers, restaurants, … Ø Garage Entrances Ø …

Why software quality matters? Ø We are so dependent on software for critical tasks Ø Traffic Control Ø Ø Transportation Control Ø Ø 12 Bank, stock, online shopping Power supplies Ø Ø Ship / plane / car / elevator /… Financial transactions Ø Ø Trains, air traffic, … Electricity, Power stations, gas, … Medical equipments

What can caused by software bugs Ø 13 From inconvenience to disasters Ø Ugly presentation Ø Not showing some results or output Ø Unable to do operation Ø Lost of a hour’s / day’s / week’s work… Ø Privacy leaks, lost accounts Ø Money loss Ø Injuries, Life-threatening dangers Ø Disasters

Some well-known disasters caused by software bugs Ø Mars Climate Orbiter ($165 M, 1998) Ø Sent to Mars to relay signal from the Mars Lander Ø Smashed to the planet: failing to convert between English measure to metric values Ø Ø Shooting down of Airbus A 300 (290 death, 1988) Ø US CG-49 shoot down a Airbus A 300 Ø Misleading output of the tracking software THERAC-25 Radiation Therapy (1985) Ø 2 cancer patients at East Texas Cancer center received fatal overdoses Ø 14 Miss-handling of race condition of the software in the equipment

Facts about software bugs Ø Ø On average, 1 -5 bugs per KLOC (thousand lines of code) Ø In mature software Ø More than 10 bugs in prototypes Windows 2000 Ø 35 MLOC Ø 63 K known bugs at the time of release Ø 2 bugs per KLOC Ø $59. 5 B loss due to bugs in US 2002 (estimation by NIST) Ø It is not feasible to remove all bugs Ø 15 But try to reduce critical bugs

Why so many bugs in software? Ø Ø Errors can happen in any engineering discipline Software is one of the most error-prone products of all engineering areas Ø Ø Ø Requirements are often vague Software can be really complex, undecidable problems are everywhere Result Ø Ø 16 Most product are put into the market with no or very few errors Almost all software in the market has some number of bugs (we will see that later)

Absolute Software Quality Ø Impossible Ø Ø Ø Free of bugs can never be verified even for some simple programs Practice on critical software Ø 17 Vague Requirements The NASA example

NASA Example Ø Each execution handles $4 Billion equipments, human lives, dreams Ø Ø Ø 18 No prototypes 420 k lines of code, 17 errors in 11 versions (is it a large software? ) Commercial equivalent would have at least 1000 bugs

NASA Example Ø A third of the effort before coding starts Ø Long specifications, written down, fully discussed 19 Ø 40 k pages of specification (longer than the code) Ø Change to add GPS support (2500 pages more specification) Ø Specification is almost pseudo code

NASA Example Ø Ø When fixing bugs, fix what allowed the mistake Ø Unclear API Ø Insufficient tests Ø Improper use of tools Validation and review at all levels Ø 20 85% of bugs revealed before testing

NASA Example Ø Cost Ø 260 people Ø $32 million Ø A year Ø TOO EXPENSIVE!!! Ø Overkill for normal software 21

Quality Economics Tradeoff of cost and requirements 22

Approaches to Enhance Software Quality Ø Bug Prevention Ø Ø Bug Removal Ø Ø Remove bugs after coding, but before release Bug Tolerance Ø 23 Prevent bugs during requirement collection, designing, coding Runtime mechanisms to avoid known or unknown bugs in the software

Approach to prevent bugs Ø Ø Ø 24 Requirement Engineering Ø Remove ambiguity in requirements Ø Formal Specification Design Ø Reduce dependency between modules Ø Concentrate cross-cut concerns Coding Ø Proper coding styles Ø Well written comments Ø Avoid misleading interfaces

Approach to remove bugs Ø Testing Ø Ø Ø Limitations Ø Impossible to cover all cases Ø Test oracles (what is expected) Static checking Ø Ø 25 Feed input to software and run it to see whether its behavior is as expected Identify specific problems (e. g. , memory leak) in the software by scanning the code or all possible paths Limitations Ø Limited problem types Ø False positives

Approach to remove bugs Ø Formal Proof Ø Ø Ø 26 Formally prove that the program implements the specification Limitations Ø Difficult to have a formal specification Ø The proof cost a lot of human efforts Inspection Ø Manually review the code to detect faults Ø Limitations: Ø Hard to evaluate Ø Sometime hard to get progress

Approach to tolerate bugs Ø Exception handling Ø Ø Ø Spend time on writing exception handlers Ø Consider exceptions in the exception handler Redundant code Ø Ø 27 Specify what the program should do when unexpected results are generated Have redundant modules to avoid runtime bugs by checking consistency of results Voting for a correct results when bug happens

In our course Ø Bug prevention: Ø Ø Bug removal: Ø Ø The focus of this course Bug tolerance Ø 28 something you should learn in software engineering course, briefly introduce it at the end of our course We will introduce guidance for exception handling at the end of our course, redundant code will not be introduced because it is rarely used

Aspects of software quality Ø Dependability Ø Ø Efficiency Ø Ø appropriate user interface and adequate documentation Maintainability Ø 29 processing time, memory utilization, responsiveness Usability Ø Ø availability, reliability, safety ease of change

Dependability (I) Ø Availability Ø Ø Ø Recover from a crash? Reliability Ø Ø 30 For how large proportion of time the software system is working For how large proportion of inputs the software is giving correct outputs Fault tolerance?

Dependability (II) Ø Ø Safety Ø For critical software, e. g. insulin control Ø To what extent the software is threatening people? Ø Runtime fault detection? Security Ø Ø 31 To what extent the software is able to protect itself from malicious inputs Input sanitization?

Efficiency Ø Ø Ø Processing time for time-consuming tasks Ø Algorithm Complexity Ø Optimization Maximal Memory utilization Ø Avoid memory leaks Ø Release memory at proper time Responsiveness Ø 32 Interleave UI threads and back-end threads

Usability Ø Ø 33 User Interface Ø Looks attractive Ø Easy to understand convenient to use Help Documentation Ø Easy to search and read Ø Cover all the features of the software

Maintainability Ø Ø Ø Ease to read Ø Code quality Ø Comments Ease to extend Ø Design quality Ø Suitable design patterns Ease to transplant Ø 34 Avoid hard coded elements