Software Testing QA I Kerry Zhu Kerrygmail com

Software Testing & QA (I) Kerry Zhu. Kerry@gmail. com

Zhu. Kerry@gmail. com Introduce Myself p A senior programmer - 1990 p A professor in HFUT - 1996 p Senior p QA Director in Cisco-webex Http: //blog. csdn. net/Kerry. Zhu. Kerry@gmail. com

Zhu. Kerry@gmail. com Background survey How many guys who had testing knowledge or experience? Who can tell us your experience in software testing ? What is Testing?

Zhu. Kerry@gmail. com What is the software testing …

Zhu. Kerry@gmail. com Traditional industry Verify, check, test, ……

Zhu. Kerry@gmail. com What is testing? It is the process of assessing software functionality and behavior at a range of architectural levels and in a multitude of situations. n (Gosh that sounds impressively academic!)

Zhu. Kerry@gmail. com Testing also is 1. Verification & Validation of Software Products against product specifications and customer requirements 2. Detect the software bugs

Software Testing

Zhu. Kerry@gmail. com TESTING Space system Level of detail integration module unit security robustness performance white box usability reliability functional behaviour Characteristics black box Accessibility

Zhu. Kerry@gmail. com Test Cycle Test cases Regression tests Executi on of Additional cases tests Suspected causes Corrections Identified causes Debugging Results Evaluation

Zhu. Kerry@gmail. com Software Testing career path

Zhu. Kerry@gmail. com Chance & Challenge 目前国内软件业的弱点正是发展的前沿 Test Engineer QA/Supervisor Project/Quality

Zhu. Kerry@gmail. com 软件测试 vs. 软件开发

Zhu. Kerry@gmail. com Job marketing 1450 Positions are waiting for … Only in one web site - 51 job. cob

Zhu. Kerry@gmail. com Career Path

Zhu. Kerry@gmail. com Back to course

Zhu. Kerry@gmail. com Schedule 8 weeks (9/8 – 11/14) 3 credits ; 50 periods - 2 times per week. - Tue. 19: 00 - 21: 25 - Sat. 14: 20 – 16: 45 Practice: test in web sites (open source projects), use TA tools End of formal lessons: 11/7 Practice Presentation: 11/10, 11/14 Examination : TBD

Zhu. Kerry@gmail. com Course score Be composed of 4 parts: • Class duty - 5% • Regular exercise - 10% • Practice – semester assignment - 30% • Final Examination - 55%

Zhu. Kerry@gmail. com Book 3

Zhu. Kerry@gmail. com Book 4

Zhu. Kerry@gmail. com Software Testing (II) Part 1 The Big Picture 1 Software Testing Background 2 The Software Development Process 3 The Realities of Software Testing Part II Tesing Fundamentals 4 Examining the Specification 5 Testing the Software with Blinders On 6 Examining the Code 7 Testing the Software with X-Ray Glasses Part III Applying Your Tesing Skills 8 Configuration Testing 9 Compatibility Testing 10 Foreign-Language Testing 11 Usability Tesing 12 Testing the Documentation 13 Testing for Software Security 14 Website Testing Part IV Supplementing Your Testing 15 Automated Testing and Test Tools 16 Bug Bashes and Beta Testing Part V Working with Test Documentation 17 Planning Your Test Effort 18 Writing and Tracking test Cases 19 Reporting What You Find 20 Measuring Your Success Part VI The Future 21 Software Quality Assurance 22 Your Career as a Software Tester

Zhu. Kerry@gmail. com Practice-1

Zhu. Kerry@gmail. com Practice-2

Open Source Test Tools http: //seleniumhq. org/ http: //jakarta. apache. org/jmeter/ http: //www. autoitscript. com/autoit 3/ http: //sourceforge. net/index. php http: //www. opensourcetesting. org/

Business Test Tools n HP Quick. Test Professional n Load. Runner n Rational Functional Tester n Rational Performance Tester n n …… Web. King

Zhu. Kerry@gmail. com Schema I. THE BIG PICTURE. ( 6 学�) II. TESTING FUNDAMENTALS. ( 8 学�) III. APPLYING YOUR TESTING SKILLS. ( 10 学�) IV. SUPPLEMENTING YOUR TESTING. ( 7 学�) V. WORKING WITH TEST DOCUMENTATION. ( 8 学�) VI. THE FUTURE. ( 3 学�) Practice Presentation ( 6 学�) Final Examination ( 2 学�)

Zhu. Kerry@gmail. com Much Benefits from this course You will benefits from this course as - Test engineer - QA Engineer/Manager - The member of SEPG - Project manager - Programmer - Software Analyst - Software Consultant - ……

Zhu. Kerry@gmail. com Goals for this course • How Dose Software testing fit into the software development process • Basic and advanced software testing techniques • Applying testing skills to common testing tasks • Improving test efficiency with automation • Planning and documenting your test efforts

Zhu. Kerry@gmail. com Goals for this course (2) p Effectively reporting the problems you find p Measuring your test efforts and your product’s progress p Knowing the differences between testing and quality assurance (QA) p Getting the thought and methodology in QA from Silicon Valley of USA p Capture the experience accumulated in testing & QA work for many years

Zhu. Kerry@gmail. com Kick off …

Zhu. Kerry@gmail. com Part 1 The Big Picture 1. Software Testing Background a. What is the software defect? b. Why do bugs occur? c. The Cost of bugs? d. What exactly does a tester do? e. The making of a software tester 2. The Software Development a. What is a software product? b. Software Project staff c. Software Lifecycle Models Process 3. The Realities of Software Testing a. Testing Principles b. Terms and Definitions for software testing

Zhu. Kerry@gmail. com Ch. 1 Software Testing Background a. What is the software defect? 1. Software Error Case Studies 2. Terms for Software Failures 3. A formal Definition for Software Defect b. Why do bugs occur? c. The Cost of bugs? d. What exactly does a tester do? The goal of a software tester is to find bugs, find them As Early As Possible, and make sure they get fixed. e. The making of a software tester

Zhu. Kerry@gmail. com Who like to tell us the serious error cases happened in software ?

Zhu. Kerry@gmail. com Pentium division bug (4195835 / 3145727) * 3145727 – 4195835 0 Test engineers found bug Management decided it was not severe A user reported it 10/30/1994, Intel attempted to play down the bug. Offered to replace chips if you could prove you were affected Ultimately, they apologized; replaced all chips - cost $450 million. Interestingly, 1. 13 MHz Pentium III caused software freezes.

Zhu. Kerry@gmail. com NASA’s Mars polar lander 1999, the lander disappeared during landing attempt A switch on the landing gear’s foot set a bit intended to cut off fuel to the landing thruster at touch down; But the switch was tripped already when the legs snapped open for landing Team I tested the leg fold-down procedure, never checked the bit as it was not in the area Team II tested the rest of the landing procedure; reset the computer, hence clearing the bit Both parts worked perfectly separately, but failed when they needed to work together --- and that was never tested! They never noticed that the touch-down bit remained set after fold-down

Zhu. Kerry@gmail. com Bug

Zhu. Kerry@gmail. com Patriot missile defense 1991 The Patriot missile defense system failed to defend against several missiles A small timing error in the system’s clock accumulated so that after 14 hours, the tracking system was no longer accurate In the attack on Dhahran, KSA, the system had been in operation for over 100 hours The system was mobile, so the developers did not expect it to remain in one place for 14 hours

Zhu. Kerry@gmail. com Millennium bug (Y 2 K) n In the 1970 s and on, programmers saved precious memory and disk space by storing years in a two-digit format n They (we) never dreamed that their programs would still be running in 2000

A non-horror story: Zhu. Kerry@gmail. com The random number generator n Worked just fine for most simulation apps n Simulate a solitaire card game that was very difficult to win n In the simulation, you never won n It turned out that the random number generator’s cycle was way too short n There was a sign error in the assembler subroutine Return

Zhu. Kerry@gmail. com Bug

Zhu. Kerry@gmail. com Radiation machine kills four: 1985 to 1987 Faulty software in a Therac-25 radiation-treatment machine resulted in several cancer patients receiving lethal overdoses of radiation. Four patients died. An investigation by independent scientists found that accidents occurred even after Atomic Energy of Canada Limited (AECL) thought it had fixed particular bugs. They wrote in their report. "The basic mistakes here involved poor software-engineering practices and building a machine that relies on the software for safe operation. "

Zhu. Kerry@gmail. com AT&T long distance service fails: 1990 Switching errors in AT&T's call-handling computers caused the company's long-distance network to go down for nine hours, the worst of several telephone outages in the history of the system. The meltdown affected thousands of services and was eventually traced to a single faulty line of code.

Disney’s “Lion King Animated Storybook” 1994 Zhu. Kerry@gmail. com • Disney released its first multimedia CD-ROM for children. • A major ad campaign touting it as “the game to buy”. • After Christmas Day, many angry parents were on the phone to Disney with crying children in the background. • The problem was the game worked on only a few platforms and nothing was said about this on the wrapper as Disney hadn’t realized the problem.

Zhu. Kerry@gmail. com Bug

Zhu. Kerry@gmail. com New Denver airport misses its opening: 1995 The Denver International Airport was intended to be a state-of-the-art airport, with a complex, computerized baggage-handling system and 5, 300 miles of fiberoptic cabling. Unfortunately, bugs in the baggage system caused suitcases to be chewed up and drove automated baggage carts into walls. The airport eventually opened 16 months late, $3. 2 billion over budget, and with a mainly manual baggage system.

Intuit's Mac. In. Tax leaks financial secrets: 1995 Zhu. Kerry@gmail. com Intuit's tax software for Windows and Macintosh has suffered a series of bugs, including several that prompted the company to pledge to pay any resulting penalties and interest. The scariest bug was discovered in March 1995: the code included in a Mac. In. Tax debug file allowed Unix users to log in to Intuit's master computer, where all Mac. In. Tax returns were stored. From there, the user could modify or delete returns. Intuit later ended up winning Bug. Net's annual bug-fix award in 1996 by responding to bugs faster than any other major vendor.

Zhu. Kerry@gmail. com Java security holes; browsers simply crash This is not a single bug but a veritable bug collection. In 1996, a series of security holes in Java, for example, allow hackers to download personal information from someone's PC. To date, the possibility existed prompted many to disable Java in their browsers. a swarm of bugs, ranging from Java. Script flaws in Netscape's Communicator to a reboot bug in Microsoft's Internet Explorer.

Zhu. Kerry@gmail. com Deregulation of California utilities has to wait: 1998 Two new electrical power agencies charged with deregulating the California power industry postponed their plans by at least three months. The delay let them debug the software that runs the new power grid. Consumers and businesses were supposed to be able to choose from 200 power suppliers as of January 1, 1998. The project was postponed after a seven-day simulation of the new system revealed serious problems. The delay may cost as much as $90 million--much of which may eventually be footed by ratepayers, and which may cause some of the new power suppliers to go into debt or out of business before they even start.

Zhu. Kerry@gmail. com What did you get from these cases?

Key issues Zhu. Kerry@gmail. com • No enough test • Not in the various platforms • Incorrect test environment • No integration Test • No performance test • No stress test • Not enough reliability test • Not do exception test • No enough attention on known issues or bugs • ……

Zhu. Kerry@gmail. com What are your ideas about Quality ?

Zhu. Kerry@gmail. com What’s quality? Quality = Customer Satisfaction Customer to our QA? Internal: receivers in next procedure (PM, FT, TEO) External: users (TEO Tool) or Web. Ex customer Break down quality of Web. Ex service • Usability (ease to install; ease to use; friendly UI) • Reliability (foundation for enterprise customer) • Performance • Capacity • Scalability • Service manageability • Compatibility • Extensibility

Zhu. Kerry@gmail. com What is Good Software ? Quality software is reasonably bug-free ①, delivered on time② and within budget③, meets requirements and/or expectations④, and is maintainable⑤. However, quality is obviously a subjective term. It will depend on who the 'customer' is and their overall influence in the scheme of things. Perspectives/Opinions on Quality The quality of the product, process, business environment

Zhu. Kerry@gmail. com Views of Good Software - Transcendental View Something we can recognize but not define, Beyond common thought or experience, like Plato’s description of the ideal, Aristotle’s concept of form - User View Fitness to purpose - Manufacturing view Conformance to specification - Product review be tied to inherent product characteristics - Value-based view Depend on the amount the customer is willing to pay for it

Zhu. Kerry@gmail. com Good Software’s type Quality of the product: The characteristics of software, intertwined functionality -> identify particular aspects of the system Minor, major, critical, fatal/catastrophic Mc. Call model, Boehm Model, ISO 9126 Quality of the process: CMM ( Capability Maturity Model). ISO 9000 SPICE( Software Process Improvement and Capability d. Etermination) Quality in the Context of the Business Environment: Training, Schedule, Risk, Productivity, Customer, Costs, Business

Zhu. Kerry@gmail. com Terms for Software Failures Defect Variance Fault Failure Problem Inconsistency Error Feature Incident Bug Anomaly Return

Zhu. Kerry@gmail. com Formal Definition for Software Defect 1. Product Specification Defines all the functions/features of the product - detailing what it would be, how it will act, what it will do, and what it won’t do. ‘spec’ or ‘functional spec’ 2. Bugs definition, one or more of the 5 rules is ‘True’ a) The software doesn’t do sth. that the spec says it should do. b) The software does sth. that the functional spec says it shouldn’t do. c) The software does sth. that the spec doesn’t mention. d) The software doesn’t do sth. that the spec doesn’t mention but should. e) The software is difficult to understand, hard to use, slow, or—in the software test’s eyes---will be viewed by the end user as just plain not right

Zhu. Kerry@gmail. com Bug Terminology Quality The totality of features and characteristics of the product, process or service that bear on its ability to satisfy stated or implied needs. (ISO 8402) Bug Be a mistake in interpreting a requirement, a syntax error in a piece of code, or the (as-yet-unknown) cause of a system crash. Standard terminology is IEEE 1983 of IEEE Standard 729 - fault, error, reside in any development/maintenance product <inside view> - failure is a departure from the system’s required behavior <outside view>

Zhu. Kerry@gmail. com Bug – IEEE 94 Fault n. Any problem/disfigurement/limitation in product design & development n n n n n Feature or function can’t work Unreasonable design Partly realization in function Data error Run error Limitation in features Difference between actual results and expected results Unfriendly UI, Low performance Others

Examples Zhu. Kerry@gmail. com Difference between errors, faults and failures - Inside vs. outside - code/design/data vs. Function/feature - white box vs. black box Error that leads to a fault in the requirement Misunderstand in MRD or Specification Fault in the requirements that leads to a failure The feature can’t be realized or error feature in MRD Fault in the design that leads to a failure Fault design cause the conflicts in new and old features Fault in the test data that leads to a failure Didn’t consider the boundary condition ( 0, -1, Max, Min) or incorrect parameters, cause client crush

Zhu. Kerry@gmail. com Where are the most bugs from?

Zhu. Kerry@gmail. com No. 1 - Cause is the Spec n 1. The No. 1 Cause is the spec n 2. The next largest source is the design n 3. Not a bug, duplicate, bydesign … Return

Zhu. Kerry@gmail. com Why Spec is No. 1 error producing culprit p not written. p not thorough enough. p constantly changing. not communicated to the entire team in a timely manner. p n “If you can’t say it, you can’t do it. ”

No. 2 error producing culprit is design Zhu. Kerry@gmail. com n Same reason they occur in the Spec. It’s rushed, changed, or not well communicated

Zhu. Kerry@gmail. com WHY DOES SOFTWARE HAVE BUGS? (Some reasons --- but not all) n Miscommunication n Poorly documented code n No communication n Misuse of tools n Software complexity n Errors in system support n Programming errors software n Poor requirements n Poor planning n Changed requirements n Egos n Time pressures n Lack of experience n Changed environments n Inadequate testing

Zhu. Kerry@gmail. com COMPLEXITY CONTRIBUTES TO BUG COUNT p GUI interfaces p MDI environments p Client/server applications p Distributed applications p Data communications p Enormous databases p Sheer size p Object-oriented techniques (if not well-engineered) p Event-driven logic p Heterogeneous hardware p Parallelism

Zhu. Kerry@gmail. com The Cost of bugs Return

Zhu. Kerry@gmail. com What are Tester’s job ?

Zhu. Kerry@gmail. com What Exactly Does a Software Tester Do? The goal of a software tester is • To find bugs • Find them ASAP • Make sure they get fixed • Learn testing techniques discussed

Zhu. Kerry@gmail. com DEBUGGING vs. TESTING TESTER DEBUGGER • Developer Developer • Test team Analyst • QA team • End user THERE IS A FUNDAMENTAL CONFLICT BETWEEN THESE ROLES

Zhu. Kerry@gmail. com TESTING n The operation of a system or application under controlled conditions and the evaluation of results with the intent of finding errors. n Should include normal and abnormal conditions n Testing intentionally attempts to make things go wrong to determine: n if things happen when they shouldn’t n if things don’t happen when they should n Oriented towards “detection”

Zhu. Kerry@gmail. com DEBUGGING p DEBUGGING starts with an identified error and is the process of locating what is causing the bug and correcting the flaw. p It is NOT the process of showing that a bug exists. p Oriented towards “correction”.

The making of a software testing engineer Zhu. Kerry@gmail. com 1. Traits that most software tester should have, n Explorer n Perfectionists n Troubleshooter n Good judgment n Relentless （无情的） n Tactful and Diplomatic (机智老练) n Creative n Persuasive （善说服的） n 2. Advices a. A good listener b. A good learner n 3. Enjoy your work, enjoy testing

Zhu. Kerry@gmail. com Ch. 2 Software Development Process a. Product Components b. Software Project staff c. Software Lifecycle Models

Zhu. Kerry@gmail. com Standish Group CHAOS Figures 1994 2000 Successful 28 K (16%) 78 K ( 28%) Failed 54 K (31%) 65 K ( 23%) Challenged 93 K (53%) 137 K ( 49%) Totals 175 K (100%) 280 K (100%)

Zhu. Kerry@gmail. com WHAT IS SOFTWARE PRODUCT? • Not just a program, software products are made up of software components normally termed deliverables • These are typically organized into categories that correspond to some of the typical phases in a life cycle • Software testers need to be familiar with these as they define and explain the software product.

Zhu. Kerry@gmail. com Product Components 1. Customer Requirements (Surveys, competitive info, Feedback) 2. MRD (Task/Feature description, Usability Data) 3. Specifications 4. Schedule 5. Design Docs (Architecture, Feature implementation, Interface agreement) 6. Test Documents (Plans, test cases, QA Milestone reports) 7. Online help 8. Release Notes / Read Me 9. Release packages Return

Zhu. Kerry@gmail. com MANY OTHER PARTS OF A SOFTWARE PRODUCT • Help files • Samples and examples to illustrate points • Product support information • Error messages • Setup and installation instructions • User manual(s) • Label and stickers • Icons and art • Ads and marketing material • Readme files • etc. Testers are often expected to deal with the appropriateness and completeness of these materials.

Zhu. Kerry@gmail. com REQUIREMENTS Definition: A requirement is a condition of capability needed by a user to solve a problem or achieve an objective. (IEEE) Properties of good requirements: • Visible • Unambiguous • Complete • Consistent (or prioritized if conflicting) • Achievable • Measurable (quantifiable, testable) • Modifiable • Traceable • Dependent requirements are identified • Relevant • Free of unwarranted design detail. • Free of unstated assumptions.

Zhu. Kerry@gmail. com SPECIFICATIONS (or specs) • Requirements are raw data, although organized. • Specifications take the requirements and formalize them so the product is defined exactly. • Many of the desirable characteristics for requirements are needed here. • Different formats are used to present specs. • Level of detail depends upon the company producing the software and the customer • Some require very elaborate checks and balances.

Zhu. Kerry@gmail. com Writing MRD AND SPEC a project management issue - but a difficult one For our purposes, verifiability is critical: • (bad) The system should be easy to use and user errors should be minimal. • (better) Experienced controllers should be able to use all the system functions after a total of two-hours of training. After training, the average number of errors made by the experienced controllers will not exceed two per day. • But there are still problems. What are they?

Zhu. Kerry@gmail. com SOME WAYS TO QUANTIFY Speed: Processed transactions/second; user-response time; screen refresh time. Size: Number of bytes; number of records; number of transactions. Ease of use: Training time; number of help frames; define classes of users objectively

Zhu. Kerry@gmail. com SOME WAYS TO QUANTIFY Reliability: Mean time to failure; probability of unavailability; rate of failure occurrence; availability. Robustness: Time to restart after failure; percentage of events causing failure; probability of data corruption on failure. Portability: Percentage of target-dependent statements; number of target systems.

Zhu. Kerry@gmail. com SCHEDULES • Can be very elaborate. （精细） • There are software tools for producing these and many methods for estimating time needed (as well as costs). • The software testers are concerned with test plans and schedules- when, where, and how the software products will be tested.

Zhu. Kerry@gmail. com SOFTWARE DESIGN DOCUMENTS Level of detail and formality varies. Some possibilities include: • Architectural design • Data flow diagrams • State transition diagrams • Entity-relational diagrams • Decision tables etc. (See CPSC 359: System Analysis and Design)

Zhu. Kerry@gmail. com TEST DOCUMENTS Documents specific to the testing of the software product. Some examples (which will be explored in detail later) are: • Test plans • Test cases • Bug reports • Metrics, statistics, and summaries • Projections as to how many of the bugs have been found.

Zhu. Kerry@gmail. com SOFTWARE PROJECT STAFF • Project manager • Architects (software and hardware, possibly) • System analysts • System designers • System engineers • Programmers or coders • Testers • Quality Assurance persons • Technical writers • User assistance writers • User education trainers • Manual writers • Configuration managers And these just include the technical side!

Zhu. Kerry@gmail. com Software Project staff (1) 1. PMs (Project manager, Program manager) Work with the team to define the product's vision, goals, marketing requirements, development requirements, and core features. n Note that, while some PM positions are more technical and some are business-oriented, the PM job always includes working with marketing to coordinate research and identify the customer and appropriate feature set and pricing. n PM also coordinate the feature set with the Software Design Engineers and work to determine what features can be built in what time frame. n They work with Software Design Engineers in Test and Software Test Engineers to determine how difficult the product is to test and to determine whether they have enough team resources. n

Zhu. Kerry@gmail. com Software Project staff (2) 2. Architects or System Engineer n Technical experts on the product team. Qualify to design the overall systems architecture or design for the software. n Work closely with Programmers 3. Programmers, Developers or Coder (Dev) n Design and implement features, Coding, Fix bugs

Zhu. Kerry@gmail. com Software Project staff (3) 4. QA, Testers n Tester is Responsible for finding and reporting product defect, and make sure bugs get fixed. n QA definition is a little bit different with tester, n To examine and measure the current software development process and find ways to improve it with a goal of preventing bugs from ever occurring.

Zhu. Kerry@gmail. com Software Project staff (4) 5. Technical writer, Manual writes, Online help maker … n Documentation that comes with a software product 6. Configuration management, build engineer

Zhu. Kerry@gmail. com SD Team DEV manager Tester Programmer DOC Management 图DEV-oriented model PM Test Lead DEV lead 图PM-Oriented model PM DOC Test Manager DEV Manager Tri-Model

Zhu. Kerry@gmail. com A Example for Development Team p Windows 2000 Team n 程序经理 450 n 开发人员 900 n 测试人员 1800 n 技术支持人员 600 n 技术传播人员 1120 n 本地化人员 110 n 培训人员 115 n 文档人员 100 n 市场人员 100 n 内部IT 50 n 合计 5345 p Web Matrix Team n n n 程序经理 2 开发组长/架构师： 1 开发人员： 7 测试组长 1 测试人员 13 合计 24

Zhu. Kerry@gmail. com 测试组织中的角色

Zhu. Kerry@gmail. com Why need to build the Models in SDLC ?

Zhu. Kerry@gmail. com Software Lifecycle Models SDLC ( Software Development Life Cycle) n Big-Bang Model n Code-and-Fix Model n Waterfall Model n V-model n Spiral Model n Phased, Incremental and Iterative models

THE SOFTWARE LIFE CYCLE Zhu. Kerry@gmail. com Begins when an application is first conceived. Ends when it is no longer in use. Includes: Initial conception Requirements/ specification analysis Functional design Internal design Documentation planning Test planning Coding Documenting Integration Testing Maintenance Updates Retesting Phase-out Many different models are possible for sequencing these activities!

Zhu. Kerry@gmail. com Activities in SDLC Project Planning Initiate project & Organize Project Definition & Planning Management Review & Approval Analysis User Req’ts Req’t Analysi s s Quality Requirement s Design Implement Test Roll-out UI Design Technic Detaile Codin al d g Design Quality Verificati on Test Planning & Preparation User Procedure & Training Roll-out Planning & Preparation Perform Integratio n Test Perfor m Syste m Test Conve rt Site

BIG BANG MODEL Zhu. Kerry@gmail. com Bring together a huge amount of people and money. Expend a lot of energy and over hours with minimal planning, scheduling, or formal development processes. And we have. .

Zhu. Kerry@gmail. com CODE AND FIX MODEL Typically starts with an informal product specification. Code, fix, and repeat until someone is satisfied. Not a formal model, but one that teams often adopt inadvertently. “There’s never time to do it right, but there’s always time to do it over. ”

Zhu. Kerry@gmail. com EVALUATING THE SOFTWARE DEVELOPMENT MODELS FROM THE VIEWPOINT OF THE TESTER Big Bang Model Testing is nothing more than reporting problems to the customers. Code and Fix Model • You will be constantly changing your testing approaches as the project changes constantly. • You may not be finished testing one version when you receive the next one. • When you work on several of these projects, you will begin to appreciate more formal models.

Overall SDLC Process Zhu. Kerry@gmail. com

Zhu. Kerry@gmail. com Waterfall Model Return

Waterfall Model with Prototyping Zhu. Kerry@gmail. com Requirements Analysis System Design Program Design Coding Unit Test Integration Testing Prototyping User/Customer System Testing Acceptance Testing Operation maintenance

Zhu. Kerry@gmail. com Rapid Application

Zhu. Kerry@gmail. com Rapid Application Development (RAD) - V Model Operation maintenance Validate requirements Requirements Analysis Acceptance Testing System Design System Testing Verify design Unit Test Integration Testing Program Design/Analysis Verify/testing Coding

Zhu. Kerry@gmail. com RAD - V Model (Improved) Validate requirements customer, user, PM, technical support Verify design Engineers Design/Analysis Verify/testing

RAD - V Model (richen by me) Validation of Customer Requirements Operational or Business Needs customer, user, PM, technical support Verification of system design Test Objectives Test Planning Engineers Define Requirements Test Design System Test Execution (Static) Design/Analysis Test Execution (Dynamic) Verify/testing Build System

Zhu. Kerry@gmail. com Evaluation Modified Waterfall • Everything is carefully and rigorously specified. • Accurate plans and a schedule can be established. • Testing is not really begun until the end and, consequently, bugs can creep in early and not be detected until the end. • The result is higher cost for fixing bugs. Rapid Application • Doesn’t specifically provide for testing, but is usually done in the repeat stages. • Still, lacks formality. • This entire model can degenerate into a Code and Fix Model easily with all the problems for testers in that model

Zhu. Kerry@gmail. com Prototyping Methodologies

Zhu. Kerry@gmail. com Spiral Model Return

Zhu. Kerry@gmail. com Spiral Method • Most testers like these models as they can impact the final product early in the development cycle. • It is easy to see where the product came from and where it is going--- this helps testers choose appropriate test plans and schedules. • Normally there is not a last minute push to do a major amount of testing. • Both develop requirements and specifications, but ones that evolve over time.

XP-e. Xtreme Programming n Focus on “the simplest thing that could possibly work. ” n An approach to programming particularly appropriate for: Small team (2 -10 programmers) n “High risk” n Rapid changing or unstable requirements n Requires “testability” n n Main focii n “Communication, simplicity, feedback, courage” Kent Beck

XP - e. Xtreme Programming

XP Lifecycle http: //www. pctest. com/

TDD - Test-Driven Development Start Write a test for new capability Refactor as needed Compile Run the test And see it pass Fix compile errors Write the code Run the test And see it fail

TDD – sub-cycle

Phased development models Developers Zhu. Kerry@gmail. com Build Release 1 Build Release 2 Build Release 3 Time Users USE Release 1 USE Release 2 USE Release 3

Zhu. Kerry@gmail. com Incremental and Iterative models Incremental Development Iterative Development

Ch. 3 The Realities of Software Testing Zhu. Kerry@gmail. com a. Testing Axioms b. Terms and Definitions for software testing

Zhu. Kerry@gmail. com MYTHS ABOUT TESTING • Testing can prove that code is correct. • A system has only a finite number situations in which • • • errors are exhibited. The best tester is the developer as that person “knows” the code. Code must be made bug free or it is worthless. The key to good testing is to run many tests. Testing should be done after the coding is complete, otherwise test cases can’t be run. The testing team is responsible for assuring quality.

Zhu. Kerry@gmail. com MORE MYTHS ABOUT TESTING • Usability problems are not considered valid bugs. • All bugs are equally important. • Finish one testing task before moving onto the next. • Testing is a transitional job for new programmers. • Testers should be recruited from the rank of failed programmers. • Only programmers can be testers. • Programmer’s can’t test their own code. • Testing is best done by a lone person.

Zhu. Kerry@gmail. com Testing Axioms 1. It’s impossible to test a program completely - The number of possible inputs is very large -The number of possible outputs is very large - The number of paths through the software is very large - The software specification is subjective, you might say that the bug is in the eye of the beholder 2. Testing can’t show that bugs don’t exist it is very difficult to prove that the software is 100% correct à 3. Software Testing is a risk-based exercise Figure 3. 2

Zhu. Kerry@gmail. com Testing Axioms 2 4. The more bugs you found, the more bugs there are - Programmers have bad days - Programmers often make the same mistakes - Some bugs are really just the tip of the iceberg (fundamental problem) 5. The pesticide paradox - Build up resistance and the pesticide no longer works - new, various , different test cases 6. Not all the bugs you found will be fixed - There is not enough time - It’s really not a bug - It’s too risky to fix - It’s just not worth it - It can’t be fixed - It is cause by the 3 rd side

Zhu. Kerry@gmail. com Testing Axioms 3 7. When a Bugs’ a bug is difficult to say 8. Product Specifications are never final 9. Software Testers aren’t the most popular members of a project team - Find bugs early -Temper your enthusiasm - Don’t always report bad news 10. Software Testing is a disciplined Technical profession

Terms and Definitions for software testing Zhu. Kerry@gmail. com n Precision and Accuracy n Verification and Validation n Quality and Reliability n Testing and Quality Assurance (QA)

Zhu. Kerry@gmail. com Precision vs. Accuracy n Answers are accurate if it they are within a close tolerance of the “true” answer. n Answers are precise if they are all close together, even if they are not close to the “true” answer. n Example, if the true answer is 0. 123, an answer of 0. 125 would be accurate if the tolerance is. 003, but it would not be precise to 3 decimal places. n Often, of course, we want both.

Zhu. Kerry@gmail. com Precision vs. Accuracy (2)

Zhu. Kerry@gmail. com VERIFICATION The process of evaluating a system or component during the development process to determine whether it satisfies the specifications. THE CRITICAL QUESTION IS: - Did we build what we planned to build? Typically involves reviews and meetings to evaluate: • • • documents plans code requirements specifications

Zhu. Kerry@gmail. com VALIDATION The process of evaluating a system or component during the development process to determine whether it satisfies the user's requirements THE CRITICAL QUESTION IS “Did we build what the user wanted? ”

Zhu. Kerry@gmail. com Verification vs. Validation n Verification confirms that the software meets its specification. - Are we building the product right？ n Validation confirms that the software meets the user’s requirements. - Are we building the right product? n n Never assume the specifications are correct when testing, Both V&V tests must be performed. Example: The Hubble Space mirror was ground incorrectly (because the specs were wrong) and, consequently, it failed to produce clear pictures when originally launched.

Zhu. Kerry@gmail. com Quality vs. Reliability • Quality refers to user satisfaction with things such as the breadth of features, upper compatibility with other software, software support, etc. • Reliability refers to things such as stability and dependability. • Software developers often equate reliability with quality when, in truth, it is one aspect of quality.

SQA - SOFTWARE QUALITY ASSURANCE Zhu. Kerry@gmail. com n Involves the entire software development PROCESS n Monitors and improves the process n Makes sure any agreed-upon standards and procedures are followed n Ensures that problems are found and handled n Oriented towards “prevention” of problems.

Zhu. Kerry@gmail. com Testing vs. Quality Assurance (QA) n The goal of a testing person is to find bugs, find them as early as possible, and make sure they are fixed. n The goal of a QA person is to create and enforce standards and methods to improve the development process so that bugs are prevented. n In smaller organizations, the same people often wear both hats. n But, larger organizations often separate the roles.

Black box testing vs. White box testing Zhu. Kerry@gmail. com Black box testing tests software without knowing how it is coded. Given certain inputs, what will be produced We don’t care HOW it is produced. White box testing uses knowledge of the code to provide clues as to what should be tested and how to test it. Caution must be used to not overlook cases just because the code doesn’t handle them. There is also a gray box testing approach which is handy with web page testing.

Black box vs. White box Zhu. Kerry@gmail. com requirements Structure Test Logic-driven Test output input events Function Test Data-driven Test

Zhu. Kerry@gmail. com Static testing vs. Dynamic testing Static testing involves examining and reviewing code and other support materials that are not running. Dynamic testing refers to testing that is done while the code is running. Many people equate testing with dynamic testing, and they shouldn’t.

Zhu. Kerry@gmail. com Static testing vs. Dynamic testing review leader standards bearer (SQA) producer maintenance oracle reviewer recorder Informal user rep Formal Peer review Walkthrough Inspection

Zhu. Kerry@gmail. com OVERVIEW OF CLASSIFYING TESTS BLACK-BOX: No knowledge of the internal logic of the code is utilized. Tests are based on requirements and functionality. WHITE-BOX: Tests are designed based on the internal logic of the code, code coverage considerations, and analysis of branches, paths, loops, and conditions UNIT: Test at the function or module level. May require test drivers or test harnesses. INCREMENTAL INTEGRATION: Continuous testing as new functionality is added. May need test drivers. Testing combined parts of an application to see if the parts function together properly. • REGRESSION: Re-testing after fixes or modifications of software or the environment.

Zhu. Kerry@gmail. com TEST PHASES COMPARISON: Compares weaknesses and strengths to competing products. ALPHA: Testing when development is nearing completion; minor design changes may be required. BETA: Development and testing viewed as completed and looking for final bugs and problems before final release.

Zhu. Kerry@gmail. com CLASSIFYING TESTS FUNCTIONAL: Black-box testing geared to check functional requirements of an application. SYSTEM: Black-box testing that is based on overall requirements and specifications. STRESS: Often used interchangeably with ‘load’ and ‘performance’. But, others believe tests should check functionality under extreme conditions. PERFORMANCE: Often used interchangeably with “load” and ‘stress’. LOAD: Test under heavy loads to determine at what point a system’s response time degrades or fails END-TO-END: Testing that mimics real-world use.

Zhu. Kerry@gmail. com CLASSIFYING TESTS (continued) ACCEPTANCE: Final testing based on specifications of the end-user or customer. USABILITY: Testing for “user-friendliness”. This is clearly subjective and will depend on the targeted end-user or customer profile. RECOVERY: How well can the system recover from crashes, hardware failures, or other catastrophic problems. INSTALL/UNINSTALL: Test full, partial, or upgrade install/uninstall processes. SECURITY: How well is the system able to protect against unauthorized internal or external access, willful damage, etc. COMPATABILITY: How well does the software perform in a given hardware/software/operating system/network environment.

Zhu. Kerry@gmail. com CLASSIFYING TESTS (summary) By purposes • Correctness testing – Black-box – White-box • Performance testing • Reliability testing - Robustness/strong testing - Exception handling testing - Stress/load testing • Security testing By life cycle phase By scope • implied in • Requirements – Unit testing phase testing – Component testing • Design phase – Integration testing – System testing • Program phase • or in testing – Unit testing • Evaluating test – String testing results • Installation phase – System testing (a test) testing • Acceptance testing – Acceptance testing (b test) • Testing changes: maintenance

Zhu. Kerry@gmail. com Questions and Answers? In Part I, We mainly focus on following problems in software testing, 1. What is a software defect and the causes? 2. What does a software tester do? 3. Software project staff 4. Software development models 5. Basic knowledge in software testing

Zhu. Kerry@gmail. com Q&A

Zhu. Kerry@gmail. com Exercise Page 22: 3. 7. Page 36: 2. 5. Page 50: 3. 4. 6.