Testing Computer Software Mike Stewart Microsoft Fox Pro

  • Slides: 31
Download presentation
Testing Computer Software Mike Stewart Microsoft Fox. Pro User Group World Tour

Testing Computer Software Mike Stewart Microsoft Fox. Pro User Group World Tour

Who Am I and Why Should You Care? u u Software Test Lead for

Who Am I and Why Should You Care? u u Software Test Lead for the Visual Fox. Pro team Numerous Knowledge Base articles and white papers for Microsoft Consultant developing custom applications using VFP, C++, Delphi and VB Unix sysadmin and systems programmer using C++ Fox. Pro User Group World Tour 2002

What this Session is Not u u A lot of code examples – no

What this Session is Not u u A lot of code examples – no example can fit every situation Testing the “Microsoft way” – this is all based on software QA industry methodologies One size fits all – everyone’s testing needs are different For people with aversions to blue Power Point slides Fox. Pro User Group World Tour 2002

Why Test? u u u Customer satisfaction – testers are customer advocates Cost –

Why Test? u u u Customer satisfaction – testers are customer advocates Cost – the later a bug is found, the more it costs to fix Quality – the earlier a bug is found, the more likely it is to be fixed Fox. Pro User Group World Tour 2002

What is the goal of testing? u u Find bugs – more specifically, get

What is the goal of testing? u u Find bugs – more specifically, get bugs fixed Verify that it works Prevent errors before they get into the product Provide information on risk – how solid is the product? Fox. Pro User Group World Tour 2002

Important Things to Remember u u “A test that reveals a problem is a

Important Things to Remember u u “A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time. ” - Kaner You will not find all of the bugs n n n Myers’ 20 line program: 100 trillion paths Program takes two inputs of four-digit number: almost 400 million possibilities Test something 10, 000 iterations: what if it fails at 10 K + 1? Fox. Pro User Group World Tour 2002

Exhaustive Testing n “The only exhaustive testing there is, is so much testing that

Exhaustive Testing n “The only exhaustive testing there is, is so much testing that the tester is exhausted!” — Hetzel Fox. Pro User Group World Tour 2002

“White Box” vs. “Black Box” Testing u u u White box testing treats the

“White Box” vs. “Black Box” Testing u u u White box testing treats the software as a box that you can see into. n You are looking into the box to see how it works n Assumes you have source code Black box testing treats the software as a box you cannot see into. n Put in data (input) and see what comes out of the box (output). n No source code needed. Black box testing on finds about 55% of the bugs Fox. Pro User Group World Tour 2002

Why White Box Testing isn’t Covered u u Doesn’t test the software in the

Why White Box Testing isn’t Covered u u Doesn’t test the software in the manner in which it will be used It doesn’t do spec testing – it has no way of knowing if it is functioning as designed or not It does not find missing code – some estimate this accounts for 30% of bugs Not enough time! It’s a discipline of its own, and we’d be here all day. Fox. Pro User Group World Tour 2002

Steps in testing u Test planning n n u u u Write test plans

Steps in testing u Test planning n n u u u Write test plans Spec and code reviews Build verification – is it even stable enough to test? Function and system testing – run the full suite of test cases Media verification n No viruses Bits on the media are the release bits The product can be installed from the media Fox. Pro User Group World Tour 2002

Essential Testing Issues u Establish a bug tracking system n n u Establish processes

Essential Testing Issues u Establish a bug tracking system n n u Establish processes n n u ATS Web – free at http: //msdn. microsoft. com/vstudio No sticky notes Use cases Test plans Test cases Bug entry and resolution Automate, automate! Fox. Pro User Group World Tour 2002

Bug Tracking u Accountability n n u Metrics n n n u Assures that

Bug Tracking u Accountability n n u Metrics n n n u Assures that the bug does not fall through the cracks When closed, we are assured that it is fixed. How great is the severity of bugs found? How fast are we closing bugs? Are we ready to ship? Liability n CYA – cover yourself against legal liability (I am not a lawyer, however) Fox. Pro User Group World Tour 2002

Test Plans u u u A test plan is a general “plan of attack”,

Test Plans u u u A test plan is a general “plan of attack”, not a detailed list of test cases. List areas to test, and how they will be tested. Notepad File/Open example “Notepad’s File/Open functionality will be tested for various file types, file size limits, file name limits and regional encoding. Various media need to be considered, such as removable media and networks. ” Fox. Pro User Group World Tour 2002

Test Cases u u u Each task can be compartmentalized into separate automation programs,

Test Cases u u u Each task can be compartmentalized into separate automation programs, or conglomerated into one big program. Keep test cases atomized: test one particular feature at a time. Create a matrix, and build test cases from that n u Could be something as simple as an Excel spreadsheet Example: the Notepad File/Open dialog Fox. Pro User Group World Tour 2002

Test Cases for File/Open/Filename u u u u Long file names – boundary testing

Test Cases for File/Open/Filename u u u u Long file names – boundary testing Single letter file names – boundary testing Long extensions – boundary testing Edit box text length – boundary testing Invalid characters in file names – functional Valid file names – functional Wild card characters – functional Edit keys (Ctrl-C, etc. ) – functional or spec Paste in binary from clipboard - pathological MRU dropdown – functional Path completion – functional Large fonts – functional Gain and lose focus – functional High ASCII chars - functional Fox. Pro User Group World Tour 2002

Bug Entry and Resolution u Minimum to track: n n u Severity Priority The

Bug Entry and Resolution u Minimum to track: n n u Severity Priority The tester who found it The dev who fixed it Enter good bugs: n n n The easier to reproduce, the more likely it is to get fixed Check on a second machine – could be a configuration/OS issue If it only happens on one OS, say so Fox. Pro User Group World Tour 2002

Bug Entry and Resolution u u u If you’re org is tracking metrics, close

Bug Entry and Resolution u u u If you’re org is tracking metrics, close resolved bugs as quickly as possible Set severity appropriately, and leave the priority to the project manager(s) Your org will have to decide how to set severity and priority – everyone has different needs Fox. Pro User Group World Tour 2002

Testing Methodologies u u Smoke testing – basic functionality Exploratory testing n n u

Testing Methodologies u u Smoke testing – basic functionality Exploratory testing n n u Testing while learning the product Misses large areas of the product Boundary conditions n n Cannot test every value, so test for the most important values Hex boundaries and the boundary of field length Boundary -1, boundary, and boundary +1 Also test largest value allowed, if it is not a hex value Fox. Pro User Group World Tour 2002

Testing Methodologies u Stress n n What happens when a large number of users

Testing Methodologies u Stress n n What happens when a large number of users hit it? Low memory, low disk space, large amounts of data, repetitive Bugs can often not be reproduced Web Application Stress Tool is great for web apps. Fox. Pro User Group World Tour 2002

Testing Methodologies u Functional n n u Performance n u Does it do what

Testing Methodologies u Functional n n u Performance n u Does it do what the spec says it should do? How does it handle error conditions? A discipline unto itself, worthy of an entire session Regression testing n n Make sure nothing got broken Does not find a lot of bugs Fox. Pro User Group World Tour 2002

Testing Methodologies u u Code Coverage – for VFP, use the Coverage Profiler Code

Testing Methodologies u u Code Coverage – for VFP, use the Coverage Profiler Code coverage testing tells you what code has not been hit, and therefore needs to have test cases that hit this code. You are unlikely to get 100% coverage The VFP Coverage Profiler can also reveal bottlenecks. Fox. Pro User Group World Tour 2002

Testing Methodologies u Race conditions n n u u Two pieces of code are

Testing Methodologies u Race conditions n n u u Two pieces of code are colliding Especially important in multi-threaded and multi-user apps Error conditions - does it fail gracefully? UI n n Tab order is easy to miss if you are a mouseaddicted developer When testing as a dev, do not use the mouse, because you know the mouse will work. Fox. Pro User Group World Tour 2002

Testing Methodologies u Ad Hoc testing – “guerrilla testing” n n n Great for

Testing Methodologies u Ad Hoc testing – “guerrilla testing” n n n Great for checking basic functionality Can give broader testing than more formal means Hard to verify coverage Fox. Pro User Group World Tour 2002

Automation u u The Active Accessibility test harness included with VFP 7. 0 Other

Automation u u The Active Accessibility test harness included with VFP 7. 0 Other automation products: n n Rational – Visual Test Mercury Interactive – Testing suite with various products for web and thick client applications Fox. Pro User Group World Tour 2002

Automation Tips u The test should be self-documenting n n u u Do not

Automation Tips u The test should be self-documenting n n u u Do not use “pass” or “fail” as the test result Test should say what it expected and what it received, and whether it passed or failed. Write “drivers” or hooks into your software if needed – for example, VFP’s Active Accessibility or Project Manager hooks Tests should be self-sustaining – create your own data, and clean up. Expect nothing. Fox. Pro User Group World Tour 2002

When Do You Ship? u u You will ship with bugs if your testing

When Do You Ship? u u You will ship with bugs if your testing is adequate. Define show-stoppers n n n u Define risk n u Will customers find the bug? Does it crash your product? Will only pathologic testers find it? Fixing a bug has a 60% risk of introducing new bugs When the show-stoppers are fixed, ship Fox. Pro User Group World Tour 2002

Selling Testing to Client u u Bulk contracts – sell the job, not the

Selling Testing to Client u u Bulk contracts – sell the job, not the hours to develop it, and the hours to test it. Once the software is in the customer’s hands, bugs are much more expensive to fix How much does it cost to have 30 users find a bug vs. a qualified, dedicated test effort at $x/hour, that assures quality? Other ideas? Fox. Pro User Group World Tour 2002

“Things They Should Teach in the Testing Stuff Class, but Don’t” u u u

“Things They Should Teach in the Testing Stuff Class, but Don’t” u u u Run a high-color scheme like Marine High. Color Use large fonts (Display/Settings/Advanced) Throw away your mouse Fox. Pro User Group World Tour 2002

More information u u Testing Computer Software – Kaner The Art of Software Testing

More information u u Testing Computer Software – Kaner The Art of Software Testing - Myer Software Quality Testing Engineering magazine: http: //www. stickyminds. com STAR Conference – the Dev. Con for professional software testers (www. stqe. com) Fox. Pro User Group World Tour 2002

ADVISOR DEVCON Web Update Page http: //www. Advisor. com/CMF 0109 update This session WILL

ADVISOR DEVCON Web Update Page http: //www. Advisor. com/CMF 0109 update This session WILL have updates. Fox. Pro User Group World Tour 2002

Thank you! Please remember to fill out your evaluation. Fox. Pro User Group World

Thank you! Please remember to fill out your evaluation. Fox. Pro User Group World Tour 2002