Testing Computer Software Mike Stewart Microsoft Fox Pro
- Slides: 31
Testing Computer Software Mike Stewart Microsoft Fox. Pro User Group World Tour
Who Am I and Why Should You Care? u u Software Test Lead for the Visual Fox. Pro team Numerous Knowledge Base articles and white papers for Microsoft Consultant developing custom applications using VFP, C++, Delphi and VB Unix sysadmin and systems programmer using C++ Fox. Pro User Group World Tour 2002
What this Session is Not u u A lot of code examples – no example can fit every situation Testing the “Microsoft way” – this is all based on software QA industry methodologies One size fits all – everyone’s testing needs are different For people with aversions to blue Power Point slides Fox. Pro User Group World Tour 2002
Why Test? u u u Customer satisfaction – testers are customer advocates Cost – the later a bug is found, the more it costs to fix Quality – the earlier a bug is found, the more likely it is to be fixed Fox. Pro User Group World Tour 2002
What is the goal of testing? u u Find bugs – more specifically, get bugs fixed Verify that it works Prevent errors before they get into the product Provide information on risk – how solid is the product? Fox. Pro User Group World Tour 2002
Important Things to Remember u u “A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time. ” - Kaner You will not find all of the bugs n n n Myers’ 20 line program: 100 trillion paths Program takes two inputs of four-digit number: almost 400 million possibilities Test something 10, 000 iterations: what if it fails at 10 K + 1? Fox. Pro User Group World Tour 2002
Exhaustive Testing n “The only exhaustive testing there is, is so much testing that the tester is exhausted!” — Hetzel Fox. Pro User Group World Tour 2002
“White Box” vs. “Black Box” Testing u u u White box testing treats the software as a box that you can see into. n You are looking into the box to see how it works n Assumes you have source code Black box testing treats the software as a box you cannot see into. n Put in data (input) and see what comes out of the box (output). n No source code needed. Black box testing on finds about 55% of the bugs Fox. Pro User Group World Tour 2002
Why White Box Testing isn’t Covered u u Doesn’t test the software in the manner in which it will be used It doesn’t do spec testing – it has no way of knowing if it is functioning as designed or not It does not find missing code – some estimate this accounts for 30% of bugs Not enough time! It’s a discipline of its own, and we’d be here all day. Fox. Pro User Group World Tour 2002
Steps in testing u Test planning n n u u u Write test plans Spec and code reviews Build verification – is it even stable enough to test? Function and system testing – run the full suite of test cases Media verification n No viruses Bits on the media are the release bits The product can be installed from the media Fox. Pro User Group World Tour 2002
Essential Testing Issues u Establish a bug tracking system n n u Establish processes n n u ATS Web – free at http: //msdn. microsoft. com/vstudio No sticky notes Use cases Test plans Test cases Bug entry and resolution Automate, automate! Fox. Pro User Group World Tour 2002
Bug Tracking u Accountability n n u Metrics n n n u Assures that the bug does not fall through the cracks When closed, we are assured that it is fixed. How great is the severity of bugs found? How fast are we closing bugs? Are we ready to ship? Liability n CYA – cover yourself against legal liability (I am not a lawyer, however) Fox. Pro User Group World Tour 2002
Test Plans u u u A test plan is a general “plan of attack”, not a detailed list of test cases. List areas to test, and how they will be tested. Notepad File/Open example “Notepad’s File/Open functionality will be tested for various file types, file size limits, file name limits and regional encoding. Various media need to be considered, such as removable media and networks. ” Fox. Pro User Group World Tour 2002
Test Cases u u u Each task can be compartmentalized into separate automation programs, or conglomerated into one big program. Keep test cases atomized: test one particular feature at a time. Create a matrix, and build test cases from that n u Could be something as simple as an Excel spreadsheet Example: the Notepad File/Open dialog Fox. Pro User Group World Tour 2002
Test Cases for File/Open/Filename u u u u Long file names – boundary testing Single letter file names – boundary testing Long extensions – boundary testing Edit box text length – boundary testing Invalid characters in file names – functional Valid file names – functional Wild card characters – functional Edit keys (Ctrl-C, etc. ) – functional or spec Paste in binary from clipboard - pathological MRU dropdown – functional Path completion – functional Large fonts – functional Gain and lose focus – functional High ASCII chars - functional Fox. Pro User Group World Tour 2002
Bug Entry and Resolution u Minimum to track: n n u Severity Priority The tester who found it The dev who fixed it Enter good bugs: n n n The easier to reproduce, the more likely it is to get fixed Check on a second machine – could be a configuration/OS issue If it only happens on one OS, say so Fox. Pro User Group World Tour 2002
Bug Entry and Resolution u u u If you’re org is tracking metrics, close resolved bugs as quickly as possible Set severity appropriately, and leave the priority to the project manager(s) Your org will have to decide how to set severity and priority – everyone has different needs Fox. Pro User Group World Tour 2002
Testing Methodologies u u Smoke testing – basic functionality Exploratory testing n n u Testing while learning the product Misses large areas of the product Boundary conditions n n Cannot test every value, so test for the most important values Hex boundaries and the boundary of field length Boundary -1, boundary, and boundary +1 Also test largest value allowed, if it is not a hex value Fox. Pro User Group World Tour 2002
Testing Methodologies u Stress n n What happens when a large number of users hit it? Low memory, low disk space, large amounts of data, repetitive Bugs can often not be reproduced Web Application Stress Tool is great for web apps. Fox. Pro User Group World Tour 2002
Testing Methodologies u Functional n n u Performance n u Does it do what the spec says it should do? How does it handle error conditions? A discipline unto itself, worthy of an entire session Regression testing n n Make sure nothing got broken Does not find a lot of bugs Fox. Pro User Group World Tour 2002
Testing Methodologies u u Code Coverage – for VFP, use the Coverage Profiler Code coverage testing tells you what code has not been hit, and therefore needs to have test cases that hit this code. You are unlikely to get 100% coverage The VFP Coverage Profiler can also reveal bottlenecks. Fox. Pro User Group World Tour 2002
Testing Methodologies u Race conditions n n u u Two pieces of code are colliding Especially important in multi-threaded and multi-user apps Error conditions - does it fail gracefully? UI n n Tab order is easy to miss if you are a mouseaddicted developer When testing as a dev, do not use the mouse, because you know the mouse will work. Fox. Pro User Group World Tour 2002
Testing Methodologies u Ad Hoc testing – “guerrilla testing” n n n Great for checking basic functionality Can give broader testing than more formal means Hard to verify coverage Fox. Pro User Group World Tour 2002
Automation u u The Active Accessibility test harness included with VFP 7. 0 Other automation products: n n Rational – Visual Test Mercury Interactive – Testing suite with various products for web and thick client applications Fox. Pro User Group World Tour 2002
Automation Tips u The test should be self-documenting n n u u Do not use “pass” or “fail” as the test result Test should say what it expected and what it received, and whether it passed or failed. Write “drivers” or hooks into your software if needed – for example, VFP’s Active Accessibility or Project Manager hooks Tests should be self-sustaining – create your own data, and clean up. Expect nothing. Fox. Pro User Group World Tour 2002
When Do You Ship? u u You will ship with bugs if your testing is adequate. Define show-stoppers n n n u Define risk n u Will customers find the bug? Does it crash your product? Will only pathologic testers find it? Fixing a bug has a 60% risk of introducing new bugs When the show-stoppers are fixed, ship Fox. Pro User Group World Tour 2002
Selling Testing to Client u u Bulk contracts – sell the job, not the hours to develop it, and the hours to test it. Once the software is in the customer’s hands, bugs are much more expensive to fix How much does it cost to have 30 users find a bug vs. a qualified, dedicated test effort at $x/hour, that assures quality? Other ideas? Fox. Pro User Group World Tour 2002
“Things They Should Teach in the Testing Stuff Class, but Don’t” u u u Run a high-color scheme like Marine High. Color Use large fonts (Display/Settings/Advanced) Throw away your mouse Fox. Pro User Group World Tour 2002
More information u u Testing Computer Software – Kaner The Art of Software Testing - Myer Software Quality Testing Engineering magazine: http: //www. stickyminds. com STAR Conference – the Dev. Con for professional software testers (www. stqe. com) Fox. Pro User Group World Tour 2002
ADVISOR DEVCON Web Update Page http: //www. Advisor. com/CMF 0109 update This session WILL have updates. Fox. Pro User Group World Tour 2002
Thank you! Please remember to fill out your evaluation. Fox. Pro User Group World Tour 2002
- Domain testing in software testing methodologies
- Motivational overview in software testing
- Du path testing
- Globalization testing
- Decision table testing in software testing
- Control structure testing in software engineering
- Decision table testing in software testing
- Decision table advantages and disadvantages
- Decision tables testing
- Rigorous testing in software testing
- Testing blindness in software testing
- Software domain examples
- Visualfox pro
- Visual fox pro 6
- Visual fox pro
- Visual fox pro
- Visual fox pro 6
- Computer aided software testing
- Computer-aided software engineering
- Catt testing
- Computer aided software testing
- Macro pro
- What music software does toby fox use
- Medical testing on animals pros and cons
- Positive testing and negative testing
- Cs 3250
- Functional testing vs unit testing
- Black-box testing disebut juga sebagai behavioral testing
- Behavior testing adalah
- Component testing is a black box testing
- Stewart clark durham
- David r stewart