Exploratory Test Management Outline 1 Introduction 2 Test

Exploratory Test Management Outline 1. Introduction 2. Test Management and Techniques 3. ET Planning, Exec. and Documentation 4. ET Styles 5. ET Management ET Workshop v. 1. 20 Test Management • Exploratory Test Team Management • Risk-Based Test Management • Session Based Test Management © 2002 Amland Consulting 1

Exploratory Test Management These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http: //creativecommons. org/licenses/bysa/2. 0/legalcode. ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 2

1. 2. 3. 4. 5. Exploratory Test Management 5. 1 Exploratory Test Team Management 5. 2 Risk-Based Test Management 5. 3 Session Based Test Management ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 5 -3

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Traditional Test Teams… …are built on… Available knowledge and skills Business needs Employee’s ambitions …provide a career path through… Execution coordination planning Defect reporting analysis Strategy This might not work for Exploratory Testing ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 4

1. 2. 3. 4. 5. How to build and lead an effective test team Slides and Questionnaire from Lloyd Roden, Grove Consultants, Lloyd@grove. co. uk ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (Lloyd@grove. co. uk) © 2002 Amland Consulting 5 -5

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The questionnaire Things to note No right or wrong answer Try not to think too much Helps us assess our strengths Can be used with other management questionnaires (Belbin, 16 PF etc) Plot values on the graph ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 6

Introduction Test Management and Techniques ET Planning and Documentation ET Styles The questionnaire - how to complete ET Management X - Axis Friendly Approachable Casual Open Unstructured Social Intuitive Random Warm Perceptive ET Workshop v. 1. 20 Test Management X __ __ X __ __ X 6__ Formal Retiring Business Like Guarded Organised Introvert Logical Focused Cool Insensitive __ X __ __ X __ Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 7

Introduction Test Management and Techniques ET Planning and Documentation ET Styles The questionnaire - how to complete ET Management Y - Axis To the point Challenging Quick Insistent Lively Impatient Adventurous Confronting Competitive Strong Minded ET Workshop v. 1. 20 Test Management __ __ X __ __ __ 2__ Indirect Accepting Leisurely Thoughtful Relaxed Patient Cautious Receptive Co-operative Analytical X __ __ X __ Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 8

Introduction Test Management and Techniques ET Planning and Documentation The Grid ET Styles ET Management 10 9 8 7 6 0 1 2 3 4 5 6 7 8 9 10 4 3 2 1 X (6, 2) 0 ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 9

Introduction Test Management and Techniques ET Planning and Documentation The Model ET Styles ET Management The Pragmatist The Analyst ET Workshop v. 1. 20 Test Management The Pioneer The Facilitator Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 10

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Pragmatic’ Style Tester Likes Dislikes strategic / goals positive results / brief practical efficiency tasks ET Workshop v. 1. 20 Test Management indecision vagueness time-wasting unproductive Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 11

Introduction Test Management and Techniques ET Planning and Documentation ET Styles The ‘Pragmatic’ style tester will… ET Management be good for setting and monitoring short/long term goals for the team be good at documenting factual ‘test reports’ remain positive through pressure be keen to adopt ‘Most Important Tests’ first principle be a strong driving force - ensure a task is done want to implement efficiency into the team be self-motivated and task oriented will make quick decisions enjoy challenging testing tasks ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 12

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Pioneer’ Style Tester Likes Dislikes new / ideas change openness results/efficiency involving others risks ET Workshop v. 1. 20 Test Management standards detail ‘norm’ paper-work Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 13

Introduction Test Management and Techniques ET Planning and Documentation The ‘Pioneer’ style tester will… ET Styles ET Management be good at ‘ad-hoc’ testing / bug hunting / errorguessing/ exploratory testing be good at challenging and improving things to make more efficient and effective enjoy “GUI” type testing/lateral tester have good ideas be good at brainstorming Test Conditions share ideas about different ways to approach testing identify and take necessary risks when required have creative test ideas - how to find more faults ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 14

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Analysing’ Style Tester Likes Dislikes accuracy attention to detail proof standards reliable all alternatives ET Workshop v. 1. 20 Test Management new / change untested / risks brief / speed letting go Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 15

Introduction Test Management and Techniques ET Planning and Documentation ET Styles The ‘Analysing’ style tester will… ET Management be good at defining and documenting test cases be good at producing test standards and procedures analyse problems and finding root cause produce work which is accurate and complete enjoy logical tests scenarios provide proof when faults are found document thorough test reports complete work regardless of what it takes challenge requirements ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 16

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Facilitating’ Style Tester Dislikes Likes networking positive team oriented consensus / sharing building bridges status quo ET Workshop v. 1. 20 Test Management pressure / deadlines confrontation isolation dictated Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 17

Introduction Test Management and Techniques ET Planning and Documentation The ‘Facilitating’ style tester will… ET Styles ET Management be good in a RAD environment or a ‘buddy’ test team often ask opinion before raising issues be good at documentation co-operate well with other departments often see the ‘other side’ be good at defusing ‘us’ v ‘them’ syndrome be popular make things happen - eventually! will provide support in testing to other team members ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 18

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Tester Style - patterns usually operate within a certain boundary what if you are on the line/centre flexible between styles can be difficult to manage opposites repel maybe reasons for team tension! analysts & pragmatists tend towards ‘Tasks’ facilitators & pioneers tend towards ‘People’ ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 19

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The Ideal Test Team… A MIXTURE IS THE BEST BUT IT DEPENDS. . . ET Workshop v. 1. 20 Test Management Copyright © 2002 Lloyd Roden, Grove Consultants (www. grove. co. uk) © 2002 Amland Consulting 20

1. 2. 3. 4. 5. Exploratory Test Management 5. 1 Exploratory Test Team Management 5. 2 Risk-Based Test Management 5. 3 Session Based Test Management ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 5 -21

1. 2. 3. 4. 5. ”Ordinary” Project Plan including Testing Plan: Specification Design Coding Test In the Real World: Specification ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting Test 22

1. 2. 3. 4. ”Incremental” Project Plan with Testing 5. Plan: Spec. Design Coding Test In the Real World: Spec. Design Coding Spec. Test? Design Spec. ET Workshop v. 1. 20 Test Management Coding Design © 2002 Amland Consulting Test? Coding Test. . . 23

1. 2. 3. 4. 5. Why Risk Based Testing? It’s about controlling: Time Resources New Technology Lack of knowledge Lack of experience Quality Demands Scope ? ? ? ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 24

1. 2. 3. What is Risk? 4. 5. “A risk is an unwanted event that has negative consequences. ” Shari Lawrence Pfleeger (2000) In other words, a Shari risk is a problem waiting to happen. © 2002 Amland Consulting ET Workshop v. 1. 20 Test Management 25

1. 2. 3. 4. 5. What is Risk Management? Plans to avoid these unwanted events or, if they are inevitable, minimize their negative consequences. Shari Lawrence Pfleeger ET Workshop v. 1. 20 Test Management Shari Lawrence Pfleeger (2000) © 2002 Amland Consulting 26

1. 2. 3. Problems vs. Risk 4. 5. Problem (Issue): Something that has or will happen Risk Something that might happen (in the future) ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 27

1. 2. 3. Types of Risk 4. 5. Project / Process Risk Resourcing Planning Contracts Etc. Business / Product Risk Stability Performance Quality / Errors (Quality Risk = Potential Errors) Etc. ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 28

1. 2. 3. 4. 5. Business Risk – Why is it so difficult? (1) Flaw in the Pentium chip (1994) 6 million PCs sold, potential USD 300 loss per chip Intel’s risk impact USD 1. 8 billion Intel: ”Average” computer user: wrong answer every 27. 000 years ”Heavy user”: every 270 years Conclusion: The flaw is not meaningful to most users IBM: ”Average” computer user: a problem every 24 days A large company (500 PCs): 20 problems per day! Conclusion: stopped selling Pentium PCs IBM’s assessment 400. 000 times worse than Intel’s! ET Workshop v. 1. 20 Test Management Pfleeger (2000) © 2002 Amland Consulting 29

1. 2. 3. 4. 5. Business Risk – Why is it so difficult? (2) Evaluating the risk of an accident at a small ammonia storage plant (1988 to 1990) 11 EU countries (national teams) + several private firms The national teams’ assessment varied by a factor of 25. 000, reaching wildly different conclusions. ET Workshop v. 1. 20 Test Management Pfleeger (2000) © 2002 Amland Consulting 30

1. 2. 3. 4. 5. How to identify risk? 1. A loss associated with the event 2. The likelihood that the event will occur 3. The degree to which we can change the outcome ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 31

1. 2. 3. 4. 5. What is Risk-Based Testing? ”Risk-based testing carries at least two major meanings: ” Risk-Based Test Management To determine what things to test next (prioritisation) Doing Risk analysis for the Purpose of Finding Errors ET Workshop v. 1. 20 Test Management Lessons Learned in Software Testing, Cem Kaner et al 2001 b © 2002 Amland Consulting 32

1. 2. 3. 4. 5. RBT: The purpose of finding errors Make a prioritised list of risks. Perform testing that explores each risk. As risks evaporate and new ones emerge, adjust your test effort to stay focused on the current crop. ET Workshop v. 1. 20 Test Management James Bach 1999 a © 2002 Amland Consulting 33

1. 2. 3. 4. 5. Risk-Based Test Management 1. Define all requirements to be tested 2. Based on risk assessment – prioritise the requirements 3. Plan and define tests according to requirement prioritisation (coverage to be defined in test plan) 4. Execute test according to prioritisation and acceptance criteria (as defined in the test plan) ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 34

1. 2. 3. 4. 5. Risk-Based Test Management Statistical Risk Analysis (Hans Schaefer) Failure Mode and Effect Analysis (FMEA) Software Reliability engineering, John D. Musa Risk based test strategy (TMap, Iquip. nl)) Decision Theory (Bayesian Belief Nets) Rational Unified Process (RUP). . . and many, many more ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 35

1. RBT Management - Statistic Risk Analysis – Hans Schaefer 2. 3. 4. 5. Statistical Risk Analysis – Theory Equation: Re(f) - Risk Exposure of function f P(f) - Probability of a fault in function f C(f) - Cost related to a fault in function f ET Workshop v. 1. 20 Test Management Schaefer 1998, Amland 1999 © 2002 Amland Consulting 36

RBT Management - Statistic Risk Analysis - Hans Schaefer Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Statistical Risk Analysis – Matrix Weighted ET Workshop v. 1. 20 Test Management Schaefer 1998, Amland 1999 © 2002 Amland Consulting 37

Introduction Test Management and Techniques ET Planning and Documentation ET Styles Statistical Risk Based Testing ET Management Plan: Identify Elements to be Tested Logical or physical Functions, Modules etc. Identify Risk Indicators What is important to predict the probability of faults? Identify Cost (consequence) of faults Identify Critical Elements I. e. functions, tasks, activities etc. based on Risk Analysis (Indicators and Cost) Execute Improve the Test Process and Organization: Schedule and Track ET Workshop v. 1. 20 Test Management Schaefer 1998, Amland 1999 © 2002 Amland Consulting 38

RBT Management - Statistic Risk Analysis Introduction Test Management and Techniques ET Planning and Documentation ET Styles Example – Statistical Risk Analysis Matrix ET Management Cost * C(s) C(c) Avrg. Probability New Func. 5 Desgn Qual. Size = Re Com- Weigh. Risk plexity Sum Exposure 5 1 3 Interest Calc. 3 3 3 2 3 37 111 Close Account 1 3 2 2 3 31 62 Customer Profitablty 2 1 1, 5 3 3 2 3 41 61, 5 Other Probability Factors might include: Function Points, Frequency of Use etc. ET Workshop v. 1. 20 Test Management Schaefer 1998, Amland 1999 © 2002 Amland Consulting 39

Introduction Test Management and Techniques Risk Based Testing - Reporting ET Planning and Documentation ET Styles High Low Medium Probability ET Management Low Medium High Consequence ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 40

1. 2. 3. 4. 5. Exploratory Test Management 5. 1 Exploratory Test Team Management 5. 2 Risk-Based Test Management 5. 3 Session Based Test Management ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 5 -41

1. 2. Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing 3. 4. 5. Introducing the Test Session 1) Charter 2) Time Box 3) Reviewable Result 4) Debriefing ET Workshop v. 1. 20 Test Management vs. From Rapid Software Testing, copyright © 1996 -2002 James Bach © 2002 Amland Consulting 42

1. 2. Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing 3. 4. 5. Charter – summary… “Architecting the Charters” i. e. Test Planning Brief information / guidelines on: Mission: Why do we test this? What should be tested? How to test (approach)? What problems to look for? Might include guidelines on: Tools to use Specific Test Techniques or tactics to use What risks are involved Documents to examine Desired output from the testing ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 43

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing Time Box: Focused test effort of fixed duration Short: 60 minutes (+-15) Normal: 90 minutes (+-15) Long: 120 minutes (+-15) Brief enough for accurate reporting. Brief enough to allow flexible scheduling. Brief enough to allow course correction. Long enough to get solid testing done. Long enough for efficient debriefings. Beware of overly precise timing. ET Workshop v. 1. 20 Test Management From Rapid Software Testing, copyright © 1996 -2002 James Bach © 2002 Amland Consulting 44

Introduction Test Management and Techniques Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing ET Planning and Documentation ET Styles Reviewable Result: The session sheet ET Management Charter #AREAS Start Time Tester Name(s) Breakdown Test Notes Bugs #BUG Issues #ISSUE #DURATION #TEST DESIGN AND EXECUTION #BUG INVESTIGATION AND REPORTING #SESSION SETUP #CHARTER/OPPORTUNITY Data Files ET Workshop v. 1. 20 Test Management CHARTER -----------------------Analyze Map. Maker’s View menu functionality and report on areas of potential risk. #AREAS OS | Windows 2000 Menu | View Strategy | Function Testing Strategy | Functional Analysis START -----------------------5/30/00 03: 20 pm TESTER -----------------------Jonathan Bach TASK BREAKDOWN ----------------------- #DURATION short #TEST DESIGN AND EXECUTION 65 #BUG INVESTIGATION AND REPORTING 25 #SESSION SETUP 20 From Rapid Software Testing, copyright © 1996 -2002 James Bach © 2002 Amland Consulting 45

Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing: Measurement begins with observation The manager reviews session sheet to assure that he understands it and that it follows the protocol. The tester answers any questions. Session metrics are checked. Charter may be adjusted. Session may be extended. New sessions may be chartered. Coaching / Mentoring happens. ET Workshop v. 1. 20 Test Management From Rapid Software Testing, copyright © 1996 -2002 James Bach © 2002 Amland Consulting 46

1. 2. 3. 4. 5. Exploratory Test Management Summary Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles • Exploratory Test Team Management • Risk-Based Test Management • Session Based Test Management ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 47

Exercise 5 Based on available information; Discuss how you would manage and track a test team working on testing Star. Office. What would your ideal test team look like? What management strategy would you use? Why? Outline a risk matrix Select a few areas or functions Identify probability indicators How to define Consequence of an error? ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 48

Summary (1) Strengths: Customer-focused, risk-focused Takes advantages of each tester’s strengths Responsive to changing circumstances Well managed, it avoids duplicative analysis and testing High Bug find rates Blind Spots: The less we know, the more we risk missing Limited by each tester’s weaknesses (can be mitigated with careful management) This is skilled work, juniors aren’t very good at it ET Workshop v. 1. 20 Test Management Copyright © 1996 – 2002 Cem Kaner © 2002 Amland Consulting 49

Summary (2) Exploratory testing in pairs, is a very effective test approach Use it as a complementary testing approach (it depends…) Skilled testers can become very good exploratory testers. Novice testers need mentoring to do exploratory testing. ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 50

Learn more… Test Training www. testingeducation. org Exploratory Testing: Rapid Software Testing, by James Bach, www. satisfice. com james@satisfice. com Black Box Software Testing, by Cem Kaner, www. kaner. com kaner@kaner. com Amland Consulting, www. amland. no, stale@amland. no People Issues, test techniques, inspections: Grove Consultants, www. grove. co. uk + many, many more ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 51

Presentation - Summary Introduction: Exploratory Testing and Risk, Why and when to use Exploratory Testing. 1. Introduction 2. Test Management and Techniques Test Management, Test Techniques and Test Process 3. ET Planning, Exec. and Documentation ET Planning, Documentation and Execution. How to do Exploratory Testing 4. ET Styles 5. ET Management ET Workshop v. 1. 20 Test Management Exploratory Test Styles How to do Exploratory Testing, continued Exploratory Test Management How to Manage Exploratory Testing and Teams © 2002 Amland Consulting 52

References/Bibliography (1) Amland, S. , 1999. Risk Based Testing and Metrics, Euro. STAR '99 Preceding, Barcelona, Spain, http: //www. amland. no/articles Amland, S. , 2000. ”Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study”, Journal of Systems and Software, Volume 53, Issue 3, 15. 9. 2000, Elsevier publishing, UK. Argus, C. & Johnson, B. , Ad Hoc Software Testing; Exploring the Controversy of Unstructured Testing, http: //www. testingcraft. com/ad_hoc_testing. pdf Bach, J. , 1999 a. Risk-Based Testing. How to conduct heuristic risk analysis, Software Testing & Quality Engineering Magazine, November/December 1999, vol. 1, issue 6, http: //www. stqemagazine. com. See also satisfice. com Bach, J. , 1999 b. What is Exploratory Testing? www. satisfice. com Bach, Jonathan, 2000, Session-Based Test Management, STQE Magazine, 11/00, www. satisfice. com Beizer, Boris, Software Testing Techniques, Van Nostrand Reinhold. NY. 1990. Beizer, Boris, 1995, Black Box Testing, Wiley ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 53

References/Bibliography (2) Bereza-Jarocinski, B. , 2000, Test and Decision Theory, Enea Data AB, bogb@enea. se, +46 50 709 714 293 (www. bbj. com. pl) Black, Rex, 1999. Managing the Testing Process, Microsoft Press. Buwalda, Hans, Janssen, Dennis, and Pinkster, Iris, 2001. Integrated Test Design & Automation Using The Test. Frame Method, Addison Wesley, Copeland, Lee, Exploratory Planning, Sep. 3, 2001, Sticky. Minds. com Collard, Ross, 2002, to be published, rcollar@attglobal. net. Fenton, N. E. & Pfleeger, S. L. , 1997. Software Metrics, a rigorous & practical approach, 2 nd edition, International Thomson Computer Press. Fewster, M. , Graham, D. , 1999, Software Test Automation: Effective Use of Text Execution Tools, Addison-Wesley Gerrard, P. , 1999, Risk-Based Testing, TEST Congress, London, UK, www. evolutif. co. uk Gilb, T. , Principles of Software Engineering Management, Addison-Wesley, Wokingham ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 54

References/Bibliography (3) Kaner, C. , Falk, Nguyen, 1999, Testing Computer Software, John Wiley & Sons Kaner, C. , 2000. Rethinking Software Metrics, Evaluating measurement schemes, Software Testing & Quality Engineering Magazine, March/April 2000, vol. 2, issue 2. Kaner, C. , 2001 a, Black-Box Software Testing, training at Satisfice July 2001. Kaner, C. , Bach, J. , Pettichord, B. , 2001 b, Lessons Learned in Software Testing, John Wiley & Sons; http: //www. testinglessons. com/, ISBN: 0471081124 Kaner, C. , Bach, J. 2001 c, Exploratory Testing in Pairs, presentation at STAREast, Orlando, FL, www. kaner. com Karolak, Dale Walter, “Software Engineering Risk Management”, IEEE Computer Society Press, 1996. Keefer, G. , 2002. Extreme Programming Considered Harmful for Reliable Software Development, www. avoca-vsm. com Keith, Geordie, 2002, “All Hands on Deck: How we used our whole company to test”, STQE Magazine, vol. 4, issue 4, July / August 2002. Lyndsay, James and Eeden, Niel van, 2002, “Adventures in Session-Based Testing”, paper at Sticky. Minds. com presented at Euro. STAR 2002 ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 55

References/Bibliography (4) Marick, Brian, 1995, The Craft of Software Testing, Prentice Hall PTR, ISBN 0 -13 -177411 -5 Myers, Glenford, 1979. The Art of Software Testing, John Wiley & Sons Neumann, P. G. , Computer-Related Risks, Addison-Wesley/ACM Press, ISBN 0201 -55805 -X, 1995, http: //www. csl. sri. com/users/neumann/ Nguyen, 2000, Testing Applications on the WEB, John Wiley & Sons Ottevanger, Dr. I. B. , 1999. A Risk-Based Test Strategy, Presented at STARWest 1999, http: //www. sqe. com/startwest. Part of TMAP, IQUIP Informatica B. V. PO Box 263, 1110 AG Diemen, The Netherlands (www. iquip. nl). Pfleeger, S. L. , 2000. Risky Business: what we have yet to learn about software risk management, Journal of Systems and Software, issue 11, 2000, Elsevier Publishing, UK. Robinson, H. , Microsoft, Exploratory Modeling, http: //www. testingcraft. com/exploratory-robinson. html, home page: http: //www. geocities. com/model_based_testing/ Roden, Lloyd, 2001. How to build and lead an effective test team, tutorial Euro. STAR 2001, Grove Consultants, www. grove. co. uk, Lloyd@grove. co. uk ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 56

References/Bibliography (5) Schaefer, H. , 1998. Surviving under time and budget pressure, keynote STAR West '98, schaefer@c 2 i. net, http: //www. sqe. com/starwest, http: //home. c 2 i. net/schaefer/ Våga, J. , Amland, S. , 2002. High Speed Web Testing, contribution to the book ” Software Quality and Testing in Internet Times”, Editor: Dirk Meyerhoff et al, Springer-Verlag, Berlin Heidelberg, Germany 2002, ISBN: 3 -540 -42632 -9 Whittaker, J. and Jorgensen, A. , 1999. Why software fails. ACM Software Engineering Notes, July. http: //se. fit. edu/papers/ and www. stickyminds. com Whittaker, J. and Jorgensen A. , 2002, “How to Break Software: A Practical Guide to Testing”, Addison-Wesley, ISBN: 0 -201 -79619 -8, presented at Euro. STAR 2000 Øvstedal, E. Ø. and Stålhane, T. , 1992. A goal oriented approach to software testing, Reliability Engineering and System Safety. Elsevier Science Publishers Ltd. , UK. ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 57

Links www. stickyminds. com www. satisfice. com www. kaner. com www. testingeducation. org www. pettichord. com www. amland. no home. c 2 i. net/schaefer/ www. bbj. com. pl http: //www. stqemagazine. com http: //www. testinglessons. com/ http: //www. context-driven-testing. com/ www. grove. co. uk ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 58

1. 2. 3. Contact Details 4. 5. Ståle Amland, Amland Consulting Hulda Garborgsv. 2, N-4020 STAVANGER Norway Phone: +47 905 28 930 Fax: +47 51 58 55 24 e-mail: stale@amland. no WEB: www. amland. no ET Workshop v. 1. 20 Test Management © 2002 Amland Consulting 59