Acceptance Testing for ROME Pete Castle Test Quality

Acceptance Testing for ROME Pete Castle Test & Quality Manager

Agenda • • • What is software testing/ Who does it? Why software testing is important Some fundamentals of testing Test Plans & Scripts Sample Testing Techniques

What is software Testing? • “Software testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test” Professor Cem Kaner - Director of Florida Tech's Center for Software Testing Education & Research • Empirical - derived from experiment, experience, and observation Technical - Having special skill or practical knowledge Investigation - A detailed inquiry or systematic examination • •

Why testing is Important • • • All Software has defects (bugs) All software products are ‘prototypes’ (in my view) Software products are getting larger and more complicated - Vista 40% larger than XP @ over 50 million LOC Software Engineering is not as mature as other disciplines e. g. Civil Engineering Software is written by people – people make mistakes Software testing looks to find the most important defects as early as possible – increasing confidence that the software meets specification

Who’s involved in testing? • • Requirements Analysts – Inspections, Peer Reviews Developers – Code Inspection, Unit Testing Testers – System & Integration Testing Trainers – Training materials production Users – User Acceptance Testing Project Managers – Scheduling, Resourcing, Risks, Issues, Defect Stats Everybody is responsible for quality - NASA

Fundamentals of Software Testing Plan • Specify Execute Record Check Software testing needs planning, tests need specifying, once executed they need results recording, and post completion should be easily auditable

The importance of a planned approach Important to map out a strategy that will give the greatest level of confidence in the product • ‘Ad hoc’ testing may find errors, but may not be cost effective • Testing should focus on areas where defects are most likely • All testing should have a reason • • • Question “Is a test that doesn’t find an error a good test or not? ” Essential to plan what needs to be done and then itemise how it is to be achieved.

Testing Mantra • Mantra - Spiritual conduit, words or vibrations that instil concentration in the devotee. • Test as early as possible • Gather as much knowledge of the application under test as possible • Look for vulnerabilities • Build ‘Bug Taxonomies’ (Classification) • Use Quicktests (and publicise the fact)

Testing Mantra • You can always think of another test – but should you? • • • Concept of ‘Good enough Testing’ Practicality over dogma Everybody has responsibility for ‘shipping’ the product • Record all tests/defects/issues/recommendations • Testers are not the sole arbiters of quality • • Testing only shows problems exist – not their absence Never, ever make it personal • • Defects are issues with products and process not people Good working relationship is essential for good products

Document Hierarchy - Test Plan Test Policy Test Strategy Project Test Plan Unit Phase Integration Phase Acceptance Phase

What is a Test Plan - 1 Ø Test plan is l l Ø tool to help plan the testing activity product to inform others of test process Includes l l l Document control Objectives Scope Approach – Schedule, Priorities, Deliverables, Resources, Responsibilities Risks/Contingences Sign-off/Approval

What is a Test Plan - 2 • • Produced by Test Lead/Project Manager Published to Project/Programme Not constrained by format – living document Enough information to be used by anyone to test the product

Rome Test Plan Ø Ready for review Ø Written by Tim Wells

Document Hierarchy -Test Scripts Test Policy Test Strategy Project Test Plan Unit Phase Integration Phase Acceptance Phase

Test Scripts Test Script - Is a collection of test cases for the software under test (manual or automated) • Test Case - A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement. •

Pre-conditions/Information • • • Browsers – IE, Firefox, Safari O/S – Linux, Windows Access Control – Logins, Roles Test Data requirements Date/Time considerations Other document references

Example Test Script - 1 • System Test of input of numeric month into data field Ref. Field/Button Action 001 Month Enter Data 002 Month 003 Input Expected Result Pass/Fail 0 Data rejected. Error Message 'Invalid Month' Fail Enter Data 1 Data Accepted, January Displayed Pass Month Enter Data 06 Data Accepted, June Displayed Pass 004 Month Enter Data 12 Data Accepted, December Displayed Pass 005 Month Enter Data 13 Data rejected. Error Message 'Invalid Month' Fail

Example Test Script - 2 Search Researcher Page Test Ref. Reqs Ref. Function Inputs Expected Result Actual Result Pass/ Fail 2. 001 REF 003 Search Researchers 1. Forenames = John 2. Surname = <Blank> 3. e. Mail = <Blank> All UCL researchers with forenames starting Pete displayed in alphabetic order, 23 records per page List comprises Name, Department, Occupation Type All data items hyperlinked 427 matches - paging working correctly, data displayed correctly and in reasonable time (5 secs) Pass 2. 002 REF 003 Search Researchers 1. Forenames = <Blank> 2. Surname = Smith 3. e. Mail = <Blank> All UCL researchers with surnames starting Smith displayed in alphabetic order, 23 records per page List comprises Name, Department, Occupation Type All data items hyperlinked 61 matches - paging working correctly, data displayed correctly and in reasonable time (5 secs) Pass

Example Test Script - 3 Type Scenario Test To test? New/ Change? Covered (Y/N) Pass / Fail Who Date tested 39 Admissions Basic Licence Admissions - Setup Y Y Y P SM 06/01/08 40 Basic Licence Admissions - Criterion Setup Y Y Y P SM 06/01/08 41 Basic Licence Admissions - Admissions Policy Y Y Y P SM 06/01/08 42 Basic Licence Admissions - ADT Import from EMS Y Y Y P SM 06/01/08 43 Basic Licence Admissions - ASL Export to EMS Y Y Y F SM 06/01/08 No.

Test Scripts Ø Readable Ø Accessible Ø Usable by anyone – standards Ø Can vary depending upon the type of testing being undertaken

Testing Techniques Quicktests • Negative Testing • Integration Testing •

Techniques 1 - Quick Tests • Quicktests - Investigation more important than Confirmation • A quicktest is a cheap test that has some value but requires little preparation, knowledge, or time to perform • Focus on common coding errors

Techniques 1 - Quick Tests • Things that have failed before – Defect data • • • Tab order (particularly when adding new functions) Addresses (BFPO, new Post Codes) Short cut keys • Boundaries – Ages, Dates, Values, Increments, Page Breaks • Interrupts, Duplications, Ordering of tasks • • • Generate Order before setup – no cost codes, cost centres, suppliers, budgets Exit/Interrupt before completion Consume resource (Dog Piling) • • Never close a window Never exit an option

Techniques 1 - Quick Tests • Force all error messages • • Informative messages - Spelling Debug information? • Capacity/Files – Files to fill available space, large files, empty files, incorrect format • Dependencies – e. g. same student many functions • • Integration Quicktest Comparisons – screens, data, reports • Tools e. g. Beyond Compare, Screen Capture, Redgate Toolset, In. Ctrl 3

Negative Testing the system using negative data – to generate exceptions that cause the software to fail • Going against knowledge of ‘How the system should work’ • For Example - Testing the password where it should be minimum of 8 characters - so test it using 7 and 9 characters • Emphasis on breaking not confirmation •

Negative Testing • • • Embedded single quote and other ‘special’ characters e. g. John’s Car, John & Erin, 99%, Straße (German Addresses) Required Data Input – Don’t Field Size – Shoe test (also Quick Test) Field ‘Types’ – Characters in numeric field Boundaries (Upper/Lower) – underage job applications, 101 lines on an order with a maximum of 100 lines Invalid dates e. g. 31/04/08 Addresses – BFPO, Hong Kong Addresses, ‘New Post Codes’ Web Session Testing – Access web page but not logged in Switch off during upgrade – what happens, does the application know there is a problem?

Integration Testing (In the large) • • • “Testing performed to expose faults in the interfaces and in the interaction between integrated components and products” Sue Myler – Integration Team Lead Usually scenario based rather than low level test cases Relies upon testers having system knowledge & testing expertise – ability to think ‘outside of the box’, develop new tests during testing Relies on successful unit, integration in the small and system testing Can mimic business processes

Integration Testing (In the large) Integration Test Cases • 3 Applicants • 1 applies for 1 post • 1 applies for 2 posts - also applies for the same post twice (by accident) • 1 applies for 3 posts • do their records appear correctly across ROME • Delete a Vacancy – what happens to that applicant records?

Integration Testing (In the large) Short list applicant for post he entered twice, deleting one application • Invite for interview but candidate withdraws • Candidate then re-applies • Data exported, ROME updated, then re-exported, does data appear correctly in target application •

Test Scenarios Type Scenario Test To test? New/ Change? Covered (Y/N) Pass/ Fail Who Date tested 39 Admissions Basic Licence Admissions - Setup Y Y Y P SM 06/01/08 40 Basic Licence Admissions - Criterion Setup Y Y Y P SM 06/01/08 41 Basic Licence Admissions - Admissions Policy Y Y Y P SM 06/01/08 42 Basic Licence Admissions - ADT Import from EMS Y Y Y P SM 06/01/08 43 Basic Licence Admissions - ASL Export to EMS Y Y Y F SM 06/01/08 44 Basic Licence Admissions - ATF Import from EMS Y Y Y F SM 06/01/08 45 Basic Licence Admissions - ATF Re. Import from EMS with additions and amendments Y Y Y F SM 06/01/08 No.

Review • • • Software Testing is important for increasing confidence that the software meets specification To get the best results from testing certain fundamentals should be followed Testing is part of software development Different software testing techniques enhance our ability to test Many different types of software testing exist – which we can combine into single test cases/scenarios

• Test Example – Data Entry Screen Create Test cases to negatively test (break) the data entry screen Description Data/Validation Title 20 Chars, mandatory, pick list provided Forename 25 Chars, mandatory Surname 25 Chars, mandatory email Address 50 Chars, mandatory, validated Home Telephone Number 25 Chars

Next Steps Ø ROME - Kick off meeting l l l Testing required who/when Test Script Template Mantis - Issue/Defect Logging