Jama Options for Complex Testing variations 2013 Jama

  • Slides: 13
Download presentation
Jama Options for Complex Testing variations 2013 © Jama Software, Inc Confidential www. jamasoftware.

Jama Options for Complex Testing variations 2013 © Jama Software, Inc Confidential www. jamasoftware. com | 1. 800. 679. 3058

Test Run vs Test Case Status Test Plan Test Case Test Case Test Run

Test Run vs Test Case Status Test Plan Test Case Test Case Test Run Cycle 1 Blocked Test Run Cycle 2 Failed Test Run Cycle 3 Passed Confidential 2013 © Jama Software, Inc

Test Run vs Test Case Status Test Plan Test Case Test Case Test Run

Test Run vs Test Case Status Test Plan Test Case Test Case Test Run Test Case status shows is the most recent Test Run status Cycle 1 Blocked Test Run Cycle 2 Failed Test Run Cycle 3 Passed Confidential 2013 © Jama Software, Inc

Complex Testing Scenario Operating Systems • Mac OSX • Windows 7 • Windows 8

Complex Testing Scenario Operating Systems • Mac OSX • Windows 7 • Windows 8 Browser • Internet Explorer (MS only) • Safari (Mac only) • Chrome • Firefox Browser Version • Internet Explorer 11, 10, 9 • Safari 7, 6, 5 • Chrome 33, 32, 31 • Firefox 28, 27, 26 Each test needs to be run against each variation above This equals 27 different variations for each TC Confidential 2013 © Jama Software, Inc

Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test

Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test Plan per Variation Confidential 2013 © Jama Software, Inc

Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test

Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test Plan per Variation PRO • Report on the true % of passing vs failing test cases • Multiple cycles and test runs per variation • Tweak test case per variation • See the real complexity of your testing in Coverage view • Filter TCs based on variation and see roll-up stats • Can organize test cases by variation AND functional area (e. g. Login tests, Email tests, etc) CON • Longer setup effort: author/import test cases once - then reuse. • Each time a new test cycle begins, Test Manager must create a new cycle per variation (27 in this sample) • For any new test cases added must reuse into each set and then add to 27 test plans • Users must constantly switch between Test Plans Confidential 2013 © Jama Software, Inc

Option 2 – Unique TC per Variation Single test plan - use Folders and

Option 2 – Unique TC per Variation Single test plan - use Folders and Test Groups for Variation Confidential 2013 © Jama Software, Inc

Option 2 – Unique TC per Variation Single test plan - use Folders and

Option 2 – Unique TC per Variation Single test plan - use Folders and Test Groups for Variation PRO • Report on the true % of passing vs failing test cases • Multiple cycles and test runs per TC variation • Tweak test case per variation • See the real complexity of your testing in Coverage view • Filter TCs based on variation and see roll-up stats • Same authoring/import effort as Opt 1 but less Test Execution setup time • Organize everything in a Single Test Plan – Managers can add a new cycle for all variations in one process – Testers have a single place to see all test variations – Can still assign by variation (e. g. Becky does all Mac OS Safari 7 tests) CON • Cannot organize test cases by functional area, only by variation • For any new test cases added must reuse into each folder Confidential 2013 © Jama Software, Inc

Option 3 – Generic TC, Unique Test Plans Confidential 2013 © Jama Software, Inc

Option 3 – Generic TC, Unique Test Plans Confidential 2013 © Jama Software, Inc

Option 3 – Generic TC, Unique Test Plans PRO • Less setup time to

Option 3 – Generic TC, Unique Test Plans PRO • Less setup time to author/import test cases • Report on the true % of passing vs. failing test runs • Organize and assign testing by variation (e. g. Becky - Mac OSX Safari 7 tests) CON • Same Test Execution setup effort as Option 1 – Test Manager must create 27 new cycles, one per variation – Test Manager must update 27 cycles if a new test case is added • • • Cannot see total complexity in Coverage Explorer view No TC variations per OS/Browser Different test run results may conflict the roll-up test case status. Example: – Test in Chrome, Fails. – Test in IE, Passes. – Overall Test Case Status will be Passed Confidential 2013 © Jama Software, Inc

Option 4 – Generic TC and Runs Confidential 2013 © Jama Software, Inc

Option 4 – Generic TC and Runs Confidential 2013 © Jama Software, Inc

Option 4 – Generic TC and Runs PRO • Quick setup time • Simple

Option 4 – Generic TC and Runs PRO • Quick setup time • Simple management when updating or adding a Test Case (just add once) • Defects are the source of what is failing on which OS/Browser ? Organize and assign by test, not by variation (e. g. Becky does single test for ALL OS and Browser versions) CON • When a test fails, hard show which variations passed vs failed. • Cannot show true metrics of % passed vs % failed • Test Cases do not allow for variations depending on the browser, os, etc • Cannot see total complexity in Coverage Explorer view • Bug relationships don’t go away upon new Test Cycles so bugs from old cycles will still show in Coverage View Confidential 2013 © Jama Software, Inc

LOE Test Authoring • • • Add Import Update Test Planning • • Add

LOE Test Authoring • • • Add Import Update Test Planning • • Add Cycle New Release Assign Update Plans w/ new Test Cases Test Execution • • Execute Log Defects Test Result Reporting • % Pass/Fail • Traceability Option 1 Option 2 Option 3 Option 4 Unique TC Unique Test Plans Unique TC Single Plan, Unique Test Groups Generic TC Unique Test Plans Generic TC Generic Test Runs Medium Easy High Easy Medium High Easy Medium High Confidential 2013 © Jama Software, Inc