Department of Computer Science Regression Testing copyright A

Regression testing • Testing changed software – Retest all – Selective regression testing •

Retest all • Assumption: – Changes could have introduced errors anywhere in the code

Selective regression testing • Impact analysis • Only code impacted by change needs to

Regeneration strategy • Often for defect fixes when unsure whether defect is really fixed

Regression testing process • • Identify changes Determine which tests will remain valid: T’

Regression testing process (2) • T’: – test whether modifications have broken anything –

Classification of tests in T • Reusable: – does not test software modification –

Practical regression testing Approach Quality Change Impact Schedule Full new Low Key Code Extensive

Approaches • Full new test cycle – Redesign entire test plan – Rebuild whole

Approaches (2) • Expanded scope of regression testing – Reuse part of test plan

Attributes of regression tests • Safe: – All tests that could possibly exhibit different

Attributes of regression tests (2) • Efficiency: cost of identifying a reduced regression test

Regression test patterns • Retest changed code – Class, cluster, subsystem scope – Identify

Retest risky use cases • Suspicious use cases – Use cases that depend on

Retest within Firewall • Firewall: set of components whose test cases will be included

Slides: 17

Download presentation

Department of Computer Science Regression Testing copyright A. Andrews, 2003

Regression testing • Testing changed software – Retest all – Selective regression testing • Regression test suite • Types of changes – – Defect fixes Enhancements Adaptations Perfective maintenance copyright A. Andrews, 2003

Retest all • Assumption: – Changes could have introduced errors anywhere in the code – Expensive, prohibitive for large systems – Reuse existing test suite – Add new tests as needed – Remove obsolete tests (discrepancies between expected and actual output) copyright A. Andrews, 2003

Selective regression testing • Impact analysis • Only code impacted by change needs to be retested • Select tests that exercise such code • Add new tests if needed • Remove obsolete tests copyright A. Andrews, 2003

Regeneration strategy • Often for defect fixes when unsure whether defect is really fixed • Develop new tests – Probe around defect fix – Test affected functionality • Comes in retest all, and selective regression testing flavors copyright A. Andrews, 2003

Regression testing process • • Identify changes Determine which tests will remain valid: T’ Test modified software with T’ If T’ does not meet test criterion, generate new test cases T’’ • Execute modified software with T’’ copyright A. Andrews, 2003

Regression testing process (2) • T’: – test whether modifications have broken anything – Determined either by retest all, or selective strategy • T’’: – test whether modified code works • Use scope appropriate criteria copyright A. Andrews, 2003

Classification of tests in T • Reusable: – does not test software modification – Should produce same result – Need not be rerun • Retestable: – Tests software modification – Must be rerun • Obsolete: no longer applies copyright A. Andrews, 2003

Practical regression testing Approach Quality Change Impact Schedule Full new Low Key Code Extensive None Minimal High Isolated Localized Very tight Expanded Moderate Key Code Extensive Moderate Full reuse Low Series of fixes Extensive Moderate copyright A. Andrews, 2003

Approaches • Full new test cycle – Redesign entire test plan – Rebuild whole test suite • Minimum regression test – Reuse test plan as much as possible – Rerun minimum number of retestable test cases – Generate minimum number of new test cases for changes copyright A. Andrews, 2003

Approaches (2) • Expanded scope of regression testing – Reuse part of test plan – Rerun all retestable test cases – Generate new test cases on the full scope of changes • Full reuse of existing test suite – Reuse test plan and test suite copyright A. Andrews, 2003

Attributes of regression tests • Safe: – All tests that could possibly exhibit different outputs on the modified software – Is 100% inclusive • Precision: – Percentage of tests in a reduced test suite that cannot reveal regression faults, and – Are not selected for the reduced test suite copyright A. Andrews, 2003

Attributes of regression tests (2) • Efficiency: cost of identifying a reduced regression test suite • Generality: range of application for the selection strategy copyright A. Andrews, 2003

Regression test patterns • Retest changed code – Class, cluster, subsystem scope – Identify segment coverage by test case – Use version control to identify changed segments – Select tests covering changed/deleted segments – Develop new tests for new segments • Safe copyright A. Andrews, 2003

Retest by profile • • Assumes operational profile Unsafe Include critical use cases Include other use cases by frequency in operational profile • No impact analysis • Better: determine impact on use case copyright A. Andrews, 2003

Retest risky use cases • Suspicious use cases – Use cases that depend on components, objects, middleware, resources that are • • • Unstable, unproven Have not been shown to work together before Implement complex business rules Are complex Were fault-prone during development • Critical use cases • Unsafe copyright A. Andrews, 2003

Retest within Firewall • Firewall: set of components whose test cases will be included in a regression test • External interface, contract change • Internal implementation change copyright A. Andrews, 2003