March 30 2004 Exploratory Testing Testing Approaches n

March 30, 2004

Exploratory Testing

Testing Approaches n n Analytical Information-driven Intuitive Exploratory n n n Design the tests and test concurrently Learn the system and test it as you go Structure creative testing Think while testing Risk-based Analogous to Extreme Programming

Exploratory Testing Tasks n Explore n n n Design Tests n n n Which elements Speculate on possible quality problems Execute Tests n n n Elements of the product How the product should work Observe behavior Evaluate against expectations All with test design techniques best suited for the product

Exploratory Testing Practice n n Used to probe for weak areas Especially useful when n n Weak specifications and requirements Little domain knowledge Time pressures Less appropriate when n Well-defined test requirements Strong need for regression testing Repeatable over releases n n Cost of maintenance Few new test cases

Planning n Decompose the product into elements n n n Quality criteria n n Areas of function that you can test in 1 -2 days Define charters n Decomposition into units that can be tested in 1 -2 hours Capability, reliability, usability, performance, installability, compatibility, … Select test techniques

Charter n n Provides clear mission of why this test Suggests what and how it should be tested, as well as problems to look for Not a detailed plan, but should be as specific as possible Might include risks, documents and desired output

Maintenance

Cost of Maintenance n Estimates of percentage of total life cycle cost: 40% - 90%

Software Maintenance Types n n Adaptive maintenance: changes needed as a consequence of operation system, hardware, or DBMS changes Corrective maintenance: the identification and removal of faults in the software Perfective maintenance: changes required as a result of user requests Preventive maintenance: changes made to software to make it more maintainable

Why adaptation? n Lehman’s Law: if a program doesn’t adapt, it becomes increasingly useless n n Example: programs that didn’t adapt to the web The majority of maintenance is concerned with evolution deriving from user requested changes.

Lehman’s Second Law n As an evolving program changes, its structure tends to become more complex. n n Extra resources must be devoted to preserving the semantic and simplifying the structure. For most software, nothing has been done about it, so changes are increasingly more expensive and difficult.

How is Maintenance Initiated? n n Most common: user request Types of requests n Bug reports n n n Accepted Working as designed Documentation fix Feature requests New environments

Steps for handling a change n n n n Understand the problem Design the changes Analyze impact Implement changes Update documentation Regression test Release

Patches n What is a patch? n n When should it be used? n n Quick fix that doesn’t go through the full process Error that is preventing use of the system Problems with use n n Multiple patches can be order dependent Users can barely track which ones have been applied Code version explosion Permanent fix may or may not be compatible

Problems of Maintenance n Organizational n n n Process n n Alignment with objectives Cost benefit analysis Impact Documentation Regression testing Technical n Building software that is maintainable

Different Objectives over Time n n n At release: bug-free Six months later: competitive or competition-leading features Two years later: reduce maintenance cost

Outsourcing of Maintenance n Outsourcing: inside or outside the company n Advantages n n Less expensive Uses less skilled people Frees developers for the next release Disadvantages n n n Changes can be slower May still require developer help Code can become less maintainable

Cost Benefit Analysis (Risk Analysis) n n n Will this problem reduce the number of programs that I sell? Will this problem impact future sales/ How many people will it affect? How important are the customers it will affect? Is it a “show stopper” or an annoyance?

Process Issues n n n Impact analysis Updating documentation Reengineering Legacy systems Reverse engineering

Impact analysis n n n How much of the system will it impact? How complex are the changes? Heuristics: n n Percentage of requirements that are impacted Where was the defect injected (requirements, design, code)

Updating documentation n A requirement defect can impact n n n Requirements Use cases Architecture Design Code Where in the process do you update? n No matter when, there is a period of inconsistent documents!

Reengineering n Code gets messy over time n n At some point, quality suffers n n n Extreme programming re-factoring Changes slow Fixes introducing errors Need to invest in the code! n n n Rules as to when to rewrite a module Abstractions: variables -> methods Harder: when is REDESIGN needed?

Legacy Systems n Existing systems that are still useful n May not want to invest in enhancements n n Future functions will use new process May not be able to easily modify n n n Unsupported language or libraries Lack of skills No source code available!

Handling Legacy Systems n Incorporation n n Business as usual Encapsulation n n Accessed from new system Adapters n n Wrapper around the legacy system Adapters in new system

Technical Issues n n Building maintainable software What features have we talked about in code and documentation that help?

Technical Issues n n Building maintainable software What features have we talked about in code and documentation that help? n Well documented code n n Names, headers, style, … All the documentation n Architecture, design documentation, use cases, requirements, …

Today’s Tip n You need to make sure that when you turn in your final project, everything is consistent. n n Contract can’t be changed Other documents

For your enjoyment… n “Yes, the world needs bankers and artists and even professional athletes. They, among countless others, create the breadth of society and culture. But if you want tomorrow to come – if you want to spawn entire economic sectors that didn’t exist yesterday – those are not the people you turn to. It’s technologists who create that kind of future. ” (Neil de. Grasse Tyson, Director of Hayden Planetarium)