Politehnica University of Timisoara Mobile Computing Sensors Network

Politehnica University of Timisoara Mobile Computing, Sensors Network and Embedded Systems Laboratory Embedded Systems Testing instructor: Assist. Prof. Razvan BOGDAN razvan. bogdan@cs. upt. ro

Administrative • Laboratories • 312, Vector Lab • CANoe, Vector tools and devices • Students counseling • Wednesday 16 -18. 30 in B 413 A/B 414 • Evaluation • Final mark: • Exam (30%) + lab (40%) + Online activity (30%) • Embedded Systems, Embedded Systems Testing, Ambient Intelligence, Ambient-Assisted Living, Complex Networks, Internet-of-Things, e-Learning, Big Data • razvan. bogdan@cs. upt. ro, B 413 A/B 414

Course objectives • Learning skills for planning tests • Applying software testing skills to your projects • Selecting appropriate testing techniques and objectives • Learning a common terminology • Learning practical tools for automotive testing

Textbook • Main textbook: • Andreas Spillner, Tilo Linz, Hans Schaefer, Software Testing Foundations, O'Reilly Media, 2014 • Rex Black, Erik van Veenendaal, Dorothy Graham, Foundations of Software Testing, 3 rd Edition, 2012, Cengage Learning -> check students resources • International Software Testing Qualification Board, The Certified Tester Foundation Level in Software Testing, 2011, http: //www. istqb. org/downloads/syllabi/foundation-levelsyllabus. html

Content 1. Fundamentals of testing • • • Why is testing necessary? What is testing? Testing principles Fundamental test process The psychology of testing 2. Testing Throughout the Software Life Cycle • • Software Development Models Test Levels Testing Types Maintenance Testing 3. Static Techniques • Static Techniques and the Test Process • Review Process • Static Analysis by Tools

Content 4. Testing Design Techniques • • • The test development process Categories of Test Design Techniques Specification-based or Black-box Techniques Structure-based or White-box Techniques Experience-based Techniques Choosing Test Techniques 5. Testing Techniques in Automotive • Transaction Flow Modeling, All Round-Trip Paths, Loop Testing, Data Flow Testing, Equivalence Partitioning, Boundary Value Analysis, Regression Testing, Negative Testing, Error Guessing, Error Handling Testing, Recovery Testing, Stress Testing, Load Testing 6. Test Management • Test Organization, Test Planning and Estimation, Test Progress Monitoring and Control, Configuration Management, Risk and Testing, Incident Management 7. Test Tools

Politehnica University of Timisoara Mobile Computing, Sensors Network and Embedded Systems Laboratory Embedded Systems Testing Fundamentals Of Testing instructor: Razvan BOGDAN

Outlines 1. Why is testing necessary? 2. What is testing? 3. Testing principles 4. Fundamental test process 5. The psychology of testing

Why is testing necessary? • A. The economic importance of software • The present machines and equipment largely rely on software • B. Software Quality • The quality of software has become the determining factor for the success of technical or commercial systems; such systems are to be found in different commercial products

Why is testing necessary? • Growing importance of Embedded Systems • As of January 2014: • • 90% of American adults have a cell phone 58% of American adults have a smartphone 32% of American adults own an e-reader 42% of American adults own a tablet computer • As of May 2013, 63% of adult cell owners use their phones to go online. • 34% of cell internet users go online mostly using their phones, and not using some other device such as a desktop or laptop computer

Why is testing necessary? • Testing is necessary because we all make mistakes. • Some of those mistakes are unimportant, but some of them are expensive or dangerous. • We need to check everything and anything we produce because humans make mistakes all the time

Why is testing necessary? • some mistakes come from bad assumptions and blind spots, so we might make the same mistakes when we check our own work as we made when we did it. • we may notice the flaws in what we have done. Ideally, we should get someone else to check our work - another person is more likely to spot the flaws. • => C. Testing for quality improvement • Testing and reviewing insure the importance of the quality of software products, but also the quality of the software development process itself.

Why is testing necessary? • Failure example 1: Ariane 5 Launch

Why is testing necessary? • Failure example 2: Patriot Missile

Why is testing necessary? • Failure example 3: USM Application System

Software systems context Testing Principle - Testing is context dependent Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site. • Not all software systems carry the same level of risk and not all problems have the same impact when they occur. • A risk is something that has not happened yet and it may never happen; it is a potential problem • When we discuss risks, we need to consider how likely it is that the problem would occur and the impact if it happens

Software systems context • For example: • whenever we cross the road, there is some risk that we'll be injured by a car. • The likeli-hood depends on factors such as how much traffic is on the road, whethere is a safe crossing place, how well we can see, and how fast we can cross. • The impact depends on how fast the car is going, whether we are wearing protective things, our age and our health. • The risk for a particular person can be worked out and therefore the best road-crossing strategy can be offered.

Causes of software defects • Human Error • A defect was introduced into the software code, the data or the configuration parameters • Causes of human errors: time pressure, complex code, complexity of infrastructure, changing technologies, and/or many system interactions • Environmental Conditions • Changes of environment conditions • Causes of negative environmental conditions: radiation, magnetism, electronic fields, and pollution cause faults in firmware or influence the execution of software by changing the hardware conditions

Causes of software defects

Causes of software defects • Error (IEEE 610) • A human action that produces an incorrect results; • Example: a programming error • Defect (also known as Fault or Bug) • A flaw in a component or system that can cause the component or system to fail to perform its required function • If executed, a defect can cause a failure • Example: an incorrect statement or data definition • Failure • The physical or functional manifestation of a defect. A defect, if encountered during execution, may cause a failure • Deviation of the software from its expected delivery or service [after Fenton] => Work of testers is to produce failures Defects cause failures

Causes of software defects

When do defects arise?

When do defects arise? • Unfortunately, requirements and design defects are not rare; assessments of thousands of projects have shown that defects introduced during requirements and design make up close to half of the total number of defects • Defects should be detected as early as possible in the process

What is the cost of defects? • the old English proverb 'a stitch in time saves nine'. This means that if you mend a tear in your sleeve now while it is small, it's easy to mend, but if you leave it, it will get worse and need more stitches to mend it.

What is the cost of defects?

Role of Testing in Software Development, Maintenance and Operations • Increasing software quality • Testing helps to furnish the software with the desired attributes, namely to remove defects leading to failures • Reduction of the risk of encountering errors • Appropriate test activities will reduce the risk that errors are encountered during software operation • Meeting obligations • Tests might be mandatory because of client’s or legal regulation as well as to meet industrial standards

Testing and quality • Definition Software: Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system. [IEEE 610] • Definition Software Quality: The totality of functionality and features of a software product that bear on its ability to satisfy stated or implied needs. [After ISO 9126] • Definition Quality: The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations. [After IEEE 610]

Testing and quality • Testing helps us to measure the quality of software in terms of the number of defects found, the tests run, and the system covered by the tests. • Projects aim to deliver software to specification. • For the project to deliver what the customer needs requires a correct specification. • Additionally, the delivered system must meet the specification. • This is known as validation ('is the right specification? ') and verification ('is the system correct to specification? ').

Testing and quality • According to ISO/EC 9126 software quality consists of: • Functional Quality-attributes • Functionality • Non-functional Quality-attributes (how is the software running) • • • Reliability Usability Efficiency Maintainability Portability • Types of Quality Assurance (QA) • Constructive activities to prevent defects, for example: through appropriate methods of software engineering • Analytical activities for finding defects, for example: through testing leading to correcting defects and preventing failures, therefore increasing the software quality

Testing and quality • Constructive quality assurance • Quality of process – Quality management • Defects that were made need not to be repeated => prevent defects

Testing and quality • Analytical quality assurance • • Quality of product – Verification and test procedure Defects should be detected as early as possible in the process Static testing – examination without executing the program Dynamic testing – includes executing the program

Test Goals • Gain knowledge about defects in the test object • Defects contained in the test objects must be detected and be described in such a way as to facilitate their correction • Proof of functionality • System functionality should be implemented as specified • Generating information • Before handing over a software system to the users, information about possible risks has to be provided. Gaining such information might be one of the test goals • Gaining confidence • Software that has been well tested is trusted to meet the expected functionality and to have a high quality level.

How much testing is enough? • Exit criteria • Not finding any more defects is not an appropriate criterion to stop testing activities. There might be other metrics that are needed to adequately reflect the quality level reached. • The set of generic and specific conditions, agreed upon with the stakeholders for permitting a process to be officially completed. • The purpose of exit criteria is to prevent a task from being considered completed when there are still outstanding parts of the task which have not been finished. Exit criteria are used to report against and to plan when to stop testing. [After Gilb and Graham] • Time and budget testing • The amount of resources available (personnel, time and budget) might determine the extent of testing efforts

How much testing is enough? • Instead we need a test approach which provides the right amount of testing for the project. • We do this by aligning the testing we do with the risks (risk-based testing) for the customers, the stake-holders, the project and the software. • Assessing and managing risk is one of the most important activities in any project, and is a key activity and reason for testing. • Deciding how much testing is enough should take account of the level of risk, including technical and business risks related to the product and project constraints such as time and budget

Test Case, Test Basis • Test Case (according to IEEE 829) • Test case definitions include at least the following information • • • Pre-conditions Set of input values Set of expected values Expected post-conditions Unique identifier Dependence on other test cases Reference to the requirement that will be tested How to execute the test and check-in results Priority What about exit criteria? Integration tests? • Test Basis • Set of documents (e. g. Requirements doc) defining the requirements of a component or system. Used as the basis for the development of test cases

Software development and reviews • Code: computer instructions and data definitions expressed in a programming language or in a form output by an assembler, compiler or other translator. [IEEE 610] • Software – Development: a complex process/sequence of activities aiming at implementing a computer system. It usually follows a software development model. • Requirement: • A requirement describes a functional attribute that is desired or seen as mandatory • A condition or capability needed by a user to solve a problem or achieve an objective that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document. [After IEEE 610]

Why is Testing Necessary? - Summary • Software failures may cause enormous damage • Software quality is the sum of attributes that refer to the capability of the software to meet given requirements • Constructive software quality assurance deals with preventing defects • Analytical software quality assurance deals with finding defects and correcting them • Functional and non-functional quality attributes define the total quality of the system • Each test has to have predefined exit criteria. Reaching the exit criteria will conclude testing activities • Testers look for failures in the system and report them (testing) • Developers look for defects and correct them (debugging)

Outlines 1. Why is testing necessary? 2. What is testing? 3. Testing principles 4. Fundamental test process 5. The psychology of testing

WHAT IS TESTING? • The driving test - an analogy for software testing • In a driving test, the examiner critically assesses the candidate's driving, noting every mistake, large or small, made by the driver under test. The examiner takes the driver through a route which tests many possible driving activities • Some of the activities must be tested. For example, in the UK, an emergency stop test is always carried out • At the end of the test, the examiner makes a judgment about the driver's performance • The examiner bases the judgment on the number and severity of the failures identified, and also whether the driver has been able to meet the driving requirements.

WHAT IS TESTING? • Testing means more than running tests • Running tests is only one part of testing • Test process includes: • • Planning and control Choosing test conditions Designing and executing test cases Checking results Evaluating exit criteria Reporting on the testing process and system under test Finalizing or completing closure activities • Reviewing documents, source codes and conducting static analysis also help to prevent defects appearing in the code

WHAT IS TESTING? • Testing Objectives • Finding defects it may arise: • In development testing: to cause as many failures as possible • In acceptance testing: to confirm that the system works as expected • Gaining confidence about the level of Quality • Providing information for decision-making • To assess the quality of software, to give information to stakeholders of the risk of releasing the system at a given time • Preventing defects • Maintenance testing that no new defects have been introduced during development of the changes

WHAT IS TESTING? • Terms: Software Development • Debugging • The process of finding, analyzing and removing the causes of failures in software. • Review • An evaluation of a product or project status to ascertain discrepancies from planned results and to recommend improvements. Examples include management review, informal review, technical review, inspection, and walkthrough. [After IEEE 1028]

WHAT IS TESTING? • Testing and Debugging Test Debugging Correcting defects Retest • Test and re-test are test activities • Testing shows system failures • Retesting proves, that the defect has been corrected • Debugging and correcting defects are developer activities • Through debugging, developers can reproduce failures, investigate the state of programs and find the corresponding defect in order to correct it Developer tasks

WHAT IS TESTING? - Summary • The fundamental test process comprises • • • Test planning and control Test analysis and design Test implementation and execution Evaluating on exit criteria and reporting Test closure activities • Testing objectives • May be: finding defects, the level of quality, information for decision-making, preventing defects • The thought process and activities involved in designing test early in the life cycle • Dynamic testing • Means execution the software, static testing means reviews of documents • Shows failures that are caused by defects, debugging finds, analyses and removes the cause of the failure

Outlines 1. Why is testing necessary? 2. What is testing? 3. Testing principles 4. Fundamental test process 5. The psychology of testing

SEVEN TESTING PRINCIPLES • Principle 1: Testing shows presence of defects • Principle 2: Exhaustive testing is impossible • Principle 3: Early testing • Principle 4: Defect clustering • Principle 5: Pesticide paradox • Principle 6: Testing is context dependent • Principle 7: Absence-of-errors fallacy

SEVEN TESTING PRINCIPLES

SEVEN TESTING PRINCIPLES Principle 1 - Testing shows presence of defects • Testing can show that defects are present, but • cannot prove that there are no defects. • Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness.

SEVEN TESTING PRINCIPLES Principle 2 - Exhaustive testing is impossible • Testing everything (all combinations of inputs and preconditions) is not feasible except for small cases. • Test case explosion • Defines the exponential increase of efforts and costs when testing exhaustively • Instead of exhaustive testing, we use risks and priorities to focus testing efforts or we can use: • Sample Test • The test includes only a (systematically or randomly delivered) subset of all possible input values • Under real life conditions, sample tests are generally used; testing all combination of inputs and preconditions is only economically feasible in trivial cases.

SEVEN TESTING PRINCIPLES Principle 3 - Early testing • Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined objectives. • The earlier a defect is discovered, the less costly is its correction • Highest cost effectiveness when errors are corrected before implementation • Concepts and specifications may already be tested • Defects discovered at the conception phase are corrected with the least effort and costs • Preparing a test is time-consuming also • Testing involves more than just test execution • Test activities can be prepared before software development is completed • Testing activities (including reviews) should run in parallel to software specification and design

SEVEN TESTING PRINCIPLES Principle 4 - Defect clustering • A small number of modules contain most of the defects discovered during pre-release testing or show the most operational failures • Find a defect and you will find more defects nearby • Defects often appear clustered like mushrooms or cockroaches • It is worth screening the same module where one defect was found • Testers must be flexible • Once a defect is found, it is a good idea to reconsider the direction of further testing • The location of a defect might be screened at higher details level, e. g. starting additional tests or modifying existing tests

SEVEN TESTING PRINCIPLES Principle 5 - Pesticide paradox • If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new bugs. • To overcome this 'pesticide paradox‘: • the test cases need to be regularly reviewed and revised • new and different tests need to be written to exercise different parts of the software or system to potentially find more defects. • It is necessary to repeat a test after changes have been made in the code (bug fixing, new functionality) • Automating tests can be an advantage if a group of test cases is used regularly

SEVEN TESTING PRINCIPLES A typical test improvement initiative will initially find more defects as the testing improves and then, as the defect prevention kicks in, we see the defect numbers dropping, as shown in Figure 1. 3.

SEVEN TESTING PRINCIPLES Principle 6 - Testing is context dependent • Testing is done differently in different contexts. • Different test objects are tested differently • For example, safety-critical software, engine controller of a car is tested differently from an e-commerce site. • Test environment (test bed, test bench) vs. production environment • Tests take place on an environment other than the production environment; the test environment should be very similar to the production environment • There will always be deviations between the test environment and the production environment; these deviations impact the conclusions drawn after testing

SEVEN TESTING PRINCIPLES Principle 7 – Absence-of-errors fallacy • Finding and fixing defects does not help if the system built is unusable and does not fulfil the users' needs and expectations. • Successful testing finds the most serious failures • In most cases, testing will not find all defects of the system (see principle 2), but the most serious defects should be found • This alone does not prove the quality of the software • The functionality of the software may not meet the needs and expectations of the users • You cannot test quality into the product, it must be built in from the very beginning

SEVEN TESTING PRINCIPLES • Debugging removes defects • When a test finds a defect that must be fixed, a programmer must do some work to locate the defect in the code and make the fix. • In this process, called debugging, a programmer will examine the code for the immediate cause of the problem, repair the code and check that the code now executes as expected. • The fix is often then tested separately to confirm the fix. • When maintaining software by enhancing it or fixing bugs, we are changing software that is already being used. • In that case an objective of testing may be to ensure that we have not made errors and introduced defects when we changed the software. This is called regression testing • following the debugging cycle the fixed code is tested independently both to retest the fix itself and to apply regression testing to the surrounding unchanged software.

Class work List three of the test principles

SEVEN TESTING PRINCIPLES - Summary • Testing can show that defects are present, but cannot prove that there are no defects (absence of defects) • Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases (exhaustive testing) • To find defects early, testing activities shall be started as early as possible (early testing) • Defects show up clustered • If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new defects (pesticide paradox) • Testing is done differently in different contexts • Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’ needs and expectations

Outlines 1. Why is testing necessary? 2. What is testing? 3. Testing principles 4. Fundamental test process 5. The psychology of testing

FUNDAMENTAL TEST PROCESS • We can divide the activities within the fundamental test process into the following basic phases: • Test Planning and Control; • Test Analysis and Design; • Test Implementation and Execution; • Evaluating Exit Criteria and Reporting; • Test Closure Activities. • Test phases may overlap

FUNDAMENTAL TEST PROCESS • Testing throughout the software development process • Testing is more than test execution • Each phase of the testing process takes place concurrent to the phases of the software development process

FUNDAMENTAL TEST PROCESS • 1. Test planning - major tasks: • Determine the scope and risks and identify the objectives of testing • Determine the test approach (techniques, test items, identifying and interfacing with the teams involved in testing) • Implement the test policy and/or the test strategy • Determine the required test resources (e. g. people, test environment, PCs) • Schedule test analysis and design tasks, test implementation, execution and evaluation • Determine the exit criteria

FUNDAMENTAL TEST PROCESS • Master Test Plan • A document describing the scope, approach, resources and schedule of intended test activities. • It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process. [After IEEE 829] • Test Strategy • A high-level description of the test levels to be performed and the testing within those levels for an organization or program (one or more projects). • Test Approach • The implementation of the test strategy for a specific project. It typically includes the decisions made that follow based on the (test) project’s goal and the risk assessment carried out, starting points regarding the test process, the test design techniques to be applied, exit criteria and test types to be performed • Exit Criteria • The set of generic and specific conditions, agreed upon with the stakeholders for permitting a process to be officially completed. The purpose of exit criteria is to prevent a task from being considered completed when there are still outstanding parts of the task which have not been finished. Exit criteria are used to report against and to plan when to stop testing. [After Gilb and Graham]

FUNDAMENTAL TEST PROCESS • 2. Test control - major tasks • Measure and analyse the results of reviews and testing • Monitor and document progress, test coverage and exit criteria • Provide information on testing • Initiate corrective actions • make decisions or enable others to make decisions: to continue testing, to stop testing, to release the software or to retain it for further work • An ongoing activity influencing test planning. The master test plan may be modified according to the information acquired from test controlling • The status of the test process is determined by comparing the progress achieved against the test plan. Necessary activities will be started accordingly

FUNDAMENTAL TEST PROCESS • 3. Test Analysis and Design- major tasks /1 • we take general testing objectives identified during planning and build test designs and test procedures (scripts) • In this phase we only need specifications, not the software itself, the SWDs can program here • Review the test basis (such as the product risk analysis, requirements, architecture, design specifications, and interfaces), examining the specifications for the software we are testing. • Designing tests/test cases before the code exists, as we can use the test basis documents to understand what the system should do once built. • Create and prioritize logical/high level test cases (test cases without specific values for test data) • Positive tests give proof of the functionality, negative tests check the handling of error situations • representative tests that relate to particular aspects of the software which carry risks or which are of particular interest

FUNDAMENTAL TEST PROCESS • 3. Test Analysis and Design- major tasks /2 • Identify test conditions and required test data based on analysis of test items, their specifications, and what we know about their behaviour and structure. • Evaluate the availability of test data and/or the feasibility of generating test data • This gives us a high-level list of what we are interested in testing. • Evaluate testability of the requirements and system. The requirements may be written in a way that allows a tester to design tests • Example: If the requirements just say 'the software needs to respond quickly enough' that is not testable, because 'quick enough' may mean different things to different people. A more testable requirement would be 'the software needs to respond in 5 seconds with 20 people logged on'

FUNDAMENTAL TEST PROCESS • 3. Test Analysis and Design- major tasks /3 • Design the test environment set-up and identify any required infrastructure and tools. • This includes testing tools and support tools such as spreadsheets, word processors, project planning tools, and non-IT tools and equipment - everything we need to carry out our work. • (Exclusive) availability of the test environment, time windows etc. • Define the operation of the test environment, including user administration • Processes, procedures and responsibility choosing • Creating bi-directional traceability between test basis and test cases

FUNDAMENTAL TEST PROCESS • Test Data • Data that exists (for example, in a database) before a test is executed, and that affects or is affected by the component or system under test • Input Data • A variable that is read by a component (whether stored within the system or outside) • Test Coverage • The degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite (expressed as a percentage). Used mostly on white box tests to determine code coverage • The total number of activities in relation to executed activities • Test oracle • A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), other software, a user manual, or an individual’s specialized knowledge, but should not be the code. [After Adrion]

FUNDAMENTAL TEST PROCESS • Testware • Artifacts produced during the test process required to plan, design, and execute tests, such as documentation, scripts, inputs, expected results, setup and clear-up procedures, files, databases, environment, and any additional software or utilities used in testing. [After Fewster and Graham]

FUNDAMENTAL TEST PROCESS • 4. Test Implementation & Execution /1 • Finalizing, implementing and prioritizing test cases • including the identification of test data • Developing and prioritizing test procedures, creating test data • optionally, preparing test harnesses and writing automated test scripts • Creating test suites from the test procedures for efficient test execution • Verifying that the test environment (test bed) has been set up correctly • Verifying and updating bi-directional traceability between the test basis and test cases • Executing tests (test procedures) either manually or by using test execution tools, according to the planned sequence

FUNDAMENTAL TEST PROCESS • 4. Test Implementation & Execution /2 • Logging the outcome of test execution and recording the identities and versions of the software under test, test tools and testware • Comparing actual results with expected results • Reporting discrepancies as incidents (PRs) and analyzing them in order to establish their cause • Example: a defect in the code, in specified test data, in the test document, or a mistake in the way the test was executed • Repeating test activities to confirm a fix • Retest (after defect correction) • Regression test: • execution of a corrected test and/or execution of tests in order to ensure that defects have not been introduced in unchanged areas of the software or that defect fixing did not uncover other defects • *PR = Problem Report, CR = Change Request

FUNDAMENTAL TEST PROCESS • Test suite/test sequence • A set of several test cases for a component or system under test, where the post condition of one test is often used as the precondition for the next one. • Test procedure specification (test scenario) • A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script. [After IEEE 829] • Test execution • The process of running a test on the component or system under test, producing actual result(s) • Test log • A chronological record of relevant details about the execution of tests. [IEEE 829] • Regression testing: • Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed • Re-testing (confirmation testing) • Repeating a test after a defect has been fixed in order to confirm that the original defect has been successfully removed (see the process of testing a PR)

FUNDAMENTAL TEST PROCESS • 5. Evaluating Exit Criteria • Evaluating exit criteria is the activity where test execution is assessed against the defined objectives. • This should be done for each test level, as for each we need to know whether we have done enough testing. • Based on our risk assessment, we'll have set criteria against which we'll measure 'enough'. • Evaluating exit criteria has the following major tasks: • Check test logs against the exit criteria specified in test planning • Assess if more tests are needed or if the exit criteria specified should be changed • Write a test summary report for stakeholders

FUNDAMENTAL TEST PROCESS • 6. Test Closure Activities – major tasks • collect data from completed test activities to consolidate experience, testware, facts and numbers • occur at project milestones such as when a software system is released, a test project is completed (or cancelled), a milestone has been achieved, or a maintenance release has been completed • Checking which planned deliverables have been delivered • Closing incident reports or raising change records for any that re main open • Documenting the acceptance of the system • Finalizing and archiving testware, the test environment and the test infrastructure for later reuse, handing over the testware to the maintenance organization • Analyzing “lessons learned” to determine changes needed for future releases and projects • Using the information gathered to improve test maturity

FUNDAMENTAL TEST PROCESS - Summary • The testing process can be divided into different phases • Test planning covers the activities defining test approach for all test phases as well as planning resources (time, personnel, machines) • Test design (specification) covers designing the test cases and their expected results • Test execution covers defining test data, performing test execution and comparing results • Test evaluation and reporting covers exit criteria evaluation and recording of test results in written form • Test conclusion covers closure of incident report and lesson learned • Test control consists of controlling activities covering all the above phases of the testing process

Outlines 1. Why is testing necessary? 2. What is testing? 3. Testing principles 4. Fundamental test process 5. The psychology of testing

THE PSYCHOLOGY OF TESTING • Roles and Responsibilities • Developer Role • Tester Role • implements requirements • develops structures • designs and programs the software • creating a product is his success • plans testing activities • designs test cases • is concerned only with finding defects • finding an error made by a developer is his success Perception Developers are constructive! Testers are destructive! Wrong! Testing is a constructive activity as well, It aims at eliminating defects from a product!

THE PSYCHOLOGY OF TESTING • Personal attributes of a good tester /1 • Curious, perceptive, attentive to detail • To comprehend the practical scenarios of the customer • To be able to analyze the structure of the test • To discover details, where failures might show • Skepticism and has a critical eye • test objects contain defects – you just have to find them • do not believe everything you are told by the developers • one must not get frightened by the fact that serious defects may often be found which will have an impact on the course of the project

THE PSYCHOLOGY OF TESTING • Personal attributes of a good tester /2 • Good communication skills • to bring bad news to the developers • to overbear frustration states of minds • both technical and issues of the practical use of the system must be understood and communicated • positive communication can help to avoid or to edge difficult situations • Praise them from the beginning • to quickly establish a working relationship with the developer • Experience • personal factors influencing error occurrence • experience helps identifying where errors might accumulate

THE PSYCHOLOGY OF TESTING • Differences: to design – to develop – to test • Testing requires a different mindset from designing or developing new computer systems • common goal: to provide good software • design mission: help the customer to supply the right requirements • developer’s mission: convert the requirements into functions • tester’s mission: examine the correct implementation of the customer’s requirements • In principle, one person can be given all three roles to work at • differences in goals and role models must be taken into account • this is difficult but possible • other solutions (independent testers) are often easier and produce better results

THE PSYCHOLOGY OF TESTING • Independent testing • The separation of testing responsibilities supports the independent evaluation of test results • Degree of independence ? The higher the value, the greater is the independence

THE PSYCHOLOGY OF TESTING • Types of test organization /1 • Developer test • The developer will never examine his “creation” unbiased (emotional attachment) • However, the developer knows the test object better than anybody else • extra costs result for the orientation of other persons on the test object • human beings tend to overlook their own faults • the developer runs the risk of not recognizing even self-evident defects • errors made because of misinterpretation of the requirements will remain undetected • setting up test teams where developers test each other’s products helps to avoid or at least lessen this shortcoming Degree of independence ?

THE PSYCHOLOGY OF TESTING • Types of test organization /2 • Team of developers • developers speak the same language • costs for orientation in the test object are kept moderate, especially when the teams exchange test objects • danger of conflicts’ generation among developing teams • One developer who looks for and finds a defect will not be the other developer’s best friend • mingling development and test activities • frequent switching of ways of thinking • makes difficult to control project budget Degree of independence ?

THE PSYCHOLOGY OF TESTING • Types of test organization /3 • Test teams • Creating test teams covering different project areas enhances the quality of testing • It is important that test teams of different areas in the project work independently Degree of independence ?

THE PSYCHOLOGY OF TESTING • Types of test organization /4 • Outsourcing tests • The separation of testing activities and development activities offers best independence between test object and tester • Outsourced test activities are performed by personal having relatively little knowledge about the test object and the project background • Learning curve brings high costs, therefore unbiased party experts should be involved at the early stages of the project • External experts have a high level of testing know how: • An appropriate test design is ensured methods and tools find optimal use • Designing test cases automatically • Computer aided generation of test cases, e. g. based on the formal specification documents, is also independent Degree of independence ?

THE PSYCHOLOGY OF TESTING • Difficulties /1 • unable to understand each other • developers should have basic knowledge of testing • testers should have basic knowledge of software development • especially in stress situations, discovering errors that someone has made often leads to conflict • the way of documenting defects and the way the defect is described will decide how the situation will develop • persons should not be criticized, the defects must be stated factually • defect description should help the developer find the error • common objectives must always be the main issue

THE PSYCHOLOGY OF TESTING • Difficulties /2 • Communication between testers and developers is missing or is insufficient. This can make it impossible to work together • testers seen as “only messengers of bad news” • Improvement: try to see yourself in the other person’s role. Did my message come through? Did the answer reach me? • A solid test requires the appropriate distance to the test object • An independent and non-biased position is acquired through distance from the development • But: too large a distance between the test object and the development team will lead to more effort and time for testing

THE PSYCHOLOGY OF TESTING - Summary • People make mistakes, every implementation has defects • Human nature makes it difficult to stand in front of one’s own defects (error blindness) • Developer and tester means two different worlds meet each other • developing is constructive – something is created that was not there before • testing seems destructive at first glance – defects will be found • Together, development and test are constructive in their objective to ensure software with the least defects possible • Independent testing enhances quality of testing • Instead of developer teams, use tester teams or teams with external personnel for testing