Lecture 8 Tool Support for Testing Tool Support

Lecture 8 Tool Support for. Testing

Tool Support for Testing Types of Test tool Effective Use of Tools – Benefits and Risks Integration Tools Examples

7. 1 Types of Test Tools Test Tool Classification Tool Support for Management of Testing Tool Support for Static Testing Tool Support for Test Specification Tool Support for Execution and Logging Tool Support for Performance and Monitoring Tool Support for Specific Application Areas Tool Support using other tools

Types of Test Tool 7. 1. 1 Test Tool Classification (1) • Tools are classified according to the test activities they support (e. g. test management, static testing, etc. ) • Some tools may support multiple activities while others just one • Tool vendors may provide a suite of (hopefully) integrated tools for the testing lifecycle. (Major vendors include Mercury, Compuware, Borland, etc. ) • Tools can be any piece of purchased or locally developed software • Tools can be tremendously helpful in improving the efficiency and quality of testing… • …but can also be a waste of time and effort

Types of Test Tool Classification (2) • Some tools may be intrusive – ﺗﺪﺧﻼ i. e. they affect the test outcome itself – this is the Probe Effect – For example, some tools that measure the performance of a piece of code may insert code to count or time instructions – though very detailed information can be obtained, the timings will be affected simply because of the extra code • Some tools offer greater support to developers rather than testers e. g. , Static Analysers are more commonly used by developers • Some tools offer greater support for different parts of the software development lifecycle (e. g. Unit Testing, System Testing) • Computer Aided Software Test Tools (CAST Tools) is a common term used in testing circles

Tool Support for Testing Test management tools Static testing tools Test specification tools Test execution and logging tools Test management Requirements management Incident management Configuration management Static analysis Review process support Modelling Test design Test data preparation Test execution Test harnesses Test comparators Coverage measurement Security Performance, monitoring tools Application specific tools Other tools Dynamic analysis Performance/load/stress Monitoring Embedded systems Web testing Language specific Word Processors Operating system utilities Spreadsheets SQL, Code debugging

Types of Test Tool 7. 1. 2 Tool Support for Management of Testing (1) • Management tools can be applied across the entire software development lifecycle: – during unit testing, acceptance testing, etc. – and by all project staff - by managers, developers, testers, etc. Visit: http: //www. testlodge. com/signup Online test case management tool, allowing you to manage your test plans, test cases and test runs with ease. Visit: . . List of notable test management tools. doc

Types of Test Tool Support for Management of Testing (2) Test Management Tools: • Help Manage tests and activities – Provide management and control of test documentation, test cases, specifications and results – Support scheduling of tests, logging results, recording test environments • Interface to other tools – Link to test automation tools to run tests automatically – Link to defect tracking tools to allow cross referencing to incidents/defects – Link to requirements management tools to cross reference tests and requirements • Support Version control – Allow test cases, logs etc. to be baselined, “checked-out”, modified, retrieved and controlled (often via another ‘linked’ tool)

Types of Test Tool Support for Management of Testing (3) Test Management Tools – continued • Allow Traceability – Allow traceability to be recorded between test and requirements – Allow traceability to associate defects/incidents • Support Logging – Allow recording of results within the tool – Have results linked to ‘useful’ information (who, when, on what hardware, etc. ) • Provide Analysis – Produce analysis reports on testing progress and test coverage – Produce reports on tests passed and failed – Produce reports on incidents raised – Tools differ in their ease of use, accuracy of the information and portability of results for distribution

Types of Test Tool Support for Management of Testing (4) Requirements Management Tools • Primary purpose is to store requirements (as simple text, as Use Cases, formal language, etc. ) • Generally provide a means of recording the priority of each requirement • To support testing, such tools can: – Verify and validate requirements (consistency checking, missing links to other requirements) – Allow tests cases to be linked to the requirements • Test Teams should review requirements early in the lifecycle to check for consistency, testability and to allow test cases to be constructed

Types of Test Tool Support for Management of Testing (5) Incident Management Tools • Also known as defect tracking tools • Record defects/problems/incidents/anomalies with virtually any aspect of the project • E. g. operational problems, testing failures, documentation errors. Fortunately, it is not common to use them for reporting problems with people • Widely used by all members of a project team – particularly testers • They support the prioritisation of incidents (e. g. High for incidents that must be fixed in 4 hours through to Low for incidents that may never get fixed. )

Types of Test Tool Support for Management of Testing (6) Incident Management Tools - continued • Allow incidents and resulting actions to be assigned to project members. E. g. to Investigate, to test, to cost, etc. • Record current status and history of an incident. E. g. New, or Rejected, or Resolved, or Waiting for test, etc. • Some generate analysis reports on trends in incident creation, resolution, effort recorded, etc. • Most can be tailored to specific project needs e. g. to define project specific states or priorities, to specify the layout of information, to construct predefined reports, etc.

Types of Test Tool Support for Management of Testing (7) Configuration Management Tools • Allow software baselines to be kept and managed (See notes for background information) • Not strictly testing tools, but are heavily used by test teams to: – obtain specific version of software (code, documents, etc. ) – maintain their own test material • Traceability between testware and software can be achieved by applying build/release labels to corresponding items • Useful (often essential) when managing different version of a system for different target environments or different operational releases

Types of Test Tool 7. 1. 3 Tool Support for Static Testing (1) Review Process support tools • During a review: – Documents are issued, people are nominated, roles are assigned – Templates and guidelines are distributed – Feedback is produced, times are recorded. • Tools support these activities by storing the documents, providing communication mechanisms, recording metrics • Some support on-line reviews where people are geographically dispersed. (Within EDS, e. Room is used to support online reviews)

Static Test Tools • These tools do not involve actual input and output. Rather, they take a symbolic approach to testing, i. e. they do not test the actual execution of the software. • These tools include the following: 1) Flow analyzers: ensure consistency in data flow from input to output. 2) Path tests: find unused code and code with contradictions. 3) Coverage analyzers: ensures that all logic paths are tested. 4) Interface analyzers: examines the effects of passing variables and data between modules.

Types of Test Tool 7. 1. 4 Tool Support for Test Specification (1) Test Design Tools • Test design tools generate some or all of the following automatically: – Test inputs (i. e. the test data) this would be data from external system or user – Test cases – Expected results • They use requirements, analysis models, data models, state models, code • Tools will require the model to be in a format they can ‘read’ Perhaps as a formal language or as part of the tool itself • They may look for input boundary conditions, state transitions, paths through the system.

Types of Test Tool Support for Test Specification (2) Test data preparation tools • Enable data to be selected from existing databases, files or captured transmissions • Enable data to be created, generated, manipulated and edited for use in tests • The most sophisticated tools can deal with a range of file and database formats • The application under test is often used to ‘Drive’ the data into databases, etc. • They have the benefit of generating anonymous data for data protection and security purposes.

Types of Test Tool 7. 1. 5 Tool Support for Execution and logging (1) Test execution tools • Provide capture, replay and comparison facilities • Tools simulate mouse movement, button clicks and keyboard inputs • Test scripts are normally captured in a programmable script language (VB, C, Java, etc. ) • Additional code can be added to the script for data manipulations, verifications, iterations, etc. • When executed, tests run automatically, following the recorded/ programmed script using the defined input data • Compare results with the predefined expected values and record the logs in the tool (i. e. they use a Comparator)

Types of Test Tool Support for Execution and logging (2) Test harness/unit test framework tools • test harnesses, drivers and stubs simulate other parts of the system or other connecting systems (or people) • They are used to: – Simulate other components that are not yet available – so testing can continue – Simulate other components to allow detailed, controllable testing of a specific component – Simulate events that may be difficult (even dangerous) to produce with the whole system (e. g. database failures or nuclear meltdown) • Commercially available tools exist, but custom-written programs also fall into this category • Simulators also come under this category

Types of Test Tool Support for Execution and logging (3) Test harness ﺍﺳﺘﻔﺎﺩﺓ /unit test framework tools – continued • Test harnesses inject data into a component and capture outputs from the component • Comparisons with previous runs and pre-defined expected outcomes can be performed • Some frameworks are designed for developers for unit/component testing -these are unit test frameworks (e. g. JUnit, HTTPUnit)

Types of Test Tool Support for Execution and logging (4) Test comparators • Comparison tools are used to detect differences between actual results and expected results • Standalone comparison tools normally deal with a range of file or database formats (i. e. Generic tool ) • Test execution tools usually have built-in comparators that deal with character screens, GUI objects or bitmap images • These tools often have filtering or masking capabilities - they can ‘ignore’ rows or columns of data or areas on screens (e. g. volatile date fields)

Types of Test Tool Support for Execution and logging (5) Coverage measurement tools • These tools are dynamic test tools and allow test coverage to be monitored • Most commercial tools are designed for code coverage i. e. how much of the code (statements, branches) has been executed • Provide objective measures of test coverage when tests are executed • Some tools add extra code to the source code (called instrumentation) to allow detailed information to be obtained – but recall the Probe Effect • After execution, the log file is analysed and coverage statistics generated • Statistics are provided on the most common coverage measures such as statement or branch coverage

Types of Test Tool Support for Execution and logging (6) Security tools • With the rise of the internet, the need for preventing malicious attacks on systems has increased • Tools have emerged which test systems vulnerabilities ﻧﻘﺎﻁ ﺍﻟﻀﻌﻒ to attacks • ( in web applications, password are encrypted in the URL address rather than transferring as it is to prevent unauthenticated users from accessing DB system containing the confidential data related to passwords) Yahoo e-mail example: https: //login. yahoo. com/config/login? . slogin=&. intl=xe&. src=maktoob&. bypass=&. partner=&. done=https %3 a//login. yahoo. com/config/validate%3 f. src=maktoob%26. intl=xe%26. pc=3783%26. done=https%253 A %252 Flogin. maktoob. com%252 Fylogin. php%253 Fjump%253 DYToz. Ontz. Oj. Y 6 Ik. No. Tm. Ft. ZSI 7 czo 4 Oi. Jo b 21 lc. GFn. ZSI 7 czo 1 Oi. JDa. FVSTCI 7 czoy. ODoia. HR 0 c. Dov. L 2 Vu. LW 1 ha 3 Rvb 2 Iue. WFob 28 u. Y 29 t. Ly. I 7 czo 0 Oi. JMYW 5 n. I jtz. Oj. I 6 Im. Vu. Ijt 9%26. scrumb=0&pkg=&owd= • Denial of service attacks ( )ﻣﺤﺘﺎﻝ – Tools examine code and attempt to cause a failure by injecting rogue data. – They may look for weaknesses in code (in HTML, . NET, etc. ) intentionally trying to damage the service • Penetration testing – trying to break in: Tools will look for security loopholes, monitor traffic, or password weaknesses, etc. and attempt to gain access to machines or services

Types of Test Tool 7. 1. 6 Tool Support for Performance & monitoring (1) Dynamic Analysis tools • These are used to find defects that are apparent only when the software is actually running • These tools are most commonly used to: – identify unassigned pointers – monitor the allocation use and de-allocation of memory to flag memory leaks • These tools are generally used by developers

Types of Test Tool Support for Performance & monitoring (2) Performance testing tools ( e. g. Load. Runner from Mercury for EDS company) • Performance test tools have two main facilities : Ø load generation measurement Ø test transaction measurement • Load generation can simulate either multiple users or high volumes of input data • Load generation is done either by driving the application using its user interface or by test drivers • Records of the numbers of transactions executed are logged • Response times are taken and logged for selected transactions • Performance testing tools normally provide reports based on test logs and graphs of load against response times • Tools in this category are also called load test tools or stress test tools

Types of Test Tool Support for Performance & monitoring (3) Monitoring tools • These tools are not strictly test tools, but enable failures or potential failures to be detected • Monitoring tools monitor: Ø the systems memory, Ø CPU usage, Ø file spaces, Ø network traffic, Ø disk I/O, Ø current processes running, Ø database optimisation (e. g. monitoring embedded into products such as Oracle) • They can be ‘tuned’ to report potential failures – e. g. file space 80% full, process XYZ should always be running.

Types of Test Tool 7. 1. 7 Tool Support for Specific Application areas • Tools exist which are specialised for a specific task, environment or application • For example: – Harnesses, monitoring tools, etc for embedded systems – where their specialised hardware and application requires specialised tools – hyperlink testing tools (Spiders) are used to check that no broken hyperlinks are present on a web site – Tools for a specific operating system (Windows, UNIX, etc. ) – Tools for a specific language (JAVA, C++, . Net, etc) – Performance tools specifically for WEB based user interfaces

Types of Test Tool 7. 1. 8 Tool Support using other tools • Many other tools are used to support the test effort. • For example: – Word processing tools – widely used for producing test plans, test reports, to record test cases, etc. – Spreadsheets – also widely used for test cases, cross-referencing, analysis of data, reporting, etc. – Database interrogation tools (e. g. SQL, Toad, Raptor) – database interrogation is performed to validate the contents of tables. Simple SQL statements can be used or more sophisticated tools (Toad, Raptor, etc. which allow the ‘novice’ to interrogate a database) – Operating system utilities and scripts to monitor resources, generate data, run tests – Code debugging tools – to step through code, monitor variables, etc.

7. 2 Effective Use of Tools – benefits and risks Potential Benefits and risks – all tools (1) • Some tools bring quick and easy benefit to the test effort, but… • Some tools require considerable investment in time and thought to realise their benefits • Don’t just look at the adverts or listen to the salesman ( you have to taste and test )

Effective Use of Tools – benefits and risks Potential Benefits and risks – all tools (2) Potential Benefits of using tools • • Repetitive work reduced – “Relieve the boredom” – re-running the same test many times can be extremely boring and error prone – Tools can reduce the time we spend on repetitive tasks Consistency and repeatability – “Errare Humanum Est” (to Err is Human) – • Objective assessment – tools have no “axe to grind” – • they are objective with their assessments and results Ease of access to information – • Test identified by a tool or run by a tool can be more repeatable and consistent. Tools do what they are told tools can give easy access to test information stored and present it in multiple formats for distribution Automated tests can be run out of hours

Effective Use of Tools – benefits and risks Potential Benefits and risks – all tools (3) Risks of using tools - every silver lining has a cloud • Unrealistic expectation - Don’t just look at the adverts or listen to the salesman. Tools may be rich in features – but are they useful to your test effort? • Underestimating how much effort/money is required to purchase, install, get trained and simply get started with the tool • Underestimating how much effort is required to gain experience so that benefits can be realised • Underestimating the effort to incorporate into the test process • Underestimating the effort required to maintain the assets (automated scripts may need to change for every change in the data or user interface) • Over-reliance – not considering other tests that need to be found or test that need to be run

7. 3 Integration Tools Example 1: HP Quality Center Tool • is a set of web-based test management software offerings from the HP Software Division of Hewlett-Packard, many of which were acquired from Mercury Interactive Corporation. • HP Quality Center offers software quality assurance, including requirements management, test management and business process testing for IT and application environments. • HP Quality Center and HP Performance Center are components of the HP Application Lifecycle Management software solution set. • http: //www. youtube. com/watch? v=6 b. Pmk. H_5 Tt. Q

Example 2: Star Team Tool • It is a revision control system used in software development, especially when a project involves multiple teams in different locations. Star. Team is an SCM and SDLC software application, created by Starbase Corporation, acquired by Borland in January 2003 which was acquired by Microfocus meanwhile. • The application is client-server, backed by a relational database that retains all changes made to a project during its evolution as well as the project requirements, task assignments, threaded discussions and bug tracking. Microsoft SQL Server and Oracle database are supported database servers.

Star Team eclipse Tool Snapshot