IBM Software Group IBM Software Development Platform Software

® IBM Software Group IBM Software Development Platform Software Quality Sales 201 Training 2005 © 2003 IBM Corporation

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 2

IBM Software Group Imperatives in Today’s World: Innovation & Productivity Innovating the business to capture new value. Business IT Improving the productivity of resources deployed. 3

IBM Software Group IT is the lever to maximize flexibility and responsiveness while containing cost n tio ra eg Inf Ma rast na ruc ge tu me re nt On Demand On Operating Environment Int An on demand operating environment is an integrated infrastructure aligned to business goals and processes in a resilient and secure manner Business-Driven Development All designed with an architecture that allows you to manage services as components Results: Simplification and Optimization of IT to meet the needs of the business responsively 4

IBM Software Group The challenge: Poor visibility, lack of cohesion across business and technology domains Business View § Poor visibility and governance over IT investments § Lack of actionable information § Blind decision-making Operations View Business Analysts § Inadequate service levels § Inability to rapidly deploy applications § Complex, multi-tier operating Operations environments Application Developers Application Development View § Overwhelming complexity § Relentless time-to-market pressure § Uncontrolled change 5

IBM Software Group The solution: Breaking down the silos A shared view of the development lifecycle Business View § Clear view of technology ROI § Top-down and bottom-up visibility into technology projects § Objective decision-making support Operations View § Improved service and quality compliance § Predictable deployments § Accelerated diagnosis and repair Business Analysts Operations Application Developers Application Development View § Rapid application development and deployment § Improved collaboration § Asset reuse 6

IBM Software Group Transform and simplify software development Business-driven development An integrated approach to software development that aligns line-of-business, development and operations teams to improve business performance Development as a business process • Align Technology and Business priorities • Improve efficiency and responsiveness • Create innovative products • Higher productivity - 50% + increase in developer productivity • Improved quality - 80% fewer bugs • Greater predictability Software development becomes a driver of competitive advantage 7

IBM Software Group Discover, develop, and deploy assets Prioritize Plan Manage Measure § Discover business & technology assets 4 Business priorities Business 4 Requirements 4 Middleware and software assets § Develop at the speed of business 4 Rapid application development 4 Model-driven architecture 4 Asset-based development 4 Direct-to-middleware productivity § Deploy to closed-loop environments Operations Optimize Iterate 4 Automated applications deployment 4 Streamlined composite application management 4 Direct-to-operations productivity 8

IBM Software Group The business-driven development lifecycle Prioritize Plan Manage Measure Business Executive Model the Business Optimize End User Define Requirements Govern Analyze & Design Operations Manager Project Manager Manage change & assets Manage Application Support Deploy Operations Deployment Manager Optimize Analyst Architect Implement Developer Tester Development Iterate 9

IBM Software Group The IBM Software Development Platform A complete, open, modular, and proven solution Analyst Model, simulate, assemble, and monitor processes Project Manager Executive Architect Visually model applications and data Developer Tester Rapidly construct, transform, integrate and generate code Design, create, and execute tests Deployment Manager Provision, configure, tune and troubleshoot applications § Follow a common § Manage change process and assets § Manage and measure projects and portfolios § Manage quality § Manage requirements § Align investments with business objectives § Analyze and monitor project portfolios 10

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 11

IBM Software Group Focus # 1: Discover Assets Optimize Business Investments DISCOVER IBM SOFTWARE DEVELOPMENT PLATFORM Identify and analyze ü Business processes ü Requirements ü Software Assets üPackaged applications üExisting applications ü Middleware ü Impact of potential change Business process optimization is the #1 priority in 2004 — Information Week Study 2 Q 04 — 400 business-technology executives 12

IBM Software Group Focus # 2: Develop Assets Improve Time-to-Value IBM SOFTWARE DEVELOPMENT PLATFORM Dramatically accelerate the development cycle ü Rapid application development ü Model-driven architecture ü Asset based development ü Direct-to-middleware productivity ü Service-oriented architecture ü Tightly integrated run-time test and deploy ü Anytime, anywhere, team collaborative environments 13

IBM Software Group Focus # 3: Deploy Assets Deliver Business Results Enable closed-loop deployment IBM SOFTWARE DEVELOPMENT PLATFORM ü Unify development and operations teams ü Accelerate deployment and provisioning ü Automate applications monitoring ü Monitor business performance and analyze ROI ü Verify and manage change 14

IBM Software Group A team-based approach All tools and team members support all development imperatives Analyst Model, simulate, assemble, and monitor processes Project Manager Executive Architect Visually model applications and data Developer Tester Rapidly construct, transform, integrate and generate code Design, create, and execute tests Deployment Manager Provision, configure, tune and troubleshoot applications § Follow a common § Manage change process and assets § Manage and measure projects and portfolios § Manage quality § Manage requirements § Align investments with business objectives § Analyze and monitor project portfolios 15

IBM Software Group IBM Software Quality Offerings Product Description Capabilities § Runtime analysis § Component, functional and performance testing § Cross-platform test execution support § Test & planning reporting § Ensures reliability, functionality, scalability § Reduces cycle time § Accommodates multiple skill sets, roles § Shares responsibility for quality across team Business Analyst Tester Developer IBM Rational Functional Tester IBM Rational Performance Tester IBM Rational Robot IBM Rational Purify. Plus Key Products IBM Rational Manual Tester IBM Rational Test Real. Time IBM Rational Test. Manager 16

IBM Software Group From the Industry Analysts IBM Rational Software Quality Products § Ovum 4 "IBM Rational has been focusing on strategic integration of the ASQ* toolset with the complete IBM Software Development Platform, including the use of the Eclipse framework for software tool integration. We expect over the next 18 months IBM Rational's ASQ tools will again provide best-of-breed, singlepoint test tools, which will excel by operating within a formidably integrated environment across the development, test and delivery lifecycle. As a bonus, the fact that the platform is based on Eclipse open standards means that you will not be locked into IBM Rational-only tools. “ – Source Paul Herzlich (Ovum) 08. 04 - Ovum Evaluates: Software Testing Tools § IDC 4 - Noted Rational’s “extremely well established and complementary product set to IBM’s. ” § Source: Worldwide Distributed Automated Software Quality Tools 2004 -2008 Forecast and 2003 Vendor Shares M. Webster (IDC) 08. 04 * Automated Software Quality 17

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 18

IBM Software Group Challenges of Functional Testing § Control cost of test automation 4 Inexperienced testers face steep learning curve for tools “Our script language limits our ability to overcome some automation obstacles. ” 4 Test maintenance grows with project § Testing Tools Lack Flexibility 4 Power users hands are tied by tool limitations “I need one tool for both my expert and novice testers. ” 19

IBM Software Group Functional Testing with IBM Rational Functional Tester Test automation for the novice and the professional § IBM Rational Functional Tester 4 Automated regression testing for Web, J 2 EE, . Net and terminal-based applications § Key Benefits 4 Minimize test maintenance with scripts resilient to application changes 4 Wizard enhanced automation to speed test creation for the new user 4 Powerful scripting language and IDE for the professional tester 4 Supports Team oriented parallel development 20

IBM Software Group Challenge 1: Controlling the Cost of Automation § Challenge 4 Overcome the complexity of test automation to ensure teams spend more time on high value quality assessment rather than on low return test maintenance. § Resolution 4 Provide technologies to enable novice testers to craft complex tests with no programming § Data driven wizard § Dynamic data verification 4 Reduce script maintenance § Wizard driven global updates to centralized object map 21

IBM Software Group Challenge 2: Tools Lack Flexibility § Challenge 4 Fulfill both standard test needs as well as expert requirements to ensure usage limited only by the imagination of the project team. § Resolution 4 Provide two test script development languages § Java § Visual Basic. NET 4 Give professional testers the flexibility to overcome testing challenges in the language of their choice 22

IBM Software Group Challenge 3: Test Team Integration § Challenge 4 Accommodate the requirements of both dedicated, independent QA teams as well as integrated project teams to ensure high operational integrity. § Resolution 4 Provide version control support in the box with Clear. Case LT 4 Provide integrations for standard version control features § Check In/Out § Compare with Previous Versions § Show change history 23

IBM Software Group What is Automated Testing? § Using automation tools to record actions taken against an application. § Test scripts are derived from these recordings which are then used for subsequent test execution. Record Enhance Execute 24

IBM Software Group Challenge of Manual Testing with Short Test Cycles Tests . . . the new Reruntests. . . the first tests and. . . Manual testing can’t keep pace!. . . take more time Time Build 1 Build 2 Build 3, 4, 5 … § Manual testing alone can’t keep pace with the rapid development of applications 4 Quality will be sacrificed to keep on schedule

IBM Software Group Effective Test Automation Creating an automated test is a three steps process Record § Script Recording 4 Capture user interactions with the system under test 26

IBM Software Group Effective Test Automation Creating an automated test is a three step process Record Enhance § Script Recording 4 Capture user interactions with the system under test § Script Enhancing 4 Modify scripts for situation specific challenges 27

IBM Software Group Effective Test Automation Creating an automated test is a three step process Record Enhance Execute § Script Recording 4 Capture user interactions with the system under test § Script Enhancing 4 Modify scripts for situation specific challenges § Script Execution 4 Ensure reliable playback for local or remote execution 28

IBM Software Group IBM Rational Functional Tester Overview GUI Technologies • Automates regression testing of Java, . NET, Web and terminal emulator-based applications Data-driven Testing • Simplified data-driven testing enabled by wizards and Excellike data tables Eclipse availability • When using Java scripting, runs inside Eclipse/RSA/RAD shell • Full Eclipse JDT features are accessible • Accessible to developers from within their IDE Clear. Case. LT in the box • Supports parallel development of manual tests • Enables remote test access for distributed teams IBM Rational Functional Tester Tests Java, . NET, Web and terminal emulator-based apps Java and VB. NET-based test scripts Automated functional regression testing Test Script Language • User has choice of two test script languages – Java and VB. NET • Use of Java implies editing and debugging in Eclipse/RAD/RSAbased editor • Use of VB. NET implies usage of Microsoft VS. NET-based editor/debugger • Identical capabilities in both editors Wizard and table-assisted data-driven tests Linux-based test authoring Improved Linux support • Users can author and modify test scripts on Linux (no script recorder) • Scripts can also be executed on Linux Eclipse Java Development Tools Manual Tester Clear. Case LT Manual Tester in the box • Customers receive one Manual Tester license per Functional Tester license • Improve return on manual testing efforts while adopting automated test methods 29

IBM Software Group IBM Rational Functional Tester New Positioning ØLowering the Cost of Functional Testing ØOffers Extensive Customization Options ØSupports Integration of Test Team with Rest of the Project Team What’s New in Atlantic § Broader environment support 4 Adds support for testing VS. NET Winforms applications 4 Adds ability to edit and playback scripts on Linux (no recording) § Advances in ease of use 4 Choice of script languages 4 Wizard for data-driven testing 4 Improved object map maintenance capabilities Ships with full-use copies of IBM Rational Manual Tester and IBM Rational Clear. Case. LT and customers can download Rational Test. Manager 30

IBM Software Group Rational Functional Tester – Key Features and Benefits Test Java, Web & VS. NET Winform apps Lower TCO with a single tool supporting a broad array of industry standard application development technologies Java or VB. NET scripting language Easier adoption of the tool – Teams select the language they are most familiar with. Helps sell to Microsoft customers Add-on for Terminalbased application testing Central QA and IT teams can support client-server, web and legacy application testing with single tool Wizards for datadriven testing Eliminates manual coding required to vary data in test script Expanded dynamic verification Ability to use regular expressions in data pools created with new wizard for datadriven testing Advanced Object Map Maintenance Global Find and Modify tools further reduce test maintenance 31

IBM Software Group Rational Functional Tester – Key Features and Benefits Version Control “inthe-box” Collocated teams can support multiple versions of test scripts and parallel development paths with no additional tools to buy. Distributed teams can upgrade to Clear. Case to facilitate remote testing, whether offsite or off-shore. Ability to install, edit and execute tests on Linux Developers and testers in diverse, heterogeneous environments benefit from a common toolset 32

IBM Software Group Summarizing Key Features in IBM Functional Tester § Broad Environment Support 4 Test Web, Java. . net and terminal-based applications 4 Create data driven tests without coding 4 Static data and properties verification 4 Dynamic data validation without coding § Flexible and Extensible Scripts 4 Flexible coding language 4 Powerful, professional debugger 4 Object map editing flexibility 4 Version control of scripts § Script Execution Options 4 Reliable playback with Script. Assure 4 Remote and local playback on various platforms 33

IBM Software Group Reduce Test Script Maintenance Reliable Playback with Script Assure Version 1. 0 Version 2. 0 Tester Sees No User Intervention Required With Tool TM Sees Script. Assure Determines Match 34

IBM Software Group Case Study SAS Institute, Inc. § Challenge 4 Improve communication between developers and testers § Solution 4 IBM Rational Functional Tester § Enables best-of-breed automated functional and regression testing from within an Eclipse environment, including testing from within IBM Web. Sphere Studio and Rational XDE Developer § Benefits § Productivity and quality improvements – Efficiency has gone up and the quality of software delivery has gone up § Bring quality issues to the surface faster – fix these issues and deliver software on time § Minimize script maintenance through resilient reusable test scripts § Achieved better results by using a consistent methodology § More information 4 http: //www-306. ibm. com/software/success/cssdb. nsf/CS/JENS-5 WVSPY? Open. Document&Site=software 35

IBM Software Group Competitive Advantages IBM Rational Functional Tester § Part of a proven, complete, open and modular software development platform 4 Single vendor solution 4 Integration with other development tools § Provides a shared development/test environment and common language on application development projects using Eclipse or Visual Studio. net 4 Choice of a Java or Visual Basic. net scripting language 4 Allows testers to leverages existing Java or VB. net skills on program teams § Product wizards to help build complex tests without coding § Provides a single tool to test Java, Web, . net and terminal-based applications § Script. Assure technology for building robust, resilient test scripts 36

IBM Software Group Resources IBM Rational Functional Tester § IBM Rational Functional Tester Web page 4 http: //www-306. ibm. com/software/awdtools/tester/functional/index. html § Rational Functional Tester evaluation copy 4 http: //www 14. software. ibm. com/webapp/download/preconfig. jsp? id=2005 -0119+04%3 A 01%3 A 18. 402144 R&S_TACT=104 CBW 71&S_CMP= § Introduction to IBM Rational Functional Tester 4 http: //www-128. ibm. com/developerworks//rational/library/04/r-3228/ § developer. Works testing resources 4 http: //www-130. ibm. com/developerworks/rational/products/xdetester § Success stories 4 http: //www. ibm. com/software/success/cssdb. nsf/topstories. FM? Open. Form&Site=rational § Training 4 http: //www-306. ibm. com/software/sw-training/ § Events 4 http: //www-306. ibm. com/software/sw-events/ 37

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 38

IBM Software Group Effective Test Automation Creating an automated test is a three step process Record Enhance Execute § Script Recording 4 Capture user interactions with the system under test § Script Enhancing 4 Modify scripts for situation specific challenges § Script Execution 4 Ensure reliable playback for local or remote execution 39

IBM Software Group Recording Scripts § Create a Test Script that captures all system-user interaction 4 Test scripts are recorded on the fly, as user navigates application 4 Verification points are inserted to validate system response § Key considerations when recording scripts 4 Environment Support 4 Flexible verification ability 40

IBM Software Group Environment Support for Recording Scripts § IBM Rational Robot has special environment support for key development languages 4 Java, HTML, . NET, Visual Basic, MFC 4 Power. Builder, Delphi, Oracle Forms § Robot also has a general recording engine which can be used against other windows and terminal emulation environments 41

IBM Software Group Verification Points in Recording Scripts Robot Sees Data You See… § Automated Validation Robot Sees Properties 4 Robot captures data and properties that can be invisible to users 4 During script execution, current results are compared to stored baselines 4 Discrepancies are flagged and reported to user 4 13 different verification points are available to validate system behavior including Rational Site. Check for web site link checking 42

IBM Software Group Enhancing Scripts § Enhancing Scripts with basic coding extends their value and reach 4 SQA Basic code is added to perform a variety of functions 4 Typical Modifications: Conditional branching, datapooling, refactoring § Key considerations when enhancing scripts 4 Flexible coding language 4 Extensibility of language and playback engine 43

IBM Software Group Flexible Coding Language for Enhancing Scripts § IBM Rational Robot uses the SQA Basic Language 4 Uses Visual Basic syntax 4 Augments VB commands with test specific functions 4 Easy to read language very suitable to those will little programming experience § Flexible power to enhance scripts 4 Programmatic access to all GUI objects 4 Datapool facility enables data driven tests 4 Programmatic access to Windows API 4 Programmatic access to COM objects 44

IBM Software Group Solving Custom Problems with Extensibility § Robot has an extensibility interface to allow support for custom controls 4 Development partners have been trained 4 Can manufacture support for Active. X, . NET Windows Forms, Java & Win 32 including MFC and ATL GUI controls § Eliminates the work around 4 Allowing testers to focus on the task of testing 45

IBM Software Group Executing Tests § Scripts are executed, discrepancies are noted 4 Scripts are executed and test logs created 4 Test logs are highlight differences between actual and expected results § Key considerations when executing scripts 4 Multiple execution points 4 Coordination of scripts 46

IBM Software Group Automated Testing Test Run from Robot or Command Line Remote Tests Run From Test Manager Master § Script Execution Options 4 Scripts can be run from Robot GUI 4 Scripts can be executed from Command line 4 Scripts can be run from Test. Manager § Offers ability to run scripts on remote machines 47

IBM Software Group Effective Test Automation Agent Master § Powerful and flexible scheduling 4 Test. Manager will coordinate timing and dependencies 4 Schedule execution on remote computers from central control station 4 Test. Manager is configuration aware, i. e. : § Tests designated as Windows XP will only be run on XP machines 48

IBM Software Group Summarizing Key Features in IBM Rational Robot § Broad Environment Support 4 Support for all major development environments § Flexible and Extensible Scripts 4 Simple, yet powerful, scripting language for enhancing scripts 4 Support available for custom control § Script Execution Options 4 Run scripts from Robot, Test. Manager or command line 49

IBM Software Group Case Study Corporate Express § Challenges 4 Cut testing time and challenges in manual testing process § manual regression testing found to be tedious, time-consuming and error-prone § Improve repeatability of tests 4 Improve confidence in software quality at release § Solution 4 Automated the manual process with IBM Rational Robot § Benefits 4 Cut testing time from days to weeks, over 75% § 31/2 weeks of testing in 3 days 4 Team confidence that each regression test is executed accurately and with a consistent process from one release to the next § More information 4 http: //www-306. ibm. com/software/success/cssdb. nsf/CS/JENS-5 WPKR 2? Open. Document&Site=software . 50

IBM Software Group Resources § IBM Rational Robot Web page 4 http: //www-306. ibm. com/software/awdtools/tester/robot/index. html § IBM developer. Works page for IBM Rational Robot 4 http: //www-136. ibm. com/developerworks/rational/products/robot § Success stories 4 http: //www. ibm. com/software/success/cssdb. nsf/topstories. FM? Open. Form&Site=rational § Training 4 http: //www-306. ibm. com/software/sw-training/ § Events 4 http: //www-306. ibm. com/software/sw-events/ 51

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 52

IBM Software Group What is Manual Testing? 4 Using human labor to validate the business use cases and fitness for use of an application, system or device 4 Includes labor to plan, design, execute and evaluate the status of the application or device under test 4 Experience indicates over 75% of the testing done today is performed manually 53

IBM Software Group Why Teams use Manual Testing? Recruitment of subject-matter experts and end-users to test Strict regulatory or process requirements aren’t met with automation tools Poor or non-existent test automation-ready interfaces 54

IBM Software Group Rational Manual Tester Positioning ØAdvances Manual Testing ØSatisfies Diverse Needs ØSimplifies Through Assistance Key Capabilities § Supports manual testing done by analysts, subject-matter experts and technical/non-technical testers § Develop detailed, easy to follow manual tests § Minimize test maintenance and ensure test consistency § Reduce human error during data entry and data verification § Support diverse usage models and processes § Ships with full copy of Clear. Case. LT for version control of tests 55

IBM Software Group Challenges of Manual Testing § Labor Intensive 4 Too much time is spent creating and maintaining manual test documentation “More time is spent maintaining test documentation than on executing tests. ” “We struggle to document even a minimum set of tests. ” 56

IBM Software Group Challenges of Manual Testing § Labor Intensive 4 Too much time is spent creating and maintaining manual test documentation “Our results are unreliable due to human error. ” § Unreliable 4 Inadvertent human-error often skews test results “Our tests aren’t detailed enough for the testers to consistently execute them. ” 57

IBM Software Group Challenges of Manual Testing § Labor Intensive 4 Too much time is spent creating and maintaining manual test documentation “Each project has different standards, which are hard to manage manually. ” § Unreliable 4 Accidental human-error often skews test results § Resistant to Change 4 Each project has diverse processes, tester skill sets and types of applications and devices under test “No single tool or process works for the different software and hardware systems we must test. ” 58

IBM Software Group Challenge 1: Labor Intensive § Challenge 4 Creating and maintaining manual tests is labor intensive and difficult with tools such as Microsoft Word or Excel. § Resolution 4 Implement a solution that promotes the best practice of modular tests which can share common content 4 Leverage existing Word or Excel test documents to quickstart test development Share test content 59

IBM Software Group Challenge 2: Unreliable § Challenge 4 Human error often skews tests results § Resolution 4 Assist testers with the tasks most prone to error – data entry and data verification 4 Build manual tests that are clear, concise and easy for the tester to follow 60

IBM Software Group Challenge 3: Resistant to Change § Challenge 4 Each project has diverse processes, tester skill sets and types of applications and devices under test § Resolution For all skill levels 4 Provide a solution that testers of all skills levels can leverage 4 Customize the tool to fit each project’s process metrics and values 4 Leverage a solution that is independent of the software or hardware under test For any process For any target of test 61

IBM Software Group IBM Rational Manual Tester Overview Rich Text Editor • Modify test font, size, color; attach images and files • Supports creation of both test steps and verification points • Ensures clarity of test step direction and consistency of test execution IBM Rational Manual Tester Rich test editor Test step reuse library Tool Assistance • Automated data entry • Automated data comparison and results storage • Reduces opportunity for manual error during test execution Data entry & validation assistance Test Import / Results Export Manual test authoring and execution Reuse Library • Library for common test procedures • Drag-and-drop construction of new tests • Automatic global update when modifying any one instance of linked content Import/Export • Imports manual test source files from Microsoft Word, Microsoft Excel and Rational Test. Manager • Exports test results to CSV format for additional analysis Custom test fields Customizable • Add new data fields to individual test steps • Apply internally-defined naming conventions Clear. Case LT Clear. Case. LT in the box • Supports parallel development of manual tests • Enables remote test access for distributed teams 62

IBM Software Group Rational Manual Tester – Key Features and Benefits Rich Text Editor Reduces the ambiguity that often plagues other methods used to define and deliver test step directions Import From Word, Excel or Test. Manager Easier adoption of the tool Multiple Validation Points Allows test results to be reported analyzed at either a high level or a granular level Reuse Palette Promotes shared content which results in time & cost saving in test maintenance & greater control over test quality and consistency Custom Data Fields Easily adapts to existing processes and metrics collection standards Assisted Data Entry and Data Verification Reduce the likelihood of human error in the areas most prone to mistakes 63

IBM Software Group Rational Manual Tester – Key Features and Benefits Translucent Execution UI The translucent window used to display test instructions makes test completion faster by eliminating the need to move the test instruction window in order to see the application under test Supports Team Use Supports individuals, teams, and mobile works through private and shared test data stores – ensuring test assets remain easily accessible and reportable – Clear. Case. LT in the box Generates Spreadsheet-Ready Results Testers can use any reporting tool to analyze results - ensuring that Rational Manual Tester can easily work with a team’s processes and standards for metrics collection and reporting 64

IBM Software Group Summarizing Key Features in Manual Tester § Advances manual testing 4 Rich text editor for test authoring 4 Test step reuse palette 4 Test script import from Microsoft Word or Excel § Satisfies diverse needs 4 Supports usage by distributed teams 4 Generates spreadsheet-ready results data 4 Multiple validation points 4 Custom data fields § Simplifies through assistance 4 Assisted data entry and data verification during test execution 65

IBM Software Group Resources IBM Rational Manual Tester § IBM Rational Manual Tester Web page 4 http: //www-306. ibm. com/software/awdtools/tester/manual/index. html § Rational Manual Tester evaluation copy 4 http: //www 14. software. ibm. com/webapp/download/preconfig. jsp? id=2005 -0116+03%3 A 32%3 A 04. 405210 R&S_TACT=104 CBW 71&S_CMP=&s= § Rational Manual Tester evaluation guide 4 http: //www-106. ibm. com/developerworks/edu/i-dw-r-manualtest-i. html § Article 4 A simple tool to manage manual testing § http: //www-128. ibm. com/developerworks/rational/library/04/r-3232/ § developer. Works testing resources 4 http: //www-128. ibm. com/developerworks/rational/products/manualtester/ § Success stories 4 http: //www. ibm. com/software/success/cssdb. nsf/topstories. FM? Open. Form&Site=rational § Training 4 http: //www-306. ibm. com/software/sw-training/ § Events 4 http: //www-306. ibm. com/software/sw-events/ 66

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 67

IBM Software Group IBM Rational Performance Tester Overview New load testing technology • New load test recorder and execution engine • Supports HTTP/S; additional protocol support (e. g. ERP) planned for Q 2 2005 High scalability • High scalability, low footprint due to hyperthreaded architecture • Significant improvement over Robot technology Platform Support • Multiple platform support for load generation Clear. Case. LT in the box • Supports parallel development of manual tests • Enables remote test access for distributed teams Load and performance testing IBM Rational Performance Tester HTTP/S load testing Visual test construction and editing High scalability hyper-threaded execution architecture Graphical test scheduler and user profiler Windows and Linux load generation support Eclipse Java Development Tools Visual Test Editor • Test viewer requires no coding for test modification and execution • Automated identification and variation of dynamic data • Insertion of Java code is supported for advanced customization needs Test scheduler • Graphical user profiling and transaction modeling • Eases modeling of user populations Eclipse-based • Runs inside Eclipse/RSA/RAD shell • Accessible to developers from within their IDE Clear. Case LT 68

IBM Software Group IBM Rational Performance Tester New Positioning ØBuilt for Day One Productivity ØProvides Advanced Data Capabilities ØLowers Cost of Performance Testing What’s New in Atlantic § Completely re-architected performance testing solution for testing complex web application § Designed for novice and advanced performance testers § Real-time monitoring for immediate feedback during test execution § Easy to build data driven tests § Eclipse-3 integration 69

IBM Software Group Rational Performance Tester – Key Features and Benefits Visual test editor Easy for a novice to understand, with drilldown to detailed data for more expert users. Simplifies test creation and maintenance Automatic identification of variable data Allows tests to automatically re-use data sent by server and to prepare for data -driven testing Eclipse based Provides a single environment for development as well as testing activities Rich scheduling editor Novice users can easily assemble and schedule tests Built-in datapool editor for data-driven testing Easily vary data for more realistic load tests 70

IBM Software Group Rational Performance Tester – Key Features and Benefits Real-time reporting Know immediately about performance problems Remote system monitoring Collect and visualization of remote server information. Find resource constraints that cause poor performance High Scalability Low memory and processor footprint required to generate large multi-user loads. Allows larger tests with less hardware Java execution engine Tests may call custom Java provided by the user. Provides highlycustomizable and flexible performance testing 71

IBM Software Group Resources IBM Rational Performance Tester § IBM Rational Performance Tester Web page 4 http: //www-306. ibm. com/software/awdtools/tester/performance/index. html § Success stories 4 http: //www. ibm. com/software/success/cssdb. nsf/topstories. FM? Open. Form&Site=rational § Training 4 http: //www-306. ibm. com/software/sw-training/ § Events 4 http: //www-306. ibm. com/software/sw-events/ 72

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 73

IBM Software Group The Problem with Trying to Pinpoint Bugs Crash Errors triggered Symptoms Crash 74

IBM Software Group What Makes Bugs Difficult to Find? § Many memory-related bugs have no immediate visible symptoms § Often the symptoms are difficult, if not impossible, to reproduce § Manually tracing symptoms back to problems in the source code can be a daunting task Crash 75

IBM Software Group Why Is My Program So Slow? § Do you deliver slow code that’s “good enough? ” § Is it too difficult to find performance problems? § Can you get repeatable, useful performance data? § Can you quantify the effect of performance fixes? § Does your performance profiling data help you manage your development project? What about Predictability at runtime? 76

IBM Software Group Code Coverage Challenge § Can you guarantee the code that you deliver is reliable? § How complete is your test suite? § Are you sure that you have tested the entire application? § Can you understand the summary of several tests? How Reliable is Untested Code? 77

IBM Software Group IBM Rational Purify. Plus: Developing Fast, Reliable Code § To develop fast, reliable code, you need a tool that 4 Automatically pinpoints hard-to-find bugs 4 Highlights performance bottlenecks 4 Keeps you from shipping untested code § You need a multi-platform and multi-language tool for Unix, Windows, Linux, Java, . NET, VB 6, C/C++, and more § IBM Rational Purify. Plus combines three industry leaders into one box: 4 IBM Rational Purify 4 IBM Rational Quantify 4 IBM Rational Pure. Coverage The x-ray for software Instantly become more productive 78

IBM Software Group Detect Memory Problems: Rational Purify ü Automatically identifies the problem while the application is running! ü Finds errors in third-party code, even without source 4 Bugs in libraries that you deliver are still bugs in your application! ü No recompile required 4 Much faster and easier to use 4 Incorporate into your existing project quickly and easily ü Test what you are delivering 4 No need to maintain multiple source baselines 4 QE knows that they are testing what you are delivering Detects memory corruption and memory leaks 79

IBM Software Group Improve Code Performance: Rational Quantify ü Leading product that provides repeatable performance data 4 Build better quality C/C++, Java, Visual Basic 6. 0, and. NET software 4 Get your team solving the correct problems ü Fast and easy to use 4 Provides concise, graphical views of performance data 4 “River of Time” – a straightforward way to view and understand performance profiling data 4 Easily integrates into your nightly builds ü Collects performance data from code you intend to deliver 4 No need to maintain multiple source baselines 80

IBM Software Group Identify Untested Code: IBM Rational Pure. Coverage ü Helps developers deliver higher quality code 4 You know the unit testing is complete 4 Helps identify “dead code” or unexercised paths in legacy code ü Helps improve your QA process 4 Easily see what has and has not been tested 4 Validate the quality and completeness of your test suite 4 Easy to learn and easy to use out of the box ü With Rational Pure. Coverage, you know that you are covered! Avoid shipping untested code 81

IBM Software Group IBM Rational Purify. Plus § Key components of IBM Rational Purify. Plus 4 Provides run-time data collection and analysis capabilities 4 Key for proactive debugging § Supports Unix, Windows, Java, . NET and Linux environments Linux § C/C++ § Red. Hat § Full 64 bit Windows § § C/C++ VB 6. NET Java Solaris SGI-IRIX HP-UX § C/C++ § Java § Full 64 bit § C/C++ § Full 64 bit § Quantify § Pure. Coverage Not Available Yet 82

IBM Software Group IBM Rational Purify. Plus Summary § To develop fast, reliable code, you need IBM Rational Purify. Plus that: 4 Automatically pinpoints hard-to-find bugs 4 Highlights performance bottlenecks 4 Keeps you from shipping untested code § Powerful tool for developers and testers 4 Strong ROI when used as “medicine” 4 Even better ROI when used as “vitamin” § Used as part of the development process Instantly become more productive 83

IBM Software Group Case Study Ensemble Systems § Challenge 4 Ensemble Systems was contracted to develop an easy-to-maintain software system to perform data analysis twice as fast as an existing solution, while significantly reducing analysis errors. § Solution 4 Rational Rose, Rational Purify and Rational Quantify § able to effectively model the system, detect and fix memory leaks and enhance the overall performance of the application under development. § Benefits 4 Reduced the time the system required to analyze defects by 85%, and reduced the system's error rate by 90%. 4 Quickly pinpointed memory leaks and identified opportunities to improve performance § More Information 4 http: //www-306. ibm. com/software/success/cssdb. nsf/CS/JENS-5 PFQXR? Open. Document&Site=software 84

IBM Software Group Case Study SAS Institute, Inc. § Challenge 4 Improve software quality § Solution 4 Rational Purify and Rational Quantify to help find performance bottlenecks and memory problems in the software 4 Integrated Rational Purify and Rational Quantify into testing process through product APIs § Benefits 4 Achieve higher level of confidence in the quality of software releases § More information 4 http: //www-306. ibm. com/software/success/cssdb. nsf/CS/JENS-5 WVSWK? Open. Document&Site=software 85

IBM Software Group Competitive Advantages IBM Rational Purify. Plus § Part of a family of tools that comprise the IBM Software Development Platform 4 Integration across tools 4 Can form a more complete development and testing solution § Multilanguage support 4 Java, but also Visual C++, Visual Basic. NET and Visual C#. NET in one package § Module and test level coverage § Offline analysis of data § Merging test results 86

IBM Software Group Resources IBM Rational Purify. Plus § IBM Rational Purify. Plus Web page 4 http: //www-306. ibm. com/software/awdtools/purifyplus/ § developer. Works: Rational 4 http: //www. ibm. com/developerworks/ 4 Getting Started Guides 4 Training – Web-based or Instructor-led 4 Articles - whitepapers – forums § Success stories 4 http: //www. ibm. com/software/success/cssdb. nsf/topstories. FM? Open. Form&Site=rational § Training 4 http: //www-306. ibm. com/software/sw-training/ § Events 4 http: //www-306. ibm. com/software/sw-events/ 87

IBM Software Group Agenda § Strategy IBM Software Development Platform e-Business on demand § Overview of IBM Software Quality products and offerings § IBM Rational Functional Tester § IBM Rational Robot § IBM Rational Manual Tester § IBM Rational Performance Tester § IBM Rational Purify. Plus § IBM Rational Test Real. Time § Exercises § Additional References 88

IBM Software Group IBM Rational Test Real. Time Fix your code before it breaks § Rational Test Real. Time is a cross-platform testing solution designed for those who write code for real-time and embedded software products 4 Automates testing, analysis and bugs solving early during development cycle 4 Provides a low-overhead technology for enabling targetindependent tests code code code code code code Resolve Test Analyze 4 Extends model-driven development to include runtime analysis capabilities Fix the defects, enhance your tests and move on! 89

IBM Software Group Embedded and real-time software testing challenges Embedded and real-time systems are complex by nature § Application Complexity 4 Strong timing constraints 4 Low memory footprints 4 Concurrent/Distributed/Networked § Environment Complexity 4 Multiple RTOS/IDE/Chips vendors 4 Limited host-target connectivity 4 Low built-in debugging capabilities § Process Complexity 4 Requirements 4 Design translation errors 4 Difficult to maintain 4 Poor performance 90

IBM Software Group Embedded and real-time software testing challenges It’s hard to set up and deploy an effective testing framework § Are you frustrated by manual testing because of its tendency to be 4 Labor-intensive? 4 Resistant to reuse? § Are you concerned that test creation and execution will 4 Pull you out from your comfort zone? 4 Add overhead to an already crowded schedule? 4 Require to much efforts to ensure quality at both source code and system level? § Are you convinced that your embedded environment is too specific for a commercial testing solution because of 4 Design constraints? 4 Heterogeneous build/execution tools? 4 Strict testing techniques and activities recommended by certification standards? 91

IBM Software Group Embedded and real-time software testing challenges You need a way to test on the host AND on the actual target! § If your target platform is not available, you can test on the host 4 To simulate the operational environment and discover generic bugs § But tests must be performed in the final target environment as well 4 Since some errors are only detected under the operational conditions § Cross-platform testing is difficult because the system conception… 4 Is frequently based on multiple chips for different execution machines or boards 4 Combines more than one programming language 4 Presents very constrained architectural characteristics 92

IBM Software Group Embedded and real-time software testing challenges How to bring runtime analysis information at the UML level? § You want to use the Unified Modeling Language (UML) to visually lay out your application's design and tests but you still don’t know how 4 State machines and code coverage of model-level execution are related? 4 If there any memory leaks in the hand-coded parts of the application? 4 To measure the performance profile of the generated code? 93

IBM Software Group How can your team overcome these challenges? Would this accelerate quality? § Automated component testing at all levels of complexity 4 Full automation of the creation and deployment of test scripts, stubs and drivers for the simplest functions up to distributed systems § Faster debugging thanks to extensive runtime analysis capabilities 4 Memory and performance profiling, code coverage and runtime tracing features § Make testing an integral part of development 4 Dynamic links between tested code, test results and potentially visual models § Prioritize your tests 4 Code complexity and static metrics calculation § Fully repeatable test sessions 4 Complete automation of regression testing 94

IBM Software Group How can your team overcome these challenges? Would this solve the verification constraints in the real target? § An integration framework flexible enough to accommodate your selection of compiler, linker, debugger, and target architecture 4 To generate test harnesses executable in the real operational conditions 4 To avoid changing test scenarios when target environment does § Automatic test download, execution and results upload on target 4 To free developers from target management restraints 4 To help your development team concentrating primarily on test design § Low-overhead and minimally intrusive verification probes and drivers 4 To cope with low memory footprints and limited CPU resources 4 To encourage in-situ verification of the software despite target constraints 95

IBM Software Group How can your team overcome these challenges? Would this empower your model-centric development process? § Develop, debug and analyze at the model level § Graphically visualize reached and un-reached application states and transitions § Track memory errors and performance in the hand-coded portions of the application while staying in the modeling environment § Trace application execution flow in a UML-compliant graphical format 96

IBM Software Group Test, analyze and resolve during development § Test as you code 4 Automatic component test template and data generation 4 Black- and white-box testing 4 All levels of complexity: from single functions to distributed systems 4 Static metric calculation for § tests prioritization § complexity estimation code code code code code code Resolve Test Analyze 4 Full regression testing 97

IBM Software Group Test, analyze and resolve during development § Test as you code § Analyze while you test 4 Code coverage analysis 4 Memory profiling 4 Performance profiling 4 Runtime tracing code code code code code code Test 4 Thread profiling Resolve Analyze 98

IBM Software Group Test, analyze and resolve during development § Test as you code § Analyze while you test § Resolve what you have uncovered 4 Test execution integrated with your debugger 4 Consolidated, detailed to-the-point test reporting 4 Test data hyperlinked to runtime analysis results and code code code code code code Resolve Test Analyze Fix the defects, enhance your tests and move on! 99

IBM Software Group Test, analyze and resolve during development Overview of IBM Rational Test Real. Time features Component Testing C, C++, Ada, Java Code Covera ge Memory. Performance. Runtime Thread Profiling Tracing Profiling C, C++ Ada, Java C, C++ Java Unit, Integration, Validation, Regression and System Testing C Target Deployment Port § Built to achieve standards compliance 4 DO-178 B (Avionics), MISRA (Automotive), DEF STAN 00 -55 (Defense) 100

IBM Software Group Test, analyze and resolve during development Test Real. Time provides a mean for achieving pervasive quality ü Flexible to minimally impact preexisting development processes 4 Powerful and source-code aware test languages drive robust data-driven tests 4 Process agnostic, so you can stay in your comfort zone 4 Shortens time-to-problem-resolution, maximizes time-to-code ü White-box and black-box testing for all your software modules 4 Component testing of C/C++, Ada or Java (J 2 ME/J 2 SE) functions and classes 4 Message-based testing of C threads, tasks, processes and system nodes 4 Full set of runtime analysis features to enhance your debugging capabilities ü Enables quality verification from a single and unified testing tool 4 Handles all test enablement activities so you test early and often 4 Creates shared debug and test vocabulary for your team 4 Produces testing information required for certification processes conformance 101

IBM Software Group Test and debug both host and target Target Deployment Port technology § A versatile feature which fully automates 4 Test harness building and downloading on the target 4 Test execution and monitoring 4 Test results uploading on the host § A low-overhead feature which leverages 4 Your choice of compiler/linker/debugger 4 Your host-target connectivity (serial link, Ethernet connection, JTAG probe…) Test Harness Downloading Target Execution And Monitoring Hosted Build Process Test Results Uploading 102

IBM Software Group Test and debug both host and target 4 -Bit to 64 -Bit Cross-Development Environments Used By Our Customers § Wind. River § Montavista § Tasking § Green. Hills § TI § CAD-UL § ARM § NEC § Cosmic § Enea § Hitachi § Hiware § Windows CE § Apex § Hitex § Lynux. Works § Sun § Symbian § Lauterbach § Microtec § …… Languages § § C C++ Ada Java (J 2 ME/J 2 SE) Platforms § § § Windows Solaris Linux HP-UX AIX 103

IBM Software Group Test and debug both host and target Target Deployment Port provides full target independence ü Customizable to support a complete range of embedded targets 4 Assures tool adoption 4 Reduces your ramp-up time when target configuration changes 4 Guarantees the reuse of test assets despite target constraints ü Host, build and target environment agnostic 4 Ensures portability of test and runtime analysis processes 4 Simplifies multiple team deployment 4 Optimizes ROI in comparison to home-grown test solutions ü Size and speed optimized to limit target impact 4 Enables full control to minimize instrumentation overhead 4 Frees your tests from having to compensate for target restraints 4 Avoids overtasking your system 104

IBM Software Group Harness the power of model-driven testing IBM Rational Test Real. Time - IBM Rational Rose Real. Time integration § Move from code-centric to model-centric development § Perform test and runtime analysis at the model level 4 Rose Real. Time sequence diagrams for test scripting and Quality Architect Real. Time for test generation 4 Test Real. Time plug-in for the runtime analysis of your C++ models § From within Rose Real. Time for C++, activate the following runtime analysis features 4 State-machine coverage 4 Memory and performance profiling 4 UML runtime tracing 105

IBM Software Group Harness the power of model-driven testing From within IBM Rational Rose Real. Time Covered UML state UML Sequence Diagram as a test case Uncovered UML transition 106

IBM Software Group Harness the power of model-driven testing From within IBM Rational Rose Real. Time ü Develop and debug at the model level ü Seamlessly integrate 4 Test Real. Time runtime analysis capabilities 4 Rose Real. Time application execution ü Visualize covered states/transitions in your UML state machines 107

IBM Software Group Summary Fix your code before it breaks with IBM Rational Test Real. Time § A unique cross-platform solution for component testing and runtime analysis 4 Automated functional testing at all levels of complexity to ensure your software reliability 4 Memory and performance profiling, code coverage measurement and runtime tracing abilities to diagnose run-time issues 4 Dynamic links between code, test assets and visual models to make testing an integral part of the development cycle 4 Low-overhead technology for enabling target-independent tests and runtime analysis All in one tool – for any IDE and hosted on any target! 108

IBM Software Group Case Study Alcatel Space § Challenge 4 When building a communication system for a satellite project, Alcatel needed to shorten testing time yet meet stringent testing standards and overcome significant testing challenges. § Solution 4 Rational Test Real. Time enabled them to successfully achieve these objectives. § Benefits 4 Increased development productivity as a result of validation testing improvements 4 Heightened confidence in the overall quality of the application under test 4 Improved the predictability of the application's performance further ensuring customer satisfaction § More information 4 http: //www-306. ibm. com/software/success/cssdb. nsf/CS/JENS-5 MHK 6 Z? Open. Document&Site=software 109

IBM Software Group Competitive Advantages IBM Rational Test Real. Time § Full and flexible solution in a single tool § Integrated with lifecycle development tools 4 IBM Rational Clear. Case 4 IBM Rational Clear. Quest 4 IBM Rational Test Manager 4 IBM Rational Rose Real. Time 4 Rational Unified Process § Integrated with a UML model-driven development solution § Test deployment at all levels of verifications and granularity § Component testing and runtime analysis capabilities at the source code level § Test Real. Time Target Deployment Port feature 4 Deploy tests on any target from 4 -bit to 64 -bit architecture 110

IBM Software Group Resources § Rational Test Real. Time Web page 4 http: //www-306. ibm. com/software/awdtools/test/realtime/index. html § Real-Time and embedded solutions webinars 4 http: //www. ibm. com/software/rational/info/literature/whitepapers. jsp § Rational Developer Network 4 http: //www-136. ibm. com/developerworks/rational/products/testrealtime § Success stories 4 http: //www. ibm. com/software/success/cssdb. nsf/topstories. FM? Open. Form&Site=rational § Training 4 http: //www-306. ibm. com/software/sw-training/ § Events 4 http: //www-306. ibm. com/software/sw-events/ 111

IBM Software Group Quiz § Which roles on the software development team can benefit from IBM Software Quality products and offerings? § What are the benefits of an automated functional testing product such as IBM Rational Functional Tester? § Name three of the major features in Rational Functional Tester. § Which development languages does Rational Functional Tester support for creating test scripts? § What is the function of IBM Rational Robot? § Name three major features of Rational Manual Tester. § What does the assisted data verification feature in Rational Manual Tester do? § What does the visual test editor in Rational Performance Tester do? § What are three areas of code analysis in IBM Rational Purify. Plus? § What are two of the competitive advantages of IBM Rational Purify. Plus? § What does Target Deployment Port technology do in Rational Test Real. Time? 112

IBM Software Group © Copyright IBM Corporation 2004. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, the e-business logo and other IBM products and services are trademarks or registered trademarks of the International Business Machines Corporation, in the United States, other countries or both. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries or both. Microsoft, Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries or both. All other trademarks, company, products or service names may be trademarks, registered trademarks or service marks of others Disclaimer: NOTICE – BUSINESS VALUE INFORMATION IS PROVIDED TO YOU 'AS IS' WITH THE UNDERSTANDING THAT THERE ARE NO REPRESENTATIONS OR WARRANTIES OF ANY KIND EITHER EXPRESS OR IMPLIED. IBM DISCLAIMS ALL WARRANTIES INCLUDING, BUT NOT LIMITED TO, IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IBM DOES NOT WARRANT OR MAKE ANY REPRESENTATIONS REGARDING THE USE, VALIDITY, ACCURACY OR RELIABILITY OF THE BUSINESS BENEFITS SHOWN. . IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGES, INCLUDING THOSE ARISING AS A RESULT OF IBM'S NEGLIGENCE. WHETHER THOSE DAMAGES ARE DIRECT, CONSEQUENTIAL, INCIDENTAL, OR SPECIAL, FLOWING FROM YOUR USE OF OR INABILITY TO USE THE INFORMATION PROVIDED HEREWITH OR RESULTS EVEN IF IBM HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE ULTIMATE RESPONSIBILITY FOR ACHIEVING THE CALCULATED RESULTS REMAINS WITH YOU. 113