Lightning Talks Presented at STAREast 2005 By Matt

Lightning Talks Presented at STAREast 2005 By Matt Heusser … and the gang mheusser@charter. net

Pamela Wagner CUNA Mutual Group Risk Analysis in five minutes Pamela. Wagner@cunamutual. com

Simple RA Form Requirement or issue Likelihood of failure Impact of failure I need to walk down the hall without tripping. High Medium Think about your experiences, use old bug reports, trust your gut! CONTEXT CHANGES EVERYTHING! Instead of a hall- the red carpet at the Oscars where I am nominated for Best Actress, Director, Producer and Screenwriter. High HIGHEST POSSIBLE !

Jennifer Brendlinger Concurrent Technologies Corporation CMMI for testing: A critical look brendlij@ctc. com

What is CMMI? n Capability Maturity Model (CMM) Integration n n Disciplines – Bodies of knowledge n n n Serve as a guide for improvement of organizational processes http: //www. sei. cmu. edu/cmmi/ March 2003 – Concurrent Technologies Corporation (CTC) was appraised at Maturity Level 3 of the CMMI-SE/SW, Version 1. 1, Staged Representation – http: //www. ctc. com Systems Engineering – Development of total systems Software Engineering – Development of software systems Staged Representation n Proven sequence of improvements, each serving as a foundation for the next

Maturity Levels n Maturity Level 1: Initial n n Maturity Level 2: Managed n n Processes are well defined and understood and are described in standards, procedures, tools and methods. Maturity Level 4: Quantitatively Managed n n Requirements are managed, and processes are planned, performed, measured and controlled. Maturity Level 3: Defined n n Processes are ad hoc and chaotic. Success depends on heroics of the people, not on the use of proven processes. Selected subprocesses are controlled using statistical and other quantitative techniques. Maturity Level 5: Optimizing n Processes are continually improved based on a quantitative understanding of the common causes of variation inherent in processes.

Organizational Processes n Process Areas n Related practices that satisfy a set of goals n n n Specific Goals (SG) n n Describe what must be implemented to satisfy the process area Specific Practices (SP) n n Verification – Ensure that selected work products meet their specified requirements Validation – Demonstrate that a product or product component fulfills its intended use when placed in its intended environment Activities performed to achieve a specific goal Generic Goals (GG) n Enable the organization to institutionalize best practices

Verification & Validation n Verification n n Validation n SG 1 – Preparation for verification is conducted. SG 2 – Peer reviews are performed on selected work products. SG 3 – Selected work products are verified against their specified requirements. SG 1 – Preparation for validation is conducted. SG 2 – The product or product components are validated to ensure that they are suitable for use in their intended operating environment. GG 3 – The process is institutionalized as a managed process. n Company processes and procedures

Work Products Used to Enhance Testing and Comply with CMMI n n n Verification and Validation (V&V) Plan Operations & Maintenance (O&M) Manual System Test Procedures Requirements Traceability Matrix Discrepancy Reports Change Requests Test Run Logs Peer Review Reports Meeting Minutes Trip Reports Metrics

Bottom Line - Benefits of CMMI n For the Testing Process n n n For the Project n n Established plans to follow “Enforced” cooperation, dialog, reporting and follow-up Quality work product that meets its requirements and fulfills its intended use For the Organization n Everyone buys in to the processes and procedures No grassroots efforts No chaos

Questions Jennifer C. Brendlinger Software Test Lead Concurrent Technologies Corporation brendlij@ctc. com

Reversing the Outsource/Offshore Trend with Technology Rick Vatcher CEO, Accord. SQA

The Promise – The Truth • SQA Outsourcing Saves Money – Outsourcing Yields Poor Results (May 4, 2005 USATODAY) – 70% had a bad experience and 44% didn’t save money. • SQA skilled workforce more plentiful in developing countries – India experiencing skilled labor crisis (Ostfold SW)

Take a Look at the SQA Industry • SQA Tools developed by SQA Professionals • Automated, Easy to use, Easy to learn, LESS TEDIOUS • Knowledge based tools replacing Manual Repetition • Business Process Part of Process – Elevating Roles • Integral Part of the Development Process

Next-Generation SQA Accelerate the Process – Eliminates Offshore Gains MAN-HOURS Next Generation Technology 2 nd Generation Tool Test Development * 480 3000 Maintenance Cycles 50 150 Costs @ $40/Hour $21. 2 K $126. 0 K • 1500 test cases/scripts (2 hours per script) equivalent to 45 Smarte. Script test scenarios (10. 5 hours per scenario) 83% Less Cost! - 83% Less Time!

Software Tool Advancements Driving Best Practices in Resource Allocation • Next-Gen Tools Close the Gap –Save money –Speed Time-to-Market –Elevate Resources …Ultimately Reversing Outsourcing Trend

Sheryl Stockless The Mathworks, Inc. Moving Quality Engineering Up-Front Sheryl. Stockless@Mathworks. com

Verification New bug (Customer, internal) Team Triage Need more info Not a bug Close (Eng, QE, Doc, Support) (QE) Submit bug fix (Eng) Pass Verify bug fix Fail (QE) Add a test (QE) Close (QE) Sheryl Stockless, The Math. Works, Inc. Star East 2005

Validation and Verification New bug (Customer, internal) Need more info Validate bug (QE) Team Triage Not a bug Close (Eng, QE, Doc, Support) (QE) Submit bug fix (Eng) Pass Verify bug fix Fail (QE) Close (QE) Sheryl Stockless, The Math. Works, Inc. Star East 2005

Jennifer Bonine & Marie George Express Scripts Sarbanes-Oxley in the testing shop Jennifer. Bonine@Express-Scripts. com

What Any QA department Need to meet SOX Requirements • Defined Control Points – – – Requirements Test Plan Test Cases Go/No Go document Implementation Plan • Processes around the Control Points – Don’t over process • Accountability and Traceability – Sign off and acceptance from required stakeholders – Traceability of Control Points, acceptance to projects/releases *Applicable for anyone who is testing applications to get involved and understand audit requirements

Automated Testing using Distributed Computing Darshan Domah Senior Software Test Engineer Vital Images Inc. 5/19/2005 STAREAST 2005

Distributed Test Environment DTE tclient/tserver Tcl • Set up Test • Start Test • Monitor Test Python • Results Reporting Win. Runner • Drive GUI of Application • Uses Analog Mode

Overview of tclient/tserver • • 2 Tier Client/Server TCP/IP and Sockets Multiple servers, dedicated ports Used to: – – – reset box bring SW execute queue of tests monitor tasks get results • Some Commands – Start application remotely • tclsh 83 tclient. tcl –c –bgnd –h HOST 1 vitrea. exe – Start a winrunner test remotely • tclsh 83 tclient. tcl –c –wrun –n e: /Orientation –i 2 –t 3600 –h HOST 5 –o z: /orientation/testresults

Overview of Application • 3 D Visualization for medical imaging-Vitrea • Uses: diagnosis, surgery planning, post operation analysis CT and MRI data from scanners Vitrea interprets data Recreates 2 D images and 3 D volumes of anatomy Medical personnel (Radiologists & Physicians) use software capabilities to analyze rendered images, make diagnosis and create reports – Testing • Patient Orientation • Reliability Uptime

Communicating the Value of Exploratory Testing David Gilbert President Sirius Software Quality Associates

Exploratory Testing IS High Value • Due to lack of co-dependencies, it can begin much earlier. – – No requirements needed Full functionality not needed Can work around instabilities Formal Stubs and Harnesses not always needed

Exploratory Testing IS High Value • Due to flexibility, it should have the highest defect to time testing ratio of any other testing process – Not business process constrained – Looks for the cause and classification of defects, not just the repeatable specific case

But Value is Relative… • …and based on PERCEPTION! – A painting from your child is very valuable to you. – The Mona Lisa has value universally. • What is the difference? • Quality of Presentation!

Why is Automation Held in Such High Esteem? • Even though we all know many automation efforts do not successfully test their projects well, they are perceived as – – – Organized Structured Professional Repeatable Reportable Manageable

We Must Present Our Results Professionally • To be taken seriously and given respect, we must meet the same level of professional management and presentation. • This can be VERY labor intensive, using – – – Screen. Grabbers Spreadsheets Home. Grown Databases Word Processors Lots of cut and paste Reams of paper for printing results

We Must Present Our Results Professionally • Or use a tool, with features such as … – – – Keylogging Screenshots Video Organized Test Cases Integrated Issue Management Graphs and Reports

Anibal Sousa Microsoft Too many bugs to fix, too little time … asousa@microsoft. com

Motivation • 5000+ bugs during product development life cycle • Subjective and not deterministic process for triaging bugs • Testers were confused about what kind of bugs were still being accepted – they wanted to be able to focus on the right set of problems to investigate • Bad for morale and team engagement • Bugs will occur through out the whole project • Quality should go up and not down • Risk of regression gets higher with time • Not all features get ready at the same time • Aspects to be considered: cost, risk, impact to teams (test, UA, etc. )

Proposal Priority Name Definition 1 Wish These might not be even considered bugs. 2 Improvement This bug is hard to find, noticeable, causes minor problems (or none) and can be ignored. 3 Bug This is a bug, but might be obscure, rare or have small impact. Normally it has easy workaround. 4 KB This bug might be very visible and affect functionality significantly. In case it has a workaround, it is not obvious or simple. 5 PSS This bug will cause Support calls. Publishing KB article is not enough, since workaround may not exist or be too complicated. 6 QFE Serious bug. If a customer finds this bug, we will have to issue a QFE. Ship Stopper Reasonable region of a feature is not working as expected because of the bug, and there is no workaround. We can not ship the product with this bug active. 7 • • • Rules of game clear and public to all team members Bug is opened, field is set in the bug database by tester, might be updated by developer, feature team decide on it, escalating to Management team if necessary Bug bar considers: feature status, moment in project schedule and bug details, like scope, risk, cost, etc. Triage process and sets right priority for all accepted bugs accordingly Clear communication of results

Example 7 Wish 6 Improvement X 5 Bug X 4 KB X X X 3 PSS X X 2 QFE X X X X 1 Ship Stopper X X X X Areas Reports Import Export Forms Performance SBA Integration PDA User Assistance 5 Bug 4 KB X X 3 PSS X X X 2 QFE X X 1 Ship Stopper X X X X Areas Reports Import Export Forms Performance SBA Integration PDA User Assistance 3 PSS 2 QFE X X 1 Ship Stopper X X X X Areas Reports Import Export Forms Performance PDA SBA Integration User Assistance

After … • Team engagement and commitment – high morale • Number of regressions (metrics) went down significantly • Number of opened bugs did not go down, but accepted bugs did • Product was shipped in timely manner

Questions? • Anibal Sousa – asousa@microsoft. com or anibal_sousa@hotmail. com

In the Backroom, No One Can Hear You Scream Harry Robinson, Google harryr@google. com

Does Your Boss’s Boss Know Your Value?

Sure, Finding Bugs is Important But, I bet your work could help others. And helping them will help you!

Publicizing Your Contribution is Important And the pen is mightier than the bug report.

So Start. Write Now. • Start a blog. • Post a paper to Stickyminds. • Give brown bag presentations to your team. • Offer to speak at local QA group meetings. • Send in an abstract for STAR West 2005!

Here’s A Poem To Inspire You! The codfish lays ten thousand eggs, The homely hen lays one. The codfish never cackles To tell you what she's done. And so we scorn the codfish, While the humble hen we prize, Which only goes to show you That it pays to advertise. - Anonymous

Lightning talks will be at the better software conference. Call for presentations! http: //www. sqe. com/lightningtalks. asp