Chicago Quality Assurance Assocation Secrets of Successful Automation

Chicago Quality Assurance Assocation Secrets of Successful Automation Projects Presented by Paul M. Grossman Software Quality Assurance Analyst QTP Automation Framework Designer

§ § § Portfolio of Custom Software § Software Development: . Net, Java, C# § e. Commerce (Oracle ATG) § Mobile Applications: i. OS, Android, Windows Mobile/WP 7, Blackberry Quality Assurance Testing § Load. Runner, QTP, HP ALMS § Custom Tool Development and Frameworks § Performance Testing § Mobile Testing: i. OS, Android, Windows Mobile, Tablets Developed Applications in the following sectors § Games & Entertainment § Healthcare § Social Networking 4 tegroup. com

Paul M. Grossman § 11 years automation experience with HP Tools § HP Quality Center 9. 0 Certified Product Consultant § HP Quick Test Professional 9. 0 Certified Product Specialist § Beta Tester of QTP 9. 5 – 11. 5 § Speaker at Mercury/HP Events § Became Automation Engineer due to Perfect Timing! Manag e magic r caught m e t system ricks while practicing waitin to reb g for oot.

Session Goals § § Key success factors in test automation today Demonstrating test automation ROI Effective framework design techniques Successful script design for Agile environments

Automation Challenges § Lack of communication of changes/details to testing group § Lack of understanding of should be automated § Lack of variation from plan in test scripts § Insufficient standards § Insufficient Analysis & Design § “Silver bullet syndrome” (automated testing is the “silver bullet” that will end all testing problems on a project) § Lack of change control

Case Study Automation Project Failure at a Medical Device Manufacturer Year 1: QA Manager wants automated testing. Buys Win. Runner, accepts offer at new company. Year 2: New QA Manager wants automated testing. Hired Lead Developer (C++), given an Assistant (VBscript). Goal for automation: 100% coverage. Year 3: Lead Developer moves to new department. Assistant told automation is high-priority, spare-time project. Manag Year 4: QA Manager moves to new division of d er promo tes m efunc t A utoma e to Team Assistant retires. tion P roject Lead…. Results: 0 Defects found. Negative Return On Investment. Unsuspecting manual tester assigned to automation team. New Division Manager expresses interest in automation.

Top 5 Project Pitfalls 1. Lack of research & planning 2. Lack of apparent progress 3. Lack of experienced resources 4. Not a full time project 5. No buy-in § 100% automation goal resulted in development in all product areas § 20% development in all areas with nothing to show § Project handed over to assistant with no product training § Misperception: FT not needed because automation by it’s definition is automatic, right? § Testers & developers fueled rumors of tool’s stability

Mitigation Strategies 1. Lack of research & planning 2. Lack of apparent progress 3. Lack of experienced resources 4. Not a full time project 5. No buy-in § 10% Automation goal § POC Pilot Project § Training § Full-time commitment § Repeat defects manually on a non-automated system

Tips And Tricks Avoiding "100% Automation" Trap § 90% Automation can be approached but it is generally not cost effective 20 - 40% Automation makes a significant impact on the testing process. § New releases will have more functionality with constantly decreasing code coverage § A percentage of the code could be unreachable code, such as uncalled functions § Pick one: Functionality or Browsers Do the math: You won't have time for everything! Math: A Do th pp Develop ey ha ve 6 -1 ers outnum Do th 8 b ey ha ve mo Months he er your tea a re tha m? n one d start? tool?

Case Study The Pilot Project – Lessons Learned § Automate the Acceptance Test (“Smoke Tests”) — Short tests, but too wide of coverage § Commitment to six weeks of full-time development — Became four weeks, 10 days of manual testing § Used “Record & Playback” to create the entire test script — Overwhelming maintenance § Demonstration to follow at the end of the project — Management convinced they purchased the right tool — Asked for training on Winrunner Request to get training approved!

Case Study Functional Decomposition § Crafted functions from Demo Script to create “Time. Saver” utilities § Wrote discrete state change functions: TRUE, Worked! FALSE, Uh-Oh! § Modular functions made maintenance manageable § Automated the common interfaces of two additional products § Proposed automating easy features first – Show. Stopper #1: Memory Leak. First 13 -hours of endurance test. – Developers stop blaming tool, now ask for ad-hoc scripts Request to attend Mercury World Conference Approved!

Estimated Return On Investment “Best Bang For The Buck”: Project Time. Savers § Automate tedious manual tasks — Create 64 Annotations - 30 Seconds — Perform 128 End to End Procedures - 30 Minute Drive Space Test — Register 128 New Patients - 30 Minute Boundary Test § These tests were small, little maintenance § Gets testers on your side, reduces their “pain” § Scripts were easy to change into reusable functions First Quantifiable Return On Investment: $40 (1 Hour) Final estimated ROI for the company: $2 Million

Estimated Return On Investment Manu al includ Tester: $4 0 ing ov ertime Avg. hourl y and b enefit wage, s.

Estimated Return On Investment ROI Formula: Estimate Equivalent Cost of Automation How much would it cost to hire someone to do this work manually? Estimated hourly wage (Plus OT & Benefits): $40 Per Hour Average Determine hours of runtime 10 x 5 = 50 Hours per system 50 x 3 = 150 Hours per week - 10 hour runs, 5 nights a week - on 3 systems ------------ 150 x $40 = $6, 000 Extrapolated to the end of the year X 52 Weeks ------------ Four years of run time: $312, 000 $1, 248, 000

Estimated Return On Investment What would some of the worst defects found by automation have cost the company had they gotten to the field? § Case Study Examples — Show. Stopper #2: Transient Boot Up Failure – Replicated with automation and high end UPS Us — Show. Stopper #3: Data Transfer Problem unsced Functional over N T etwor est Tool to – Duplicated issue within five days. k Load issue? – Reason: Related to Network Load limitation. § We pegged the ROI of the worst detected defects by estimating the cost of a single lost customer sale: $100, 000 § Potential revenue loss from 7 Show. Stoppers: $700, 000 § Plus 1. 2 Million not paid to a manual tester: 1. 9 Million Est. ROI

Estimated Return On Investment That was then…this is now What would some of the worst defects found by automation have cost your company had they gotten to the field? — Show. Stopper #1: Credit cards fail to authenticate? – Store is closed world wide. — Show. Stopper #2: Search path leads to infinite loop? – Customers frustrated with site jump to competitor. — Show. Stopper #3: 12 -second+ response time? What – Customers abandon order if Stagin it's Dev, Q A or g Env – Is customer support being effected? iron nment ? – – – Slows down automation code development and debugging Slows down development programmers Slows down manual testers

Let's take a break! When we come back… § Tips And Tricks § Dealing with Failure § What is a Framework anyway?

What is a Framework? Framework Options § Capture/replay § Exhaustive § Function-based § Random § Data-driven § Lexical Analyzer § Keyword-driven § Hybrid

What is a Framework? § Driver § Loads & Executes Multiple Tests § Facilitates Result Reporting § Object Repository or Identification § Identifies objects by Class and Property Values § Uses Regular Expressions for dynamic values § Data § Pre-Defined § Generated § Recovery Mechanism § Browser closes § Object invalid § Error generated A framework is a set of techniques: • Reduce Maintenance • • Abstract Complexity Extend Coverage

Tips And Tricks Initial Project Design § Return To Home State Functionality — Get back to the starting point from anywhere in the application, even if it is not running. § Navigation and Population — Low level custom actions to Click buttons, Open menus, Select item from lists. § get. Valid. Object (s. Class, s. Name) — A single function that returns a valid object reference from many close matches. — Object exists (height, width & Abs_x > 0), is enabled, above underlay, closest to text. — Uses Descriptive Programming or x. Path to identify object § Tokens — Stores reusable values, such as an Order Number — Can return intrinsic values like Today's Date. § Assume Failure, Prove Success — Use a global variable & Return Codes for overall success. This creates a single exit point for script execution. § Modify, Recover, Repeat

Tips And Tricks Modular Design § Functions… but No Subs — Don’t waste time deciding which to use — Subroutines have no built in functionality for return codes — Sub File. Exists (str. File, rc) ‘ Return Code Workaround § All Functions need Return Codes — File. Exists = TRUE — IF File. Exists(str. File) THEN Process. File (str. File) § Enter_CC "Declined" Use Option Explicit in VBscript — Reduces errors due to variable misspellings § Three Level Keyword Framework — Components (Param 1, Param 2) o Keywords (Param 1, Param 2) v. All other supporting functions CLICK "Ok" get. Valid. Object

Tips And Tricks Global Variables and Dictionaries § Minimize global variables: — glo. Object: Reference set by Get. Valid. Object § Use a Dictionary Object — Global — No Naming Convention — Stores Object References — Can be Persistent! — Case Sensitive Key Names dic. Framework Key Vaue "Already. Failed" False "Next. Month" "June" "Object. Class" "Web. Button" "glo. Object" [OK Button]

Tips And Tricks Dealing with Failure dic. Framework("Already. Failed") = FALSE Function CLICK (s. Button. Name) If dic. Framework("Already. Failed") = TRUE Then Exit Function End If bln. Found = get. Valid. Object "Web. Button", s. Button. Name If bln. Found = TRUE then glo. Object. Click Else dic. Framework("Already. Failed") = TRUE End If End Function 'Sets glo. Object That's an od d nam "glo" = "Gl obal O ing conven tion? bject

Tips And Tricks Dealing with Failure dic. Framework("Already. Failed") = FALSE Function CLICK (s. Button. Name) … If bln. Found = TRUE then On Error Resume Next glo. Object. Click report. Err ("Click") On Error Goto 0 Function report. Err (s. Func) If err. Number <> 0 Then Else dic. Framework("Already. Failed") = TRUE End If … End Function Report err. Description & " in " & s. Func End If End Function

Thinking Outside The Box get. Valid. Object (s. Class, s. Text) § get. Valid. Object ("Link", "Login") 'Sets glo. Object § Three Attempts — Descriptive Programming (Exact Match) – o. Browser. o. Page. Link("innerhtml: =(. *)? Login(. *)? ") – False? Add "Index: =0" - True if there multiple objects — Search Object Collection (First Valid Match) – o. Desc("class") = "Link" : o. Desc("Name") = ". *Login. *" – Set o. Links. Collecton = o. Browser. o. Page. Child. Objects(o. Desc) — Switch Class – o. Browser. o. Page. Web. Button("innertext: =Login") – o. Browser. o. Page. Web. Element("innertext: =Login")

Thinking Outside The Box get. Valid. Object (s. Class, s. Text) § get. Valid. Object ("Link", "Login") 'Sets glo. Object § Validation – Is this object Visible? – Width > 0 – Height > 0 – Abs_x > 0 – Source_index: Appears over an underling object § This is not the List object you are looking for — List contains the string we want to select

Agile environments § Version Control — Not vulnerable to local system crash — Code Change Comparison — Merge Changes — Revert Changes

Agile environments § Continuous Integration with Jenkins — Run scripts when changes are made or at Night We ar e — Report overall Pass / Fail We sh real progra m o proce uld use sim mers: — Replaces Driver sses a i s the lar tools an App D d — Customizable evelo opers.

Secrets of Successful Automation Projects § Successful Projects Require — Modular Design, Demonstration, Low Maintenance — Plan for Common Pitfalls — Leverages Agile Processes § Automated tests — Keywords & KW-Based Components — Supporting functions — Dynamic Reusable data — Object identification — Recovery & Cleanup § Potential loss can be quantified as ROI § Training Increases ROI § Always think outside the box!

Resources § § Training Classes HP Certification classes HP Discover Linked. In Users Group Meetings On the Web § § § cqaa. org stickyminds. com sqaforums. com advancedqtp. com wilsonmar. com

Thank you! Learn more at CQAA. org Questions? QTPmgrossman@gmail. com Additional Thanks to Lee Barnes Utopia Solutions www. Utopia. Solutions. com