Building an automation framework around open source technologies

Building an automation framework around open source technologies

Agenda • • Challenges Choosing an automation framework Generic Architecture How does it work? Features of an ideal test automation framework Additions to Selenium framework Other components of test automation framework Beyond test automation 12/11/2010 Datamatics Confidential for Limited Circulation Only 2

Challenges today ROI Technology Automation Tools 12/11/2010 Datamatics Confidential for Limited Circulation Only 3

ROI Facts ROI Technology Frequency of usage Coverage of Tests Reusability in the system Automation Tools Creation efforts Maintenance efforts 12/11/2010 Datamatics Confidential for Limited Circulation Only 4

Technology Facts ROI Technology Automation • Over 73% of applications developed today are web enabled Tools applications • Prevalence of Firefox, IE, Safari and Chrome as the major browsers, with multiple versions • Incompatibilities between these browsers primarily due to javascript implementation • Testing needs for Web Applications is multiplied due to permutations of these challenges 12/11/2010 Datamatics Confidential for Limited Circulation Only 5

Test Automation Facts ROI Technology • New requirements are introduced on a daily basis Automation Tools • Testing window is getting smaller day by day • Test automation can never replace human testers • Not all test can / should be automated • Test automation is a supplement to help / boost overall testing • Use test automation for area that are least prone to changes and yet require frequent tests 12/11/2010 Datamatics Confidential for Limited Circulation Only 6

Test Automation Tool Facts ROI Technology • Only people that claim 100% automation • Learning curve is very higher with every variant • Product support for emerging technologies • Extendibility 12/11/2010 Datamatics Confidential for Limited Circulation Only Automation Tools 7

Myth#1: Test automation is a goal never reached • Automation is a project on top of our goal list • Many testers intimidated with test automation • Developers don’t want to learn test automation • GUI automation is hard so lets not do automation • Automate everything and fire the testers • For some - panacea and others - waste of time Automation testing is an investment – will take time and effort to bring to fruition 12/11/2010 Datamatics Confidential for Limited Circulation Only 8

Myth#2: Record / Playback • Main feature recommended by any test automation tool vendors – Capture user actions and later on play them back for regression testing. • Simple steps to follow – – – Design test case in test automation tool Use capture feature to record user actions Enhance script with verification points Add delay and wait state points for synchronization Playback and observe results in log file 12/11/2010 Datamatics Confidential for Limited Circulation Only 9

Record / Playback Myth • Learning’s – – – Scripts resulting in hard-coded values Costs associated with maintenance is astronomical Scripts are often not reliable (pop-up, messages) If tester makes a mistake – re-record the script If application changes – re-record the script Scripts cannot be recorded until software works Nothing is ever tested by this approach 12/11/2010 Datamatics Confidential for Limited Circulation Only 10

Test Automation using Open Source Framework Automation Approach Technology Layer Automation Tool 12/11/2010 Datamatics Confidential for Limited Circulation Only 11

Choosing a test automation approach • Test script modularity • Test library architecture • Keyword or table driven testing • Data driven testing • Hybrid-Keyword & Data driven both. 12/11/2010 Datamatics Confidential for Limited Circulation Only 12

Choosing a technology layer • • Platform Independence Execute unattended runs Distributed test execution Extensible and Customizable Ease of debugging and logging Status reporting Repeat tests as needed (CI) Version control 12/11/2010 Datamatics Confidential for Limited Circulation Only 13

Generic Architecture Data Manager Test Manager Automation Tool Version Control Test Artifacts Application Under Tests (AUT) 12/11/2010 Datamatics Confidential for Limited Circulation Only Integration (CI) 14

Choosing a Test Automation Tools • • • • • Abbot Java GUI Test Framework Anteater Apodora Arbiter Autonet Auto. Test. Flash Avignon Blerby Test Runner Canoo Web. Test Capedit Concordion Crosscheck csvdiff Cubic. Test Cucumber DBFeeder Db. Fit 12/11/2010 • • • • • Eclipse TPTP • EMOS Framework • Enterprise Web Test • Expect • expect-lite • Feed 4 Test. NG • Frankenstein GITAK • Harness • httest • Id. MUnit • Imprimatur • It. IN - Infopath • testing in. Net • ITP • Its. Nat, Natural • AJAX • Web. Driver Web. Inject Web. Tst ivalidator • Jacobie • Jameleon • j. Diff. Chaser • Jemmy • JFunc: JUnit Functional Testing • Extension • JSystem • Latka • Linux Test Project • Logi. Test • LReport • MActor • Marathon • Maveryx Max. Q • mi. Tester for SIP • • Datamatics Confidential for Limited Circulation Only SIP Testing Tool Mockito Moz. Unit OLVER Om org. tigris. mbt PAMIE Pounder pywinauto QAliber QAT QMTest Rasta Robot Framework Sahi Samie Sauce Labs • • • • • Selenium Sharp. Robo soapui STAF Solex SWAT SWTBot Systin T-Plan Robot tclwebtest Test. Plan Text. Test Tomato Toster Watij Wati. N Watir WET 15

Test automation tool 12/11/2010 Datamatics Confidential for Limited Circulation Only 16

How does it work? 12/11/2010 Datamatics Confidential for Limited Circulation Only 17

Why Selenium? • • Support for Windows, Linux and Mac OS Supports browsers – Explorer, Firefox, Safire Supports AJAX calls Non-developers can develop test cases Graphical Selenium IDE Export test cases as JUnit test cases When test fails, Selenium can send an email w/ screenshot Selenium Grid enabled test execution on multiple machines in parallel reducing overall execution time 12/11/2010 Datamatics Confidential for Limited Circulation Only 18

Features of an ideal test automation framework Scripting • Script Recording • Parameterization • Control Statements • Database Integration • Modal Dialog Support • Keyword Support 12/11/2010 Execution • Across Multiple OS and Browsers • Across Multiple machines in parallel • Ability to recover testing from failures Analysis • Detailed Reporting for failures • Snapshots on failures • Multilayer Reporting Datamatics Confidential for Limited Circulation Only 19

Features Missing within the selenium suite Scripting • Script Recording • Parameterization • Control Statements • Database Integration • Modal Dialog Support • Keyword Support 12/11/2010 Execution • Across Multiple OS and Browsers • Across Multiple machines in parallel • Ability to recover testing from failures Analysis • Detailed Reporting for failures • Snapshots on failures • Multilayer Reporting Datamatics Confidential for Limited Circulation Only 20

Additions to Selenium framework Keywords Datasheets Business Components Reusable Framework Utility / Generic Functions Reporting Functions Recovery Functions SELENIUM Application Under Test (AUT) 12/11/2010 Datamatics Confidential for Limited Circulation Only 21

Features of an ideal test automation framework Features Best found in Scripting Phase Script recording Selenium IDE Parameterization Datasheets Control Statements Implementation specific Database Integration Utility / Generic Functions Modal Dialog Support Utility / Generic Functions Keyword Support Keywords 12/11/2010 Datamatics Confidential for Limited Circulation Only 22

Features of an ideal test automation framework Features Best found in Execution Phase Across Multiple OS and Browsers Selenium RC Across Multiple machines in Parallel Selenium Grid Ability to recover testing from failures Recovery Function Analysis Phase Detailed Reporting for Failures Selenium Snapshots on failures Selenium Multilayer Reporting Function 12/11/2010 Datamatics Confidential for Limited Circulation Only 23

Generic Architecture Data Manager Test Manager Automation Tool Version Control Test Artifacts Application Under Tests (AUT) 12/11/2010 Datamatics Confidential for Limited Circulation Only Integration (CI) 24

Continuous Integration • • Apache Continuum Apache Gump Beebox Bitten Buildbot Cabie Cerberus Control Tier 12/11/2010 • • Cruise Control c. Dash Draco Hudson Jhbuild Lunt. Build Sin Tinderbox Datamatics Confidential for Limited Circulation Only 25

Version Control • • • Aegis Ar. X GNU arch Bazaar Bonsai Cervisia Codeville Cogito CVSNT 12/11/2010 • • • Darcs Fossil Git Subversion Libre Source Mercurial Monotone Repo CVS • • Open. CVS Open. Grok Revision Control System Apache Subversion SVN Tortoise Vesta DCVS Datamatics Confidential for Limited Circulation Only 26

Looking beyond automation testing • Test reports are automatically prepared • Defects are automatically captured in defect tracking system • Test metrics are updated automatically • Traceability matrix is updated automatically • Select test cases automatically based defect status updates • Calculating code and test coverage automatically • Suggesting improvements in test coverage 12/11/2010 Datamatics Confidential for Limited Circulation Only 27

Thank You!!!