Snake oil in Test Automation 24112018 NTT Data

Snake oil in Test Automation 24/11/2018 NTT Data România Robin Molnar © 2018 NTT DATA Romania SA

A big Thank you to our Sponsors! © 2018 NTT DATA Romania SA 2

Feedback is a way to say Thank You! © 2018 NTT DATA Romania SA 3

Disclaimer © 2018 NTT DATA Romania SA 4

Snake Oil? 1. Isn’t this bad? (Please say No!) 2. What is this anyway? Hint: traps 3. Effect: test automation seems to become more and more difficult! 4. Why is this counter-intuitive? “You're giving me a million reasons” 1. Frameworks (Protractor, Spec. Flow, Serenity) 2. Tools (Appium, Selenium, Robotium) Wait, what? How can these be bad? They aren’t’. Seriously, test automation becomes more and more difficult because more and more the success of test automation depends on factors outside of test automation control. © 2018 NTT DATA Romania SA 5

First, Point-and-Shoot Test Automation © 2018 NTT DATA Romania SA 6

First, Point-and-Shoot Test Automation (Continued) • Easy • What is Point-and-Shoot? Hint: test automation oriented towards the application • Straightforward, Page Object Model (Pattern) • Fancy • TDD > test automation • TDD = process to maintain a tight quality control • BDD • Domain-Specific Language • Automation without testers • Basic form of Quality Control © 2018 NTT DATA Romania SA 7

First, Point-and-Shoot Test Automation (Continued) BDD is TA because BDD is Test and because BDD is Automation Point-and-Shoot automation shortcomings: • Takes more time • Ignores human errors To avoid this: • Top-to-Bottom software development • Bottom-to-Top software development • Sometimes: Bottom-Up software development (also called Point-and-shoot. ME) © 2018 NTT DATA Romania SA 8

First, Point-and-Shoot Test Automation (Example) Scope Methodology Test Automation Risk Mobile App Top-to-Bottom Functional 3 rd Party API Solution: Quality Gates for API 3 rd Party API • Test • Qualify Mobile App • Test • Qualify Scope Methodology Test Automation Risk Mobile App Top-to-Bottom Functional 3 rd Party API Any Smoke? Regression? © 2018 NTT DATA Romania SA 9

First, Point-and-Shoot Test Automation (Counter Example) Scope Methodology Test Automation Mobile App Top-to-Bottom Functional Backend Risk Ours, stable Too many bugs The theory looks nice, but the practice is underwhelming. Solutions: • Process: • Shift-Left is too to Left of the Process, maybe it’s even outside of it • Fix the code quality of the app • Split the Functional testing of the app into something smaller, more manageable • Rotate team members every odd number of iterations • Reduce defect clustering • Ramp-up ownership • This can be done for Dev and Test Automation engineers, in parallel • Carry a big stick and sweet carrot, some will bite. • In reality, TA engineers are contempt to only write tests for the same features. Hint: It’s a trap! © 2018 NTT DATA Romania SA 10

They see me testin’, they hatin’ One of the most frequent traps in test automation is focusing on Dev and Process, because automation is a piece of cake. Made of lava. And you are serving it atop a volcano. During an explosive eruption. While you are not hungry. Blindfolded. © 2018 NTT DATA Romania SA 11

Second, A Better Idea I have Effective Automation Calamity Zone • Innovation Learning Zone • CI/ CD • Dev. Ops Fear Zone Comfort Zone © 2018 NTT DATA Romania SA 12 • TDD • BDD • Point-and-Shoot

Second, I have a Better Idea (Requirements, Simplified) Requirement Point-and-Shoot BDD TDD CI CD Dev. Ops Atomic Tests Rock-Solid Test Data Management ± Atomic Suites ± ± Consistent across environments ± ± ± Error-proof recovery scenarios ± ± © 2018 NTT DATA Romania SA 13

Second, I have a Better Idea (Visualized) Dev Tes t Tes t t Tes t © 2018 NTT DATA Romania SA Deploy Tes t t Tes t 14 Tes t

Second, Test Automation is Just an Equation in a System of Equations • Measure Quality • Maintenance Test Automation • Measure Quality • Maintenance Test Data Management Product Evaluating the quality of the maintenance process is a killer. © 2018 NTT DATA Romania SA 15 CI Process • Measure Quality • Maintenance

Measure Iteration Z+1 Iteration Z Measure M t n i a a en c Iteration Y+4 Measure By measuring the quality of the process, a need will arise to do additional work, Failed Tests on i t a m o t Au ss e c Pro Iteration X+1 Iteration X 16 Bugs TDM Errors Iteration Y+1 Iteration Y Tests in Error Debt Iteration Y+2 st e T © 2018 NTT DATA Romania SA Process Overhead Iteration Y+3 Iterations Can be days, mostly they are execution runs of automation, as agreed by team n ce o Pr s es TA Process • • • Execute Fix Report Re-Run Cry Recovery Scenario Errors Other issues

Second, Example V. Bad Average Good V. Good Perfect 0. 5 0. 6 0. 7 0. 8 0. 9 1 Half-done Not needed Test Complexity TDM Maintenance Robustness 0. 8 1 0. 9 0. 72 0. 8 0. 9 0. 57 Robustness formula: Test Complexity x TDM Process Complexity x Maintenance Process Complexity = Risk The more layers we have, the more multiplications are involved, the worse it gets. © 2018 NTT DATA Romania SA 17

Third, about the Snail that went Woosh! The speed of test automation change, thus the speed of updating automation to reflect the desired reality (specifications) relies on the quality and the speed of your agile process. Process But both, the quality and the speed of the agile process are outside of the test automation scope. Team Test Automation © 2018 NTT DATA Romania SA 18

Third, I have this wee Tool/ Process/ Brilliant Idea Main reasons for test automation technical debt: • Too much work • Too complex test cases • Too standard a team (3: 1 ratio) The magic 3: 1 ratio works for Point-and-Shoot automation. Elsewhere, it’s called Dark Magic and it’s poisonous to human’s souls. V. Bad Average Good V. Good Perfect 0. 5 0. 6 0. 7 0. 8 0. 9 1 Half-done Not needed Scenario Test Complexity TDM Maintenance Process TAE 3 Devs 0. 8 1 0. 9 Agile 0. 9 0. 85 8 Devs 0. 8 0. 7 Kanban 0. 8 2. 8 8 Devs 0. 7 Agile 0. 7 4 © 2018 NTT DATA Romania SA 19

Third, Technical Debt Factory 8 am 9 am 10 am Dev 11 am 12 pm Develop 13 pm Idle, best case scenario TAE 1: 1 Investigate Submit Fix test overnight defects (SD) cases (FTC) errors (IOS) TAE* 1: 2 IOS SD SD FTC 15 pm 16 pm 17 pm 18 pm 19 pm Push Automation TAE 1: 1 14 pm Execute FTC Lunch Write new tests (WNT) WNT D S FTC Idle, or another run from another dev Outside of business hours WNT 6 h * This is the best case scenario I’ve seen, 1 Dev, 2 TA to avoid working exaggerated hours, but the need was such, so it had to be mitigated somehow. It was good, but not good enough. © 2018 NTT DATA Romania SA 20

Third, that Escalated Quickly © 2018 NTT DATA Romania SA 21

Third, Automation a bit Cloudy? © 2018 NTT DATA Romania SA 22

Third, ain’t nobody got euro fo’ that Application Component 1 Dev Test Component 2 TAE Dev TAE Test Feature 1 © 2018 NTT DATA Romania SA Feature 2 • Application • Component Feature 3 Dev Dev TAE TAE 23 Test • Feature

Third, interlaced augmentation Application System 1 System 2 Component 1 Component 2 TAE Feature 1 Dev Component 1 Feature 2 Dev TAE Feature 1 Dev Component 2 Feature 2 TAE Feature 1 Dev Feature 2 Dev TAE Every odd layer of a multi-layer system also is doable, almost as efficient as structure mirroring. © 2018 NTT DATA Romania SA 24

Third, I want to play a game… Application Component 1 Dev Component 2 TAE Feature 1 Dev Feature 2 Dev © 2018 NTT DATA Romania SA Feature 3 Dev Minimally viable process, under reasonable conditions. 25 TAE Dev

Third, you think this is a game? How to play: 1. Start from a topology 2. Figure out What works and Why. 3. Improve the What or the Why 4. Then improve the Why or the What. DOs Try Fail DONT s Panic Quit Hint: improperly tailored test automation teams create more problems than they solve. Regarding (3) and (4), it is possible to augment your test automation team by: • Leveraging a better process: • Re-iterate the ‘game’ above • Rotate team members: • Dev features rotation per dev/ ownership rotation • Scrum master rotation within the team • Leveraging a better technology • Use abstractions as a means to reduce complexity • Use framework enablers, to allow testers focus on testing rather than coding. © 2018 NTT DATA Romania SA 26

Third, augmentation by Technology Not Augmented Declare once, use once Declare once, use many // Java-style pseudo-code // Written in Notepad++ // Probably won’t work as is private Web. Element username. Input = find. Element(By. Id("username")); private Web. Element password. Input = find. Element(By. Id("password")); private Web. Element login. Button = find. Element(By. Id("login")); @Test public void negative. Login(){ find. Element(By. Id("username")). send. Keys("wrong"); find. Element(By. Id("password")). send. Keys("pass"); find. Element(By. Id("login")). click(); assert. True("Is the login not visible? ", find. Element(By. Id("login-error")). is. Displayed()); } public void login(String user, String pass){ username. Input. send. Keys(user); password. Input. send. Keys(pass); login. Button. click(); } @Test public void Test_01(){ login("wrong", "pass") assert. True("Are we logged in? ", login. Page. Is. Visible()); } © 2018 NTT DATA Romania SA 27

Third, 50 First Tests Want Need © 2018 NTT DATA Romania SA 28

Fourth, say, I have this Awesome Tool Here, Please use it Virtually, we have all been there, asked by customers to use a specific tool. This is a trap that can be avoided by walking into it, the same way all mushrooms can be eaten at least once. Two ways to deal with it: 1. Whine and complain 2. Win To reduce the risk of this trap: 1. Educate yourself on the tool 2. Get certifications 3. Learn how to get the most out of the tool. 4. Do everything possible with the tool. 1. By now, you should have earned enough expertise with the tool 2. By now, you should have earned enough trust with the customer 5. Explain the tool shortcomings to the customer © 2018 NTT DATA Romania SA 29

Fifth, it’s only a Matter of Testability Sometimes, testability is like tasting a chocolate without unwrapping the package, good luck with that! What you can do: • Don’t use Xpath • Use j. Query selectors, if applicable • Use CSS Selectors Testing desktop apps with any of the above will, obviously, not Work, and image-based testing is not a magical solution: • UI Automation (Microsoft) • Win. App. Driver (not mature enough, Microsoft) Testability issues add complexity to an already overly complex Process, ask Dev for help. © 2018 NTT DATA Romania SA 30

Sixth, the Quantum Continuity Some aspects of CI/ CD were covered previously. Lucky us. Unfortunately, it gets worse. Figure that. For CI/ CD we need the tests, the test suite and the System Under Test to behave in a very strange way: we need everything to be stateless and stateful at the same time: The tests usually start considering a stateless application, but end up requiring a stateful application. As in Quantum Physics, in CI/ CD, the application is both, working and not working, until observed. And as in Quantum Physics, the state of the observed system is changed by the very observation we make. As Quantum Physics, CI/ CD can be extremely counter-intuitive. One day, a CI tester will win the Nobel prize in physics. © 2018 NTT DATA Romania SA 31

Seventh, Dev. Ops and the Philosopher’s Stone To be fair, Dev. Ops alleviates some of the challenges CI/ CD bring forward. But where is test automation? • Near Dev • Near Ops • Alone in the dark What can go wrong? • Focus is on the process, not the quality • Too large a skillset, requires Full Stack Test Automation Engineers • Shift-left might be too-to-the Left (even outside of the Process) Don’t do Dev. Ops unless you really have to! © 2018 NTT DATA Romania SA 32

Eight, Rise of the Planet of the RPA 1. Robotic Process Automation can be very expensive 2. It adds a complex maintenance process to an already complex process 3. Don’t buy the low-maintenance donut, there is no magic 4. RPA might pose a compliance issue, do your research first There also are pros to RPA: 1. Fun to use: Ui. Path, Leapwork 2. It may help alleviate some of the pains of Cleanup and Recovery in CI/ CD & Dev. Ops 3. When used on systems other than SUT, RPA can be incredibly robust. © 2018 NTT DATA Romania SA • Test Automation • RPA • CI/ CD Testing Process Cleanup & Recovery Process • Dev. Ops 33

Ninth, Codeless Automation and the Prisoner of Azkaban Programmatic • Most test automation frameworks Test Automation • Tricentis Tosca Codeless Test • Katalon Studio Automation • Test. Complete © 2018 NTT DATA Romania SA 34

Ninth, Codeless Automation and the Fellowship of the Ring Programmatic Test Automation Codeless Test Automation TAE selects the elements to interact with Magic Customizations are easy Customizations? Learn programming. C#. Tests are code Tests are magic recipes Tests are easy Click, click, magic Suites are code Suites are magic tomes TAE can do everything TAE must do everything Interaction with some applications is hell Interaction is like the taming of the Shrew Maintenance is hell Sometimes you find unexpected ways Sometimes, you get unexpected results Works well with the Pareto principle Only works well with the Pareto principle Everything can be fixed Can implode the entire universe Can work with CI/ CD/ BDD/ Dev. Ops Maybe CI, if you sell your soul to Sysadmin © 2018 NTT DATA Romania SA 35

Ninth, Codeless Automation and the two Towers There are many things codeless automation – which is some form of Model-based testing – can do for you, especially if you are automating simple things There also are many other things codeless automation cannot do, while also actively blocking you from doing, such as: 1. 2. 3. 4. 5. Speedy tests Custom reports Use a custom Appium server Cloud devices are, mostly, a no-go Proper interaction with the console In reality, there is no magic, the team will need someone with good programming knowledge to help overcome some of the limitations of codeless test automation. But then, again, it would no longer be codeles. © 2018 NTT DATA Romania SA 36

Conclusions There is no magic Process, by itself, is irrelevant Most things work most of the time Technology is not always the winner, on the contrary There are no absolutes There are no guarantees Don’t be afraid to try new things. The process needs to be tailored carefully © 2018 NTT DATA Romania SA 37

Q&A © 2018 NTT DATA Romania SA 38

© 2018 NTT DATA Romania SA 39