Traditional Testing The Silent Killer of Dev Ops

Traditional Testing The Silent Killer of Dev. Ops Ingo Philipp

Businesses must continuously exploit digital technologies to both create new sources of customer value and increase operational agility in service of customers. © Forrester Research

Across industries, companies face the challenge of software-led transformation So Get Disrupted Digital Disruption Disrupt ftw ar e

Fail Fast ›› Fail Cheap ›› Fail Forward Build Solution Identify Needs Receive Money Plan Code Version Build Accelerate Value Exchange Provision Test Deploy Monitor

6+ Big Bang Waterfall Implementation Testing Acceptance Deployment an ge Design ch Requirements n to re lu ac vo n tim e to tio Incremental Re 4 Rational Unified Process Inception 2 Elaboration Construction Transition Following a Plan Contract Negotiation Copious Documentation Processes & Tools Rigid Rules Agile Event Horizon Flexible Framework Agile Individuals & Interactions Scrum, Kanban Development on Dev. Ops Digital Singularity Development Response Responding to Change i lut Continuous Working Software Customer Collaboration Evo 1 Operations

It’s not the strongest that survive, nor the most intelligent, but the one most responsive to change Charles Darwin Because software is not written, it is rewritten

Present Traditional Testing The reliance on manual testing is the top technical challenge in app development 80% Manual Testing World Quality Report, 2015/16 Test automation requires developers 20% Automated Testing UI API Load

Present Traditional Testing Future Continuous Testing Exploratory Tests Automated UI Tests 80% Manual Testing Load Tests +85% Automated Testing API Tests 20% Automated Testing Orchestrated Service Virtualization

Sprint Start Time Efficiency 100 % Gain + 4 x Creation + 6 x Sprint End API Start testing early, shift left test automation Degree of Completion Maintenance + 20 x Execution UI 0%

On average, organizations require access to Testing Present 33 systems for development or testing. Welcome to the tester’s hell. 18 Average # of systems with unrestricted access 96 % of testers have restricted test lab access Enterprise system landscapes are alike disease gene networks. *voke, market snapshot reporton service virtualization - 2012

Automation is the dictate of today. It doesn’t make testing easy, it makes testing possible. Lesson Learned © Wolfgang Platz

~100 Billion Billion Combinations A B Visiting London = 50

Testing harder isn’t the answer, testing smarter is! Lesson Learned © Wolfgang Platz

Have the right test cases. Do the right things. Low Risk Critical Limit The time needed for testing is infinitely larger than the time available. Medium 20 % Test Cases Risk High Risk 80 % Business Risk Coverage

Usability Issue Scalability Issue Reliability Issue Accessibility Issue Risks Understandability Issue Coherence Issue Security Issue Performance Issue Functional Issue Stability Issue Convenience Issue Testability Issue

Testing is exactly like washing a pig. Because it’s messy. It has no rules. No clear beginning, middle, or end. It’s kind of a pain in the ass, and when you’re done you’re not sure if the pig is really clean or even why you were washing a pig in the first place. © Luke Sullivan

That’s what we could possibly know That’s what we know Closing the gap between what we know and what we don’t know That’s what we check Problem Testing Purpose Knowledge Gap © Richard Bradshaw Confirmatory Testing Exploratory Testing

© M. Bolton, C. Kaner, J. Bach Change Detector Problem Detector High Risk Coverage Low Risk Coverage Easy to scale because it’s parallelizable Hard to scale continuously because it relies on humans Low Information Value High Information Value Monitor Known Risks Analyze Potential Risks Mechanical Testing Intelligent Testing Repeat what you have already learned Confirm what you already know Learn something new Focus on the things you don’t know Process pre-defined data in pre-designed steps Create new test ideas based on what you have learned Confirmatory Testing Exploratory Testing « Demonstrate your depth of knowledge » « Demonstrate your breadth of knowledge »

© M. Bolton, C. Kaner, J. Bach Change Detector Problem Detector Low Risk Coverage Hard to scale continuously because it relies on humans High Information Value Learn something new Analyze Potential Risks Focus on the things you don’t know Intelligent Testing Evaluate a product by applying algorithmic decision rules to specific observations of a product Create new test ideas based on what you have learned Checking Exploratory Testing « Requires Processing » « Demonstrate your breadth of knowledge »

© M. Bolton, C. Kaner, J. Bach Change Detector Problem Detector Evaluate a product by applying algorithmic decision rules to specific observations of a product Evaluate a product by learning about it through exploration and experimentation Checking Exploring « Requires Processing » « Requires Thinking »

Verify through Pay attention to Create Follow Instructions Deviations Test Cases Procedure Examine Factory Requirements Process Checking « Requires Processing » © Rich Rogers Problem Detector Evaluate a product by learning about it through exploration and experimentation Exploring « Requires Thinking »

Verify through Pay attention to Investigate through Pay attention to Create Follow Instructions Deviations Experiments Oracles Test Cases Procedure Test Ideas Clues Examine Factory Examine Adaptive Requirements Process Checking « Requires Processing » © Rich Rogers Risks Investigation Exploring « Requires Thinking »

Checking Efficient Confirmatory Testing Agile Testing Equation + Exploring Effective Exploratory Testing = Testing Thorough Testing © Elisabeth Hendrickson

If this doesn’t make an impact, then its absence doesn’t make a difference Our Proof © Trey Smith

Testing Present 100% Manual Testing Testers Coverage 11 ? Manual Test Cases Unknown Execution There’s a way to do it better. Find it! Project Charter 0% Automated Testing 4755 Manual 10 Weeks

Risk Coverage Optimization Test Cases Test Data Management Testers UI/API Test Automation Coverage Continuous Integration & Distributed Execution Coverage Execution Test Case Count 7% 92 % Automated 11 Manual 37 ? % Business Risk Unknown 89 % Business Risk 64% API Tests 53 % 34 Business Risk Minutes Smoke Testing Test Cases Execution 1193 4755 10 5 2. 5 48 8 75% Redundancy 50% Effort Test Data Manual Weeks Hours Multiple Single Agents

Risk Coverage Optimization Test Cases Test Data Management Production Defects UI/API Test Automation Coverage Continuous Integration & Distributed Execution Coverage Execution Test Case Count 7% 89 % 92 % Automated 64% API Tests Test Cases Critical Defect Reduction 1193 Manual Minutes Smoke Testing Execution 5 2. 5 8 Weeks Test Case Design & Exploratory Testing 34 Business Risk 72 % 53 % Weeks Hours Multiple Agents

The future doesn’t just happen, it gets happened, so make it happen Our Conclusion © Paul Tagliablue

? Questions Because answers exist only to questions. . . Ingo Philipp