Reinventing the Testing Role Jon Bach Quality Evangelist

Reinventing the Testing Role Jon Bach Quality Evangelist, e. Bay

3 Themes 1) What are the meanings and purposes of roles on projects? 2) What are the potential roles of skilled testers in projects, no matter the job title? 3) How can you create and maintain a testing culture? 2

Scenario: A “Never Again” Site Incident Pick a role you’d like to play … • PMs: 1 idea to design differently • Development: 1 idea to prevent issues • Testers: 1 idea to reveal issues • VPs: What is your internal messaging? • Executives: What is your external messaging? • Analysts: How to know if we’re succeeding? • Trainers: How will you keep the memory alive?

PM Role: Solutions 4

PM Role: Solutions • Meet/work more closely with the development teams and product delivery • Design with fault tolerance in mind • Understand that quality needs to be designed in from Day One, not tacked on at the end • Assemble a group of cross-industry experts to detect and defend against such problems in the future • Identify root cause of what happened • Create a team of representatives across all teams to collaborate and identify 3 proposals with risk and cost benefit; proposals to prevent issues from re-occurring; present to stakeholders for decision • Keep up with latest platforms used by internal IT • Make sure Ops are prepared about upcoming new functionality and how we can start monitoring that area of the application even before the release is done • Do team-building exercises to demonstrate the benefits of collaboration 5

Developer Role: Solutions 6

Developer Role: Solutions • Follow TDD and write more unit tests • Make sure there are tests for 3 rd party dependencies • Display test results in a big dashboard to all • Wear “tester hat” during implementation • Create a “watchdog” service to alert, restart, reboot • Automatically shut off lower priority system functions to divert more resources to critical functions • Automatically throttle memory usage, dump unused to disk • Peer review and test my code before sending it to test team • Incremental peer reviews, not just at the end • Use ATDD with test automation • Find the root cause of the issue, not just the surface failure • Use BDD 7

VP Role: Solutions 8

VP Role: Solutions • Say: “We will not make release decisions in a silo. ” • Say: “We will frame the process where we review risks as a team to make decisions” • Say: “We will never make date promises to our clients without buy-in from Development. ” • Reward the QA teams that found the issue at next conference All-Hands • Ask vendors to have troubleshooting and backup recovery plan • Challenge the teams: how can we restore faster, root cause before it happens again? • Say “I’ll get this done, ” then give the teams anything they need (experts, diagnostics, pizza) • Emphasize quality over quantity: “I would rather deliver NOTHING before delivering a defective product!” • Say: “Let’s get together for lunch today and brainstorm what we can learn from this critical incident” • Assemble a team of experts to focus on the problem. In the meantime, come to me or your manager with concerns. • Show you are working with other VPs and stakeholders • Elicit suggestions to empower change 9

Trainer Role: Solutions 10

Trainer Role: Solutions • Empathize with the user • Teach where the “Report a Problem” link is • Teach users how to be aware of their data contexts and where and how to note data metrics • Incorporate the issue into training as an example of what types of issues could happen that have nothing to do with the project • Tell more stories around bad things that happened • Emphasize root cause analysis for bad stories that are told 11

Test Role: Solutions 12

Test Role: Solutions 13

Test Role: Solutions • Weekly “Test Training” reminder session of 30 minutes • More real-life user (and-misuser) scenarios • Break out of “we do it this way” • Review test coverage across teams • Focus on critical workflows • Root cause analysis to learn from the incidents • Closer cooperation with Operations to know what changes in the environment happen from Test to Prod • Continuous demos to business during development • Map dependencies so we can refer to them when planning test cases • Put QA environment on solid state drives to avoid unnecessary system latency and avoid risk of testers not testing due to lack of patience • Automate the passing of random data to APIs and document the results, looking for failure points 14

My roles: 1995 – 2010 • 1995 – Tester: Microsoft Bookshelf (contractor) • 1996 – Tester: Contact. Builder • 1997 – Lead: Microsoft SMS 2. 0 • 2000 – Lead: Satisfice, Inc. • 2002 – Tester: Microsoft Flight Simulator • 2003 – Intellect Manager: Quardev, Inc. • 2009 – Test Manager: Lexis. Nexis • 2010 – Consultant: Quardev, Inc. (contractor) 15

My e. Bay career: 2011 to present 2011 Director QE Search (Front End) 2012 Director Live Site Quality reorg 2013 2014 QE/PM End-to-End Solutions QE/PM (Commerce. OS) reorg 2015 Quality Evangelist 2016 Engineering Excellence lead

Your role is like a feature

Director: Live Site Quality Cultivate Testing Culture 18

Call customers who participate in our NPS callback program Speak at VOICES meet-ups Go on the “radio”? 19

e. Bay Radio is a weekly podcast for Customers hosted by Jim Griffiths… ---------------------------------------From a customer: “I have attached the screen shot for you. Try as I might, I can't get the selling reminders to go to the top. The arrows don't work, but I can move sold, unsold and active selling. I even set the default to summary, but no luck. FYI- I am using Safari on my trusty Mac. Also, the default font size is too big. When I zoom it out, it's too small. It was a better size the way it was before. I sent a message to the team through the 'what do you think' for that page. ” 20

P 1 Bugs found this week (with Root Cause Analysis) Overdue bugs that need fixing What bad things happened in the last 24 hours? 21

Drive issues that threaten the notion of “site readiness” Make sure e. Watch is in the loop on issues Find out who *really* owns this bug 22

Cultivate Testing Culture Coordinate teams in end-to-end testing Host bug bashes that focus on exploration Manage crowdsourced testing efforts in production 23

Cultivate Testing Culture Open-Book Testing The act of creating open-ended questions such that you learn about the product 15 minutes at a time

“Open Book” charters for e. Bay What item for sale is nearest to you location? Submitted the most and sold the least? What item appears most with no bids? What are the most recent items posted for any given category? Which category has the most items? Which search strings create null queries? What are 3 examples of broad queries vs 3 examples of ambiguous queries? What queries create invalid results? How do you find "sold" items? What is the most common item? What is the most bizarre item? What is the most expensive item? Is it possible to restrict results to one category without the categories field? Which seller has listed the most items which have not sold? What search string takes the longest to return results? Is there a search that works differently on different browsers? What was the highest bid for an item in support of any charity? Find an item that the seller has no business selling! Find an item you find only on e. Bay, not Amazon. What item could you find that had the most bids? How many e. Bay stores are there? What is the oldest object (not item) for sale on ebay? Are results different if you are logged in? Try a search on ebay. com and then try the same on on your mobile. What are the 3 major differences? What item has the greatest difference between the highest and second highest bids? What user has bid the most but won the least? What item has been on e. Bay the longest? Is there a way to tell the age of the item ID? Who has sold the most items? Is there a way to find the most commonly misspelled item? What item can be found in the most categories?

Rapid Testing Intensive July 2012 • 80 testers from around the world – in-person and online * 4 days of testing e. Bay Motors: Tire Finder / Wheel Center • How to jump into a product and find bugs while you learn all about it, while working toward "deep coverage" testing. • Charter-based exploration, modeling, usability testing, intense coverage to deliver notions of quality and risk 26

When site visitors click this… 27

They see this… 28

A large seller wants to have a “Flash Sale”. Are we ready? 29

30

31

QE/PM: Commerce. OS API Portfolio Login Identity Catalog Messages Cart Search List Coupons Shipping Graph Pricing Experiment Campaigns OMS Evangelize the framework standards Make those standards easy to use Host office hours Triage bugs in the platform, drive closure 32

Quality Evangelist When site visitors click this… 33

Engineering Excellence lead From the book “Explore It!” by Elisabeth Hendrickson

Testing Culture Executive: “We value these kinds of things. ” Culture Staff: “I value these kinds of things. ” 35

Testing Culture We. Spire screenshot 36

Priorities Executive: “Do this. ” Your Influence Your Staff: “We have these others things to do. ” 37

Influence 38

Heuristics of quality Bad Buyer Experiences Gross Merchandise Volume (+) Net Promoter Score Earnings Per Share Social Media chatter Morning Mission Report e. Watch’s Most Wanted Re-rolls and Rollbacks Call volume Mean Time to Detect Mean Time to Mitigate 39

Heuristics of quality Overdue bug fixes Survey results Tell Us What You Think Metrics: Do. D / Wow / Mo. M / Yo. Y Key Flow Test Results e. Commerce Bytes Twitter / Facebook posts “Be The Customer” Discussion Forum topics Calls to e. Bay Radio Emails from VOICES 40

PRESENTATION TITLE GOES HERE 41