The Testing Pyramid Nancy Chacko About Me Working
The Testing Pyramid Nancy Chacko
About Me Working in software testing for over 10 years Director of Software Engineering in Test at Opower Founding member of DCAST (DC Agile Software Testing group) http: //meetup. com/d-cast
Objective We will explore how to get testers and developers to think about testing in a holistic manner. By working together to create the set of tests and determine where that testing belongs, we can increase efficiency, reliability and reduce debugging time for failures in the future.
Overview Testing and test engineers in an agile environment Layers of testing and the pyramid Uniting dev and test Pushing tests to the lowest layers
Analysis Design Build Test Maintain
Agile Testing Emphasis on rapid development forces a focus on testing TDD is often practiced by developers Automation is key to running fast Testers transitioning to agile can often feel lost If developers are testing, what is my role? Teams are not used to working together – the divide of waterfall can continue
Layers of Tests Unit tests Tests at the method level, often use mock objects Integration/service/component tests Tests at interfaces, often use real objects and DBs End-to-end automated tests GUI tests that need to be run for regression testing Manual/exploratory/ad hoc/GUI tests Used to fully exercise UI and explore unknowns
Typical Divide Development typically owns tests at the code level Unit tests Integration tests Testers often own the tests at the UI level Manual tests Teams sometimes work together to automate the UI tests End-to-end automated tests
Typical Divide Dev & Test Manual. Tests Test Manual Tests End-to-End End-To-End Automated Tests* Integration. Tests* Unit. Tests*
http: //watirmelon. com/2011/06/10/yet-another-software-testing-pyramid/ http: //blog. mountaingoatsoftware. com/the-forgotten-layer-of-the-test-automation-pyramid http: //benhutchison. wordpress. com/2008/03/20/automated-testing-2 -the-test-pyramid/ http: //www. informit. com/articles/article. aspx? p=1433906&seq. Num=3 http: //blogs. agilefaqs. com/2011/02/01/inverting-the-testing-pyramid/
Manual Tests End-to-End Automated Tests* Integration Tests* Unit Tests* Closer to Customer Closer to Code Test Pyramid
Uniting Dev and Test Lack of communication often leads to trouble Duplication of tests Tests performed at an inefficient layer Bugs are found later than they could be Need to break teams out of old mindsets and get them to see the other side of the picture
What a Tester Sees Usually sees a functional spec Simple login form that shows informative errors Test cases • Email address not associated/a password • Does the page work in IE 6? From facebook. com
Test Cases Email address associated with an account/no password Email address with multiple @ Email address associated/correct password Email address with no @ Email address associated/incorrect password Other malformed email addresses Email address not associated/no password Layout of the screen Email address not associated/a password Password with unusual characters Email address with a +/correct password What happens when JS is disabled? Is the email or password field still filled in after submitting the form that results in an error? Layout of the screen when there are errors displayed Does the page work in IE 6? What happens when cookies are disabled? No email address/no password No email address/a password
Michael Hunter’s “You Are Not Done Yet” From http: //www. thebraidytester. com/downloads/You. Are. Not. Done. Yet. pdf
What a Front End Developer Sees <html> <js on. Submit validate email and password are not null> <form on. Submit login> <input email> <input password> </form> </html>
What a Back End Developer Sees def login if email is nil redirect to login_form, set error_message to “Please enter email” if password is nil redirect to login_form, set error_message to “Please enter a password” hash_password = hash(password) db_password = query_db(email) if db_password is nil redirect to login_form, set error_message to “Invalid email or password” if hash_password != db_password redirect to login_form, set error_message to “Invalid email or password” set cookie, redirect to logged_in_page end
Test Cases Email address associated with an account/no password Password with unusual characters Email address associated/correct password Email address associated but password in DB is null Email address associated/incorrect password Email address associated but password in DB is empty string Email address not associated/no password Slow DB connection Email address not associated/a password Hashed password with unusual characters Email address with a +/correct password No email address
Dev Both Test Hashed password with unusual characters Email associated with an account/no password What happens when cookies are disabled? No email address Email associated/correct password Does the page work in IE 6? Email associated but password in DB is null Email associated/incorrect password No email address/no password Email associated but password in DB is empty string Email not associated/no password Email address with multiple @ Slow DB connection Email not associated/a password Email address with no @ Email with a +/correct password Other malformed email addresses Password with unusual characters Layout of the screen What happens when JS is disabled? Layout of the screen when there are errors displayed Is the email or password field still filled in after submitting the form
Uniting Dev and Test Simply consolidating lists can help reduce testing while maintaining quality levels May actually be able to increase quality – new perspectives spark new tests
Test Pyramid Dev & Test Manual. Tests Test Manual Tests End-to-End End-To-End Automated Tests* Integration. Tests* Unit. Tests*
What Layer for What Test Once a united set of tests is created, next step is to decide where to put the tests Based on what you want to verify, try and find the layer that allows you to do this with minimal interference
Push Testing As Low As It Can Go!
Benefits of Lower Level Testing More efficient: Tend to run in less time – no cost of starting a browser or user wait times More reliable: GUI tests fail inconsistently more often than unit/integration tests Easier debugging: less components are involved so root cause is faster to fine
Multiple Layers Some tests will belong in multiple places Time to start thinking about what to verify! Often the same setup will need to be verified at multiple layers for different reasons Sometimes a little redundancy is not a bad thing
Email address not associated/no password Unit/Integration GUI/Manual Verify that the “Please enter Verify that JS error message a password” error message is returned is shown and no request is sent to the server Verify that with JS disabled, the server error is well formatted on screen
How to pick a layer Think about what is actually being tested Find the layers of code that are actually involved Requires a good understanding of the architecture Best done as a collaboration between dev and test Push the testing as low as you can The higher the layer, the harder you should think about how to group tests together
Test Matrix Test Case U I G Email address associated with an account/no password X X Email address associated with an account/correct password X X Email address associated with an account/incorrect password X X Email address not associated with an account/no password X Email address not associated with an account/a password X Email address with a +/correct password X M X Is the email or password field filled in when an error screen is shown? X Does the page work in IE 6? X Layout of the screen X Email associated but password in DB is empty string X
Test Matrix Becomes the basis for testing Can easily divide work to be done now without overlaps Need to remember the verification may differ at the different layers Document may only live for the life of a few iterations A little redundancy is ok
Manual/Exploratory Testing Often undervalued in an agile environment Very useful for helping to answer the unknowns: “Does the page work in IE 6? ” May lead to new tests to automate Automation is for tests you want to continually run Exploration helps ensure the gaps between those tests work
Group Higher Level Tests Think about the true goal of the tests Often, they will be layout related as opposed to core functions Don’t need to test 10 error messages, just the ones that push the boundary cases Longest message, message with unusual characters …
Test Pyramid Manual Tests End-to-End Automated Tests* Integration Tests* Unit Tests*
In Practice Need a good understanding of the underlying architecture Get you dev and testers working together Bullet point the tests that you think are valuable Determine the layer for each test Aggressively group the higher level tests together Don’t underestimate the benefits of exploratory testing Divide and conquer the work!
Thank you!
- Slides: 34
