A Quick Intro to ModelBased Testing Harry Robinson

A Quick Intro to Model-Based Testing Harry Robinson, harryr@microsoft. com Test Architect Microsoft Engineering Excellence Group

Traditional Automated Testing Imagine that this projector is the software you are testing.

Traditional Automated Testing Typically, testers automate by creating static scripts.

Traditional Automated Testing Given enough time, these scripts will cover the behavior.

Traditional Automated Testing But what happens when the software’s behavior changes?

So What’s a Model? • A model is a description of a system’s behavior. • Models are simpler than the systems they describe. • Models help us understand predict the system’s behavior.

Model-Based Testing Now, imagine that the top projector is your model.

Model-Based Testing The model generates tests to cover the behavior.

Model-Based Testing … and when the behavior changes…

Model-Based Testing … so do the tests.

Model Based Testing – Overview • “Model-based testing is black box testing process for derivation of test cases from a model that describe functional aspects of the system under test and executing those test cases. ” is a partial description of Model System can be derived from can be run against see: Practical Model Based Testing- A Tools Approach, Mark Utting & Bruno Legeard Abstract Tests are abstract versions of Executable Tests

Model Based Testing Approach • Five Main Steps 1) Model the SUT 2) Generate abstract tests from the model. 3) Concretize the abstract tests to make them executable 4) Execute the tests on the SUT and assign verdicts. 5) Analyze the test results. see: Practical Model Based Testing- A Tools Approach, Mark Utting & Bruno Legeard

Why Does Model-Based Testing Work? system under test complexity model speed “… I think that less than 10 percent of most programs’ code is specific to the application. Furthermore, that 10 percent is often the easiest 10 percent. Therefore, it is not unreasonable to build a model program to use as an oracle. ” –Boris Beizer, Black Box Testing, p. 63

Benefits of Model-Based Testing • Easy test case maintenance • don’t reuse just test execution but also test generation! • Reduced costs -> more tests • Early bug detection and Increased bug count • Time savings • Start automated testing from Version 0. 1

Obstacles to Model-Based Testing • Skill set – Need testers who can build/use a model • Comfort factor – Creating a model and understanding how it is traversed to generate tests can be technical. • Expectations – Models can be a significant upfront investment – MBT will never catch all the bugs • Metrics – Bad metrics: bug counts, number of test cases – Better metrics: spec coverage, code coverage

To Learn More • Model-based testing website: www. model-based-testing. org • Books: “Black-Box Testing : Techniques for Functional Testing of Software and Systems” by Boris Beizer “Testing Object-Oriented Systems: Models, Patterns, and Tools” by Robert Binder • A real model-based testing tool: – Spec. Explorer