ModelBased Testing made easy with Spec Explorer 2010

Model-Based Testing made easy with Spec Explorer 2010 Name: Wolfgang Grieskamp Title: Software Architect Company: Microsoft Session Code: DEV 316

Modeling and Model -Based Testing

Modeling in Science and Engineering A model: Is an abstraction of the system from a particular perspective Supports investigation, construction and prediction Is not necessarily comprehensive Can be expressed as a table, graphical diagram, formal notation, etc. Ubiquitous and Universal 3

Software Modeling Styles and Notations Software Models Structure Class Diagram Component Diagram Behavior Interaction Based State Based Use Cases Guarded Update Machines (Code) Message Sequence Charts State Charts Traces/Patterns Focus of this session 4

Model-Based Testing in a Nutshell Requirements Feedback Author Model Feedback Generate Inputs (Test Sequences) Control Expected Outputs (Test Oracle) Observe Implementation Issue Verdict Feedback

Model-Based Testing with Spec Explorer

Model-Based Testing with Spec Explorer C# Model (or other. Net Language) Model Graph Explore Analyze Generate Test Suite Execute VSTT Result

Walkthrough: Modeling and Testing a Web Service Example: A Simple Web Shop Bill Web Shop Client Add. To. Cart Checkout Billing Web Shop Service Ship System Under Test Shipping

Web Shop Implementation in WCF [Service. Contract(Callback. Contract = typeof(IWeb. Shop. Callback))] public interface IWeb. Shop { [Operation. Contract] // Add item to stock of web shop void Add. Item(string item. Name, int price); [Operation. Contract] // Add item to shopping cart of user bool Add. To. Cart(string user. Name, string item. Name); [Operation. Contract] // Check out void Checkout(string user. Name); } public interface IWeb. Shop. Callback { [Operation. Contract] // Initiate billing, returns true on success bool Bill(string user. Name, int amount); [Operation. Contract(Is. One. Way = true)] // Initiate shipping void Ship(string user. Name, string item. Name); }

Understanding the Web Shop Which of the following traces is valid? (assuming an initialization of the shop with Add. Item(“wine”, 10) and Add. Item(“cheese”, 5)) 1 Add. To. Cart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”, 10)/true Ship(“Peter”, ”wine”) 2 Add. To. Cart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”, 10)/false Ship(“Peter”, ”wine”) 3 Add. To. Cart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”, 12)/true Ship(“Peter”, ”wine”) 4 Add. To. Cart(“Peter”, “wine”) Add. To. Cart(“Peter”, ”cheese”) Checkout(“Peter”) Bill(“Peter”, 15)/true Ship(“Peter”, ”wine”) Ship(“Peter”, ”cheese”) 5 Add. To. Cart(“Peter”, “wine”) Add. To. Cart(“Peter”, ”cheese”) Checkout(“Peter”) Bill(“Peter”, 15)/true Ship(“Peter”, ”cheese”) Ship(“Peter”, ”wine”)

Modeling Web Shop: Basic Approach 1. Import or declare action vocabulary in configuration script 2. Define state variables in C# model program 3. Define state update rules in C# model program 4. Define scenarios and generate state transition system

Modeling Web Shop: Action Vocabulary Adapter for the Reference to test System-Under-Test adapter method config Actions { action static void Test. Adapter. Add. Item(string item. Name, int price); action static void Test. Adapter. Add. To. Cart(string user. Name, string item. Name); action static void Test. Adapter. Checkout(string user. Name); action event static void Test. Adapter. Billed(string user. Name, int amount, bool success); action event static void Test. Adapter. Shipped(string user. Name, string item. Name); } Indicates that this action is an event raised by adapter

Modeling Web. Shop: Define Model State in C# enum Shopper. Status { Shopping, Checked. Out, Billed } class Shopper. Info { // Current status of this user internal Shopper. Status; // Items on cart so far internal Sequence. Container<string> Items; } // Foreach user shopping, contains info static Map. Container<string, Shopper. Info> shoppers; // Foreach item in the shop, contains price static Map. Container<string, int> items;

Modeling Web. Shop: Defining State Update Rules [Action(“Add. To. Cart(user. Name, item. Name)”)] static void Add. To. Cart(string user. Name, string item. Name) { Contracts. Requires(items. Contains. Key(item. Name)); Binding rule to action in action vocabulary if (!shoppers. Contains. Key(user. Name)) shoppers[user. Name] = new Shopper. Info(); } Contracts. Requires(In. Status(user. Name, Shopper. Status. Shopping)); Condition under shoppers[user. Name]. Items. Add(item. Name); which rule is enabled. Fires a state update static bool In. Status(string user. Name, Shopper. Status status) { return shoppers. Contains. Key(user. Name) State update && shoppers[user. Name]. Status == status; }

Modeling Web. Shop: Define Scenarios machine Multiple. Item. Scenario() : Config { Add. Item("Bread", 2); Add. Item("Cheese", 5); Add. Item("Wine", 10) Add. To. Cart("Paul", "Wine"); Add. To. Cart("Paul", "Cheese"); (Checkout | Billed | Shipped)* } Sequence of actions Or’ing actions in repetition (Note parameters omitted) machine Multiple. Item. Slice() : Config { Multiple. Item. Scenario || construct model program from Config } Combining scenario with C# model program

Modeling Web. Shop: Exploration of the Scenario Parameters and control flow undetermined

Modeling Web. Shop: Exploration of the Model Slice Choice point for system -under-test

Modeling Web. Shop: Execution of generated test code Model expects bill of $2, implementation bills $4

Demo The Web. Shop Example in Action

Spec Explorer @ Microsoft Open Protocol Documentation Testing

“Blue. Line” Technical Document Testing Project 222 open protocol specifications tested 22, 847 pages 66, 962 person days (250+ years) Vendor test teams Hyderabad: 250 Beijing: 100 Tool dev center in Beijing (Spec Explorer and other) ~10, 000 document bugs filed

Comparison Model-Based/Traditional Testing Vendor teams had (guided) choice which approach to use Out of 222 protocols approx. ½ modeled, ½ traditional Modeling resulted in 42% productivity gain in average (some teams reached more than 300%) Overall, modeling saved 50 person-years (12, 547 days) No special education background was required Teams of junior engineers Mostly vendors, fresh out of college

Question & Answer

Spec Explorer on the Web A Model-Based Testing tool from Microsoft Dev. Labs home page and download http: //msdn. microsoft. com/devlabs Spec Explorer team blog http: //blogs. msdn. com/specexplorer Spec Explorer forum http: //social. msdn. com/forums/specexplorer Wolfgang Grieskamp’s blog http: //blogs. msdn. com/wrwg

Resources www. microsoft. com/teched www. microsoft. com/learning Sessions On-Demand & Community Microsoft Certification & Training Resources http: //microsoft. com/technet http: //microsoft. com/msdn Resources for IT Professionals Resources for Developers

Complete an evaluation on Comm. Net and enter to win an Xbox 360 Elite!

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U. S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.