Technology Model Based Platforms Approach to Software Testing

Technology Model Based. Platforms Approach to Software Testing 22. 5. 2007 1 Company Confidential Author: Mikko Mäkinen Supervisor: Professor Heikki Hämmäinen Instructor: M. Sc. Antti Heimola

Agenda • Current state of S 60 UI testing • Background to model based testing • Model based testing tool implementation • Results of the comparison between testing methods (traditional vs. model based) • Conclusions Company Confidential 2 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

Current State of S 60 UI testing • Three different views on how the applications should work – UI designer – Software developer – Tester • Test cases are not in sync with UI specifications • A lot of man-hours is spent on testing and modifying the test cases into always changing applications – Hiring more people to perform testing is not a long-term solution – Another solution is needed Company Confidential 3 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

Model Based Testing - Background • Test cases are not created by hand, but generated automatically from a model • If the model is a thorough presentation of an application, the application can also be tested thoroughly with the test cases generated • Model must be a high-level abstraction of the application – If they are on the same level, the efforts of validating that the model works correctly are exactly the same as validating that the application works correctly Company Confidential 4 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

Model Based Testing – A New Idea • Where does the model come from? – All researches on model based testing suggest so-called test model, which is created by testers – The problem of updating the test cases is simply transferred to a problem of updating the model • Import the model from the UI specifications – No separate test models needed – The model is in sync with the specs, and so are the automatically created test cases • No suitable commercial model based test tools available to try out this idea – Implementation was needed Company Confidential 5 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

KENDO – A Model Based Testing Tool (1/3) • Model represented as a finite-state machine – Contains user- and system-states, and transitions between them • States contain test automation parameters – User – what the user does (action keywords) – System – how the system responds to user actions (verification keywords) • Transitions connect the states together – May contain probabilities • Model can be created from a scratch, or imported from UI specifications – Also possibility to export the model into UI specification Company Confidential 6 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

KENDO – A Model Based Testing Tool (2/3) • Test case generation by traversing the finite state machine – Randomly – 100 % model coverage with Chinese Postman algorithm (visit each transition once) • Generated test case sent to test engine • Test engine executes the test case on either a real device (Nokia smart phones with Symbian OS), or on an emulator – Results logged Company Confidential 7 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

KENDO – A Model Based Testing Tool (3/3) Company Confidential 8 © 2007 Nokia

Comparison of the Testing Methods • Two S 60 applications used – Image Viewer – Voice Recorder • The Symbian source code was instrumented – Measurements with decision-coverage (if-else branches, case branches in switch statements, exception catching, control transfers) • Manual testing vs. automated tests generated with KENDO Company Confidential 9 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

Results of the Comparison • Manual testing – Image Viewer, 48 man-hours, 79 % code coverage – Voice Recorder, 15 man-hours, 60 % code coverage • Model based testing – 8 hours of random-walk execution – Image Viewer, 71 % code coverage – Voice Recorder, 55 % code coverage • Several reasons for the differences in code coverage – Hand-crafted models (no suitable UI specifications available) – Manual test cases contain also non-UI tests Company Confidential 10 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä

Conclusions • On yearly basis, model based testing provides huge savings – S 60 applications are released and tested for regression bi-weekly – In the case of Image Viewer, over 1000 man-hours would be saved yearly – Also higher code coverage during the regression testing, since only 30 % of manual cases executed regularly (the whole set used in the comparison executed rarely) • Even more would be saved if the UI specifications would be formal enough to be used as a base for the models – The idea was tested and proved to work – Next step is to try to convince the Nokia organization to change the UI specification format Company Confidential 11 © 2007 Nokia Mäkinen_220507. ppt / 2007 -05 -22 / MMä