Modelbased testing of complex manufacturing systems A case

Model-based testing of complex manufacturing systems: A case study 10 th Dutch Testing Day, October 8, 2004 Niels Braspenning Systems Engineering Group, Eindhoven University of Technology

Outline • • ASML and Tangram Model-based testing (MBT) MBT framework Case: ASML laser subsystem Conclusions Future work Questions 2

ASML: TWINSCAN Key figures: Supervisory Control (SUN) 10 -15 sub-systems (Power PC) Lower level CPUs (firmware) 50 processors 400 sensors, 500 actuators, 12, 5 MLOC Language: C (Java, Python, Matlab) Presentation ‘Testing High Technology Developments at ASML’ by Tom Brugman, Dutch Testing Day 2003 3

effort Tangram Development activities effort time Test Shipment date Development activities Test time Shipment date 4

Model-based testing (MBT) • Testing is an operational way to check whether a system implementation is correct • There is a need for unambiguous specifications and for test automation MBT: – Specification is described in a formal model (unambiguous) – Tests can be generated from this formal model (automation) • However: – Unambiguous specification takes a lot of time (thus modeling also) – Current (worldwide) automatic testing covers small and specific test domains 5

MBT framework documentation, mental models System interface access formal, suitable input for test tool unambiguous description of correct system behavior automatic generation and execution of tests 6

Case: laser subsystem communication laser beam • Functional black box testing of laser subsystem • Objectives: – Show applicability of MBT with Tor. X (RU/UT) within ASML – Show usability of (TU/e) specification models for MBT – Investigate limitation/shortcomings of the approach 7

Case: approach ASML docs Specification model ( ) Specification model (Promela) E EP model validate/verify Mental model translate validate/verify L LP Spin ert Informal specification nv verification properties co Test environment TDRV = model LT Tor. X Test rack Test tool Laser = tooling = ASML Test model (Trojka, laser only) SUT 8

Case: informal specification • Interface specifications • Operational sequences • State diagrams (Confidential) 9

Case: specification model • Environment process: – Closed system – Specific traces for analysis • Laser processes: – IO: handle communication – LS: process commands, execute actions, create responses – CLS: hold current laser state • Error handling • Configurable behavior – Initially: Cymer ELS 7600 with laser state behavior only 10

Case: different models • – Simulation only – Modeling expressivity (data, time, functions, stochastics, hybrid) – Reasonably easy to modify/configure • Promela – Simulation and verification – Less modeling expressivity (workarounds) – Less easy to modify/configure • Trojka – Testing only (open system) – Same characteristics as Promela 11

Case: results 1/2 • Validation and verification – Mainly good weather (operational) behavior – Simulation with and SPIN – Model checking with SPIN • Testing – Also bad weather (exceptional) behavior – Discrepancies are detected by Tor. X, with different sources: • Documentation, model incompleteness, tooling, SUT 12

Case: results 2/2 13

Conclusions • Proof of concept delivered: – Applicability of MBT with Tor. X within ASML – Usability of specification models for MBT • Shortcomings/limitations: – – Manual translation from to Promela Tested functionality is limited due to limited interface access Problems with data and timing Error injection in SUT is not possible 14

Future work • Laser case: – Test more functionality – Test real laser • Tangram: – – Direct connection between and Tor. X Extensions for data, timed and hybrid testing Using simulation models for (early) integration testing Research in test strategy, test infrastructure, and model-based diagnosis 15

Questions/discussion 16