Automating the Analysis of Simulation Output Data Katy

Automating the Analysis of Simulation Output Data Katy Hoad (kathryn. hoad@wbs. ac. uk), Stewart Robinson, Ruth Davies, Mark Elder Funded by EPSRC and SIMUL 8 Corporation

The Problem • Prevalence of simulation software: ‘easy-to -develop’ models and use by non-experts. • Simulation software generally have very limited facilities for directing/advising user how to run the model to get accurate estimates of performance. • With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

3 Main Decisions: • How long a warm-up is needed? • How many replications should be run? • How long a run length is needed?

Enter Analyser AUTOMATIC SIMULATION OUTPUT ANALYSER Replications Trial Calculator Warm-up analyser Warm-up period specified Replication s or one long run? Recommend number of replications One long run Run-length calculator Recommend run-length EXIT Analyser

Warm-up Analyser • MSER-5 most promising method for automation – Performs robustly and effectively for the majority of data sets tested. – Not model or data type specific. – No estimation of parameters needed. – Can function without user intervention. – Quick to run. – Fairly simple to understand.

MSER-5 warm-up method MSER-5 test statistic 0. 018 Estimated warm-up period Estimated truncation point, Lsol Rejection zone 0. 016 0. 014 Test Statistic 6 Output data (batched means values) 5 4 0. 012 0. 01 3 0. 008 2 0. 006 0. 004 1 0. 002 0 0 0 50 100 150 200 250 Truncation Point 300 350 400 Batch Means 0. 02

Heuristic framework around MSER-5 Includes: • Iterative procedure for procuring more data when required. • ‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated. • Graphical feedback to user.

Trial calculator

Confidence Interval Method with ‘look-ahead’ Precision > 5% Precision ≤ 5% 95% confidence limits Precision ≤ 5% Cumulative mean, f(k. Limit) Nsol 1 Nsol 2 + f(k. Limit)

Run length calculator • Batch Means Method. • Want a robust automatable method that estimates the minimum run length needed to achieve a required precision in the output point estimator. • Currently investigating literature and testing methods.

Enter Analyser AUTOMATIC SIMULATION OUTPUT ANALYSER Replications Trial Calculator Warm-up analyser Warm-up. BEING period specified IMPLEMENTED IN SIMUL 8 Replication s or one long run? Recommend IMPLEMENTED number of replications IN SIMUL 8 One long run Run-length calculator Recommend run-length EXIT Analyser

ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (Auto. Sim. OA) project (http: //www. wbs. ac. uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D 033640/1). The work is being carried out in collaboration with SIMUL 8 Corporation, who are also providing sponsorship for the project. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School SW 08