The use and abuse of simple models when

The use and abuse of simple models- when is a model not a model? 2 September 2003 Glenn Richards & Mike O’Connor

Section 1 Introduction

3 Introduction • Qineti. Q have examined a large number of logistic and engineering problems within the British Armed Forces, using 2 basic types of model: – Large and complex • source code / workings are not always visible – Simple • often made ad hoc for a specific problem by analyst

4 Aim • The purpose of this presentation is to discuss based on our experience – what is a model? – what is a simple model? – the characteristics of simple models – software packages – conclusions • To answer the question posed by the title – when is a model not a model?

Section 2 What is a model?

6 What constitutes a model? • A simple OR model definition “a representation of [a] system in which only those properties believed to be relevant to the problem being examined are represented explicitly” RMCS c 1970 • Perhaps not 100% valid in an age when computing has made far greater complexity achievable in a model • Reuse of existing code may be the most economical way to provide that complexity

7 Why do analysts use models? • Because they – enable the analyst to argue by analogy from the model to the way the real system is likely to behave – allow extrapolations to be made from known to unknown conditions – indicate how a real system can best be controlled and decisions taken about it

8 The role of a model in an OA study Real life ‘Implementation’ ‘Boundary conditions’ System under examination ‘Verification and Validation’ Recommended Solution ‘Abstraction’ Model(s) ‘Prediction’ Pay Off Matrix

9 The role of a model in an OA study • Many references warn against model use unless – confident that it reproduces behaviour of the real system – any limitations to the model’s usage are known • Rarely possible in defence to follow scientific method (theory, prediction, observation, revision) • Important that the model is “right” first time

10 Verification and Validation • To determine if a model is a true representation of the real life system it is necessary to – verify its workings and – test and validate the model against known data • This should include sensitivity analyses of assumptions – to determine how critical they are to model behaviour

Section 3 What is a simple model?

12 What is a simple model for? • Simple models might be used to – examine a number of subjects superficially – provide rough answers or increase understanding – examine a single subject in detail (logistics models)

13 What is a simple model? • There is no unambiguous definition • In general a simple model will – contain a relatively small number of variables and manipulate them according to simple rules – represent processes which are easy to understand • making them easy to reject as well as easy to accept! – probably apply to a fairly narrow range of problems

14 What is not a model? • If a model can be defined the question “what does not constitute a model? ” springs to mind 1) If performance data are manipulated to prepare them for input to a model, does the spreadsheet (or calculator) constitute a model? 2) If technical characteristics of a TI are processed in a spreadsheet to produce prob(detection) as a f(range), is that a model? • Many would answer ‘No’ and ‘Yes’ respectively, but where does the boundary lie? • Must a model describe something that is recognisably a system? • Perhaps there is a useful distinction between – a model (which needs validation) and – a tool (which does not)

15 Simple vs complex? - An OA example 1 • A force's water requirement on operations can be represented simply if the need is accepted – policy gives water demand for personnel and equipment – total demand the equipment needed to meet it can be determined from the ORBAT and policy • A model which can do this need not represent other battlefield processes – i. e. a simple model • But. . .

16 Simple vs complex? - An OA example 2 – if the demand for water varied with the activity of individuals, – and even more if the availability of water constrained their activity – or if the need for water were questioned • It would be necessary to represent water supply in a more complex battle model • The level of model complexity required is related to the assumptions acceptable to decision makers

17 KISS - Keep it simple, stupid! • The mantra is repeated time and time again – yet when dealing with technical issues people persist in developing increasingly complicated models – there at least 3 reasons why

18 KISS 2 1) The processes and activities that are modelled are essentially complex – as the understanding of processes increases, human nature wants to include more in any model – however, acceptable simplicity stems from genuine understanding – but, it is easier to represent detail than to know which details matter

19 KISS 3 2) Model quality is often associated with the degree to which it faithfully represents the system it is representing – simplifying assumptions are often seen as a loss, or a compromise between what we would like to do and what we can do – intuitively it is known that some form of simplification is required, but the analyst is frequently loath to accept any

KISS 4 3) The adversarial nature of procurement processes drives models to greater complexity – disgruntled stakeholders may wish to dismiss results and simplification provides an excuse • Analyst response – add the detail to the model • increased cost of ownership of the model – reject the complaint for a valid reason • requires understanding – add the feature, examine effect and remove if required • rarely done and the result is to introduce a ratchet effect toward ever increasing complexity 20

Section 4 Pros and cons of simple models

22 Discussion • Cheap – logistics studies are often for small-scale procurements where money and deadlines are tight – lack of models means simple and cheap models have to be created – possible conclusion: simple models are cheaper and quicker to produce – yet to be examined in detail: for many purposes not sensible to create a model for each study – OA on OA? : what is the most cost-effective modelling policy? – who guards the guards?

23 • Quick – A realisable benefit is the quickness of producing a result • as long as suitable data are available • simple models require little setup time and less run time • the time required to undertake sensitivity analysis (in the validation process) is reduced

24 • Easier to test, validate, verify – if no real data exists, stakeholder ‘buy in’ is used – the simpler a model the easier to get stakeholder ‘buy in’ – imperative if results are to be trusted by decision makers – need for V&V in simple models is no less – time MUST be made to achieve the appropriate level i. e. to show that the model is adequate for its purpose. – results from an unvalidated simple model are as flawed as those from an unvalidated large model • Documentation – often simple models are undocumented

25 • Easier to discard – ‘cheap and cheerful’ models • Match the model to data – build model to fit the data – often simpler to investigate boundary conditions (cf large models) as relationships are less complex – enables analysts and decision makers to concentrate on areas which make an impact on decisions • Avoiding shoehorning – answers specific question

26 • Easy to change • Choice – increase the choice of the analyst as more can be easily developed – but the interactions between phenomena examined in separate models may need to be assessed together – don’t normally have agreed architecture or shared assumptions • Problem scoping – aids understanding

27 • Transparency – if model and results self evident: answers easier to sell! • Data are fresh – vs hard wired into black box • Not made here/ licence fees • Resolution – Trade between resolution and breadth • Outputs/inputs not compatible with other models – often built ad hoc without thought to any existing modelling hierarchy

Section 5 Software Packages

29 Pros and cons of software packages • Excel – Pros: Flexible, powerful, easy to use & verify, widely available – Cons: can be hard to demonstrate, not sexy, config control • Access – Pros: great for large amounts of data, front end – Cons: knowledge not as widespread, VBA black box • Simul 8 – Pros: visualising, scoping problems, ease of use – Cons: ease of use, optimisation requires module

Section 6 Conclusions

31 Conclusions • It's easier to define what a model is compared to what is not a model – but this has implications for cost of ownership • The simplicity of a model is related to the willingness of decision makers to accept assumptions • The choice of simple or complex models should be based on a programme of work, not a single study – there is always pressure to add complexity • Easily used packages can lead to uncritical use

32 Final thought • When is a model not a model? – It depends. . . – when it's not validated – when it’s a tool… – maybe? – seek guidance from scrutineers – Guidelines for the Verification and Validation of Operational Analysis Modelling Capabilities - D/DG(S&A)/7/23/11. December 2002 – I’d appreciate your thoughts

33 “Things should be as simple as possible, but no simpler” Einstein (attributed)

Questions