NATO Guide for JudgementBased Operational Analysis in Defence

NATO Guide for Judgement-Based Operational Analysis in Defence Decision Making NATO SAS-087 presented by Neville J Curtis Research Leader, Operations Research Defence Science and Technology Organisation OR 54 (2012)

Dramatis personae Australia – NJ Curtis Canada – M Halbrohr France – O de Baysier (until Autumn 2010) Germany – G Mihelcic, C Wittmann Netherlands – ICL Bastings, DJD Wijnmalen (Chair) Sweden – J Frelin United Kingdom – SM Lord, GA Pickburn United States – YH Wong (from December 2010) NATO/ACT – S Collins, A Smethurst Workshop participants at ISMOR 2010 and OR 52 Academic consultants: LA Franco and EAJA Rouwette Defence consultants: RA Forder, J-H Pay, FS Ordean and I Psomas

The products NATO Guide for Judgement-based Operational Analysis in Defence Decision Making: § Analyst-oriented volume (142 p. A 4) § Client-oriented volume (56 p. A 5) § Executive-oriented volume (4 p. A 4) the Code of Best Practice for ‘Soft’ Operational Analysis

Aim of the Guidance as to how to set up & conduct a judgement-based (‘soft’) OA/OR study that meets quality criteria including client expectations: validity, credibility, acceptance § Rules of the road for analysts, creating clarity & focus § Clarification and expectation management for clients This presentation will describe the contents of the guide

Analyst-oriented volume (Code of Best Practice)

Ch 2 (12 p) - Puzzles, Problems, Messes and soft OA/OR

Relating ‘p’, ‘m’ to ‘hard’, ‘soft’ OA/OR (1) Increasing opportunity for use of judgement-based (‘soft’) OA

Relating ‘p’, ‘m’ to ‘hard’, ‘soft’ OA/OR (2)

Ch 3 (10 p) - The model – how to do the right thing right

Ch 4 (21 p) - Roles and responsibilities analysis of (Ch 4, 6) skills and caveats (Ch 4) how to cope with (Ch 4, 7)

Ch 5 (15 p) – The process

Ch 6 (21 p) – Methodology - how to recognise a mess? (1) q. Not much is initially known about the nature of the problematic situation and its boundaries (what matters and what does not). q. Not much is initially known about defining the elements of the problematic situation and how they may be interrelated. q. Not much is initially known about who the stakeholders are and in what manner they may be directly or indirectly affected, their viewpoints and what they are worried about. q. Not much is initially known about the goals, objectives and measures of effectiveness or merit that may be relevant. q. Not much is initially known about what can and should be changed towards improvement of the problematic situation, and under what conditions or according to what criteria a change will be regarded as an improvement. q. Not much is initially known about the data needed, its relevance, availability and reliability. q. Not much is initially known about the way in which changes in context will affect the problematic situation, its improvement and the study design to achieve it. q. Power, emotion, politics and ethics will most likely come into play, but not much is initially known about how and to what effect. q. Different people tell different things (or express different views) about the same issue. q. Grand-scale issues are at stake that have no clear end-points (e. g. terrorism) and/or where any possible resolution will most likely have side-effects attached to them that are undesirable by stakeholders.

Ch 6 - How to recognise a mess? (2) None ticked: puzzle A few ticked: à problem Most ticked: à mess Anonymous client quotation: “I want to exclude unforeseen outcomes”

CH 7 (13 p) – Data collection eg Subject Matter Experts Stage Pros of involving a SME Cons of involving a SME requirement Appreciation Detailed inside knowledge. Could be tasked to obtain specific information. Incomplete knowledge of the system. Need to balance analyst team contribution against SME In depth knowledge of how the system should operate and its actual behaviour Ability to work with analysts. Analysis Gives peer review. Provides sources of data to test the model. Provide personal experience on current deficiencies. May provide useful speculation on new ideas Might bias the rest of the study through a personal or institutional mind set. Formulation of a model that is too literal with little opportunity to explore innovative changes. Embarrassment to the study team if the model is poor and this is the first time the SME sees it. May be reluctant to provide useful speculation on new ideas. Acceptance of another’s perception of his system. Ability to work with analysts. Ability to generalise on the nature of data and its applicability. Ability to speculate on ‘out of the box’ interventions. Ability to work with analysts. Assessment May identify the study team’s unknown obstacles to change. May apply their own filters if the changes do not fit their value systems. Ability to provide an honest sanity check. Ability to work with analysts. Action Provides credibility to the study. Identifies pragmatic courses of action Would probably have to be involved in most of the process – time constraint. Needs to speak with authority and be recognised as such by the executive decision makers.

Ch 8 (9 p) - Using the outcomes

Analyst-oriented volume (Code of Best Practice)

Client-oriented volume (1) Addresses 7 key questions: 1. What is judgement-based OA? 2. Which problematic situations require judgementbased OA? 3. How does judgement-based OA add value? 4. What does a judgement-based OA study look like? 5. What is expected of me, the client? 6. What does the analyst bring to achieve validity, credibility and acceptance? 7. How can a Co. BP protect the client from threats to the study?

Client-oriented volume (2) In answering the key questions it stresses 3 key aspects of judgementbased OA: • • • It enables progress to be made for some otherwise intractable and complex decisions. It involves a creative journey of discovery and learning that can be used to the advantage of decision makers. The inherent uncertainty of complicated decision situations that the defence sector faces, leads the client for judgement -based OA to what are perhaps his most pressing concerns – its validity, credibility and acceptance. Study methods must therefore be well documented to withstand scrutiny.

Threats (Ch 7) and benefits (Ch 3) § disagreement with a specific part of the study leading to a dismissal of the rest of the material. § scenarios treating judgement-based OA study results with too much certainty (e. g. as a prediction). § too rapid a progression from an illformed concern to a rigid plan for change (e. g. an acquisition). § biased preference of some forms of evidence (e. g. from more quantitative sources). § selective interpretation to support a specific argument. § an improved shared understanding, through a recognition to consider all sides of the issue. § an enhanced sense of common purpose. § greater commitment to a general way forward. § the discovery and consideration of alternative options. § the development of acceptable solutions or ways forward to improve the problematic situation. § the systematic gathering and analysis of information. § use of an approach that recognises people’s different cognitive viewpoints and belief systems.

Executive-oriented leaflet Addresses: • • • What is judgement-based OA? How can defence decision makers be supported? When should judgement-based OA be used? What can you do with facilitated workshops? What are the value and the benefits of judgement-based OA?

Downloads www. cso. nato. int/abstracts. aspx Goes to “Search STO Scientific Publications” Enter “SAS-087” in top box The three files are then listed in order of: 1. client-oriented volume 2. analyst-oriented volume 3. executive volume