The Evidence Framework Approach Practical Ways of Thinking

Operational Analysis (OA) Focus Dstl’s Defence & Security Analysis Division applies OA to a

The Evidence Framework Approach Problem Formulation Quality of Evidence Quality Assurance Become better system

Problem Formulation (Enhancing Study Design). • Through conducting an Analysis Estimate David Snowden http:

Quality of Evidence - Using the EPT Assessment of an analytical programme of work

Quality of Evidence - Using the VPT Assessment of an analytical programme of work

Quality of Evidence – Using the CAT Assessment of an analytical programme of work

Example Application – LEOA Insights EPT only Study • Target level of evidence required

Quality Assurance • Uses the Evidence Quality Questionnaire (EQQ). • The EQQ is a

EFA Summary • The Evidence Framework Approach will improve the quality of OA to

What Next? • The EFA is being successfully applied and is continuing to develop:

Further Reading February 2022 © Crown copyright 2017 Dstl UK OFFICIAL

Slides: 13

Download presentation

The Evidence Framework Approach: Practical Ways of Thinking About Evidence 34 th International Symposium on Military Operational Research 18 th to 21 st July 2017 Royal Holloway, University of London Paul Pearce Simulation & Numerical Methods Group Defence and Security Analysis Division Defence Science & Technology Laboratory Portsdown West pvpearce@dstl. gov. uk February 2022 © Crown copyright 2017 Dstl UK OFFICIAL DSTL/PUB 103348

Operational Analysis (OA) Focus Dstl’s Defence & Security Analysis Division applies OA to a range of questions using a variety of ‘soft’ and ‘hard’ analysis techniques…. presenting a number of evidence challenges. 2017 m t s e v In 2017 - 2027 February 2022 © Crown copyright 2017 Dstl e c n e d i v e s e r i u q e r t en 2027 - 2037 UK OFFICIAL 2037+

The Evidence Framework Approach Problem Formulation Quality of Evidence Quality Assurance Become better system thinkers through enhanced Study design approach Employ a means to characterise quality of evidence using the Evidence Profile Table (EPT), Validation Profile Table (VPT) and Confidence Assessment Table (CAT) Think about analytical quality assurance by assessing RIGOUR* using an Evidence Quality Questionnaire (EQQ) Results in better engagement, greater understanding and more robustly defendable evidence * Repeatability Independence Grounding in reality Objectivity Uncertainty-managed Robustness February 2022 © Crown copyright 2017 Dstl UK OFFICIAL

Quality of Evidence - Using the EPT Assessment of an analytical programme of work using the Evidence Profile Table Increasing certainty Problem Space Obvious All uncertainty explored System can be described Problem Space chaotic Unknowableunknowns perspective February 2022 © Crown copyright 2017 Dstl UK OFFICIAL

Quality of Evidence - Using the EPT Assessment of an analytical programme of work using the Evidence Profile Table Comprehensiveness Relevance February 2022 © Crown copyright 2017 Dstl Objectivity UK OFFICIAL Quantity Consistency Target 10 Actual 11

Quality of Evidence - Using the VPT Assessment of an analytical programme of work using the Evidence Profile Table Face Validity Criterion Validity February 2022 © Crown copyright 2017 Dstl Construct Validity UK OFFICIAL Content Validity Target 6 Actual 7

Quality of Evidence – Using the CAT Assessment of an analytical programme of work using the Confidence Assessment Table • • • February 2022 © Crown copyright 2017 Dstl UK OFFICIAL Moderate warrant Strong Validity Medium Confidence

Example Application – LEOA Insights EPT only Study • Target level of evidence required • Multiple experiments – the production of capping paper to consolidate findings • Draw on further expertise to validate assumptions before use and after findings February 2022 © Crown copyright 2017 Dstl Method • Coverage of relevant factors • Good track record for problem. • Strong direct support and indirect support for insights. • Simulation mode: Possible to assert that A is very likely to cause B for some of the insights. UK OFFICIAL Insights • Insight 1 has arisen in other studies and in other scenarios. Further analysis unlikely to change the findings. • Insight 2 has emerged over several LEOA activities. • Insight 3 new or emerging insight not been seen before.

Quality Assurance • Uses the Evidence Quality Questionnaire (EQQ). • The EQQ is a comprehensive framework for assessing analytical quality against key criteria, helping users to understand how the evidence should be interpreted and to identify areas for improvement. • Structured around the mnemonic RIGOUR as used in the Aqua Book. • EQQ has sections corresponding to the generic phases of a study. February 2022 © Crown copyright 2017 Dstl UK OFFICIAL

EFA Summary • The Evidence Framework Approach will improve the quality of OA to inform decision making • Extensive research has been conducted to provide the conceptual underpinnings • Its utility has being tested through experimentation and application • Guidance is in place to aid its use on studies • Development continues and your help in taking it forward and improving it is welcome February 2022 © Crown copyright 2017 Dstl UK OFFICIAL

What Next? • The EFA is being successfully applied and is continuing to develop: – The EFA is transitioning to ‘business as usual’ influencing a number of communities of interest. – Engagement with academia, industry and analytical communities continues to test and develop the EFA, improving the language and broadening the applicability. • Engagement with wider UK Government: – Aiming to provide input for updates to the Governments guidance on providing quality analysis (AQUA Book) • ISMOR Workshop tomorrow (19 th) at 0900 and 1330 to introduce the ideas February 2022 © Crown copyright 2017 Dstl UK OFFICIAL