Reasoning with Incomplete Information A Bayesian Approach Tracey

Reasoning with Incomplete Information: A Bayesian Approach Tracey Enderwick Supervisor: Ken Mc. Naught Cranfield University, 04/09/07 Defence Academy of the United Kingdom

Presentation Contents • Introduction • Background information – Bayesian networks – Influence diagrams • Experiments: time vs quality trade-offs • Completeness 2 04/09/07

Introduction • Decision makers are often faced with the difficult task of deciding a course of action based on incomplete and conflicting information from multiple sources. • Bayesian Networks (BNs) and Influence Diagrams (IDs) are known analytic methods for structuring this information and answering questions such as: – How likely is some effect to occur given a mixture of evidence on its possible causes and intermediate effects? (prediction) – How likely is some possible cause to be true given a mixture of evidence on its possible effects and other possible causes? (diagnosis) – What is the best decision given what evidence we currently have and the payoffs or consequences associated with every possible outcome? (optimisation) 3 04/09/07

Introduction As well as the previous questions, we are interested in applying BNs and IDs to answer the following: When is the best time to make the decision? Can we measure how complete our information is when we make a decision? 4 04/09/07

Simple Bayesian Network Mediating (updated) Background Prior (updated) Hypothesis (updated) Background Prior (updated) Mediating (updated) Observable Instantiated Information Observable information Information (updated) 5 04/09/07

Simple Influence Diagrams extend Bayesian Networks by including Decision and Utility nodes 6 04/09/07

Optimal decision timing • Experiments conducted with a sample of potential decision makers • Influence Diagram developed to represent overall scenario • Decision Support tool using Netica VB API and a link to Excel 7 04/09/07

Trade off time v quality Deciding quickly can have many positive outcomes… – Quick arrest; catches suspect by surprise – Cancer caught quickly and is more treatable … however there is a larger chance of making the wrong decision when little or no information is available… – Raid wrong building – Intensive chemo is given when tumour is benign … waiting reduces the chance of making the wrong decision but waiting too long can also be detrimental – Suspect escapes – Cancer spreads and becomes untreatable 8 04/09/07

Experiment overview • Commander in built up area • Three alternatives: • Wait for more intelligence • Search building A • Search building B • Three intelligence indicators: • 60% reliable • 75% reliable • 90% reliable • Normal or Geometric time distributions for receiving intelligence • Utility decreases linearly over time – Waiting too long or giving an incorrect decision results in 0 utility • Ten different scenarios per time distribution 9 04/09/07

Scenario Influence Diagram 10 04/09/07

Scenario Influence Diagram 11 04/09/07

Experimental results Proportion of people who made the right decision at the optimal time step – Normal time distribution 12 04/09/07

Experimental results Proportion of people who made the right decision at the optimal time step – Geometric time distribution 13 04/09/07

Experimental Strategies • Most candidates went with the 90% reliability indicator when available • Very few trusted the 60% reliability indicator alone • More thought was given when the 75% reliability indicator was known, especially if it contradicted the 60% reliability indicator 14 04/09/07

Experimental bias and psychology • More risky decision making – “Guessing” • Military personnel tended to focus on the outcome and the consequence of making the wrong decision • Civilian personnel tended to focus on the probabilities and the chances of making the wrong decision 15 04/09/07

Analytical results Normal time Distribution for receiving information no information 04/09/07 Mean 15 minutes, Standard deviation 5 minutes 16

Analytical results Normal time Distribution for receiving information 60% reliability indicator known at t = 5 04/09/07 Mean 15 minutes, Standard deviation 5 minutes 17

Analytical results Normal time Distribution for receiving information 75% reliability indicator known at t = 5 04/09/07 Mean 15 minutes, Standard deviation 5 minutes 18

Analytical results Normal time Distribution for receiving information 90% reliability indicator known at t = 5 04/09/07 Mean 15 minutes, Standard deviation 5 minutes 19

Analytical results Geometric time Distribution for receiving information 60% reliability indicator known at t = 5 P(receiving information in next time step) = 20% 04/09/07 20

Analytical results Geometric time Distribution for receiving information 75% reliability indicator known at t = 5 P(receiving information in next time step) = 20% 04/09/07 21

Analytical results Geometric time Distribution for receiving information 75% reliability indicator known at t = 5 P(receiving information in next time step) = 30% 04/09/07 22

Analytical results Geometric time Distribution for receiving information 90% reliability indicator known at t = 5 P(receiving information in next time step) = 20% 04/09/07 23

Analytical recommendations Indicator Reliability Recommended time to wait for more information, given indicator received, after which waiting is no longer beneficial Normal Geometric 40 minutes 25 minutes with p(receiving more intel) = 0. 2 75% 25 minutes need p(receiving more intel) > 0. 3 90% 0 minutes 60% 24 04/09/07

Decision support tool demonstration 25 04/09/07

Completeness of information 26 04/09/07

Definitions • Entropy Measure of disorder in a system • Value of information A quantitative measure of the value of knowing the outcome of an uncertain variable prior to making a decision • Mutual information The expected reduction in entropy of a target variable brought about by observing another unknown variable 27 04/09/07

Completeness of information: Proposed formula The denominator in the formula answers the question: what would the expected entropy reduction be if all the IS nodes were observed? The numerator answers the question: what would the expected entropy reduction be if all the IS nodes were observed given the evidence already known? Completeness = 28 04/09/07

Influential Set D Completeness value: D known ≈ 12% B C E D & E known ≈ 25% B, C & D known = 100% Key: - Hypothesis variable - Influential Set variables - Other observable variables 29 04/09/07

Further Study • Investigating methods to extend approach to larger networks • Combine optimal decision time, completeness and other aspects of incomplete information in decision support tool to make it more comprehensive 30 04/09/07

Questions and Answers 04/09/07