Cognitive Bias in RiskReward Analysis Damon Levine CFA

Cognitive Bias in Risk-Reward Analysis Damon Levine, CFA, CRCMP, MA Atlanta RIMS Educational Conference February 9, 2017

Disclaimer • The author has not conducted original research on these biases nor has he coined any terminology related to the biases • The suggestions for mitigation or response to a particular cognitive bias may or may not be effective depending on the details of the specific circumstance, organization, or people involved • No guarantees of improved risk identification, risk mitigation, risk management, or performance are stated or implied 2

The “From the Hip” Pop Quiz (just answer!) 1. Estimate how many randomly selected people are needed to have 97% likelihood of (at least) 2 people sharing a birthday? (e. g. “March 3”) 2. A bat and ball cost $1. 10. The bat costs one dollar more than the ball. How much does the ball cost? 3. A risk manager shows Management the 95 th percentile point for potential hurricane losses for a calendar year. This loss value is $150 million. An executive new to the company asks for the probability of incurring $150 million in hurricane losses. Assuming the model is correct what should the risk manager say? 3

A Thankless Job? “Prevention is not easily perceived, measured, or rewarded; it is generally a silent and thankless activity… One can easily compute the costs [of mitigation] while the results are hard to determine. How can one tell its effectiveness, whether the measure was successful or if it just coincided with no particular accident? Job performance assessments in these matters are not just tricky, but may be biased in favor of the observed ‘acts of heroism. ’ History books do not account for heroic preventive measures. ” -Nassim Taleb 4

Availability Bias • In a randomly selected English word of three or more letters is it more likely that the word starts with an R (“rope”), or that R is its third letter (“park”)? • For a question such as the one above (tough to answer quickly in a purely scientific, well substantiated manner) we turn to heuristics and use what’s available in our memory and experience such as facts, impressions, anecdotes, intuition, etc. • Many are able to recall/list more words beginning with R and therefore reply that R more often appears as the beginning letter. The “availability” to recall those examples leads to a bias, trusting what’s “available” in memory as evidence, and the wrong answer 5

Hindsight Bias • Hindsight bias “…after learning the eventual outcome, [subjects] give a much higher estimate for the predictability of that outcome than subjects who [consider] the outcome without advance knowledge. Hindsight bias is sometimes called the I-knew-it-all-along effect. ” -Eliezer Yudkowsky, Machine Intelligence Research Institute • Risk managers must not be swayed by emotion. A recent or severe event tends to make our probability estimate larger than it should be • Mitigation: remember the context and look at the appropriate time horizon (at least capture where the conditions or environment are relatively stable) and determine an empirical probability. Blend with some amount of expert opinion and any new information. Revisit the in-place assumptions about likelihood, causality and correlations prior to the observed event and confirm, question, or revise those concepts and parameters 6

Black Swan Challenge 1. The hypothetical event X is thought of as a “black swan” and its annual probability is regarded as 1/50. Assuming year-to-year independence, what is the approximate probability that event X occurs at least once in the next 50 years? A) 2% B) 18% C) 46% D) 50% E) 63% 2. A diversified company stochastically models annual earnings for each of its 20 lines of business. Assuming independence between the lines, what is the probability that at least 1 line will have actual earnings less than its own 1 st percentile value (a very bad result) ? A) 3% B) 5% C) 12% D) 18% E) 23% 7

Hindsight & Availability Biases Cloud our View of Black Swans • We may fail to remember extreme events from the distant past that would help inform our outlook regarding likelihood • We may have vivid mental pictures of the devastation of recent “rare” disasters and our severity estimates for future event can be warped • It is possible to overestimate probability of black swans because of recent and striking examples of risk manifestation 8

“Yep, I saw that coming!’ “Mistakenly believing that the past was predictable, people conclude that the future is predictable. It has been said 1 : When we attempt to understand past events, we implicitly test the hypotheses or rules we use both to interpret and to anticipate the world around us. If, in hindsight, we systematically underestimate the surprises that the past held and holds for us, we are subjecting those hypotheses to inordinately weak tests and, presumably, finding little reason to change them. ” 2 1. Fischhoff, Baruch. 1982. “For Those Condemned to Study the Past: Heuristics and Biases in Hindsight. ” 2. “Cognitive Biases Potentially Affecting Judgment of Global Risks”, Eliezer Yudkowsky, 2008. 9

On the Average… • A bet has 2 outcomes: $10 with probability 0. 7 and $100 with probability 0. 3. It has expected value or “average value” of: 0. 7(10) + 0. 3(100) or $37 • Averages (expected values) are sometimes useful and give an idea of what the sample mean of a large set of observations might be • In a game a fair 6 -sided die is rolled. If a “ 2” or more is rolled, you win $1 M but if a “ 1” occurs, you pay $50, 000. This game has an expected value of $825, 000. Would you play the game? What would you pay to play? What about playing 1000 times before settlement? 10

A Misleading Average • Each January 1, an investor uses leverage and derivatives to set up a one-year time horizon position, P, whose maturity value at 12 -31 depends on an underlying asset A • There is very little upfront investment because of the leverage and use of deep out-of-the-money options • The asset A has a long data history and the average (mean) annual price fluctuation is +1. 4 (delta from Jan 1 to Dec 31) • Position P is such that there is a large profit for price changes of at least +1, a zero profit for a change between -2 and +1, and a very large loss results when the change is worse than -2, i. e. , (-∞ , -2) • Does the average annual price change imply this is great strategy? What role does risk appetite play here? Standard deviation? 11

Standard Deviation: A Remedy? • Question: A quantity follows a specific, but unknown, statistical distribution. It is of great interest to know when an observation three (3) standard deviations below the mean may occur. How likely is this? • Answer: We don’t know. (even knowing that the quantity follows some named distribution does not allow us to answer) • Better answer: Use Chebyshev’s inequality… 12

Chebyshev’s Inequality (one-sided version) Chebyshev’s inequality is a nearly universal result which is powerful in that very few assumptions are needed for its application. For any random variable X with finite expected value (average) μ and finite non -zero variance σ2 we have for any real number k>0: P(X ≤ μ-kσ) ≤ 1/(1+k 2) Example with k=3: the probability of an observation being 3 or more standard deviations below the mean is at most 1/(1 + 32) or 1/10. It is important to note that this is a much larger probability than under a normal distribution assumption. 13

Drawing different conclusions from the same information presented differently [towergateinsurance. co. uk] 14

Framing Effect: Mitigation • In public forums and the media, tone and terminology will influence and be influenced by public opinion • Separate facts from hearsay; separate facts from “expert opinion” • Understand motives and conflicts of interest • Remember the West Africa Ebola scare of 2014 -2015 15

The Foggy Mirror You emerge from the shower and stare into the foggy bathroom mirror. You can just make out your reflection and, being in a playful mood, outline your face with your finger. The resulting outline traced on the mirror has a size which: A) is approximately the same as your (actual) face B) depends on the distance you stand from the mirror C) is half the size of your face D) is exactly the size of your face 16

Intuition: Be Careful of What You “Know” • A bat and ball cost $1. 10. The bat costs one dollar more than the ball. How much does the ball cost? Solution: $0. 05 (5 cents) • “Mirrors Don’t Lie: Mislead? Oh, Yes. ” - NY Times, July 22, 2008. 17

What was “Known” in the Past • “Stocks have reached what looks like a permanently high plateau. ” – Irving Fisher, 3 days before the 1929 Black Thursday crash precipitated the Great Depression • Circa 1987: Portfolio insurance can protect investors: it will let them get out with minimal damage if markets ever begin to fall. They would simply sell ever-increasing numbers of futures contracts, a process known as dynamic hedging. (Then…Black Monday) • Circa 2006: housing prices always go up! (and therefore the GFC!) • “At this juncture, however, the impact on the broader economy and financial markets of the problems in the subprime market seems likely to be contained. ” - Ben Bernanke, March 2007, in testimony to the Congressional committee 18

What’s “Known” Today? • What do you assume or take as fact about your company’s risks, businesses, policies and practices? • What does management and the Board assume is true? • Does the security desk in the lobby really secure anything? Do people know procedures to follow in emergencies? Do you have backup resources and backup systems to stay in business? Run tabletops! • Don’t question everything but… 19

…You Should Question… • Key assumptions by those pitching a new product, venture, mitigation or solution • Use and extrapolation of data; consider if it’s apples to apples, if the sample size is large enough, if implicit assumptions about distribution or volatility are made • Do people know who is a risk owner and who is responsible for mitigation? • Are people aware of risk/safety policies or when/how to escalate risks? • Why is this project or this team different from those in past failures? • Why is this probability so low? Who is emotionally attached to the quality of a proposal, merger, acquisition, resource, tactic or mitigation? Is an estimate made under duress? 20

We Should Also Question “Expert Opinion” • As a society it seems we have a special fondness for expert forecasts • In many cases such predictions are wrong to an embarrassing extent and often could have been greatly improved by strong consideration of the normal or “base rate” seen in the available data • It is the temptation of too many experts (and laymen for that matter) to ignore the generic situation once a particular case comes under analysis • The previous is described as the Base Rate Neglect or the base rate fallacy 21

Base Rate Fallacy and the Acquisition • an executive pushing for an acquisition projects that all sales growth, profit and synergy projections will be met or exceeded over the next ten years while the historic record would paint a much more somber picture of post-acquisition performance • Honest M&A experts sometimes say “synergies are never realized” • Solution: run scenarios where sales do not follow the “hockey stick”, synergies fail or come very late, and/or profit margin is not improved by the (superior intellect of) new management • Solution: avoid the emotional attachment…use valuation analysis before the negotiation to determine a “walk-away” price 22

Don’t Go Away Mad, Just Go Away Former CEO of Citibank Sandy Weill says “Knowing when to get out of the game is a critical consideration. . . I’ve been in situations where we’ve got an agreement, and as time goes by the other side sees you getting anxious and raises the price, ” he says. “You have to be disciplined at that point, and it isn’t easy. Deal making is akin to dating and falling in love. If you don’t think the behavior of the other party is something you can live with from a cultural point of view, you have to grit your teeth and simply say ‘No. We’re done. ’ 23

Base Rate and Additional Information: Hit-and-Run Consider the following scenario and your gut response to the question: A cab was involved in a hit-and-run accident at night. Two cab companies, the Green and the Blue, operate in the city. (The Green company cabs are indeed all green and all Blue cabs are blue) Assume the following data: • 85% of the cabs in the city are Green and 15% are Blue. • A witness identified the cab as Blue. The court tested the reliability of the witness under the circumstances that existed on the night of the accident and concluded that the witness is 80% accurate in color identification (so is wrong 20% of the time). What is the probability that the cab involved in the accident was Blue? Example taken from Thinking, Fast and Slow by Daniel Kahneman 24

Hit-and-Run (continued) • The answer is 41% • The “base rate” information in this problem is the statement about the percentage of each cab in the city. With no other assumptions we must infer from it that Green cabs are in 85% of the accidents (hit-and -run or otherwise). • Bayes’ theorem is the right approach to combine that information with the witness testimony. If the base rate is ignored, and it is by many people, one comes up with a reply of 80%, which is very far from the correct answer of 41%. (shown in appendix) 25

The Law of Small Numbers or the Sophomore Slump • The “law of small numbers” refers to this misleading pattern which drops out as we reach samples of sufficient size • Most risk managers know the problems with small samples but still may attach significance to the behavior seen in small samples • Consider a baseball rookie whose first season batting average is. 400. This is a nearly unsustainable average and likely represents what will later be regarded as the player’s full potential. His “sophomore” season he bats. 320. This is perhaps more representative of his skill level but compared to the first season it is seen as a slump! 26

Sophomore Slump (cont’d) • In many real-world situations, the occurrence of a value very far from the average is less likely than one closer to the average • If one of the “typical” (near-the-average) values occurs after one that was seen as extreme (far from the average), this may be perceived as some type of “reversion to the mean” and this is a common, but misleading and inaccurate term, for this behavior. • As mentioned, in sports, it is common to see a rookie’s stellar first year followed by a more “normal” level of performance; in Australia they call this the “second year syndrome” 27

Sophomore Slump (cont’d) • If a bond manager has an incredible year and is among the top 1% of managers across the country, there’s a good chance the next year will be less “extreme” and will be closer to the average • No matter what name is used for this behavior, risk managers should understand it and not assume it’s a result of corrective action, praise, or year-to-year (negative) auto-correlation • Mitigation: don’t fall for “reversion to the mean”; understand likelihood of values far from the mean…what shape is the distribution? Are there fat tails? Is the distribution uniform around its mean? For product “experiments” or studies, don’t rely on statistically small samples! 28

Trees and Forest (and the Portfolio View) • Risk exposures are often analyzed in isolation and this is not necessarily wrong for certain purposes • Enterprise risk management (ERM), Management, and the Board must be aware of a holistic or portfolio view of risk • When considering stand-alone risk assessments, e. g. , focused on specific risk types or LOBs, one must not forget the enterprise view and the concepts of risk aggregation, concentration, correlation and diversification! 29

WYSIATI (“What you see is all there is”*) • The analysis of a very specific situation may naturally lull one into forgetting the bigger picture • A True Story… A LOB’s risk-reward analysis shows very low volatility in earnings, little chance of needing a capital infusion, and forecasts a modest profit. A Board member suggests the LOB should be taking on more risk. He forgot to consider what is happening in the rest of the company, and in doing so, ignored the portfolio or aggregate view that is so sought after in the ERM world. It was in fact this stable and low risk business that tempered aggressive risk taking at the other business lines in the company! (Note: he was, by all other measures, a very intelligent person!) *term coined by Daniel Kahneman 30

The desire to do the opposite of what is requested or advised, due to a perceived threat to freedom of choice [towergateinsurance. co. uk] 31

Mitigation of Reactance • A message that conflicts with your original understanding can be hard to swallow; try to put aside any pre-conceived notions or pride • We may not want to admit we misunderstood, misjudged or were unaware of key facts • Before rejecting a proposed path, decide if your objection is tied to your ego 32

Sunk Cost Fallacy “It is commonly expected that individuals will reverse decisions or change behaviors which result in negative consequences. Yet, within investment decision contexts, negative consequences may actually cause decision makers to increase the commitment of resources and undergo the risk of further negative consequences. ” - “Knee-Deep in the Big Muddy: A Study of Escalating Commitment to a Chosen Course of Action”, by Barry Staw. Organizational Behavior and Human Performance, Volume 16, Issue 1, June 1976, Pages 27– 44. 33

Sunk Cost Fallacy • The Sunk Cost Fallacy often goes away when a forward-looking cost-benefit analysis or risk-reward analysis is conducted • Many fall into the trap! o Nick Leeson: short straddle, Kobe earthquake, long-long future arbitrage, prison! o Boston’s Big Dig (Central Artery/Tunnel Project): the most expensive highway project in the US; originally scheduled to be completed in 1998 at an estimated cost of $2. 8 billion, but was completed in December 2007, at a cost of over $14. 6 billion o Sony: continued participation in electronics after $8. 5 billion in losses over 10 years o After a heated and aggressive bidding war, Robert Campeau buys Bloomingdale's with an estimated $600 million overpayment. The Wall Street Journal says "we're not dealing in price anymore but egos. " Campeau declared bankruptcy soon afterwards 34

On Board the Flying Bank • The Swiss airline Swissair was once thought to be so financially stable that it earned the title “the Flying Bank” • Despite financial troubles in the late 90 s it continued to execute the Hunter Strategy by buying up bankrupt airlines all over Europe • The management underestimated the dangers and difficulties in acquisitions and investments of partially ailing airlines. So the Belgian Sabena and the German LTU were taken despite the significant capital requirements 35

On Board the Flying Bank (cont’d) • The Hunter Strategy “was rooted in the arrogance of a company desperate to punch above its weight on the international scene” – Sepp Moser • Aaron Hermann and Hussain Rammal cite groupthink as a contributing factor in the collapse of Swissair: the belief that the group is invulnerable and the belief in the morality of the group. • They point out that before its failure, “the size of the company board was reduced, subsequently eliminating industrial expertise and the board members were lacking expertise in the field and had somewhat similar background, norms, and values…the pressure to conform may have become more prominent” 36

Mitigation of Groupthink • To the extend possible seek out assessments, risk identification, probabilities, forecasts, and parameter estimates in a one-on-one environment; employ anonymous polling and/or surveys (or pick up the phone!) • Besides looking at individual results to assess majority views, one can also use the spread or range of numerical risk assessments as a measure of understanding of the risk itself or the volatility of the “risky” quantity of interest: in other words, a wide variety of replies may indicate there is not a great understanding of the risk or it may simply indicate a risk with high volatility or uncertainty 37

Black Swans Revisited • Consider a very unlikely event with annual probability 1/n. For example, n might be 100 or 500 • In a given year the probability the event does not occur is 1 -1/n • Assuming year to year independence, the probability that it does not occur over n years is (1 -1/n)n • As n gets larger this expression approaches 1/e ≈ 0. 37 and therefore, for large n, the probability the black swan event does occur at least once in n years is approximately 1 – 1/e ≈ 0. 63 or 63% • This works well when even for “small” n such as 20 38

Black Swans Revisited A diversified company models annual earnings using a stochastic approach for each of its 20 lines of business. Assuming independence between the lines, what is the probability that at least 1 line will have actual earnings less than its own 1 st percentile value? Solution: P(no LOB has a result below the 1 st %ile) is 0. 9920 or 81. 79%, and therefore, using the complement, we have: P(at least one LOB has a result below the 1 st %ile) = 1 – 81. 79% or 18. 21% 39

The Anchor Effect • Risk quantification and financial forecasting often begin with a single initial estimate • The source for the value might be last year’s value, a result of deep analysis, or could be based on intuition • Regardless of the source, once the initial value is seen, it is very hard to mentally move far away from it in subsequent consideration or estimation of alternatives! • That initial value, whether or not of high “quality”, tends to anchor any future estimates to be close to that initial value 40

The MIT Mock Auction • A professor of management science at MIT, Dan Ariely, conducted a mock auction with his MBA students. • A CFO magazine article summarizes his behavior experiment: “He asked students to write down the last two digits of their Social Security numbers, and then submit bids on such items as bottles of wine and chocolate. ” • The half of the group with higher two-digit numbers bid “between 60 percent and 120 percent more” on the items! 41

The Anchor Effect (cont’d) • Ariely says “people are good at setting relative values” but “it’s very hard to figure out what the fundamental value of something is whether it’s an accounting system, a company’s stock, or a CEO. ” • Mitigation: when possible, make the estimate or projection without the benefit of another person’s estimate or “last year’s value” • Make the following joke false!: How many actuaries does it take to screw in a light bulb? Well, how many did it take last year? ” - “Avoiding Decision Traps”, by Edward Teach. Online at: http: //ww 2. cfo. com/human-capital-careers/2004/06/avoiding-decision-traps 42

Solution to Pop Quiz #3 • A risk manager shows Management the 95 th percentile point for potential hurricane losses for a calendar year. This loss value is $150 million. An executive new to the company asks for the probability of incurring $150 million in hurricane losses. Assuming the model is correct what should the risk manager say? • Solution: about 0%, but the probability of incurring $150 million or more in hurricane losses is 5% based on the model • This is almost a version of WYSIATI type thinking as most of us know the answer and some are simply too focused on the specifics 43

The Birthday Problem (and Today’s Winners) Given “n” randomly selected people there is a probability of p(n) that (at least) 2 people share a birthday as follows: 44

Appendix: Hit-and-Run Solution 1 One can use Bayes’ theorem in the form: P(A|B) = P(B|A)P(A) / [ P(B|A)P(A) + P(B|A′)P(A′)] where A′ is the complement of A. One can define event A as “cab was blue” and event B as “witness said cab was blue”; then the left hand side is: P(cab was blue | witness said “Blue”) and using the rest of the formula gives approximately 41. 4%. It is necessary to make use of the witness’s 80% accuracy. 45

Hit-and-Run Solution 2 46

References and Additional Information • Many of these concepts and some wording has been based on this author’s whitepaper “ERM at the Speed of Thought: Mitigation of Cognitive Bias in Risk Assessment” available here: http: //www. ermsymposium. org/2015/Additional_Research_Papers/Levine-05 -05 -15. pdf • A great source for information on cognitive bias and a great read is Thinking, Fast and Slow by Daniel Kahneman • The website https: //www. towergateinsurance. co. uk/liabilityinsurance/cognitive-biases has a quick presentation of many biases, complete with cartoons! • Website: www. ermvalue. com Email: damonlevine 239@yahoo. com 47