September 24 2007 Overconfidence in judgment Why experience

September 24, 2007 Overconfidence in judgment: Why experience might not be a good teacher Tom Stewart

Einhorn, H. J. , & Hogarth, R. M. (1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85(5), 395 -416. “How can the contradiction between the considerable evidence on the fallibility of human judgment be reconciled with the seemingly unshakable confidence people exhibit in their judgmental ability? In other words, why does the illusion of validity persist? ” (p. 396) 2

Experience, performance, confidence ? Experience Performance ? ? Confidence 3

Experience, performance, confidence Feedback ? Experience ? ? Uncertainty Performance Confidence 4

r =. 50 5

r =. 50 6

r =. 80 7

r =. 95 See speaker note 8

Judgments are continuous Decisions are discrete 9

Threshold model Threshold 1 Decision A Threshold 2 Decision B Low Decision C High Judgment 10

Decision threshold Don’t r =. 50 Act 11

r =. 50 Action is appropriate Criterion Action is inappropriate threshold 12

Decision threshold Misses r =. 50 Hits Criterion threshold Correct rejections False alarms 13

Decision threshold Misses r =. 50 Hits Criterion threshold Correct rejections False alarms 14

Decision threshold Misses r =. 50 Hits Criterion threshold Correct rejections False alarms 15

Decision threshold Misses r =. 95 Hits Criterion threshold Correct rejections False alarms 16

Research on judging contingencies between x and y based on information in 2 x 2 tables suggests that people focus on frequency of Hits. Misses Hits Correct rejections False alarms This may be due to the difficulty people have in using disconfirming information. 17

How do people learn to make decisions if feedback (knowledge of results) is incomplete? • Selective feedback example – selection task – If an employer chooses not hire an applicant, she will not learn how that applicant would have performed. • Selective feedback example – detection task – If a customs officer chooses not to conduct a search of an airline passenger entering the country, he will not learn whether the passenger is smuggling goods into the country. 18

Knowledge of results: Full feedback Misses Hits Correct rejections False alarms 19

Knowledge of results: Selective feedback Misses Hits Correct rejections False alarms 20

Typical results Full feedback Selective feedback 21

Possible explanation • Encoding of cases when no feedback is available. Two possibilities (not exhaustive): – Positivist – People assume that when feedback is missing accuracy is the same as when feedback is present. – Constructivist (optimistic) – People assume perfect accuracy when feedback is missing. Elwin, E. , Juslin, P. , Olsson, H. , & Enkvist, T. (2007). Constructivist Coding: Learning From Selective Feedback. Psychological Science, 18(2), 105 -110. 22

Selective feedback – possible types of encoding Objective results Subjective results – Positivist encoding Subjective results – Constructivist (optimistic) encoding = subjective encoding 23

Encoding and values affect the cutoff · Subjective encoding · If people assume they are correct when they don’t get feedback, the cutoff will move up (fewer cases selected). · Values of the four outcomes – There is evidence that people value hits more than other outcomes. – This could result in selecting more cases. – However, if people pay attention to the positive hit rate, they might select fewer cases. Einhorn and Hogarth, 1978 24

Hits as a function of selection rate Note that hit rate can be high even if accuracy is not. Hit rate Proportion correct Proportion of all decisions that are hits (Hit rate is number of hits divided by number of positive decisions. ) 25

Plot of expected value vs. decision cutoff Payoff matrix assumes greater value for hits Selective feedback, constructivist encoding, subjective expected value Selective feedback, positivist encoding, subjective expected value Full feedback, objective expected value 26

Summary: Selective feedback increases confidence while reducing performance • Research suggests that, with limited feedback, people will learn to select fewer cases. – This results in a decision bias that increases the error rate. • Other research suggests that people pay more attention to hits than to other outcomes. – This could result in either more cases being selected in order to increase the number of hits, or fewer cases to increase the hit rate. • The constructivist encoding hypothesis can account for the experimental results. * • Furthermore, with constructivist encoding subjective performance will be better than objective performance, accounting for overconfidence. • It appears that while selective feedback results in more decision errors, it may not affect the accuracy of judgment. *Of course, this does not prove that people are actually doing constructivist encoding, and there are certainly individual differences. 27

End 28

Confidence • People pay attention to positive hit rate. – Inability to use disconfirming information – Limited feedback when action not taken • Positive hit rate is often high, even when accuracy is not. – Positive hit rate can always be increased by reducing selection rate/increasing threshold. – Treatment effects increase positive hit rate, and this increase is greater for high selection rates. 29

If people judge their skill by the true positive rate, what affects that rate? • • Base rate Correlation Selection rate Treatment effects Illustrate with spreadsheet C: /Documents and Settings/Tom/My Documents/aa. Documents/AAPRJCTS/2005/NSF-TR-SDT-Feedback/6 -Talks/T-R 634 Assignment-Einhorn-Hogarth-treatment. xls 30

Treatment effect r =. 50 31

Treatment effect r =. 50 32