Research Prediction Markets and the Wisdom of Crowds

Research Prediction Markets and the Wisdom of Crowds David Pennock Yahoo! Research NYC

Research Outline • Prediction Markets Survey • What is a prediction market? • Examples • Some research findings • The Wisdom of Crowds: A Story

Research Bet = Credible Opinion Hillary Clinton will win the election “I bet $100 Hillary will win at 1 to 2 odds” • Which is more believable? More Informative? • Betting intermediaries • Las Vegas, Wall Street, Betfair, Intrade, . . . • Prices: stable consensus of a large number of quantitative, credible opinions • Excellent empirical track record

Research Example: March Madness

Research More Socially Redeemable Example http: //intrade. com Screen capture 2007/05/18

Research A Prediction Market • Take a random variable, e. g. Bird Flu Outbreak US 2007? (Y/N) • Turn it into a financial instrument payoff = realized value of variable I am entitled to: $1 if Bird Flu US ’ 07 $0 if Bird Flu US ’ 07

Research Why? Get information • price probability of uncertain event (in theory, in the lab, in the field, . . . more later) • Is there some future event you’d like to forecast? A prediction market can probably help

Research A Prediction Market • Take a random variable, e. g. 2008 CA Earthquake? US’ 08 Pres = Dem? =6? • Turn it into a financial instrument payoff = realized value of variable I am entitled to: $1 if =6 $0 if 6

Research Aside: Terminology • Key aspect: payout is uncertain • Called variously: asset, security, contingent claim, derivative (future, option), stock, prediction market, information market, gamble, bet, wager, lottery • Historically mixed reputation • Esp. gambling aspect • A time when options were frowned upon • But when regulated serve important social roles. . .

Research Getting Information • Non-market approach: ask an expert I • am. How entitledmuch to: $1 if would you pay for this? =6 $0 if 6 • A: $5/36 $0. 1389 • • caveat: expert is knowledgeable caveat: expert is truthful caveat: expert is risk neutral, or ~ RN for $1 caveat: expert has no significant outside stakes

Research Getting Information • Non-market approach: pay an expert • Ask the expert for his report r of the probability P( =6 ) • Offer to pay the expert • $100 + log r • $100 + log (1 -r) if if =6 6 “logarithmic scoring rule”, a “proper” scoring rule • It so happens that the expert maximizes expected profit by reporting r truthfully • • caveat: expert is knowledgeable caveat: expert is truthful caveat: expert is risk neutral, or ~ RN caveat: expert has no significant outside stakes

Research Getting Information • Market approach: “ask” the public—experts & nonexperts alike—by opening a market: I am entitled to: $1 if =6 $0 if 6 • Let any person i submit a bid order: an offer to buy qi units at price pi • Let any person j submit an ask order: an offer to sell qj units at price pj (if you sell 1 unit, you agree to pay $1 if = 6) • Match up agreeable trades (many poss. mechs. . . )

Non-Market Alternatives vs. Markets Ø Opinion poll v. Sampling v. No incentive to be truthful v. Equally weighted information v. Hard to be real-time Ø Ask Experts v. Identifying experts can be hard v. Incentives v. Combining opinions can be difficult Ø Prediction Markets v. Self-selection v. Monetary incentive and more v. Money-weighted information v. Real-time v. Self-organizing

Non-Market Alternatives vs. Markets Ø Machine learning/Statistics v. Historical data v. Past and future are related v. Hard to incorporate recent new information Ø Prediction Markets v. No need for data v. No assumption on past and future v. Immediately incorporate new information

Function of Markets 2: Risk Management Ø If is bad for me, I buy a bunch of $1 if $0 otherwise Ø If my house is struck by lightening, I am compensated.

Risk Management Examples Ø Insurance v. I buy car insurance to hedge the risk of accident Ø Futures v. Farmers sell soybean futures to hedge the risk of price drop Ø Options v. Investors buy options to hedge the risk of stock price changes

Financial Markets vs. Prediction Markets Financial Markets Prediction Markets Primary Social welfare (trade) Hedging risk Information aggregation Secondary Information aggregation Social welfare (trade) Hedging risk

Giving/Getting Information • What you can say/learn • Where % chance that – – – – Hillary wins GOP wins Texas YHOO stock > 30 Duke wins tourney Oil prices fall Heat index rises Hurricane hits Florida Rains at place/time – – – – IEM, Intrade. com Stock options market Las Vegas, Betfair Futures market Weather derivatives Insurance company Weatherbill. com

http: //intrade. com http: //tradesports. com Screen capture 2007/05/18

Research Intrade Election Coverage

Research http: //www. biz. uiowa. edu/iem

http: //www. wsex. com/ http: //www. hedgestreet. com/ Screen capture 2007/05/18

Play money; Real predictions http: //www. hsx. com/

http: //www. ideosphere. com Cancer cured by 2010 Machine Go champion by 2020 http: //us. newsfutures. com/

Research Yahoo!/O’Reilly Tech Buzz Game http: //buzz. research. yahoo. com/

More Prediction Market Games • • Biz. Predict. com Casual. Observer. net FTPredict. com Inkling. Markets. com Pro. Trade. com Storage. Markets. com The. Sim. Exchange. com The. WSX. com • Alexadex, Celebdaq, Cenimar, Bet. Bubble, Betocracy, Crowd. IQ, Media. Mammon, Owise, Public. Gyan, RIMDEX, Smarkets, Trendio, Two. Crowds • http: //www. chrisfmasse. com/3/3/markets/#Play-Money_Prediction_Markets

Catalysts • Markets have long history of predictive accuracy: why catching on now as tool? • No press is bad press: Policy Analysis Market (“terror futures”) • Surowiecki's “Wisdom of Crowds” • Companies: – Google, Microsoft, Yahoo!; Crowd. IQ, HSX, Inkling. Markets, News. Futures • Press: Business. Week, CBS News, Economist, NYTimes, Time, WSJ, . . . http: //us. newsfutures. com/home/articles. html

Does it work? Ø Yes, evidence from real markets, laboratory experiments, and theory v. Racetrack odds beat track experts [Figlewski 1979] v. Orange Juice futures improve weather forecast [Roll 1984] v. I. E. M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002] v. HP market beat sales forecast 6/8 [Plott 2000] v. Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’ 03][Schmidt 2002] v. Market games work [Servan-Schreiber 2004][Pennock 2001] v. Laboratory experiments confirm information aggregation [Plott 1982; 1988; 1997][Forsythe 1990][Chen, EC’ 01] v. Theory: “rational expectations” [Grossman 1981][Lucas 1972] v More later …

[Source: Berg, DARPA Workshop, 2002] Research Example: IEM 1992

[Source: Berg, DARPA Workshop, 2002] Research Example: IEM

[Source: Berg, DARPA Workshop, 2002] Research Example: IEM

[Source: Berg, DARPA Workshop, 2002] Example: IEM AAAI’ 04 July 2004 MP 1 -35

[Source: Berg, DARPA Workshop, 2002] Example: IEM AAAI’ 04 July 2004 MP 1 -36

[Source: Wolfers 2004] Speed: Trade. Sports Contract: Pays $100 if Cubs win game 6 (NLCS) Price of contract (=Probability that Cubs win) Fan reaches over and spoils Alou’s catch. Still 1 out. Cubs are winning 3 -0 top of the 8 th 1 out. The Marlins proceed to hit 8 runs in the 8 th inning Time (in Ireland) AAAI’ 04 July 2004 MP 1 -37

Research Does money matter? Play vs real, head to head Experiment • 2003 NFL Season • Probability. Sports. com Online football forecasting competition • Contestants assess probabilities for each game • Quadratic scoring rule • ~2, 000 “experts”, plus: • News. Futures (play $) • Tradesports (real $) • Results: • Play money and real money performed similarly • 6 th and 8 th respectively • Markets beat most of the ~2, 000 contestants • Average of experts came 39 th (caveat) Used “last trade” prices Electronic Markets, Emile Servan. Schreiber, Justin Wolfers, David Pennock and Brian Galebach

Research

Research Does money matter? Play vs real, head to head Statistically: TS ~ NF NF >> Avg TS > Avg

Real markets vs. market games IEM HSX average log score arbitrage closure AAAI’ 04 July 2004 MP 1 -41

Real markets vs. market games HSX FX, F 1 P 6 probabilistic forecasts forecast source F 1 P 6 linear scoring F 1 P 6 F 1 -style scoring betting odds F 1 P 6 flat scoring F 1 P 6 winner scoring expected value forecasts 489 movies AAAI’ 04 July 2004 MP 1 -42 avg log score -1. 84 -1. 82 -1. 86 -2. 03 -2. 32

Research 1/7 Story Survey Research A Wisdom of Crowds Story • Probability. Sports. com • Thousands of probability judgments for sporting events • Alice: Jets 67% chance to beat Patriots • Bob: Jets 48% chance to beat Patriots • Carol, Don, Ellen, Frank, . . . • Reward: Quadratic scoring rule: Best probability judgments maximize expected score Opinion

Research Individuals • Most individuals are poor predictors • 2005 NFL Season • Best: 3747 points • Average: -944 Median: -275 • 1, 298 out of 2, 231 scored below zero (takes work!)

Research Individuals • Poorly calibrated (too extreme) • Teams given < 20% chance actually won 30% of the time • Teams given > 80% chance actually won 60% of the time

Research The Crowd • Create a crowd predictor by simply averaging everyone’s probabilities • Crowd = 1/n(Alice + Bob + Carol +. . . ) • 2005: Crowd scored 3371 points (7 th out of 2231) ! • Wisdom of fools: Create a predictor by averaging everyone who scored below zero • 2717 points (62 nd place) ! • (the best “fool” finished in 934 th place)

Research The Crowd: How Big? More: http: //blog. oddhead. com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/ http: //www. overcomingbias. com/2007/02/how_and_when_to. html

Research Can We Do Better? : ML/Stats [Dani et al. UAI 2006] • Maybe Not • • CS “experts algorithms” Other expert weights Calibrated experts Other averaging fn’s (geo mean, RMS, power means, mean of odds, . . . ) • Machine learning (NB, SVM, LR, DT, . . . ) • Maybe So • Bayesian modeling + EM • Nearest neighbor (multi-year)

Research Can we do better? : Markets