Research Prediction Markets and the Wisdom of Crowds
- Slides: 54
Research Prediction Markets and the Wisdom of Crowds David Pennock, Yahoo! Research Joint with: Yiling Chen, Varsha Dani, Lance Fortnow, Ryan Fugger, Brian Galebach, Arpita Ghosh, Sharad Goel, Mingyu Guo, Joe Kilian, Nicolas Lambert, Omid Madani, Mohammad Mahdian, Eddie Nikolova, Daniel Reeves, Sumit Sanghai, Mike Wellman, Jenn Wortman
Research Bet = Credible Opinion Obama will win the 2008 US Presidential election “I bet $100 Obama will win at 1 to 2 odds” • Which is more believable? More Informative? • Betting intermediaries • Las Vegas, Wall Street, Betfair, Intrade, . . . • Prices: stable consensus of a large number of quantitative, credible opinions • Excellent empirical track record
Research A Prediction Market • Take a random variable, e. g. Bin Laden captured in 2008? (Y/N) • Turn it into a financial instrument payoff = realized value of variable I am entitled to: Bin Laden $1 if caught ’ 08 Bin Laden $0 if caught ’ 08
Research http: //intrade. com
Research Outline • The Wisdom of Crowds • The Wisdom of Markets • Prediction Markets: Examples & Research • Does Money Matter? • Combinatorial Betting Story Survey Research
Research 1/7 Story Survey Research A WOC Story • Probability. Sports. com • Thousands of probability judgments for sporting events • Alice: Jets 67% chance to beat Patriots • Bob: Jets 48% chance to beat Patriots • Carol, Don, Ellen, Frank, . . . • Reward: Quadratic scoring rule: Best probability judgments maximize expected score Opinion
Research Individuals • Most individuals are poor predictors • 2005 NFL Season • Best: 3747 points • Average: -944 Median: -275 • 1, 298 out of 2, 231 scored below zero (takes work!)
Research Individuals • Poorly calibrated (too extreme) • Teams given < 20% chance actually won 30% of the time • Teams given > 80% chance actually won 60% of the time
Research The Crowd • Create a crowd predictor by simply averaging everyone’s probabilities • Crowd = 1/n(Alice + Bob + Carol +. . . ) • 2005: Crowd scored 3371 points (7 th out of 2231) ! • Wisdom of fools: Create a predictor by averaging everyone who scored below zero • 2717 points (62 nd place) ! • (the best “fool” finished in 934 th place)
Research The Crowd: How Big? More: http: //blog. oddhead. com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/ http: //www. overcomingbias. com/2007/02/how_and_when_to. html
Research Can We Do Better? : ML/Stats [Dani et al. UAI 2006] • Maybe Not • • CS “experts algorithms” Other expert weights Calibrated experts Other averaging fn’s (geo mean, RMS, power means, mean of odds, . . . ) • Machine learning (NB, SVM, LR, DT, . . . ) • Maybe So • Bayesian modeling + EM • Nearest neighbor (multi-year)
Research Can we do better? : Markets
Research Prediction Markets: Examples & Research
Research The Wisdom of Crowds Backed in dollars • What you can say/learn % chance that • • Obama wins GOP wins Texas YHOO stock > 30 Duke wins tourney Oil prices fall Heat index rises Hurricane hits Florida Rains at place/time • Where • • IEM, Intrade. com Stock options market Las Vegas, Betfair Futures market Weather derivatives Insurance company Weatherbill. com
Research Prediction Markets With Money Without
Research The Widsom of Crowds Backed in “Points” • • • HSX. com Newsfutures. com Inkling. Markets. com Foresight Exchange Casual. Observer. net FTPredict. com Yahoo!/O’Reilly Tech Buzz Pro. Trade. com Storage. Markets. com The. Sim. Exchange. com The. WSX. com • Alexadex, Celebdaq, Cenimar, Bet. Bubble, Betocracy, Crowd. IQ, Media. Mammon, Owise, Public. Gyan, RIMDEX, Smarkets, Trendio, Two. Crowds • http: //www. chrisfmasse. com/3/3/markets/#Play-Money_Prediction_Markets
http: //betfair. com Screen capture 2008/05/07 http: //tradesports. com Screen capture 2007/05/18
[Source: Berg, DARPA Workshop, 2002] Research Example: IEM 1992
[Source: Berg, DARPA Workshop, 2002] Research Example: IEM
[Source: Berg, DARPA Workshop, 2002] Research Example: IEM
Does it work? [Thanks: Yiling Chen] Ø Yes, evidence from real markets, laboratory experiments, and theory v. Racetrack odds beat track experts [Figlewski 1979] v. Orange Juice futures improve weather forecast [Roll 1984] v. I. E. M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002] v. HP market beat sales forecast 6/8 [Plott 2000] v. Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’ 03][Schmidt 2002] v. Laboratory experiments confirm information aggregation [Plott 1982; 1988; 1997][Forsythe 1990][Chen, EC’ 01] v. Theory: “rational expectations” [Grossman 1981][Lucas 1972] v. Market games work [Servan-Schreiber 2004][Pennock 2001]
Research Prediction Markets: Does Money Matter?
Research The Wisdom of Crowds With Money Without IEM: 237 Candidates HSX: 489 Movies
Research The Wisdom of Crowds With Money Without
Research Real markets vs. market games HSX FX, F 1 P 6 probabilistic forecasts forecast source F 1 P 6 linear scoring F 1 P 6 F 1 -style scoring betting odds F 1 P 6 flat scoring F 1 P 6 winner scoring avg log score -1. 84 -1. 82 -1. 86 -2. 03 -2. 32
Research Does money matter? Play vs real, head to head Experiment • 2003 NFL Season • Probability. Sports. com Online football forecasting competition • Contestants assess probabilities for each game • Quadratic scoring rule • ~2, 000 “experts”, plus: • News. Futures (play $) • Tradesports (real $) • Results: • Play money and real money performed similarly • 6 th and 8 th respectively • Markets beat most of the ~2, 000 contestants • Average of experts came 39 th (caveat) Used “last trade” prices Electronic Markets, Emile Servan. Schreiber, Justin Wolfers, David Pennock and Brian Galebach
Research
Research Does money matter? Play vs real, head to head Statistically: TS ~ NF NF >> Avg TS > Avg
Research Discussion • Are incentives for virtual currency strong enough? • Yes (to a degree) • Conjecture: Enough to get what people already know; not enough to motivate independent research • Reduced incentive for information discovery possibly balanced by better interpersonal weighting • Statistical validations show HSX, FX, NF are reliable sources forecasts • HSX predictions >= expert predictions • Combining sources can help
Research A Problem w/ Virtual Currency Printing Money Alice 1000 Betty 1000 Carol 1000
Research A Problem w/ Virtual Currency Printing Money Alice 5000 Betty 1000 Carol 1000
Research Yootles A Social Currency Alice 0 Betty 0 Carol 0
Research Yootles A Social Currency I owe you 5 Alice -5 Betty 0 Carol 5
Research Yootles A Social Currency I owe you 5 credit: 5 Alice -5 credit: 10 Betty 0 Carol 5
Research Yootles A Social Currency I owe you 5 credit: 5 Alice -5 credit: 10 Betty 0 Carol 5
Research Yootles A Social Currency I owe you 5 credit: 5 Alice 3995 credit: 10 Betty 0 Carol 5
Research Yootles A Social Currency • For tracking gratitude among friends • A yootle says “thanks, I owe you one”
Research Combinatorial Betting
Research Combinatorics Example March Madness
Research Combinatorics Example March Madness • Typical today Non-combinatorial • • Team wins Rnd 1 Team wins Tourney A few other “props” Everything explicit (By def, small #) • Every bet indep: Ignores logical & probabilistic relationships • Combinatorial • Any property • Team wins Rnd k Duke > {UNC, NCST} ACC wins 5 games • 2263 possible props (implicitly defined) • 1 Bet effects related bets “correctly”; e. g. , to enforce logical constraints
Expressiveness: Getting Information • Things you can say today: – – (63% chance that) Obama wins GOP wins Texas YHOO stock > 30 Dec 2007 Duke wins NCAA tourney • Things you can’t say (very well) today: – – Oil down, DOW up, & Obama wins election, if he wins OH & FL YHOO btw 25. 8 & 32. 5 Dec 2007 #1 seeds in NCAA tourney win more than #2 seeds
Expressiveness: Processing Information • Independent markets today: – – Horse race win, place, & show pools Stock options at different strike prices Every game/proposition in NCAA tourney Almost everything: Stocks, wagers, intrade, . . . • Information flow (inference) left up to traders • Better: Let traders focus on predicting whatever they want, however they want: Mechanism takes care of logical/probabilistic inference • Another advantage: Smarter budgeting
Research Market Combinatorics Permutations • A>B>C • A>C>B • B>A>C . 1. 2. 1 • B>C>A • C>A>B • C>B>A . 3. 1. 2
Research Market Combinatorics Permutations • • • D>A>B>C D>A>C>B D>B>A>C A>D>B>C A>D>C>B B>D>A>C A>B>D>C A>C>D>B B>A>D>C A>B>C>D A>C>B>D B>A>C>D . 01. 02. 05. 01. 2. 01. 02. 01 • • • D>B>C>A D>C>A>B D>C>B>A B>D>C>A C>D>A>B C>D>B>A B>C>D>A C>A>D>B C>B>D>A . 05. 1. 2. 03. 1. 02. 03. 01. 02
Research Bidding Languages • Traders want to bet on properties of orderings, not explicitly on orderings: more natural, more feasible • A will win ; A will “show” • A will finish in [4 -7] ; {A, C, E} will finish in top 10 • A will beat B ; {A, D} will both beat {B, C} • Buy 6 units of “$1 if A>B” at price $0. 4 • Supported to a limited extent at racetrack today, but each in different betting pools • Want centralized auctioneer to improve liquidity & information aggregation
Research Example • A three-way match • Buy 1 of “$1 if A>B” for 0. 7 • Buy 1 of “$1 if B>C” for 0. 7 • Buy 1 of “$1 if C>A” for 0. 7 B A C
Research Pair Betting • All bets are of the form “A will beat B” • Cycle with sum of prices > k-1 ==> Match (Find best cycle: Polytime) • Match =/=> Cycle with sum of prices > k-1 • Theorem: The Matching Problem for Pair Betting is NP-hard (reduce from min feedback arc set)
Research [Thanks: Yiling Chen] Automated Market Makers • • A market maker (a. k. a. bookmaker) is a firm or person who is almost always willing to accept both buy and sell orders at some prices Why an institutional market maker? Liquidity! • • Without market makers, the more expressive the betting mechanism is the less liquid the market is (few exact matches) Illiquidity discourages trading: Chicken and egg Subsidizes information gathering and aggregation: Circumvents no-trade theorems Market makers, unlike auctioneers, bear risk. Thus, we desire mechanisms that can bound the loss of market makers • Market scoring rules [Hanson 2002, 2003, 2006] • Dynamic pari-mutuel market [Pennock 2004]
Overview: Complexity Results Permutations General Pair Boolean Subset General Taxonomy 2 -clause Restrict General Tree ? ? ? Tourney Call Market NP-hard Poly NP-hard EC’ 07 DSS’ 05 co-NPcomplete DSS’ 05 Market Maker (LMSR) #P-hard #P-hard Poly EC’ 08 EC’ 08 STOC’ 08 XYZ‘ 09 Approx STOC’ 08
• March Madness bet constructor • Bet on any team to win any game – Duke wins in Final 4 • Bet “exotics”: – Duke advances further than UNC – ACC teams win at least 5 – A 1 -seed will lose in 1 st round
Research New Prediction Game: Yoopick An Application on Facebook
Research Catalysts • Markets have long history of predictive accuracy: why catching on now as tool? • No press is bad press: Policy Analysis Market (“terror futures”) • Surowiecki's “Wisdom of Crowds” • Companies: • Google, Microsoft, Yahoo!; Crowd. IQ, HSX, Inkling. Markets, News. Futures • Press: Business. Week, CBS News, Economist, NYTimes, Time, WSJ, . . . http: //us. newsfutures. com/home/articles. html
CFTC Role • • May. Day 2008: CFTC asks for help Q: What to do with prediction markets? Yahoo!, Google entered suggestions Right now, the biggest prediction markets are overseas, academic (1), or just for fun • CFTC may clarify, drive innovation Or not
Research Conclusion • Prediction Markets: hammer = market, nail = prediction • Great empirical successes • Momentum in academia and industry • Fascinating (algorithmic) mechanism design questions, including combinatorial betting • Points-paid peers produce prettygood predictions
- The wisdom of crowds
- James 3 earthly wisdom
- Godly wisdom vs worldly wisdom
- Earthly and heavenly wisdom
- The chief determinant of popularity in high school is
- Cliques and crowds
- Faktor pembentuk kelompok sosial
- Dina katabi
- Bustling crowds
- Examples of conventional crowds
- Grow in wisdom and knowledge
- Wealth and wisdom institute
- Wisdom excels folly
- Rabbi zidni ilma dua
- The wisdom of proverbs job and ecclesiastes
- I heard myself proclaimed
- Make a prediction about kenny and franchesca
- Inference reading strategy
- Prairie storm prediction centre
- Difference between prediction and forecasting
- Difference between static and dynamic branch prediction
- Gene prediction in prokaryotes and eukaryotes
- Hát kết hợp bộ gõ cơ thể
- Lp html
- Bổ thể
- Tỉ lệ cơ thể trẻ em
- Voi kéo gỗ như thế nào
- Glasgow thang điểm
- Bài hát chúa yêu trần thế alleluia
- Các môn thể thao bắt đầu bằng tiếng chạy
- Thế nào là hệ số cao nhất
- Các châu lục và đại dương trên thế giới
- Công thức tính độ biến thiên đông lượng
- Trời xanh đây là của chúng ta thể thơ
- Mật thư anh em như thể tay chân
- 101012 bằng
- độ dài liên kết
- Các châu lục và đại dương trên thế giới
- Thơ thất ngôn tứ tuyệt đường luật
- Quá trình desamine hóa có thể tạo ra
- Một số thể thơ truyền thống
- Bàn tay mà dây bẩn
- Vẽ hình chiếu vuông góc của vật thể sau
- Nguyên nhân của sự mỏi cơ sinh 8
- đặc điểm cơ thể của người tối cổ
- V cc
- Vẽ hình chiếu đứng bằng cạnh của vật thể
- Fecboak
- Thẻ vin
- đại từ thay thế
- điện thế nghỉ
- Tư thế ngồi viết
- Diễn thế sinh thái là
- Các loại đột biến cấu trúc nhiễm sắc thể
- Các số nguyên tố là gì