Presenter Lin SinYan Human Computation ESP Game 20081219

Presenter: Lin, Sin-Yan Human Computation & ESP Game 2008/12/19 1

References • Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems. • Ahn, L. V. , Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems. • Ahn, L. V. (2007) Human computation, paper presented at the Proceedings of the 4 th international conference on Knowledge capture. • Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58 -67. • Chen, L. -J. , Wang, B. -C. , Chen, C. -Y. , King, I. & Lee, J. (2008) An analytical study of puzzle selection strategies for the esp game. Report for Institute of Information Science, Academia Sinica (Taipei, Taiwan, ROC). • Weber, I. , Robertson, S. & Vojnovi´C, M. (2008) Rethinking the esp game. Report for Microsoft Research, Microsoft Corporation. 2

Outline • Human Computation • Human Computation System and Application • Human Computation in E-Learning • The ESP Game • Mathematical Model in ESP Game • System Gain • Image selection • Automatic playing the ESP Game • Problems of ESP Game • Conclusion and Comments 3

Human Computation • Concepts are started at 2001: The Open Mind Initiative • Some task are trivial for humans, but continue to challenge computer algorithms • Treat human brains as processors in a distributed system, each can perform a small part of a massive computation • But humans require some incentive to become part of a collective computation • By Games (Ahn and Dabbish, 2004) (Ahn and Dabbish, 2008) Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems 4 Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58 -67.

Human Computation System - Peekaboom Ahn, L. V. , Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems. 5

Human Computation System - Photo. Shoot (1/2) Cheng, K. -Y. (2006). Photo. Shoot: A Web-Game for User Assisted ROI Labeling. Information Management. Taipei, Taiwan, National Taiwan University. Master. 6

Human Computation System - Photo. Shoot (2/2) 7

Human Computation System - Image. Hunter Chen, L. -H. (2008). Accuracy Analysis in Productivity Games: A Case Study on Landmark Annotation. Department of Computer Science and information Engineering. Taipei, Taiwan, National Taiwan University. Master: 62. 8

Human Computation System - Photo. Slap Chang, T. -H. (2007). Productivity Game Design and Gameplay Analysis: Play and Annotate with Photo. Slap. Department of Computer Science and Information Engineering. Taipei, Taiwan, National Taiwan University. Master: 52. 9

Human Computation System - re. CAPTCHA • Proposed by Ahn (2008) • Gathering the characters that optical character recognition (OCR) software has been unable to read 10

Human Computation in E-Learning - why tag • Free, uncontrolled vocabulary • Social relations of TAGs image photo color design tagging art research 12

Agenda • Human Computation • Human Computation System and Application • Human Computation in E-Learning • The ESP Game • Mathematical Model in ESP Game • System Gain • Image selection • Automatic playing the ESP Game • Problems of ESP Game • Conclusion and Comments 13

The ESP Game • The first human computation system to take advantage of people’s desire to be entertained and provide useful metadata Randomly selected image Input Keywords Keyword Match Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems 14

Mathematic Model in ESP Game • There are many parameters in ESP Game • We can use those parameters to modeling many things that we want to know • Ex. • Number of players • Number of labels per game • Number of guesses to reach a consensus • Good and bad word assumption • (Human Computation) System Gain • Image selection • User scoring model 15

Mathematic Model in ESP Game - System Gain Modeling (1/2) • The performance of human computation system • The purpose of ESP Game tries to collect images’ tags • Two aspect of thinks • Take as many tags as possible for each image • Playing the maximum number of image • A metric to evaluate the system gain 16

Mathematic Model in ESP Game - System Gain Modeling (2/2) 17

Mathematic Model in ESP Game - Image Selection (1/3) • Different image selection method • Random • Fresh-first • Maximize ln(N) • Optimal (OPSA) • Largest possible system gain • P 0: all images that have not been played • P 1: all images that have been played at least once, but less than r times • P 2: other images 18

Mathematic Model in ESP Game - Image Selection (2/3) T/N 19

Mathematic Model in ESP Game - Image Selection (3/3) M: Total number of images in the system Wang, B. -C. , C. -W. Lin, et al. (2008). An Analytical Study and Modeling of the ESP game. Taipei, Taiwan, Institute of Information Science, Academia Sinica. 20

Mathematic Model in ESP Game - Automatic Playing the ESP Game (1/4) • Given enough instances of labeled images as training data • Similar with Language model • Assign probability to the next label to be added • Plays the ESP Game without looking at the image • Limitation: at least one “off-limits” in the image Weber, I. , Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 21

Mathematic Model in ESP Game - Language Modeling (1/2) • Model the sequence of word usage of a certain language • Given a Language Model: M • P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this) 22

Mathematic Model in ESP Game - Language Modeling (2/2) • P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this) • Markov assumption: • To give reasonable predictions, only the prior local context (the last few words) affects the next word • n-gram word model • Bi-gram: • P(I love this game) = P(I) P(love|I) P(this|love) P(game|this) 23

Mathematic Model in ESP Game - Automatic Playing the ESP Game (2/4) • Given enough instances of labeled images as training data • At least one “off-limits” in the image • t: next label • T: off-limits set • P(“t is next label” | “set T already present”) Weber, I. , Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 24

Mathematic Model in ESP Game - Automatic Playing the ESP Game (3/4) • P(“t is next label” | “set T already present”) • Conditional independence assumption • The problem of zero probability Weber, I. , Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 25

Mathematic Model in ESP Game - Automatic Playing the ESP Game (4/4) • The problem of zero probability Weber, I. , Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 26

Mathematic Model in ESP Game - Others • User scoring model • More informative tags 27

Problems of ESP Game • There is a lot of redundancy in the tag sets • Even when tags are not exactly synonyms, they are often “to be expected” given the other tags • There is a tendency to match on colors • People tend to add more generic labels Weber, I. , Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 28

Conclusion and Comments • The first thing of Human Computing research is finding a clear PURPOSE • How your game or method to achieve the purpose 29