Utility data annotation via Amazon Mechanical Turk X

Utility data annotation via Amazon Mechanical Turk X 100 000 = $5000 Alexander Sorokin David Forsyth University of Illinois at Urbana-Champaign http: //visionpc. cs. uiuc. edu/~largescale/

Motivation • • Unlabeled data is free Labels are useful We need large volumes of labeled data Different labeling needs: • • • Is there X in the image? Outline X. Where is part Y of X. Of these 500 images, which belong to category X? ……………. and many more ……………….

Amazon Mechanical Turk Workers Task: Dog? Answer: Yes Broker Is this a dog? o Yes o No Task $0. 01 Pay: $0. 01 www. mturk. com

Motivation X 100 000 = $5000 Custom annotations Large scale Low price

Annotation protocols • • • Type keywords Select relevant images Click on landmarks Outline something Detect features ………. . anything else ………

Type keywords $0. 01 http: //austinsmoke. com/turk/.

Select examples Joint work with Tamara and Alex Berg http: //visionpc. cs. uiuc. edu/~largescale/data/simpleevaluation/html/horse. html

Select examples $0. 02 requester mtlabel

Click on landmarks $0. 01 http: //vision-app 1. cs. uiuc. edu/mt/results/people 14 -batch 11/p 7/

Outline something $0. 01 http: //visionpc. cs. uiuc. edu/~largescale/results/production-3 -2/results_page_013. html Data from Ramanan NIPS 06

Detect features Measuring molecules. Joint work with Rebecca Schulman (Caltech) ? ? $0. 1 http: //visionpc. cs. uiuc. edu/~largescale/all_examples. html

Motivation X 100 000 = $5000 Custom annotations Large scale Low price

Issues • Quality? – How good is it? – How to be sure? • Price? – How to price it? • How does MTurk compare with others? • How do I sign up? sorokin 2@uiuc. edu http: //visionpc. cs. uiuc. edu/~largescale/

Annotation quality Agree within 5 -10 pixels on 500 x 500 screen There are bad ones. A C E G

Grading tasks • Take 10 submitted results • Create new task to verify the result • Verification is easy – Pay the same or slightly higher price • Total overhead - 10% (work in progress) http: //vision-app 1. cs. uiuc. edu/mt/grading/people 14 -batch 11 -small/p 1/

Price • $0. 01 per image (16 clicks) ~ $1500 / 100 000 images >1000 images per day <4 months • Workers suggested $0. 03 - $0. 05/img – $3500 - $5500 / 100 000 images

Is the price right? $0. 01 / 14 clicks 1. 6 hours 900 labels $0. 01 / 16 clicks 4 hours 900 labels $0. 01/ 40 clicks 15 hours 900 labels

Annotation Method Comparison Approach Cost Scale Setup effort Centralized Quality Elastic to $ MTurk $ +++ no +/+++ +++++ Yes ++ * Label. ME ++ Image. Parsing. com $$ ++ ** Yes ++++ Games with purpose (ESP++) ++++ *** Yes + + + * no +++ ++ In house $$$

How do I sign up? • Go to our web page: http: //visionpc. cs. uiuc. edu/~largescale/ • Send us an e-mail: sorokin 2@uiuc. edu • Register at Amazon Mechanical Turk http: //www. mturk. com

Acknowledgments Special thanks to: David Forsyth Tamara Berg Rebecca Schulman David Martin Kobus Barnard Mert Dikmen All workers at Amazon Mechanical Turk This work was funded in part by ONR

X 100 000 = $5000 Thank you