Insights into Mechanical Turk or Mistakes Requesters Make

What is Mechanical Turk? Amazon Mechanical Turk is a marketplace for work. © 2009

Feedback § Mechanical Turk provides a set of primitives • • • HIT properties

Law of Unintended Consequences is True © 2009 Amazon. com, Inc. or its Affiliates.

Plurality to the Rescue (? ) © 2009 Amazon. com, Inc. or its Affiliates.

When Plurality Diverges § Possible Root Causes – Worker skills/training/knowledge – Ergonomics – Worker

When Plurality Diverges § Worker Quality and Worker History – 1000 questions, 5 answers

When Plurality Diverges § Which Workers do you Trust? – “Gold Standard”, “Ground Truth”,

18 2/ 10 /2 2/ 010 25 /2 0 3/ 10 4/ 2 3/

Turnaround time for Requester X’s HITs Anyone try out these HITs. 160 140 I

1 6/30/2010 6/29/2010 6/28/2010 6/27/2010 6/26/2010 6/25/2010 6/24/2010 6/23/2010 6/22/2010 6/21/2010 6/20/2010 6/19/2010 6/18/2010

Optimizing For Best Results Price Speed Accuracy © 2009 Amazon. com, Inc. or its

By the way – We’re Hiring! turk-jobs@amazon. com © 2009 Amazon. com, Inc. or

Questions? abradley@amazon. com © 2009 Amazon. com, Inc. or its Affiliates.

Slides: 14

Download presentation

Feedback § Mechanical Turk provides a set of primitives • • • HIT properties (reward, instructions, requirements) Assignment can be approved or rejected Worker can be bonused Worker can be blocked Qualification can be assigned or revoked § You use these primitives to execute your “policies” § Your policies shape your Reputation © 2009 Amazon. com, Inc. or its Affiliates.

When Plurality Diverges § Possible Root Causes – Worker skills/training/knowledge – Ergonomics – Worker effort – Ambiguity in the Instructions – Ambiguity in the Data © 2009 Amazon. com, Inc. or its Affiliates.

When Plurality Diverges § Worker Quality and Worker History – 1000 questions, 5 answers each = 5000 independent answers – 5 Workers each answer 1000 questions § See Ipeirotis et al in HCOMP 2010 © 2009 Amazon. com, Inc. or its Affiliates.

When Plurality Diverges § Which Workers do you Trust? – “Gold Standard”, “Ground Truth”, “Trap” • “Meaningful” Sample? – Approval History…? – Is “Trust” closed or open? © 2009 Amazon. com, Inc. or its Affiliates.

18 2/ 10 /2 2/ 010 25 /2 0 3/ 10 4/ 2 3/ 010 11 /2 3/ 010 18 /2 3/ 010 25 /2 0 4/ 10 1/ 20 4/ 10 8/ 2 4/ 010 15 /2 4/ 010 22 /2 4/ 010 29 /2 0 5/ 10 6/ 2 5/ 010 13 /2 5/ 010 20 /2 5/ 010 27 /2 0 6/ 10 3/ 2 6/ 010 10 /2 6/ 010 17 /2 6/ 010 24 /2 0 7/ 10 1/ 20 7/ 10 8/ 20 10 01 /2 0 11 2/ 4/ 2 2/ /2 0 28 1/ Turnaround time for Requester X’s HITs 160 140 120 100 80 60 40 20 0 avg(min) © 2009 Amazon. com, Inc. or its Affiliates.

Turnaround time for Requester X’s HITs Anyone try out these HITs. 160 140 I stopped doing these HITS since the rejection rate has increased tremendously compared to the initial stage 120 100 thanks, for telling me about these. 80 60 I found my new favorite HIT 40 20 10 /2 2/ 010 25 /2 0 3/ 10 4/ 2 3/ 010 11 /2 3/ 010 18 /2 3/ 010 25 /2 0 4/ 10 1/ 20 4/ 10 8/ 2 4/ 010 15 /2 4/ 010 22 /2 4/ 010 29 /2 0 5/ 10 6/ 2 5/ 010 13 /2 5/ 010 20 /2 5/ 010 27 /2 0 6/ 10 3/ 2 6/ 010 10 /2 6/ 010 17 /2 6/ 010 24 /2 0 7/ 10 1/ 20 7/ 10 8/ 20 10 0 2/ 18 /2 0 01 11 2/ 4/ 2 2/ 1/ 28 /2 0 10 0 I have done about 700 with 5 rejects …. Honestly I feel like my rejects have come from being tired and selecting the wrong answer avg(min) © 2009 Amazon. com, Inc. or its Affiliates.

1 6/30/2010 6/29/2010 6/28/2010 6/27/2010 6/26/2010 6/25/2010 6/24/2010 6/23/2010 6/22/2010 6/21/2010 6/20/2010 6/19/2010 6/18/2010 6/17/2010 6/16/2010 6/15/2010 6/14/2010 6/13/2010 6/12/2010 6/11/2010 6/10/2010 6/9/2010 6/8/2010 6/7/2010 6/6/2010 6/5/2010 6/4/2010 6/3/2010 6/2/2010 6/1/2010 Average Turn Time - Minutes Comparison of Turnaround Time 10000 100 Requester X Requester Y 10 © 2009 Amazon. com, Inc. or its Affiliates.