Crowdsourcing Quality Control Mechanisms Vlad Hosu Universitt Konstanz

Crowdsourcing Quality Control Mechanisms Vlad Hosu Universität Konstanz, 21. 06. 2017 With thanks to Matthias Hirth from University of Würzburg Universität Konstanz

Agenda 1. Honesty in Online Labor Markets 2. Exemplary Quality Control Mechanisms − Filtering of workers − Screening of workers − Statistical methods − Workflow based mechanisms 3. Costs of Quality Control Mechanisms 2 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Honesty in Online Labor Markets 3 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Honesty in Online Labor Markets • Suri et al. "Honesty in an Online Labor Market. ” • User studies on MTurk with varying bonus payment • Study 1 • Fixed payment per task 0. 25 USD • Bonus payment 0. 25 USD times the roll of a die (ratio between minimum and maximum payment: 3. 5) • Users asked to use own die or random. org • 175 participants • 4 Study 2 • Fixed payment per task 0. 25 USD • Fixed bonus payment of 0. 75 USD upon completion plus 0. 05 USD times the roll of a die (ratio between minimum and maximum payment: 1. 24) • Users asked to use own die or random. org • 267 participants 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Study Results Study 1 • • 5 Study 2 In both studies: Significant trends towards higher rolls Workers tend to be dishonest even if only small benefits are expected • (Lab participants showed the same behavior) 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Influencing Factors on Poor Quality Educate Train Workers are not qualified for the task Workers do not understand instructions POOR QUALITY Workers do not work thoughtfully Workers deliberately cheat Task Design Remove QUALIFICATION 6 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms RELIABILITY Universität Konstanz

Which Workers do You Need? Trustworthy Qo. E Measurement Non. Trustworthy Qualified Surveys Data Extraction & Labeling Non-Qualified Software Testing 7 21. 06. 2017 Content Creation Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Types of Malicious Behavior • 8 Different type of malicious behavior observed in exemplary online survey (Gadiraju, Ujwal, et al. "Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. ”) • Ineligible Workers • Workers that do not conform with priory stated prerequisites • Example: Worker performs task not on the requested device • Fast Deceivers • Workers trying to maximize income supplying ill-fitting response • Example: Worker copies parts of the instruction into free text questions • Rule Breakers • Workers not sticking to the instructions • Example: Worker does not care about word limit for articles • Smart Deceivers • Similar to Fast Deceivers, but stick to instructions to void automatic detection • Example: Correctly formatted but irrelevant image tags 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Possible Attacks on Tasks • • 9 Individual attacks • Random Answers: Worker manually submits random/incorrect answer to minimize effort • Semi-Automated Answers: Worker uses script to automate parts of the spamming process • Automated Answers: Worker uses bot to automatically submit random/incorrect answer to minimize effort Group Attacks • Agree on Answers: Workers communicate submit consistent answer to unique task items, e. g. , images • Answer Sharing: Answers for gold standard data questions are stored and shared among workers • Artificial Clones: Workers create clones that act as independent workers and duplicate the workers answers 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Exemplary Quality Control Mechanisms 10 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Types of Quality Control Mechanisms • • 11 Filtering of workers • Applied before a task • Restricts access to the task to certain workers • Exemplary filter criteria: Country of origin, reputation Screening of workers • Applied during a task • Based on explicit and implicit tests integrated in the task • Examples: Test questions/consistency questions, attention tests, gold standard data, behavior monitoring Statistical methods • Applied after task completion • Based on task results • Examples: Random Clicker, Quadrant of Euphoria, Crowd. MOS, BT. 500 -13 Workflow based mechanisms • Based on combination of different task results • Examples: Repetition and aggregation, two-stage design, Find-Fix-Verify 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

FILTERING OF WORKERS Universität Konstanz

Filtering of Workers • Applied before beginning of task to restrict access for set of workers • Filter criteria often based on • Results from previous campaigns / qualification tasks • Reputation of workers / work history on platform • Properties of worker, e. g. , country of origin • Possible application: Filtering of known spammers • Issues • Reputation data on crowdsourcing platform often ‘broken’ • Potential discrimination of new workers • Ethical issues when filtering of users based on country of origin 13 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

SCREENING OF WORKERS Universität Konstanz

Screening of Workers • • • 15 Applied during the task to estimate quality of worker Screening mechanisms based on explicit tests • Content questions • Consistency questions • Trap questions • Attention tests • Gold standard data Screening mechanisms based on implicit feedback • Behavior monitoring Possible application • Assessment of worker qualification, attention, and diligence • Assessment of trustworthiness and honesty Issues • Often generation of reference data required Limited set of test items • Some methods need to be adapted to the specific tasks 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Verifiable Content Questions • Simple questions about the item/content the worker needs to process/pay attention to during a task • Content question easier to verify than actual task result • Example: Video quality assessment • Worker supposed to watch a video and give a subjective rating on the quality • Possible content question: ‘Which sport was shown in the clip? A) Tennis. B) Soccer. C) Skiing. ’ • Example: Keywords for an article • Worker need to read the article and provide meaningful keywords • Possible content question: ‘How may paragraphs does the article have’ 16 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Consistency Questions • Same question is asked multiple times in a slightly different manner • Example • Pre-task survey: ‘Please select your country of origin’ • Post-task survey: ‘Which continent do you live on? ’ • Potential issues • Subjects might not be willing to provide correct personal data • Some questions might be ambiguous or too difficult Unnecessary rejection of valid data 17 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Attention Checks • Actions to objectively assess the focus of the worker on the task • Example: Video quality assessment • Worker supposed to watch a video and give a subjective rating on the quality • Attention check: An additional button is added to the test, worker need to click the button on random requests • Example: Survey • Worker requested to complete a survey • Attention check: Asking worker to select a (unlikely/impossible) option in the next answer • Potential issues • Worker might be distracted (e. g. video test) • Checks might be too obvious (cf. survey) 18 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Trap Questions • Question and answer schemes encouraging sloppy users to select incorrect answers • Example: Information extraction task • Worker supposed to extract information from email • Given email header • • 19 Possible answers Potential issue: Correct data needs to be available 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Trustworthiness Tests • Testing basic willingness to give honest answer (important for Qo. E and other subjective tests) • Only measurable via psychological tests • Usually too complex and too long for crowdsourcing test • But: Adding some questions from psychological tests as estimation possible • General idea • Usage of questions that seem to be not-verifiable at first sight • Examples: − Number of visible cats − Recognized number of differences in two images 20 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Gold Standard Data • • 21 Questions whereof the correct results are already known • Gold standard questions are interspersed among other questions • Most common quality control approach Example: Video quality assessment • Worker supposed to watch a video and give a subjective rating on the quality • Gold standard question: ’Did you notice any stallings in the video’? Example: Image categorization • Worker needs to add an image to one of two categories • Gold standard data: Set of (unambiguous) categorized images included in the regular tasks Potential issue: • Additional costs for creating gold standard data • Limited set of gold standard data (approach to create/extend gold standard data automatically) • Gold standard data need to be robust against worker bias 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Behavior Monitoring to Identify Unreliable Users • Idea: Use objective measures on application-level for estimating the user interactions with the application • Goal: Estimate quality of the task results based on technical parameters • Challenges: Expected behavior expected measurement values Diverging measurements reconstruction of behavior • • 22 Model of “correct” behavior required (for every task different) Implementation can become very complex 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Image from sxc. hu by zenpixel Universität Konstanz

Application Layer Measurements (ALM) Prerequisite: Web based task interface Client side Server side • • Analysis of interactions with web browser • Mouse movement • Scrolling • Window focus • … Analysis using server side logging • Requests pages • Time spend on a specific page • … Privacy: Non intrusive, only task environment is observable 23 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Case Study: Language Test Hirth, Matthias, et al. "Predicting result quality in crowdsourcing using application layer monitoring. ” • Requirements for test case • Task needs to be “simple” (no special skills required) • Task evaluation must be objective • English language test with multiple choice answers • Test design • Five texts from the simplified English version of Wikipedia • Five multiple choice questions each text with four possible answers • Implicit feedback about the user via, question design, answer design, and free text question about the most interesting text • User study • 215 participants in Feb. 2013 (0. 10 USD per task) • Participants from 22 different nations (2% Native speaker) 24 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Implementation of the Language Test Trustworthiness Trap 1 2 Incentive 3 Online Translation Service 25 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Identification Suitable Qualification Threshold 26 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Evaluation of Language Test Results • Scores normalized to 100% Qualification Threshold: 50% • At minimum two answers correct • 18% of the candidates achieved the maximum score • 71% “qualified” How can we predict a worker’s test result? 27 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

ALM User Modeling • Expected behavior 1. Reading instructions 2. Start with first questions/check length of test 3. Read the text 4. Read answer the questions/recheck the text 5. Proceed to next text • 28 Expected interaction • Waiting • Scrolling • Interacting 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

ALM Implementation • • 29 Implemented measures • Time of page access and leaving • Vertical scroll position − Synchronous measure every 10 sec − Allows reconstructing user’s field of sight • Interactions with answering elements − Asynchronous measure based on interaction events − Allows reconstructing answering times Evaluated metrics • Completion time − Time between entering the test and submitting the final result − Simple metric only considering server side measures • Consideration time − Time between first time user saw question and last interaction with the related answer − Complex measure based on multiple client side measures 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Completion Time ? Plausibility Threshold • • Minimum time to read the test Automatically pre-calculated Filters only a small number of workers Temporal qualification Threshold • • Minimum completion time of qualified workers Derived from test results Requires evaluated test results No clear differentiation of workers possible in most cases! 30 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Consideration Time and Question Type Different question types • Comprehension Required information stated explicitly in text • Reorganization Combination of several explicit information from text • Inference Required information only implicitly stated in text Significant differences between qualified and unqualified workers • Qualified workers spend more time in general • Time spend by qualified workers increased with difficulty of the questions 31 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Consideration Time and Answer Type Different answer types • Plausible, in text Correct information stated in the text • Plausible, not in text Information matching question but not in the text • Not plausible, in text Information from the text but not matching the question Observations • Finding correct answer requires largest amount of time (plausible, in text) • Users guessing the answer without reading require least amount of time (plausible, not in text) • Users guessing the answer without reading the question properly lay in between (not plausible, in text) 32 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Conclusion Application Layer Monitoring Low quality results in Crowdsourcing • Different reasons for low result • Reasons hard to detect Understanding of worker-task interactions necessary Application layer monitoring • Non intrusive technical worker monitoring on client and server side - Task specific and complex to implement + Allows designing objective worker-task interaction measures Results of proof-of-concept implementation • Traditional interaction measures (completion time) outperformed by more sophisticated ALM measures (consideration time) • Support Vector Machine using ALM measures capable of predicting worker qualification with accuracy of ~89% 33 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

STATISTICAL METHODS Universität Konstanz

Random Clicker 35 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Crowd. MOS Assumptions • In subjective studies, worker ratings should converge to a global trend • Individual workers should agree with this trend Test idea: Use sample correlation to compare user ratings with global average ratings 36 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Transitivity Satisfaction Rate Assumption: Subjective ratings are transitive (A is preferred to B, B is preferred to C A is preferred to C) Test idea: • Use pair comparisons of all data points • Compute compliance with transitivity assumptions 37 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

ITU-R BT. 500 38 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Reliability of Statistical Methods Dataset • Subjective study on impact of stalling on You. Tube Qo. E • Gold standard data: Unreliable workers identified by content questions and behavior monitoring Issues in this study • Different statistical methods identify different sets of reliable and unreliable workers • Some methods show large number of false negatives (no money for those users) General issue: Filtering based on evaluated data 39 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

WORKFLOW BASED APPROACH Universität Konstanz

Workflow Based Approach • Based on combination of task results and varying tasks from different workers • Examples • Find-Fix-Verify • Majority decision • Control group decision • Two-Stage Design • Possible application • Assessment of worker quality • Increasing the quality of final outcome • Issues: Need to be tailored to the specific task 41 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Majority Decision 42 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Control Group 43 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Probability for Correct Control Group Decision Main worker Correct result Incorrect result Correct decision Control group Incorrect decision Main worker Correct result Incorrect result Control Correct decision group Incorrect decision 44 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Two Stage Design • Combined workflow of using different quality control mechanisms • Phase 1 • Pre-filtering of users according to test requirements • Simple and cheap task including training phases, qualification and trustworthiness tests (all workers paid except obvious spammers) Pseudo reliable crowd identified • Phase 2 • • • 45 Actual task, longer and more expensive Only accessible by trusted workers from phase 1 Limited application of quality control mechanism to save time and money 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Costs of Quality Control Mechanisms 46 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Costs of Quality Control • Total cost of quality related activities dividable in conformance costs and non-conformance costs • Conformance costs: Costs spend on activities to avoid poor quality • Prevention costs − Cost for activities to prevent end result from failing quality requirements − Example: Robust workflow design, easy to use task interface • Appraisal costs − Costs for finding errors − Examples: Repetition of tasks, additional work for gold standard tasks • Non-conformance costs: Costs resulting from poor quality • Internal failure − Cost due to detected quality issues − Example: Reassignment of failed tasks • External failure − Costs due to errors in the final product − Examples: Invalid translation used in article, low quality article published • Trade-off between conformance and non-conformance costs necessary 47 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Cost Model for Majority Decisions Costs incorrect Costs correct submissions Costs incorrect submissions 48 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Costs correct submissions Universität Konstanz

Impact of Cost Factors Majority Decision 49 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Cost Model for Control Group Approach Main worker Correct result Incorrect result Control Correct decision group Incorrect decision 50 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Cost Model for Control Group Approach Costs for correctly accepted main task Number of rejected main tasks 51 21. 06. 2017 Costs for incorrectly accepted main task Costs per rejected main task Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Impact of Cost Factors Control Group Approach 52 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Cost Optimal Mechanism for Routine Tasks • Assumptions • Low costs per task, wrong task not paid at all • Small/no difference of costs for main and control task • Some invalid submissions tolerable Majority decision always better than control group approach 53 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz

Cost Optimal Mechanism for Complex Tasks 54 21. 06. 2017 Crowdsourcing - Quality Control Mechanisms Universität Konstanz