microproductivity mkr prdktivt noun Jaime Teevan Microsoft Research

mi·cro·pro·duc·tiv·i·ty /ˈmīkrō prōˌdəkˈtivətē/ noun Jaime Teevan, Microsoft Research, @jteevan

Unique Solution to Focus & Multitasking • Our focus is fragmented • Constant interruptions, multitasking • Our time is fragmented • Mobile micromoments filled with Angry Birds • Current solutions promote focus mi·cro·pro·duc·tiv·i·ty /ˈmīkrō prōˌdəkˈtivətē/ noun The transformation of large productivity tasks into a set of smaller microtasks that can be completed individually in short bursts of time with limited context.

Nebeling, To, Guo, de Freitas, Teevan, Dow & Bigham. Wear. Write: Crowd-assisted writing from smartwatches. CHI 2016. Kittur, Smus, Khamkar & Kraut. Crowd. Forge: Crowdsourcing complex work. UIST 2011. Agapie, Teevan & Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015. Luther, Hahn, Dow & Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015. Hahn, Chang, Kim & Kittur. The Knowledge Accelerator: Big picture thinking in small pieces. CHI 2016. Kim, Cheng & Bernstein. Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. CSCW 2014. Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell & Panovich. Soylent: A word processor with a crowd inside. UIST 2010.

② ① ③ ① ② ③ ① Collect content ② Organize content ③ Turn content into writing

① Collect Content The Micro. Writer breaks writing into microtasks Microtasks can be shared with collaborators Microtasks can be done while mobile Collaborative writing typically requires coordination Collaborators can be known or crowd workers People have spare time when mobile Structure turns big tasks into series of small microtasks Microtasks make it easy to get started

① Collect Content

② ① ① Collect content ② Organize content ③ Turn content into writing ③ ① ① ② ③ Chilton, Little, Edge, Weld & Landay. Cascade: Crowdsourcing taxonomy creation. CHI 2013.

② Organize Content

② Organize Content collab microtask mobile tasks Microtasks can be shared with collaborators

② Organize Content collab microtask mobile The Micro. Writer breaks writing into microtasks Microtasks can be shared with collaborators Microtasks can be done while mobile Collaborative writing typically requires coordination Collaborators can be known or crowd workers People have spare time when mobile Structure turns big tasks into series of small microtasks Microtasks make it easy to get started

② ① ① Collect content ② Organize content ③ Turn content into writing ③ ① ② ③

③ Turn Content into Writing ② collab ① microtask ③ mobile The Micro. Writer breaks writing into microtasks Microtasks can be shared with collaborators Microtasks can be done while mobile Collaborative writing typically requires coordination Collaborators can be known or crowd workers People have spare time when mobile Structure turns big tasks into series of small microtasks Microtasks make it easy to get started

③ Turn Content into Writing • microtask Microd • The Micro. Writer breaks writing into microtasks • Structure turns big tasks into series of small microtasks • Microtasks make it easy to get started collab • Collab • Microtasks can be shared with collaborators • Collaborative writing typically requires coordination • Collaborators can be known or crowd workers mobile • Mobile • Microtasks can be done while mobile • People have spare time when mobile

③ Turn Content into Writing

③ Turn Content into Writing collab • Microtasks can be shared with collaborators • Collaborative writing typically requires coordination • Collaborators can be known or crowd workers Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues, or paid crowd workers.

③ Turn Content into Writing • Complete output: Structure makes it possible to turn big tasks into a series of smaller microtasks. For example, the Micro. Writer breaks writing into microtasks. These microtasks make the larger task easier to start. Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues, or paid crowd workers. People have spare time when mobile, and these micromoments are ideal for doing microtasks.

② ① ① Collect content ② Organize content ③ Turn content into writing Micro. Writer ③ ① ② ③ ③

Wear. Write

Opportunity for Microproductivity Gains • Limited productivity gains in information work • US computing capacity increased a hundredfold in the 1970 s and 1980 s • Labor productivity slowed from 3+% in 1960 s to 1% in the 1990 s • Solow computer paradox: You can see the computer age everywhere but in the productivity statistics. • Microproductivity algorithmically changes tasks microtasks • Creates an opportunity to increase productivity via: • Micromoment use • Task arbitrage • Hybrid intelligence

Productivity Gains via Micromoment Use • Idea: Convert non-productive holes into productive time by enabling people to do microtasks in their micromoments • Enabled by: Cloud and mobile technology

Studied Tasks as Macrotask or Microtasks Macrotask S M A R T Supermarket Visit us on the Internet Than you for shopping What is the total cost of all the times on the receipt? Do not type in the dollar sign. Type in the total. Microtasks K BAR CHOCOLATE ORANGE CADBURY SNACK HOBNOB SKITTLES WALLS CORNETTO BURTONS MARYLAND JAFA CAKE FRUIT SALAD FEDDO 0. 22 0. 19 0. 22 0. 24 0. 22 0. 25 0. 19 0. 18 0. 26 0. 23 Add the cost of the next item below to the previous total. (1 of 10). Do not type in the dollar sign (3 (2 Prev Total: + New Item: = Total: 0. 00 0. 41 0. 22 Total?

Macrotask Microtasks 77%

Microtasking Leads to Fewer Errors 0. 35 0. 3 Macro Micro Error rate 0. 25 0. 2 0. 15 0. 1 0. 05 0 Arithmetic Sorting Transcription

Microtasking Takes Longer 200 180 What is… 160 Macro Micro Time (seconds) 140 120 100 69 + 79 o o o 149 168 147 148 138 80 Continue 60 40 20 0 Arithmetic. Average Sorting Transcription

Microtasking Handles Interruptions 200 180 160 Macro Micro Time (seconds) 140 120 100 80 60 40 20 0 Uninterrupted Interrupted

Measure Play Write Mobile Word 3 1. 125 Stress in editing** 3. 31 4. 44 Perceived productivity* 2. 88 2 Task switching support** 2. 75 1. 81 Ease in editing**

Productivity Gains via Micromoment Use • Idea: Convert non-productive holes into productive time by enabling people to do microtasks in their micromoments • Enabled by: Cloud and mobile technology • What we have learned: • Microtasks let people take advantage of wasted time • People prefer working via microtasks to doing large tasks • Microtasks make it easier to multitask and recover from interruptions

Productivity Gains via Task Arbitrage • Idea: Maximize productive time across multiple people by allocating each microtask to the most efficient person • Enabled by: Sharing economy platforms

Guessing from Examples or Rating ?

Asking the Crowd to Guess v. Rate • Guessing Base Guess Rate Salt shakers 1. 64 1. 07 1. 43 Food (Boston) 1. 51 1. 38 1. 19 Food (Seattle) 1. 68 1. 26 • Requires fewer workers • Fun for workers • Hard to capture complex preferences • Rating • Requires many workers to find a good match • Easy for workers • Data reusable

Handwriting Imitation via Rating • Task: Write “Wizard’s Hex”

Handwriting Imitation via Guessing • Task: Write “Wizard’s Hex” by imitating this handwriting

Extraction and Manipulation Threats

Task Extraction • Target task – Text recognition • Attack task 1234 5678 9123 4567 62. 1% • Complete target task • Return answer from target: 32. 8% 1234 5678 9123 4567

Task Manipulation • Target task – Text recognition gun (36%), sun fun(75%) (26%), sun (12%) • Attack task • Enter “sun” as the answer for the attack task sun (28%)

Impact of Payment 80% 70% Response Rate 60% 50% 40% Target$0. 05 Target $0. 50 Target$0. 25 30% 20% 10% 0% $0. 05 $0. 10 $0. 25 Attack Task Payment Amount $0. 50

Productivity Gains via Task Arbitrage • Idea: Maximize productive time across multiple people by allocating each microtask to the most efficient person • Enabled by: Sharing economy platforms • What we have learned: • Microtasks reduce the overhead of collaboration • Intelligent task allocation can leverage expertise, availability of others • Care should be taken when sharing tasks with unknown collaborators

Productivity Gains via Hybrid Intelligence • Idea: Augment human intelligence by automating microtasks as usage data is collected • Enabled by: Advances in artificial intelligence • What we have learned: • Breaking a task down makes it possible to automate aspects of the task • When people perform microtasks they provide training data • Humans can provide magical solutions en route to automation

Summary of Microproductivity Gains • Micromoment Use • Microtasks let people take advantage of wasted time • People prefer working via microtasks to doing large tasks • Microtasks make it easier to multitask and recover from interruptions • Task Arbitrage • Microtasks reduce the overhead of collaboration • Intelligent task allocation can leverage expertise, availability of others • Care should be taken when sharing tasks with unknown collaborators • Hybrid Intelligence • Breaking a task down makes it possible to automate aspects of the task • When people perform microtasks they provide training data • Humans can provide magical solutions en route to automation

Learn More about Microproductivity Overview of Microproductivity • • • Teevan, Libeling, Lasecki. Selfsourcing personal tasks. CHI 2014. Teevan. The future of microwork. XRDS, 2016. Teevan, Iqbal, Cai, Bigham, Bernstein, Gerber. Productivity decomposed: Getting big things done with little microtasks. CHI 2016. Micro. Writing Systems • • Agapie, Teevan, Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015. Greer, Teevan, Iqbal. An introduction to technological support for writing. MSR-TR-2016 -01, 2016. Teevan, Iqbal, von Veh. Supporting collaborative writing with microtasks. CHI 2016. Nebeling, To, Guo, de Freitas, Teevan, Dow, Bigham. Wear. Write: Crowd-assisted writing from smartwatches. CHI 2016. Micromoment Use • • • Cheng, Teevan, Iqbal, Bernstein. Break it down: A comparison of macro- and microtasks. CHI 2015. Cai, Iqbal, Teevan. Chain reactions: The impact of order on microtask chains. CHI 2016. Teevan, Yu. Bringing the wisdom of the crowd to an individual by having them assume different roles. C&C 2017. Task Arbitrage • • • Salehi, Teevan, Iqbal, Kamar. Communicating context to the crowd for complex writing tasks. CSCW 2017. Lasecki, Teevan, Kamar. Information extraction and manipulation threats in crowd-powered systems. CSCW 2014. Organisciak, Teevan, Dumais, Miller, Kalai. A crowd of your own: Crowdsourcing for on-demand personalization. HCOMP 2014. Hybrid Intelligence • • • Toutanova, Brockett, Tran, Amershi. A dataset and evaluation metrics for abstractive sentence and paragraph compression. EMNLP 2016. Kamar, Hacker, Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. AAMAS 2012. Cranshaw, Elwany, et al. Calendar. help: Designing a workflow-based scheduling agent with humans in the loop. CHI 2017.

mi·cro·pro·duc·tiv·i·ty /ˈmīkrō prōˌdəkˈtivətē/ noun Jaime Teevan, Microsoft Research, @jteevan