TextFree User Interfaces Kentaro Toyama Assistant Managing Director

Text-Free User Interfaces Kentaro Toyama Assistant Managing Director Microsoft Research India Based on work with Indrani Medhi TCS Excellence in Computer Science January 10, 2008 – Pune, India

People Lead Researcher – Indrani Medhi Collaborators – – – Kentaro Toyama Archana Prasad Vibhore Goyal Shibili G Nimmi Rangaswamy Sean Blagsvedt Interns – Aman Sagar (Adobe) – Bharathi Pitti (CMU) – Chandima Patabandhige (Univ. of Moratuwa, Sri Lanka) – Renee Kuriyan (UC Berkeley) Photo: Indrani Medhi Indrani (holding baby) and some interns visiting a family in Bangalore

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

Illiteracy 1 -2 billion illiterate population in the world. 98% live in developing countries. India’s rate of literacy estimated at ~60%. Photo: Indrani Medhi Two residents of a slum in Jayanagar, Bangalore Many labeled “literate” are semi-literate with low functional literacy.

Target Users Women from Bangalore slums Informal sector domestic workers Income range: INR 800 -2500 (US$20 -60) per month Illiterate or semi-literate Most have never seen a PC (those who have seen have never touched) Photo: Indrani Medhi initial explorations before text-free UI work

NGO Partner Stree Jagruti Samiti – Head: Geeta Menon – Small NGO – Works with several Bangalore slums – Focus on children’s rights, women’s empowerment, fair wages for domestic laborers Photo: Indrani Medhi Geeta Menon leading a Saturday Stree Jagruti Samiti meeting

The Problem How can an application UI be converted into one that is usable by non-literate users?

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

Initial Ethnography TVs, radios, CD players common; some mobile phones; no PCs Established slums – some with 30 -year history in city; concrete housing Subjects most concerned about… Photo: Indrani Medhi Nimmi interviewing a subject – Jobs – Healthcare – Education

Text-Free UI? Design principles: – Pen or touch interface – Liberal use of imagery • No text • Semi-abstracted cartoons – Voice annotation – Aggressive use of mouseover functionality Naukri. com for domestic labourers? – Consistent help icon

Text-Free UI Details Nouns vs. Verbs Goal is to achieve exact association between image and word. Without action cues drawings tend to be interpreted as nouns. “Kitchen sink” or “washing dishes”? With action cues, drawings tend to be interpreted as verbs. Phenomenon well-understood by cartoonists and diagram designers. “Pot” or “cooking”?

Text-Free UI Details Cultural Differences Goal is to achieve as global a representation as possible. Abstracted representations are very dependent on culture. Minor misrepresentations can result in major confusion. Testing, testing! An urban family of three?

Text-Free UI Details Religious Differences Goal is still to achieve as global a representation as possible. From when to when? Religious differences can be the strongest of cultural differences. There is no “neutral” culture or religion. Testing, testing! Increasingly general representations for time indication

Text-Free UI Details Quirks of Non-Literacy Various degrees of literacy – – – Total illiteracy Numbers, but not alphabets Alphabets, but not words Words, but not sentences Semi-literate, but not fluent Etc. Many who are non-literate can read numbers – Accustomed to currency – Able to do basic arithmetic – Not necessarily 100% accurate Testing, testing! Illiteracy doesn’t necessarily mean inability to read numbers.

Text-Free UI? Original design Revised design

Text-Free UI Design principles: – Pen or touch interface – Liberal use of imagery • No text (but numbers OK) • Semi-abstracted cartoons – Aggressive use of mouseover functionality – Voice feedback – Consistent help icon Screenshot of text-free job search

Other Domains Principles transfer to other applications: Text-free maps Text-free healthcare information

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

Initial Evaluation Questions 1. Can non-literate users use traditional textbased user interfaces at all? 2. Do the proposed design principles for textfree user interfaces allow non-literate users to use computers, and to what extent? 3. Which of the design principles make the most difference for a text-free UI?

Initial Evaluation Experimental Set-Up Three versions: – Text-based – Text-free without help icon – Text-free with help icon Subjects: – Illiterate or semi-literate • (no literate subjects) – No PC experience Subject grouping: – Five-women group x 2 – Individual x 4 Total time up to one hour per session Task: For a friend who is unemployed, find the bestpaying job in her neighborhood. Measured: – Task completion (yes/no) – Time required (second) – Number of prompts required (n) All comments recorded; some trials video-recorded.

Initial Evaluation Quantitative Results Text-based UIs were completely unusable. – Obvious, but first formal such test in literature With text-free UI, 30% were able to complete task. (However, not enough participants for statistical significance. ) Tabulation of initial results

Initial Evaluation Qualitative Results Strong preference for text-free principles, in particular… – Voice annotation – Help icon Collaborative use more successful Subjects eager to engage, once comfortable with set up Various barriers to technology: fear, lack of awareness, lack of cognitive model Indrani and a subject during initial evaluation

Initial Evaluation Questions 1. Can non-literate users use traditional textbased user interfaces at all? 2. Do the proposed design principles for textfree user interfaces allow non-literate users to use computers, and to what extent? 3. Which of the design principles make the most difference for a text-free UI? NO Yes, but only partially; more work required Hypothesis: - Voice feedback - Imagery - Help

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

From initial round of research… Text-Free UI Design principles: – Pen or touch interface – Liberal use of imagery • No text (but numbers OK) • Semi-abstracted cartoons – Voice feedback – Aggressive use of mouseover functionality – Consistent help icon Screenshot of text-free job search

Goal of Study Determine what types of icon representations are best for UIs for non-literate users. – Audio: • With voice annotation • Without voice annotation – Visual: • • • Photorealistic video Hand-drawn animation Photograph Hand-drawn image Text

Domain of Testing Desired characteristics of domain: – Admits to visual representations, – Consists of information which subjects are not exposed to on a daily basis, – Has universally recognized meaning, – Offers a range of cognitive complexity, – Offers a range of visual complexity, and – Can be applied to a practical purpose. Healthcare / medical imagery – 13 medical symptoms: multimodal in nature, involving elements of color, temperature, proprioception, and some requiring motion spanning some temporal duration

Creation of Representations Ensure consistency across representations… Video Photo graph Anima tion Drawi ng

Tested Representations Video Photo graph Anima Each tion with and witho ut voice Drawi Text ng

Experimental Set-Up Representations: – One of 10 types – Evenly distributed among subjects – All 13 symptoms per subject Subjects: – Illiterate or semi-literate Threefold task: • • • Speak phrase that first comes to mind when shown a representation. Explain symptom representation is intended to convey Enact or depict the intended symptom (as explained by tester), for analysis • (no literate subjects) – No PC experience Size: – Individual; n = 200 – 20 for each representation Measured: – Accuracy (correct, close, wrong) – Response time (seconds) All comments recorded Total time up to one hour per session

Quantitative Results Voice annotation critical, with dominant effect • 30% quicker response time with audio Among cases with audio, handdrawn representations (both animation and static imagery) most accurately recognized • But, effect not dramatic Among cases without audio, animation most accurately identified • 23% more accurate than static drawing Number of accurate responses (out of 20) for various representations

Qualitative Results Test / text anxiety of subjects – Subjects uncomfortable with test-like environment of studies Short-term category conditioning – Priming for “aches” over other types of symptoms when they appeared in sequence Richer information is not necessarily better understood overall – Photorealism contains clutter; associated with actual instance Context-laying activity provides additional in dynamic imagery Socio-economic conditions correlated with cognition – Those with less formal education had difficulty processing visuals and audio at the same time; confused by multi-modal interface

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

From initial evaluation… Quantitative Results Text-based UIs were completely unusable. – Obvious, but first formal such test only!!! Help icon With text-free UI, 30% were able to complete task. However, not enough participants for statistical significance. Tabulation of initial video results UI instruction Insufficient for end goal!

From initial evaluation… Qualitative Results Strong preference for text-free principles, in particular… – Voice annotation – Help icon Collaborative use more successful Subjects eager to engage, once comfortable with set up Various barriers to technology: fear, lack of awareness, lack of cognitive model From post-experiment interviews: – Afraid to break PC – Intimidated by use of complex technology – Confusion as to value of PC for job search – Lack of understanding about how PC contains relevant information – Task better accomplished Indrani a subject during byand asking people initial evaluation – Verbal explanations by tester not convincing

New Problem! Original question: How can an application UI be converted into one that is usable by non-literate users? ILLITERACY FEAR OF TECHNOLOGY LACK OF TRUST IN TECHNOLOGY LACK OF AWARENESS OF WHAT TECHNOLOGY CAN DELIVER New question: Can a UI be developed to allow an illiterate, firsttime PC user to access information he/she needs without any assistance or prompting?

What’s the Underlying Problem? If you were told that, if you put anything in this box and spelled the object’s name in reverse 100 times out loud, it would come to life… “Any sufficiently advanced technology is indistinguishable from magic. ” – Arthur C. Clarke would you bother to do it?

Solution? Observations about TV: – Many households have TVs – Soap operas a common topic of discussion – Even households without TVs are familiar with TV Observations about subject’s cognitive styles: Photo: Indrani Medhi – Subjects engage in long linear narratives – General ideas conveyed through specific instances – Stories included real people, specific objects, or actual instances of events (Based on intuition developed through prolonged engagement with subjects!) Photo: Indrani Medhi Televisions common in households

Full-Context Solution: Video Full-Context Video A full-context video explains the broader context of the application and how it works, in addition to instructional material about how to use the application.

Full-Context Video Experimental Set-Up Two versions: – Original text-free (without fullcontext video) – Text-free with full-context video Subjects: – Illiterate or semi-literate • (no literate subjects) – No PC experience Subject grouping: – Individuals x 35 Total time up to one hour per session Task (the same): For a friend who is unemployed, find the bestpaying job in her neighborhood. Measured: – Task completion (yes/no) – Time required (second) – Number of prompts required (n) All comments recorded; some trials video-recorded.

User Studies Full-context video has clear value: Without video first, only one out of 17 (6%) was able to complete the task at all, taking 11 prompts and 8. 2 minutes With video first, 18 out of 18 completed the task, with an average of 4. 7 prompts and 6. 5 minutes Without Video (A) With Video (B) Total Task Completed (out of 35) 8 35 Prompts reqd for completion 9. 8 5. 2 9. 01 4. 59 Avg completion time (min) Sequence - AB (17 total) 1 17 Prompts reqd for completion 11 5. 9 Avg completion time (min) 8. 2 8. 6 Task Completed - AB Sequence - BA (18 total) Task completed – BA 7 18 Prompts reqd for completion 6 4. 7 10. 8 6. 5 Avg completion time (min)

Qualitative Results Other observations: Round-two subjects were incredulous that round-one subjects didn’t understand the application. Impact of video not permanent for most subjects. Many wanted to see the fullcontext video each time, even after seeing it before. Full-context video appears to increase motivation, as well as performance. Photo: Indrani Medhi Those who saw full-context video were interested in providing feedback on the specifics of the UI.

Text-Free UI Design principles: – Pen or touch interface – Liberal use of imagery • No text (but numbers OK) • Semi-abstracted cartoons – Aggressive use of mouseover functionality – Voice feedback – Consistent help icon – Full-context video playing at start of application

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

Methodology Initial Ethnography Open-ended interviews – Various combinations: focus groups and individuals – Topic: everything and anything – Follow leads around hardships Structured interviews – Focus on themes from openended interviews – Prepare questions – Be willing to deviate Photo: Indrani Medhi An open-ended interview in a subject’s home Goal: Information, intuition, rapport Time and attention critical!

Methodology Iterative Prototyping Build initial prototype, based on intuition from interviews. Iterate prototype and test. – Maintain informality – Work with small groups • 2 -4 people – Prompt whenever necessary – Record every comment • Photo: Indrani Medhi Video-record, if possible (but, high procedural overhead) – Listen to every comment – Modify prototype Testing a prototype Refine until steady state.

Methodology Participative Design Modify standard methods for non-literate partners. Encourage storytelling. Photo: Indrani Medhi Use aids that can be physically manipulated. Eliciting descriptions of medical symptoms with physical design aids Be prepared for long, counseling-like sessions. Subject enacting medical symptoms

Methodology Formal Evaluation Base on techniques from courses, textbooks – Plan according to standard practices Avoid direct counterfactuals – Place task in a story – Use friend or neighbor as protagonist, but not subject – Chavan’s “Bollywood method” Be flexible on… – Location (avoid labs) – Formal instructions – Etc. Video: Archana Prasad Scene from a full-context video: hypothetical user and NGO worker

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

Related Work Interfaces for non-literate users – Huenerfauth [2002] – Chand [2002] – Parikh et al. [2003] Visual representations – Modley et al. [1976] – Horton [1996] – Tversky et al. [2002] Video tutorials – De. Bloois et al. [1984] – Rickman et al. [1986] – Bransford et al. [1992] Much more related work in these categories (see papers) Text-Free UI novelty: – Focus on non-literate users – Rigorous evaluation – Full-context video

Publications Medhi, I. , Pitti B. and Toyama K. Text-Free UI for Employment Search. Asian Applied Computing Conference (AACC 2005). Nepal, December 2005. Medhi, I. , Sagar A. , and Toyama K. Text-Free User Interfaces for Illiterate and Semi-Literate Users. International Conference on Information and Communication Technologies and Development (ICTD 2006). Berkeley, USA, May 2006. Medhi, I. , Prasad, A. and Toyama K. Optimal Audio-Visual Representations for Illiterate Users. International World Wide Web Conference (WWW 2007). Banff, Canada, May 2007 Medhi, I. and Kuriyan R. Text-Free UI: Prospects for Social Inclusion. International Conference on Social Implications of Computers in Developing Countries (IFIP 2007). Brazil, May 2007 Medhi, I. and K. Toyama. CHI Workshop position paper on User-Centered Design and International Development, San Jose, USA, April 2007 Medhi, I. , K. Toyama. Full-Context Videos for First-Time, Illiterate Users. alt. chi forum at ACM CHI '07, San Jose, USA, April 2007 Medhi, I. User-Centered Design for Development. ACM Interactions. COLUMN: Forum: under development Volume 14 , Issue 4 (2007), pp. 12 – 14. Medhi, I. , C. R. Patabandhige, K. Toyama. Cartoon Generation for Text-Free User Interfaces. 1 st International Workshop on Computer Vision Applications for Developing Regions in Conjunction with Eleventh IEEE International Conference on Computer Vision (ICCV 2007 Workshop), Rio de Janeiro, Brazil, October 2007 Medhi, I. , Toyama, K. Full-context video for. International Conference on Information and Communication Technologies and Development (ICTD 2007). Bangalore, India, December 2007.

Text-Free UI History Text-Free healthcare information Practical Systems Online site UI Research Initial principles Optimal AV representation Full-context video Other Research Key: Completed Ongoing Future UI authoring tool (babajob. com) Text-Free UI for mobiles Semiautomated cartooning Psychology of non-literate users

Summary: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Full-context video Methodology Discussion

Photo: Indrani Medhi Thank you! http: //research. microsoft. com/research/tem {indranim, kentoy}@microsoft. com