Designing gaze interaction systems John Paulin Hansen The

  • Slides: 40
Download presentation
Designing gaze interaction systems John Paulin Hansen The IT University of Copenhagen Denmark

Designing gaze interaction systems John Paulin Hansen The IT University of Copenhagen Denmark

Short Message System (SMS) • Mobile • Maximum of 160 characters • Very popular

Short Message System (SMS) • Mobile • Maximum of 160 characters • Very popular among the 15 -24 age group • Asynchronous • Discreet • Language culture with slang and abbreviations

% SMS among mobile phone users Source: Smoreda, Z. , Thomas, F. (2000): ”Use

% SMS among mobile phone users Source: Smoreda, Z. , Thomas, F. (2000): ”Use of SMS in Europe”

The problem with SMS • Low efficiency (WPM = 7. 93) • Difficult to

The problem with SMS • Low efficiency (WPM = 7. 93) • Difficult to learn • Few and small buttons • Mode-dependent • No mental models to build on

Solutions to mobile texting ? • • Virtual QWERTY Speech recognition Hand printing (Graffiti)

Solutions to mobile texting ? • • Virtual QWERTY Speech recognition Hand printing (Graffiti) OPTI, FOCL, ABCtapping etc. • BUT: – still not truly mobile and discreet – may be difficult to learn – not always 100 % reliable – Far from QWERTYefficiency

Gazetalk • Digital cameras (web and Firewire) • Type-to-talk • Browser • E-mail •

Gazetalk • Digital cameras (web and Firewire) • Type-to-talk • Browser • E-mail • Adaptive word prediction • Freeware and Open Source • Danish, English and Japanese versions by 2003

Gaze based interaction • First system for people with disabilities in 1990 (Frey et.

Gaze based interaction • First system for people with disabilities in 1990 (Frey et. al. ) • Several efficient systems available today - used by e. g. people with ALS • Why then yet another system?

Problems with existing systems – Quite a large portion of users is not able

Problems with existing systems – Quite a large portion of users is not able to get a sufficiently good calibration – Most systems are only for stationary, indoor use – Requires a rather complicated installation by computer experts – The price is prohibitive (>50. 000 d. kr. + PC) which prevents potential users from testing whether this interaction mode would work for them

Question • Date: Tue, 06 Feb 2001 14: 41: 07 +0200 • Hi there

Question • Date: Tue, 06 Feb 2001 14: 41: 07 +0200 • Hi there • • I stay in South Africa and my father has had a brain stem stroke. He cannot move or speak. He can communicate with his eyes. We are currently using an alphabet chart, but it takes too long and he gets tired quickly. • • I'm looking for some type of Eye. Mouse. Something we can plug into a PC, Design like a virtual keyboard. • • Can you please advise me. I will appreciate any help. • • Thanks Nuno Lourenco

Answer • Dear Nuno, • • • I'm sorry to tell you but eye

Answer • Dear Nuno, • • • I'm sorry to tell you but eye trackers that are accurate enough for your father to indicate at which letter your father is looking at are much too expensive and are not available for private use. If your father still can hear, you can ask him to look left or right or up or down under certain conditions. It may work as a yes or no. • • Department of Neurology/Clinical Neurophysiology, H 2 -222 Academic Medical Centre, University of Amsterdam -Lo Bour, Ph. D

Video Demo

Video Demo

 • UI test with eye tracking system

• UI test with eye tracking system

Freeware versions 2003 Danish Japanese English

Freeware versions 2003 Danish Japanese English

Design Process • 2000: Design students made prototypes • 2001: Usability students tested first

Design Process • 2000: Design students made prototypes • 2001: Usability students tested first version with limited functionality • 2002: ALS patients to test version 1. 0

Big button browser

Big button browser

Big button browser

Big button browser

Low res head- & eye tracking Color: Mean-shift Shape: Active appearence models

Low res head- & eye tracking Color: Mean-shift Shape: Active appearence models

Gaze determination Web cam Video cam

Gaze determination Web cam Video cam

Ongoing Experiments • 12 Japanese students • 4 hours of training • Typed 20

Ongoing Experiments • 12 Japanese students • 4 hours of training • Typed 20 JP characters per minute ~10 WPM with gaze (tested with ”Quick glance”) • 12 % errors - 3 % with mouse

Comments from novice users of gaze dwell time typing: • ”A bit difficult to

Comments from novice users of gaze dwell time typing: • ”A bit difficult to get used to not dwelling at un-intended buttons. . reacts in panic” • ”Difficult to orient yourself without activating something” • ”It can be hard to keep starring at the buttons you wanth”

Loose coupling of gaze and pointer • 35 mouse activations analysed: – 40% gaze

Loose coupling of gaze and pointer • 35 mouse activations analysed: – 40% gaze remaind at dwelling key – 40% went to other key durring dwell periode – 12% went to text field durring dwell periode – 8% the eye was at another key durring all the dwell periode

Theoretical upper limit • ”Perfect” Eye tracker and trained user, utilizing all the predictions

Theoretical upper limit • ”Perfect” Eye tracker and trained user, utilizing all the predictions : – 2 characters per activation – 1, 2 seconds per activation (500 ms dwells. cf. Japanese results) = 100 characters per minute = 20 wpm, = 24 wpm, 300 ms dwells

Reaching for the moon? • • • People talk at 160 to 200 WPM

Reaching for the moon? • • • People talk at 160 to 200 WPM !!!! Non-typist write at 20 - 35 WPM AAC typing systems 2 - 26 WPM (1977 -1985) SMS (Multitap) at 8 WPM Eye typing systems from 1 to 7 WPM Birger Bergmann Jeppesen peaked at 12 WPM (Wiwik + Quick. Glance)

How fast is fast enough? • ”It should be clear that speed, in itself,

How fast is fast enough? • ”It should be clear that speed, in itself, should not be an object, but rather proficiency and ease of operation. On the other hand, when there is a lot to say, or when there is a need for extensive personal interchange, a minimum speed of 25 - 30 wpm is really needed to keep the thought moving” William G. Pierpont(2001): ”The art & skill of Radio-Telegraphy”

Other approaches • Dario D. Salvucci (1999): 822 ms pr. character, 10 seconds to

Other approaches • Dario D. Salvucci (1999): 822 ms pr. character, 10 seconds to find the right word in a data base with 1000 words. User performance 9 - 28 wpm.

DASHER • Developed by David Mackay, Cambridge University, with Stephen Hawking ”in mind” •

DASHER • Developed by David Mackay, Cambridge University, with Stephen Hawking ”in mind” • 25 WPM after one hour • 34 WPM when expert • Requires a ”Drivers license” • ”Fast hands-free writing by gaze direction” Nature 418: 838 (August 2002)

User needs Video cam Web cam • • • Fast Robust Daily expressions Mobile

User needs Video cam Web cam • • • Fast Robust Daily expressions Mobile Changes with progress of diseases • Affordable

Progress of ALS disease • • • Loss of voice Reduced control of arms

Progress of ALS disease • • • Loss of voice Reduced control of arms and legs Reduced control of fingers Reduced head movements Just eye movements

ALS Scenarios Keyboard Mouse Head Gaze

ALS Scenarios Keyboard Mouse Head Gaze

ALS Scenarios Keyboard and mouse Mouse or joystick Head Gaze

ALS Scenarios Keyboard and mouse Mouse or joystick Head Gaze

Mobile Scenarios Plane Coach Metro Strolling

Mobile Scenarios Plane Coach Metro Strolling

Classes of Mobile Devices

Classes of Mobile Devices

Gaze and voice interaction with mobile phones A. Jameson, Germany research center for artificial

Gaze and voice interaction with mobile phones A. Jameson, Germany research center for artificial intelligence.

Helmet mounted displays

Helmet mounted displays

Gaze interaction for special tasks • Industrial inspection • Fieldcommunication • Security • Medical

Gaze interaction for special tasks • Industrial inspection • Fieldcommunication • Security • Medical information systems

”Hands-free”-gaze interaction for all • Private mobilecommunication: – Faster than SMS – Note taking

”Hands-free”-gaze interaction for all • Private mobilecommunication: – Faster than SMS – Note taking during video conferences and – Dynamic indication of attention from several participants

Transfer of innovations ALS- Patient: Mobile user – No voice – Noise or need

Transfer of innovations ALS- Patient: Mobile user – No voice – Noise or need for private communication – Just head- or gaze control – Hands-free interaction – Imprecise pointing – Acceleration of textinput – Shaky environments or small displays – Word prediction on mobile units with limited number of keys

Future applications

Future applications

www. gazetalk. org: The IT-University of Copenhagen Tokyo Institute of Technology The Danish Muscular

www. gazetalk. org: The IT-University of Copenhagen Tokyo Institute of Technology The Danish Muscular Dystrophy Association …and hopefully more to come Exhibition at the IST Conference, Copenhagen, November 4 -6 , 2002.