interacting with large screens patrick baudisch microsoft research

interacting with large screens patrick baudisch microsoft research visualization and interaction research

large screens building a large display focus plus context screens interacting using the mouse high density cursor mouse ether interacting using pen & touch drag and pop tablecloth (halo) return

large screens and multimon

. . . are coming • information mural [Guimbretière, Winograd] • on large screens optical flow helps navigation [Tan 2001] • large screens help productivity tasks [Czerwinski 2003] • multi monitor setups: access palette windows in Photoshop, CAD… [Grudin 2001] Use Multimon 32% No Multimon 30% Plan to Use Multimon 38% [Jon Peddie Research Dec, 2002 N=6652]

focus plus context screens mouse across bezels reach icons mouse across distances reach anything building one conclusions

• Hardware – At least one hi res display – At least one larger low res display • Software – scaling of the display content is preserved – resolution varies

skip to dnp

setup

application scenarios video

field study: users & tasks Static documents Dynamic Subject’s task Document/view Smallest detail Ratio Web designer Page: 800 pixel Table detail: 1 pixel 800 Mechanical engineer Polybot segment: 5 cm Clearance: 0. 03 mm 2, 000 Graphic designer Poster: 1 m Align: 0. 5 mm 2, 000 Architect in remodeling Building: 50 m Accuracy: 1 cm 5, 000 Photogrammetry (2) Highway 2 miles Accuracy: 1 inch 100, 000 Geographic info. system County: 80 km Land boundaries: 0. 5 m 160, 000 Chip designers (2) Wafer: 12 cm Grid: 0. 5 m 240, 000 Air traffic ctrl. tool builder Zone: 50 km Plane distance in 25 m steps 2, 000 Ego shooter gamer Surrounding: 360º Aiming: 0. 1º 3, 000 Submarine ROV op. Surrounding: 360º Use arms: 1 mm/0. 05º 8, 000 Strategy gamers (2) Map: 30 k pixel Aiming: 1 pixel 30, 000

… and current solutions Display technology What participants used homogeneous resolution 4 Visualization Same # of pixels focus plus context screen fisheye 4 5 overview plus detail 4 wall size, hi res display Available to ½ of participants

experiment 1: • 3 interfaces: • focus plus context screen • overview + detail • homogeneous • 2 tasks • 12 subjects from Xerox PARC • Within subjects, counter balanced • Same number of pixels

task 1: closest hotel 8 maps per interface F+C screen and O+D use same magnification factor

task 2: verify connections Verify a different set of 24 connections on the board

results manually zooming takes time Average task completion times in seconds 21% faster 36% faster 700 600 500 zooming + panning 400 overview+detail 300 focus+context screen 200 100 visually switching reorientation 0 Map task Board task visually more ambiguous

experiment 2: driving simulation 120 sec sequence 100 fields of nails; 30 rocks; tradeoff

Mean number of collisions subjects caused results 25 20 Error rate only 1/3 of two monitor setup overview+detail focus+context screen 15 Subjects preferred the f+c interface 10 5 0 run over nails rocks hit • Sweet spot: flight simulation, unmanned vehicles…

observations • low res periphery is often ok • on a wall-size display: If periphery was hi res I still could not read it due to distance • focus plus context interaction toc

we observed: lots of ktop? input problems • mouse • pointer moves too fast, users lose track of it • pointer behaves in unpredicted ways when crossing screen boundaries • touch/pen input • users cannot reach their content anymore conclude

high density cursor mouse across bezels reach icons mouse across distances reach anything building one conclusions

problem: mouse across long distances • large screen • longer distances higher mouse acceleration • temporal aliasing: 500 pixels jumps • lack of visual continuity & weak stimulus users lose track of the pointer

the problem will get worse • cursor update is limited by screen refresh rate • screen refresh rate has actually decreased (LCDs) • larger screens + lower refresh rate status quo • future: even larger screens problem will get worse

how it works previous cursor position current cursor position use o m tion mo rsors u c n i fill ame r f t n curre fill pre in cu vio r us sors fra me inserts cursor image between actual cursor positions the mouse cursor appear more continuous

this is not the mouse trail video the windows mouse trail… • makes mouse trail last longer • drawback: cursor images lag behind . . . is not highdensity cursor • hd cursor makes mouse trail denser • lag-free: mouse stops => cursor stops

benefits previous cursor position current cursor position use o m tion mo ors s r u c fill in frame nt curre 1. mouse cursor appear more continuous easier to track the cursor 2. higher “visual weight” easier to re-acquire the cursor fill i pre n curs viou o s fra rs me

designs alternatives a b c d e f g h frame • reference: exponential acceleration

designs alternatives a b c d e f g h frame • motion blur with higher weight acceleration

designs alternatives a b c d e f discreet version chose 1. latest cursor position is always shown blur-free and in full opacity g 2. appearance that users are familiar with today h 3. computationally less expensive frame • temporal super-sampling vs. motion blur acceleration

designs alternatives a b c d e f g h frame • density = detectability vs. intrusiveness acceleration

transfer function (configurable) hd cu rs or ha s no ef fe ct distance between cursor images cursor trail provides no speed cues onset threshold (configurable) mouse speed

designs alternatives a b c d e f g h frame • optional cursor growth acceleration

user study • conducted pre study to define interface candidates • interfaces: control vs. high density cursor (conservative, triple. Density, plus. Scaling) • fitts’ law task • triple mon: button located at 5” to 40” distance • participants: 7 external participants, 5 coworkers • hypotheses • high density cursor faster • the greater the distance the greater the effect • triple. Density and plus. Scaling faster than conservative

time % relative to regular cursor results regular mouse cursor 102 100 98 96 94 92 90 short distance conservative + scale speedup up to 7% +3 -dense high-density cursors 125 250 500 target distance (mm) 750 1000

subjective satisfaction Liked Most Liked Least control 0 7 HD_conservative 2 3 HD_triple. Density 2 0 HD_plus. Scaling 6 0 Condition • most participants did notice that cursor was different! “did that condition use a different mouse acceleration? ”…

• lesson we learned: display frame rate is not a hard limit toc

mouse ether mouse across bezels reach icons mouse across distances reach anything building one conclusions

mouse ether start target

tasks with incorrect visual feedback

mouse across multimon problem 1: different dpi start high dpi target low dpi

mouse across multimon problem 2: the gap start target

contents • problem: why the cursor gets warped • eliminating warping with mouse ether • calibration • user study: up to 28% faster in acquisition task • conclusions: need to test fitts’ law

why the cursor gets warped system oblivious of • dpi differences start target low dpi high dpi system oblivious of • gaps start target

mouse ether now users can acquire targets on direct path with mouse ether start target start low dpi high dpi start target high dpi low dpi start target

calibration step 3 1 2

the math (ex, ey, ) on i nit +(dx, dy) every f rame (cx, cy) screen i (ex’ , ey, ’ ) (cx’ , cy’ ) screen j

off-screen travel with mouse ether start target

more off-screen travel user’s perspective system’s perspective with mouse ether start blocked target same dpi start target blocked low dpi high dpi target start name ether

taming off-screen travel • iteration 1: restrict to convex hull target • iteration 2: allow entering ether only for the purpose of transit • towards another screen • at sufficient speed • otherwise screen boundaries “hard” start

user study • 2 x 18” screens • 1280 x 1024 and 800 x 600 resolution • eight participants • Win. Fitts software • 72 movements (=4 trials x 9 paths x 2 directions)

results Movement Time (ms ±SEM) with mouse ether up to 28% faster Control 1800 1600 1400 1200 1000 800 600 400 200 0 Mouse. Ether High Low High Direction of Movement

discussion start target • multimon offers incorrect visual cues • this hinders successful target acquisition • mouse ether improves acquisition speed by matching pointer behavior to visual cues • off screen travel allows for direct target acquisition • next steps: investigate target acquisition under contradictory visual feedback ( Fitts’ law? ) toc

pen/touch input on walls-size displays • mouse input can be mapped to any screen geometry • not so for “direct” input technique, such as touch and pen input

drag-and-pop mouse across bezels reach icons mouse across distances reach anything building one conclusions

a technique for people with limited reach… • user may not be able to reach content • user may not be able to read content

drag-and-pop… • let’s focus on a specific case for a moment: extend basic actions drag and drop and picking

scenario 1: long distances dragging is designed for small screens… … but becomes time consuming on large screens

scenario 2: dragging + bezels dragging across bezels in display wall is no problem for the mouse… …but a big problem when using pen/touch input

drag-and-pop: demo • users starts dragging icon towards a distant folder or application • icons of compatible type come towards mouse cursor • user drops icon with minimal motion drag and pop works across bezels • targets retract

• drag and pop generalizes direct manipulation • bring content to the user • let the user interact with it • send content back

scenario 1: long distances dragging on large screens

scenario 2: dragging + bezels dragging across bezels in display wall

the displays we used…

design

selecting candidates • initialize • all icons are candidates • filter • • eliminate icons with non matching file types eliminate icons that are too close eliminate icons outside target angle if necessary, restrict to some hard limit

preserving layout • snap to grid • eliminate empty rows and columns • translate back • place center of bounding box in front of user • closer for experts

the rubber band • animation did not work • “frozen” motion blur • narrow midriff • suggests elasticity • clue for distance • simplified version

getting it out of the way • to rearrange icons on the desktop (overloaded): • any mouse motion moving away from the “popped up” icons de activates drag and pop • introduce flick gesture into mouse motion

pre-study • 15 single, 6 dual, and 4 triple monitor users • overall resolutions 800, 000 pixels to 3, 900, 000 pixels • (= 66% more than the display wall used in the experiment). • 3 layouts for study: sparse (11), frame (28), cluttered (35)

user study • participants: 2 female, 5 male • dyna. Wall • 3 Smartboard • 15’ long (4. 5 m) • 3 x 1024 x 768 pixels • native code not stable enough Macromedia Flash version • task: drag icons into matching folder • highlighting disappeared when started • each desktop: 11 35 icons + 10 icons to be filed

results Control Drag and pop Control Drag-and-pop 3. 7 times speedup • faster with drag and pop • error rate higher with drag and pop • most of the effect caused by the bezels

subjective satisfaction • > 6 (out of 7) • “I liked using drag and pop” • “I always understood what was happening when drag and pop was on”, • “I would use drag and pop for large displays. ” • < 3 for • “It took a long time to get used to drag and pop” • “It was hard to control what the targets did when drag and pop was on. ” • drag and pop interface causes less manual stress and fatigue than the control interface • lesson learned: the shortest connection between two points on a display wall is not a straight line (fixed)

drag-and-pick problem • launch app or open file drag-and-pick • user drags “background” • all icons in that direction move to the cursor • user drags % releases mouse over it • target is activated

current work: push-and-pop

outperforms pantographbased techniques

outperforms pantographbased techniques insert halo

tablecloth mouse across bezels reach icons mouse across distances reach anything building one conclusions

tablecloth generalizing drag-and-pop to arbitrary interactions

for users without table manners

object I would like to interact with window panning ends when window edge reaches edge of screen tablecloth panning allows panning any screen object to any screen location

design challenge • how to invoke tablecloth? • while dragging an object

1 st design: crossing menu

2 nd design: dual touch panning pan jump dragging jump back dragging video

focus-plus-context interaction • the distinction of screen space into focus and context regions is always there (focus plus context screens only emphasize it) • focus plus context interaction • bring contents to user • interact • send contents back halo toc

discussion • distance to peripheral space and human peripheral vision make peripheral content hard to read • making periphery all hi res does not solve this problem • we need to support users using appropriate interaction techniques

read more mouse across bezels CHI 2004 mouse across distances reach icons INTERACT 2003 GI 2005 INTERACT 2003 reach anything building one UIST 2001 CHI 2002 CG&A 2005 patrickbaudisch. com/projects

thanks! • try it out www. patrickbaudisch. com • thanks to: • focus plus context screens: victoria bellotti, nathan good, paul stewart, pamela schraedley • high-density cursor: ed cutrell, george robertson mouse ether: ed cutrell, ken hinckley, rob gruen • • drag-and-pop: & ed cutrell, dan robbins, peter tandler, ben bederson… • tablecloth: george robertson

can we push this further? • (unless the app has fixed focus of attention) bring content to the user on demand • that’s great—so maybe the periphery does not need to be full resolution after all • can we push this even further? can we create a visual periphery without any peripheral screen space?

halo store content in periphery

access off-screen content • make halo arcs (or city lights) clickable to pan display to that location vs. • preserve users’ spatial memory by using drag and pop instead: bring peripheral content to PDA screen

Seamless integration of displays a b

No reflections on focus screen Focus Context

If I had to commercialize today… • Build all analog immersive video link • • Immersive telepresence, Remote operated vehicles, drones Immersive VR Remote medicine

Context. Wall

Extra

(a) locate (c) traverse (b) closest (d) avoid

How does it work?

The scaling software • Display image on two display units of different resolution • Similar to two headed display • but display units are overlapping • and one of them has to be scaled down • (Related work “Flux capacitor” [Dr. Emmett Brown, 1985])

cl ip Linux/VNC app focus server e al sc input viewer context

Image viewer mouse fork ACDsee focus . gif ACDsee context e al sc input Photo shop . gif

related work • acquiring distant targets • • enhance detectability of the mouse cursor • • • move cursor with eye gaze (Sibert and Jacob, 2000), Magic pointing (Zhai et al. , 1999) flick snaps cursor to target (Dulberg et al. (1999) sticky icons capture cursor (Swaminathan and Sato, 1997) throwing gets across long distances (Geißler, 1998) expanding targets save space on screen (Mc. Guffin and Balakrishnan, 2002) drag and pop (baudisch et al 2003) <ctrl> for radar animation (Microsoft, Steve Bathiche) cursor growth (Kensington Mouseworks 2001) mouse trail for slow response LCDs (e. g. MS Windows) live. Cursor points in the direction of its motion (Ben Bederson) motion blur and temporal supersampling • • reduce temporal aliasing, such as stroboscope, e. g. wheel spokes rendering a scene multiple times (Dachille and Kaufman, 2000) improve the perceived responsiveness of graphics apps (Conner and Holden, 1997) help users anticipate motion (Chang, 1993; Thomas & Calder, 2001)

design goals • for users who track the cursor enhance the predictability of the cursor path • enhanced trail density/continuous blur • smooth interpolation of the cursor path • preservation of trail density as a cue for cursor speed. • for users who reacquire the cursor increase the detectability of the cursor (visual weight) • enhanced trail density • enhanced cursor opacity • and cursor scaling. • preserve responsiveness

designs alternatives a b c d e f g h frame acceleration • distance between cursor images as cue for mouse speed

designs alternatives a b c d e f g h frame • smooth interpolation acceleration

bezier interpolation cursor position 1. linear interpolation 2. attraction point 3. interpolate

pre-study • goal: define interfaces for user study • participants: 14 coworkers • informal procedure • try out high density cursor • try out different settings (density, onset…) • choose “favorite” setting • resulting interface parameters • 12 17 pixels/frame vs. 35 pixels/frame • distance = sqrt(n) • cursor growth on or off

touch/pen-input + multimon touch and pen input renaissance • PDAs • Tablets • Liveboards / Smartboards multi display systems • Dyna. Wall, i. Room Smartboard wall • connect tablet to external screen • … touch/pen input breaks

scenario 1: tables + screen tablet users scribble with pen… but filing icons into folder on external monitor requires mouse

scenario 1: tablet + screen filing icons into folder on external monitor

related work • techniques for transferring information • drag and drop avoids hidden clipboard (e. g. Xerox Star) • hyper dragging(Rekimoto, 1999) • pick and drop (Rekimoto, 1997)+ take and put (Streitz et al. , 2001) • overcome large distances • • magic pointing (Zhai et al. , 1999) requires an indirect input device gesture input techniques (Rubine, 1991) throwing (Geißler, 1998) for reliable target acquisition? laser pointers to acquire targets on a Smartboard (Myers et al. 2002) • mouse based interaction techniques • lodestones and lay lines (Jul, 2002) • flick (Dulberg et al. , 1999) • sticky icons (Swaminathan and Sato 1997)