Tangible Interfaces Readings Hiroshi Ishii Brygg Ullmer Tangible

Tangible Interfaces

Readings • Hiroshi Ishii, Brygg Ullmer, “Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms”, Proceedings of CHI ’ 97, March 1997, pp. 234 -241. http: //doi. acm. org/10. 1145/258549. 258715 Better figures here: http: //www. acm. org/sigchi/chi 97/proceedings/paper/hi. htm • Roy Want, Kenneth Fishkin, Anuj Gujar, Beverly Harrison, “Bridging Physical and Virtual Worlds with Electronic Tags”, Proceedings of CHI ’ 99, May 1999, pp. 370 -377. http: //doi. acm. org/10. 1145/302979. 303111

More Readings • Jun Rekimoto, Brygg Ullmer, Haruo Oba, “Data. Tiles: A Modular for Mixed Physical and Graphical Interactions”, Proceedings of CHI ’ 01, March 2001, pp. 269 -276. http: //doi. acm. org/10. 1145/365024. 365115 • Jun Rekimoto, Eduardo Sciammarella, “Tool. Stone: Effective Use of the Physical Manipulation Vocabularies of Input Devices”, Proceedings of UIST ’ 00, Nov. 2000, pp. 109 -117. http: //doi. acm. org/10. 1145/354401. 354421

Supplemental Readings • John Underkoffler, Hiroshi Ishii, “Urp: A Luminous-Tangible Workbench for Urban Planning and Design”, Proceedings of CHI ’ 99, May 1999, pp. 386 -393. http: //doi. acm. org/10. 1145/302979. 303114 • Leo Tolstoy, War and Peace, 1869, English translation by Rosemary Edmonds, Viking Press, 1982. http: //www. online-literature. com/tolstoy/war_and_peace/

Motivation • Desire to remove divide between electronic and physical worlds – Bring benefits of computation (the virtual world) beyond confines of the screen, to the rest of life • Ubiquitous computing (Weiser) • Augmented reality – Making crossing between “bits and atoms” seamless (or at least a lot easier) • Put information “in our hands” in a rather literal way

Motivation (cont) • “Spatial thinking” – Concrete – Like sketching: fluid, natural, dexterous actions – (Graspable) physical objects provide strong affordances (much closer to real affordances instead of virtual) – Different sets of thinking / reasoning skills • Used to say “visual specification for visual things” – E. g. , UI layout Ø“spatial specification for spatial things”

In “Tangible Bits” paper Ishii Promotes Three Concepts • Interactive surfaces – Transform (all? ) surfaces into an interface • (Really nothing special about “surfaces” except that they are convenient to work with at this stage) • Coupling of bits to atoms – Couple information associated with (relevant to) physical objects to the objects • Provide direct access to info from object • Right information is always “at hand” • Ambient media – Ability to use the periphery of attention and move between the periphery and center of attention

Prior Work Examples (From Ishii’s “Tangible Bits”) • Digital. Desk – Wellner, Euro. Parc – Desk with top-projected video – Camera tracking from above – Digitizing tablet – Microphone • On surface

Digital. Desk (cont) • Interact with real paper – Digitize with camera (eventually) and recognize action gestures – Augmented with video from above – Used microphone to detect “taps” • Difficult to detect contact via video alone

Prior Work Examples (From Ishii’s “Tangible Bits”) • Marble Answering Machine – Durrell Bishop, RCA – Marbles represent calls – Put marble in indentation to play message – Another spot to call back • Live Wire – Natalie Jeremijenko, Xerox PARC – String turned slightly on each Ethernet packet – Making invisible bits visible/tangible

Ishii’s work in “Tangible Bits” paper • meta. Desk – Rear projected video (the bits) – Physical objects tracked on top (the atoms) • Manipulating physical objs affects the bits (not rev)

meta. Desk

Applications of Meta. Desk • Tangible Geospace – Map manipulation via phicons, etc. • Two phicons (representing particular places) allow very easy manipulations of position, orientation, and scale of the map – Solid Plexiglas shape • Active lens gives 3 D views – LCD display on a tracked arm • Passive lens gives augmented or filtered 2 d views – Clear lens – Change the projected display under the lens!

Meta. Desk Issues • Good demo, but not a real application – Solid and compelling apps for tangible interaction still somewhat of an issue, overall – Seems very compelling, but is their really a big benefit • Seems somewhat domain specific (see URP & Toon. Town later) • Do I have to have a new physical object for every command/parameter/whatnot? – Loosing some of the extreme flexibility of the virtual world – Big research issues: How do we find the right mix of physical / virtual to get good properties of both?

Ambient. Room • Video

Ambient. Room • Interesting things here: – Ambient displays • Aesthetics important • A start on user defined mappings – Does this scale? – What about all the “hidden state” (can’t see what you’ve spec’ed)? – Physical containers for information • Issues – Hamster awareness !? ! • Again, missing a compelling application

Trans. Board • Augmented whiteboard – Tracked pens, front projected image – Looking at “surfaces which absorb information from the physical world” • Collaborage we saw earlier (but done later) was similar but probably more compelling

Additional Apps for Meta. Desk • Illuminating Light – Optical bench simulation – “I/O bulb” concept – New color bar tracking (from above) Video…

Additional Apps for Meta. Desk • URP – Urban planning app – Physical objects cast virtual shadows and reflections – Wind flow simulation – Tools for making measurements

Finding a good mix of physical and virtual: Data. Tiles (Rekimoto) • LCD with pen input – Transparent tiles • May have preprinted information • May have groves affording particular interactions – Sit on pedestals • Serve as ID readers for tiles

Data. Tiles • Appearance is combination of display and pre-printed • Grooves afford particular actions (labeled as to purpose)

Interesting tiles • Containers – Copy virtual objects, settings, etc. to a tile – Set aside until later • Parameters – Search times with parameters – Dynamic query on (preprinted) map tile • Grooves for sliders

Interesting tiles (cont) • Actions – Print • Generic print tile • Set next to object, join with gesture, fill in print dialog – Time machine • Show previous states of another tile • “Clock” grooves with appropriate dynamic feedback – Remote interaction tiles • Shared drawing surface with remote participants

Physical / virtual or generic / concrete balance • Note that a range of interaction techniques are possible along scale of very concrete and predetermined to very generic – Also static to dynamic and physical to virtual • Composition mechanism allows fairly predetermined tiles to have much more utility – General lesson: simple actions + composition • Each small and simple – Let combinatorics take over to give wide expressive range, but with (potentially) less overall complexity

Generally useful technology in this domain: Tagging objects

Tagging Objects • Several technologies (see below) for tagging objects to enable sensing of identity – Objects brought into proximity of special reader • Range varies: room to a few cm – Reader determines a unique identifier for object • Typically a unique number • Note that 64 bits is probably enough to uniquely tag every individual object ever manufactured – every CD, ever can of soup, every page of paper – Even 32 bits goes a long way (4 billion unique IDs) – 96 -128 bits is better (can partition ID space conveniently)

Fun with IDs • A number of things you can do with an ID – If you have network connectivity (or can cache) at the reader then any identifiable object can have an arbitrary amount of data attached to it • Look up the ID in a database to retrieve “attached data” • Data can be general info, commands, parameters, anything, … – Attached object can be static or dynamic • Can also do this with a local database if info is local – Object can now be a surrogate for the data • Arbitrary data (incl. arbitrary programs) very powerful

Identified Objects Can Act as • Container – Can (appear to) put in, carry away, and (later/elsewhere) take out (arbitrary) data • User ID – Can serve to represent / identify a person • Context ID – Can serve to identify a place – Can indicate the presence of a piece of equipment • Provide handle for accessing it • E. g. , walk into a room with a printer and get access to its queue • Actions to be performed • Set of attributes / parameters / settings

ID Technologies • Printed Bar Codes – Read by optical scanner • Laser or other • Usually close contact, but can be done at a distance • Typical read error rates: 1 per 10, 000 – Most common: UPC on all retail items • Most common: 10 decimal digits + prefix & check digits – Enough for every kind of soup, but not every individual can

Bar Codes (cont) • Can also store a lot more data – See: PA vehicle registration UPS “maxicode” • Advantages of bar codes – Very cheap (just print them with a printer) – No battery needed for tag – Reliable, readers relatively cheap • Disadvantages – Requires contact (and/or careful alignment) – Only work on relatively flat/smooth surfaces – Ugly

Aside: Can get rid of Ugliness • Xerox Data. Glyph technology designed to look better – Appears as a textured gray region • Relatively high density • Also survives bad faxes and other distortions well

IR Beacons • Encode ID in pulses of (invisible) infrared light – Same technology as TV remotes (cheap & easy) – ID sent periodically (typically every few seconds) • Applications – Fixed beacons to indicate place • Example: HP Cool. Town project – Emits URL of web page giving access to services (e. g. , print queue for printer, or control of projector) in room – Active Badges identify location of a person

Other Interesting ID Mechanisms • Contact tags, magnetic stripe, and smart cards • For limited sets of objects: weight them – 0. 1 gram resolution scales distinguish most objects – Issues with “wear and tear” • May want to augment with vision • Fingerprints – User fingertips as data repositories (!) • Virtually “pick up” arbitrary data, carry, and “drop” it. – Another potential app: 10 actions for 1 button

RFID Tags • Probably most generally useful technology • Reader has a typically large coil – Emits RF at a particular frequency • Tag has a typically small coil – Inductively couples with reader – Draws power from it (no batteries in tag) – Modulates its load on inductive coupling to send information back to the reader

A variety of form factors for tags

Different tag properties • Embedded processors • Persistent memory • Single read vs. multiple – Multiple requires special protocol with reader – Implies a processor • Differing read ranges – Typical 10 cm – With “active” tags (battery) can get longer (10 m)

Advantages (from Want paper) • Robustness – Designed to work in harsh environments • Inside dogs, on pigs ears, nailed into wood, under labels at K-Mart • No maintenance • Post-hoc augmentation • Can typically be hidden (no aesthetic costs) – Small and can be embedded in lots of things • Easily sensed – Doesn’t require direct contact, line of sight, or alignment

Disadvantages (from Want paper) • Need interface for associating meaning with objects – Not particular to RFID tags • Knowing what is tagged – Hidden tags mean no affordance

A good application using object identification • Toon. Town access control interface in the Some. Wire audio space Andrew Singer, Debby Hindus, Lisa Stifelman, and Sean White, “Tangible Progress: Less Is More In Somewire Audio Spaces”, Proceedings of CHI ’ 99, May 1999, pp. 104 -111, 625. http: //doi. acm. org/10. 1145/302979. 303007

Toon. Town • Shared audio system with reciprocity – Needs good access control • “Who can hear me” (awareness and easy control) – Also supported localization of the audio • Can place participants left, right, etc. • Place “cartoon characters” (small dolls) on trays to represent people – RF Tag in the bottom of each doll, readers in tray

Toon. Town • On tray can hear and be heard • Clear visibility, easy to manipulate • Downside: doesn’t scale (but not clear what would)

Toon. Town • Location on tray also controls apparent sound source location and volume • Note spots at right for other UI functions

Toolstones • An input device – Not exclusively tangible (used with GUIs), but a lot to do with grasp and touch – Device designed for use by non-dominant hand • Typically with a pen or other pointer in the dominant hand – Note: In most GUIs if you aren’t typing you might as well sit on your other hand • May be intermittent – Adjust, work with primary hand, adjust, etc.

About bimanual operations (cooperative between hands) • Hands tend to take on different roles – A lot more than a “hand preference” or “stronger hand” – Semantically different actions by dominant and non-dominant hands • Seminal work Yves Guiard, “Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model”, Journal of Motor Behavior, Vol. 19, No. 4, 1987, pp. 486 -517. http: //cogprints. soton. ac. uk/documents/disk 0/00/00/06/25/

Three key observations from Guiard (Assuming right-handed -- tendencies somewhat less clear left-handers) • Left hand sets context for right – Left provides overall positioning and orientation for detail work of right – E. g. , left holds fabric, right stitches where fabric held

Three key observations from Guiard (cont) • Tendency for larger scale and lower frequency movements by the left hand – Larger, courser, slower motions by left – Smaller, more precise, faster motions by right • Motion by left hand tend to precede those of the right – Position objects before acting on them – Pick up the cloth before sewing it

Working with these tendencies • Taking proper advantage of different roles (rather than just subdividing the work) can provide better/smoother performance • Tool. Stone device designed for nondominant hand – Will see for tasks like selecting color from palette instead of tasks like painting

Tool. Stones • Rectangular device – Each of 6 faces – Distinguishable by touch alone (via size and/or marker) • Works with a digitizing tablet – Uses components from particular type of tablet – Provides constrained six degree of freedom (6 DOF) sensing • X, Y, Z position • 3 degrees of orientation (e. g. , tilt, yaw roll) • constrained because stone must stay in contact with the surface – typically one face must be on surface, but version supporting tilting on edges also possible

Sensing • Quantize possible positions and orientations out of full 6 DOF space • 8 Directions • On 6 different faces • Also track x, y position on tablet

Sensing • Implemented using guts from 3 tablet pens

Sensing • One “pen” in contact (close proximity) with surface for any orientation • Pens are passive – No battery, just resonant coil sensed from tablet • Each pen allows sensing of – Contact (very close proximity, not “tip switch”) – Angles with respect to tablet – “Pressure” and ID of pen • • Actually location of an iron slug within the coil Each unit initially tuned to slightly different frequency Moving iron slug changes within a range Tablet can sense at least 3 different pens at simultaneously

Alternate sensing design • Two pens in contact for any face • Can sense tilt along edge

Interaction techniques • Typically each face has a different task – Important that they be distinguishable by feel – But labeling also important / useful • Palettes / menus – Orientation based menus (8 items max) • Fitts’ law says…? – Larger use motion to pick like a mouse • E. g. color palette • Potential issue: What is it? … …l …

Interaction techniques • Typically each face has a different task – Important that they be distinguishable by feel – But labeling also important / useful • Palettes / menus – Orientation based menus (8 items max) • Fitts’ law says…? – Larger use motion to pick like a mouse • E. g. color palette • Potential issue: don’t have a button – Best for modal interactions such at tool palette or as toolglass style interaction (will see this in a later lecture) – Basically piggyback off “primary” locator in dom hand

Interaction Techniques • Also can do “Action + 1 scalar parameter” – E. g. , Scale operation and magnitude of scaling – Action set by orientation • Twist to get the one you want – Parameter set by movement “up & down” (i. e, in direction of action selection) • Absolute or clutched like mouse • Could also do “Action + 2” but starting to get away from natural tendencies of non-dominant hand • Can also do things like 3 D object or camera or positioning that require more 2+ DOF

Other forms • Doesn’t have to be rectangular or even have parallel sides – Although lots of flat surfaces seems useful • More tactile differentiation of faces might be useful • Could also augment with buttons – But could easily get beyond spirit of device for non-dominant hand