Lecture 19 Physical Gadgets and their Interaction Techniques

Lecture 19: Physical Gadgets and their Interaction Techniques Brad Myers 05 -640 : Interaction Techniques Spring, 2019 © 2019 - Brad Myers 1

Definitions and Synonyms l l l Ubiquitous computing (Ubi. Comp) - computing everywhere and anywhere Pervasive computing – (no separate definition) Ambient intelligence (mostly used in Europe) – environment is instrumented so it is sensitive and responsive to people Information appliances – Smartphone or PDA Context-aware computing – mobile device that knows its surroundings, such as location, light, sound, etc. Tangible user interfaces (TUIs) -- person interacts with digital information through the physical environment l l Formerly “graspable UIs” Has its own conference series: l l TEI’ 19: 13 th International Conference on Tangible, Embedded and Embodied Interaction, The Netherlands, March 17 -20, 2019 in Tempe, Arizona Internet of Things (Io. T) 2 © 2019 - Brad Myers

Definitions, cont. l Physical Gadgets l Are to physical (tangible) user interfaces what interaction techniques are to graphical user interfaces l l An interaction technique embodied in a physical entity l l Adapted from [Greenberg’ 01] Must be reusable Many other TUIs are tabletop interactions with physical objects sensed on a table with a projector l ACM International Conference on Interactive Surfaces and Spaces (formerly ACM ITS, International Conference on Interactive Tabletops and Surfaces © 2019 - Brad Myers 3

Scope l l There are lots of interesting, cute, even useful applications of tangible and ubiquitous user interfaces Most are not interaction techniques l E. g. , Ambient displays – no interaction l l Mankoff’s Bus. Mobile E. g. , Tangible applications – not a reusable widget l Bottles that play sounds when opened 4 © 2019 - Brad Myers

Logo “Turtle” l l l From 1967 by Daniel G. Bobrow, Wally Feurzeig, Seymour Papert and Cynthia Solomon at MIT Simplified programming for children Originally drove a physical turtle on the floor with a physical pen l l Pen up / down Walk forward / turn 5 © 2019 - Brad Myers

Lego Mindstorms l Introduced in 1998 l l l Original kit contained light sensors, buttons, touch sensors, motors, etc. 1 st version programmed using “RCX code” l l l Named after Seymour Papert’s book Blocks language implemented in Macromedia Director Could also be downloaded from other languages Slow processor, low-quality sensors and actuators 6 © 2019 - Brad Myers

Phidgets l l l Saul Greenberg and Chester Fitchett. 2001. Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the 14 th annual ACM symposium on User interface software and technology (UIST '01). ACM, pp. 209 -218. http: //doi. acm. org/10. 1145/502348. 502388 “Physical widgets” Previously was very difficult to build TUIs l l Encapsulated complexities of using physical objects l l l Had to build custom hardware and microprocessors Soldering, circuit design (EE), assembly-language programming, etc. Lots of new sensors Lights, motors, sensors, cameras, switches, etc. Mostly USB Interactive since sensors for motion, light, sound, etc. l Sensor+control counts as interaction technique, not just a moving flower l Programmed (originally) in Visual Basic Simulation mode to help create the software Formed a company to market his phidgets l Video, 6: 10 (2001) l l 7 © 2019 - Brad Myers

Controllers l l l l Started about 2005 with Arduino http: //arduino. cc/ Single-board microcontroller Open source electronics prototyping platform Now about $9 to $30 each Easy to program and attach devices to Still a solid choice Current alternatives (from Chris Harrison) l https: //www. raspberrypi. org/ - cheap and super popular https: //www. hardkernel. com/ - ODroid - more powerful single board computer l http: //www. pjrc. com/teensy/ - faster/smaller version of Arduino l l l http: //leaflabs. com/devices/maple/ If you want to start from a phone, there is the IOIO-OTG board for android: l https: //www. sparkfun. com/products/12633 8 © 2019 - Brad Myers

Toys and Robots l Allison Druin. 1988. NOOBIE: the animal design playstation. SIGCHI Bull. 20, 1 (July 1988), 45 -53. http: //dx. doi. org/10. 1145/49103. 49106 l l Many toys l l Giant stuffed animal with sensors and a screen E. g. , Furby from 1998 by Tiger Electronics Many robots l l Some programmed by example Rethink Robotics Nao Boston Dynamics 9 © 2019 - Brad Myers

Research: Hiroshi Ishii l l l Tangible Media Group: http: //tangible. media. mit. edu/ He will be awarded the Lifetime Achievement award at CHI’ 2019 Dozens of projects dating back to 1990 l l But most are not “interaction techniques” One that is: l John Underkoffler and Hiroshi Ishii. 1999. Urp: a luminous-tangible workbench for urban planning and design. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI '99). ACM, pp. 386 -393. http: //doi. acm. org/10. 1145/302979. 303114 l Physical tools for measuring, changing building material, turning on wind, changing light paths, etc. 10 © 2019 - Brad Myers

Ishii, cont. l l l Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune. 2012. Radical atoms: beyond tangible bits, toward transformable materials. interactions 19, 1 (January 2012), 38 -51. http: //doi. acm. org/10. 1145/2065327. 2065337 Includes a survey of tangible Uis Lists lots of toolkits to create TUIs “Tangible design seeks an amalgam of thoughtfully designed interfaces embodied in different materials and forms in the physical world—soft and hard, robust and fragile, wearable and architectural, transient and enduring. ” Future: physical-digital “atoms” that can transform, conform and inform l E. g. , “clay” that changes its own shape based on rules, user commands, & constraints 11 © 2019 - Brad Myers

More Research: Skweezee l l l Karen Vanderloock, Vero Vanden Abeele, Johan A. K. Suykens, and Luc Geurts. 2013. The skweezee system: enabling the design and the programming of squeeze interactions. In. Proceedings of the 26 th annual ACM symposium on User interface software and technology(UIST '13). ACM, pp. 521 -530. http: //doi. acm. org/10. 1145/2501988. 2502033 Soft tangible objects, filled with conductive padding and embedded sensors (eight electrodes) Toolkit for defining squeeze gestures by example l Learns from a single example 12 © 2019 - Brad Myers

“Wearable” technology l l Fitbits, Apple Watch, etc. Project Jacquard Levi's Jean Jacket with the touch sensitive cuff: https: //atap. google. com/jacquard/ l l l “Jacquard™ by Google woven in” Video: https: //youtu. be/G 9 ADVe. Npypk $350 13 © 2019 - Brad Myers

1997 Portable Ubi. Comp l l l S. Feiner, B. Mac. Intyre, T. Höllerer, and T. Webster, A touring machine: Prototyping 3 D mobile augmented reality systems for exploring the urban environment. Proc. ISWC '97 (First IEEE Int. Symp. on Wearable Computers), October 13 -14, 1997, Cambridge, MA. Also in Personal Technologies, 1(4), 1997, pp. 208 -217 head-tracked, see-through, headworn, 3 D display, and an untracked, opaque, handheld, 2 D display with stylus and trackpad, GPS Compare to modern smartphone! 14 © 2019 - Brad Myers

Research: Hand-Held Projectors l l l Xiang Cao, Clifton Forlines, and Ravin Balakrishnan. 2007. Multi-user interaction using handheld projectors. UIST '07. ACM, 43 -52. http: //dx. doi. org/10. 1145/1294211. 1294220 Each person holds a projector with integrated camera Interact by moving projector or items in the scene Extra buttons on projector or externally Video (6: 08) Current projectors are the size of a smartphone l Sony Portable HD Mobile Projector (MPCL 1) 15 © 2019 - Brad Myers

Hybrid mobile device and physical l l Sensors on smartphones – physical manipulate the phone itself Also mobile + environment l Early example: Jun Rekimoto and Masanori Saitoh. 1999. Augmented surfaces: a spatially continuous work space for hybrid computing environments. CHI '99, 378 -385. http: //dx. doi. org/10. 1145/302979. 303113 video 3: 14 l Andy Wilson at Microsoft research has lots, e. g. : Andrew D. Wilson. 2005. Play. Anywhere: a compact interactive tabletop projection-vision system. In UIST '05. ACM, 83 -92. http: //dx. doi. org/10. 1145/1095034. 1095047 videos 16 © 2019 - Brad Myers

3 D printed physical gadgets l l l Marynel Vázquez, Eric Brockmeyer, Ruta Desai, Chris Harrison, and Scott E. Hudson. 2015. 3 D Printing Pneumatic Device Controls with Variable Activation Force Capabilities. In Proceedings of the 33 rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1295 -1304. https: //doi. org/10. 1145/2702123. 2702569 Pneumatic actuation Still need a hefty air pump (not shown) 17 © 2019 - Brad Myers

Scott Hudson’s class l l l 05 -833: “Applied Gadgets, Sensors and Activity Recognition in HCI” http: //www. hcii. cmu. edu/courses/appliedgadgets-sensors-and-activity-recognition-hci Not this year (Scott is on sabbatical) 18 © 2019 - Brad Myers