stanford hci group cs 376 VisionBased Interaction Scott

stanford hci group / cs 376 Vision-Based Interaction Scott Klemmer · 28 November 2006 http: //cs 376. stanford. edu

cs 547: Blake Ross and Asa Dotzler Mozilla: Creating simple software in a geek-driven culture 2

The first vision-based interface Myron Krueger used computer vision to create Responsive Environments (1970 s) “Reaction is the Medium” http: //www. artmuseum. net/w 2 vr/timeline/v ideoplace_video. html 3

How it works Video and background are separated in analog using chroma key techniques (think broadcast news) The first and last points of each raster are stored in the computer, and represent the person’s outline 4

Vision-based UIs: “Verbs” Detecting and Tracking elements of a certain type in a scene Capturing contents of detected objects Recognizing individual members in an object class 5

Vision-based UIs: “Verbs” Detecting and Tracking elements of a certain type in a scene 6

Vision-based UIs: “Verbs” Capturing contents of detected objects 7

Vision-based UIs: “Verbs” Recognizing individual members in a class 8

Vision-based UIs: “Nouns” People (one or multiple) Bodies Faces Hands Documents Objects 9

Vision-based UIs: “Nouns” People (one or multiple) Bodies Faces Hands Documents Objects 10

Vision-based UIs: “Nouns” People (one or multiple) Bodies Faces Hands Documents Objects 11

INFRASTRUCTURE Background Subtraction 12

Image Moments (of Inertia) 0 th moment is mass (total number of pixels) 13

Image Moments (of Inertia) 1 st moment is center 14

Image Moments (of Inertia) 2 nd moment is orientation 15

Tools for Vision apps Intel’s Open. CV C API to highly optimized image processing functions (threshold, dilate, optical flow, …) http: //www. intel. com/research/mrl/research/opencv Fast to run! Slow to develop Great for vision folks; too low-level for app folks Papier-Mâché Java API (and to some extent visual UI) for vision (and other physical input) http: //guir. berkeley. edu/papier-mache Fast to develop! Slow to run Great for app folks; ~5 fps can sometimes be too slow 16

Good Vision Books Computer Vision: A Modern Approach David Forsyth and Jean Ponce (2003) Fantastic book; but goal is more theoretical understanding than practical application Robot Vision Berthold Horn (1987) More focused on apps and interactive algorithms Somewhat out of date 17

Next Time… Software Tools Past, Present, and Future of User Interface Software Tools, Brad Myers, Scott E. Hudson, Randy Pausch Natural Programming Languages and Environments, Brad A. Myers, John F. Pane, Andy Ko 18