Workshop on Ubiquitous Computing Trip Report Jason I

Workshop on Ubiquitous Computing Trip Report Jason I. Hong Sept 28, 2001

Dagstuhl • Week-long seminars on Comp. Sci topics since 1990

Dagstuhl

Ubicomp Wkshp • Explore core issues in ubiquitous computing • Develop a research roadmap • Foster a community • Some of the topics – – – Lessons Learned Privacy Definitions of ubicomp Evaluating ubicomp Context-awareness

Background Original Ubicomp Apps • Badges emit infrared signals – Gives rough location + ID • Teleport – Redirect screen output from "home" computer to nearby computer • Phone forwarding Active Badge Olivetti / AT&T Hopper, Harter, et al – Automatically forward phone calls to nearest phone

Background Original Ubicomp Apps • Active badge + wireless – Rough location + ID • Proximate selection – Interfaces for nearby objects • Auto-diaries Parc. Tabs Xerox PARC Want, Schilit, et al – People, places, and time • Triggers – Alerts on preset events • Reconfiguration – Bind device to room

Roy Want's 10 Lessons Learned 1. People really care about how these technologies are used, be prepared for many long discussions 2. The press loves stories about ubicomp 3. Beware the press. . . 4. Only building something actually allows you to explore its full design potential. 5. Successful technology adoption is very dependent on the culture of the target users.

Roy Want's 10 Lessons Learned 6. Hard to adopt without a new level of utility 7. Build high-quality, customizable products 8. Only really get one chance to impress users with technology, second chance is rare. 9. Infrastructures are hard, lots of work to deploy and maintain. 10. Listen to user experiences, but filter desires

Privacy in Ubiquitous Computing Marc Langheinrich, ETH • • Design of technology has significant impact on what can and cannot be enforced (Lessig) Ubicomp relatively "boring" compared to AI, crypto, nanotech, and other media grabbers – But coming in fast under the radar! • • US Privacy Act, 1974 established Fair Information Principles 1995 European Union Directive on privacy – US / EU Safe Harbor program

Fair Information Practices EU Directive / Safe Harbor • Openness and Transparency – No secret record keeping • Individual Participation – See and correct your records • Collection Limitation – Data collected only for stated purposes • Data Quality – Data is relevant and up-to-date

Fair Information Practices (cont. ) EU Directive / Safe Harbor • Use Limitation – Data used only for stated purposes, by authorized personnel • Reasonable Security – Proportional to sensitivity of data • Accountability – Compliance with fair information practices • Explicit Consent – People have to agree

Privacy in Ubiquitous Computing Marc Langheinrich, ETH • Notice – – • Choice and Consent – • Temporary, unlinked identification of people Proximity and Locality – • Difficult because of scale Anonymous / Pseudonymous – • Announce what's going on Difficult because of scale, possibly filter via P 3 P Operate only when owner is nearby Access and Recourse – Trust but verify system is doing what it says

Defining Ubiquitous Computing • • In relation to more mature disciplines In relation to courses taught in universities

Evaluating Ubiquitous Computing • What's different from traditional evaluation? – Both from a systems and HCI perspective • • Evaluating many layers simultaneously Still many technological constraints – Battery life, Network bandwidth • Scale of evaluation – Space – Time – #People

Evaluating Ubiquitous Computing • Evolvability / Interoperability – Set of services, sensors, and devices changes over time, not a fixed set that you can design for • Failure Recovery – What happens if room crashes? • Introspection and Sensemaking – What just happened? Why did it happen? – What can I do here?

Dagstuhl

Some Issues in Context-Awareness • Push and Pull – "Where is Jimmy? " vs "Notify me when Jimmy enters the office" • Ambiguity, validity, probability • History of context vs garbage collection • Context hierarchies – People => Groups, Rooms => Building • Operators – Filters, interpreters, aggregators, fusion

Some Issues in Context-Awareness • Debugging and Simulation – Fake collected sensor data – Tracing sensor data, from source to transforms to sink • User interfaces for context info – Feedback, feedforward • Discovery / Replacement – Where is my data? What if I can't find it? Infer from sensors? • Security / Privacy / Trust – Who can access the information? – How am I seen by others? • Performance

Context-Awareness and AI • Is Context just Knowledge Representation? – World modeling, more dynamic – Success in narrow vs broad domains • Low-level vs high-level – In terms of sensors, inferencing, tasks – Anti-lock brakes vs "Avoid accident" • Layers in Speech recognition – Phone => Syllable => Word => Sentence – Corresponding layers for context?

Some Other Highlights • • Personal Server, Intel Disappearing Computer Initiative, EU – Physical disappearance vs mental disappearance • • Pirates Ubicomp Game, PLAY Intrabody communication, UW

Other Random Thoughts • • Invisible, Calm Computing was better received by press than "Big Brother" Location Tracking Early ubicomp research heavily systemsoriented because no standard platforms, many requirements – Devices, net access, tracking, applications • Ubicomp has many failure modes – Hardware, software, infrastructure, social • Most of ubicomp has focused on individuals versus groups

Why Ubicomp Failed (A hypothetical 10 year retrospective) • Technical Issues – – – • Couldn't scale, instability of system Power, too many batteries Modeling, couldn't get an interoperable world model Interoperability, nothing works with each other Critical mass, not enough sensors to be useful Usability and Usefulness – Ubi-maint, Ubi-beeping, Ubi-overload • Social Acceptance – Never addressed privacy adequately

Gutenberg Museum, Mainz • Lots of form factors – – Foldouts Small, large books Paper cutouts Full justify • What's missing? – Titles – Page numbers – Chapters / Headers • Like early web!

The Context Fabric Infrastructure Support for Context-Aware Computing Jason I. Hong Sept 28, 2001

Motivation • Modern computers are divorced from our reality – Unaware of who, where, and what around them – Leads to mismatch • Computers have extremely limited input – Aware of explicit input only – A lot of effort to do simple things (or to remember) • Context-Aware Computing – One line of ubicomp research – Making computers more aware of the physical and social situations they are embedded in

Why Context-Aware Computing? Existing Examples Context Types Human Concern Auto Lights On / Off Room Activity Convenience File Systems Personal Identity & Time Finding Info Calendar Reminders Time Memory Smoke Alarm Room Activity Safety Barcode Scanners Object Identity Efficiency

Some Technology Trends • Sensors – GPS, Active Badges, Active Bats – Smart Dust – Cameras and microphones • Recognition algorithms – MSR Radar, location from 802. 11 – Smart Floor footstep force • Wireless technologies – Bluetooth, 802. 11, cell phones

Why Context-Aware Computing? Potential Existing Examples Context Types Human Concern Auto Cell Phone Auto Lights On / Off In Meetings Activity Identity Convenience File Tag. Systems Photos Time Activity Location Finding Info Calendar Proximal Reminders Proximity Identity Memory Smoke Health Alarm Alert Activity Identity & Time History Safety Service Fleet Barcode Scanners Dispatching … Time Efficiency

Overview þ Background and Motivation o Some Views on Context-Awareness o Two Scenarios o The Context Fabric o Speak. Easy

What's Context? • Some views on Context – Situated Action, Activity Theory, Phenomenology, Distributed Cognition, Linguistics – Sophisticated views on how people understand interact with the world, and with each other • A computational view (bottom-up) – – Increase input channels into computer Push towards implicit acquisition Create better models to take advantage of input Using the input + models in useful ways

Desktop View We sort of know how to handle context-awareness here

Ubicomp View January

What's Different? • Harder to get input – Acquisition is distributed, not fixed set of input • Acquisition is fuzzier (sensors) – Need statistical approaches (HMMs, Bayesian Nets) • Modeling is distributed – What to model? How to model? • Dynamic set of sensors, services, and devices – On. Star custom-built with complete control • Privacy and Security of greater concern here – A lot more implicitly captured data

What's Similar? • Introspection – Why did it do that? – What does it know about me? • Bad guesses – Context for systems (Paperclip) – Context for people (Yahoo IM status) • Sensemaking – What can I do here? What limitations?

Overview þ Background and Motivation þ Some Views on Context-Awareness o Two Scenarios o The Context Fabric o Speak. Easy

Two Example Scenarios • Scenario 1 – "Where is an open alcove in Soda? " • One-time query or repeat notification? • How to model the situation? – What is the granularity of an alcove? – What data to store? Exact # people? Some people? • History – Store history? Where does it live? For how long? • Privacy – Who can make this query? Access control? Physical presence? • Where does the data live? And how to process it?

Two Example Scenarios • Scenario 2 – "Where is Jason? " • Multiple sources – Cell phone, desktop computer, Yahoo Messenger • Ambiguity, reliability, age of different sources • Where does the data live? • How is it protected? – Proximity? Group by Family, Friends, Strangers, etc? • Views – Earth, North America, Berkeley, Chez Panisse • History – Where does it live? How long to store?

Two Example Scenarios • Possible to create custom-apps in each case • Need something that: – Works independently of set of devices, sensors, services (as much as possible) – Scales space, time, #people – Provides lots of support for • Acquisition, modeling, processing, protection of context

Context Fabric • Infrastructure for building context-aware apps – Distributed data model of people, places, things – Context Specification Language (like SQL) • Other non-functional goals – ie, it would be really nice if… – Agnostic of sensor, CPU, OS, programming lang, network, discovery service, service platform – Able to evolve and incrementally deploy

Context Data Model • Question #1 - How is context represented? • Entities – Like nouns, people, places, and things • Attributes – Like adjectives or properties – Ex. identity, on / off, location, activity • Relationships – How one entity relates to another entity • Aggregates – Actions, Groups of people

Context Data Model c Person Name: Org: Role: Devices: Location: Jason Hong Xerox PARC Consultant {a, b} {c} b a

Context Data Model Action History Name: Arrive Action Time: 8: 30 AM Name: 2509 Arrive Room: Action 8: 31 AM Time: Name: Leave Action Time: 5: 32 PM Name: Leave Time: 5: 32 PM

Context Data Model • Question #2 – How long does info live? – Seems application dependant • Question #3 – Ontology? – Is there a single way of representing and talking about the world? – No universal ontology, situation and app dependant – Idea: Support multiple schemas simultaneously • PARC Schema for one set of context • Berkeley Schema for another set – What is the granularity of a place? Of a thing? • Make it like databases and XML, let specific domains define

Context Data Model • Question #4 – Where exactly does context live? – Entity, Attribute, Relationship, Aggregates are logical – Idea: Use tuple-spaces, a shared place where distributed apps can put and get data – Exists independently of sensor, process, app, device Sensor write take App read App

Context Data Model • Question #5 – How to scale up to the world? – Idea: Multiple distributed tuple-spaces – Logical entity • E. g. "Mark's context" or "HP Auditorium context" – Physical entity • E. g. "Part of Mark's context on device xyz"

Context Data Model • Question #6 – Protecting context data? – Access Control Lists on who can read, write attributes • Provide policies based on groups • Family, friends, co-workers, strangers • Temporary access too (1 hour, 1 day) – Certificates for authentication of requestor – Different views on data • Family and friends see "Jason is at Chez Panisse" • Co-workers see "Jason is out of the office" • Strangers see "Jason is in the Solar System"

Context Data Model • Question #7 – Is this the AI tarpit? – What can and cannot be modeled? • Question #8 – What does all this really buy us? – Separates acquisition, model, usage • Easier to evolve, more resilient to failure – Multiple schemas provide flexibility – Context data lives separately from process, application, device, you can always get to it – Templates for basic privacy policies • Family, friends, co-workers, strangers

Context Data Model • "Where is an open alcove in Soda? " • Each organization can define "place" locally – E. g. Berkeley Computer Science

Context Specification Language (CSL) • Problem: Difficult to coordinate data and services to get the right context data procedurally • Idea: Declaratively specify what you need • Query – "What are the nearby movie theaters? " – “How many people are in the room right now? ” • Events – “Notify me every time a person enters the room. ”

Context Specification Language What are the nearby movie theaters? Person GPS ZIP Theaters Cell ZIP Theaters Location, GPS Person Movie Theaters Location, Cell Person Proximity Context Output Theaters

Context Specification Language • Question #9 – How is context data manipulated? – Context operators, like database operators • Select, project, join, etc – Interpreters • E. g. motion => person ID, or wireless data to location – Fusers • Merge multiple streams together, e. g. sensor fusion – Data format transducers • Celsius to Farenheit – Exchange • Convert push service to pull, or pull to push – Filters • Get the top five, or eliminate things that can't be right

Context Specification Language • Question #10 – How to actually process it? – – Parse the query Generate a few equivalent plans for getting the data Find a decent plan Execute it • Databases have a nice clean algebra – This one is definitely messier, with unstructured data stored all over the place • Question #11 – Is this overkill? Maybe use templates for common types of queries? – "What are the nearest X? " "Where is Y? "

Context Specification Language • Question #12 – What does this buy us? – Introspection, "what's going on? " and "why did it that? " – More robust to changing set of sensors, services, devices do • Question #13 – Is this the AI tarpit again? – Don't want to solve Natural Language Problem! – However, it would be great to have a query language merging physical and virtual information

Context Data Model Current Status • Context Data Model – Looking at Semantic Web, Knowledge Representation – Looking at different tuplespace implementations • Context Specification Language – Working out all the basic queries – Need more work on how to assemble paths • Overall – Working out more end-to-end stories – Figuring out one or two issues to go deeply into • Privacy and scale

Overview þ Background and Motivation þ Some Views on Context-Awareness þ Two Scenarios þ The Context Fabric o Speak. Easy

Speakeasy: Supporting the Ubiquitous Computing User Experience Mark Newman, Keith Edwards, Jana Sedivy, Chris Neuwirth, Karen Marcelo, Trevor Smith, Jason Hong

Motivation The era of ubiquitous computing is upon us – many devices person, becoming interconnected

The Speakeasy Vision Enable Network Effects – analogous to phone, fax, web… Radical Interoperability – what if anything can talk to anything? – every new device or service adds value Deal with Complexity – support users’ sensemaking – what can I do in this world? – what the heck is going on?

Radical Interoperability No need to write specifically for a new component – Interact with components you’ve never heard of • Interact with types of components you’ve never heard of • Allow interoperability, defer semantics to people Our approach: mobile code + standard interfaces – (+ discovery + shared network) Identify the minimal set of interfaces. So far… - Data transfer & transformation - Context - Status & notification - User interface -…

User Experience Context - Modeled as People, Places, Things (Components) - People What components have I used before? What components belong to me? - Places What components are in this place? - Components Where am I? What can I do? How have other people used me? Key Idea - Provide key information to people, little inference

End

Questions? • "The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life. " – Mark Weiser • Context – the circumstances in which an event occurs; a setting; to join; to weave

Other Hard Questions • Can we rely on computers to make decisions? – Simple ones with little harm, e. g. Yahoo IM status – Continuous ones with high benefit, e. g. smoke alarm – What about complex ones, e. g. glass cockpits? • Fair information practices? – – – Notice, choice, access, security Privacy-Value tradeoff too Weak identity versus strong identity might help Weekly email summary of what happened "Privacy is ultimately a psychological construct, with malleable ties to specific objective conditions" (Grudin)

Other Hard Questions • How to debug and simulate applications? – Can't expect everyone to have lots of sensors

• • Push and Pull Dealing with ambiguity, validity, and probability – • History of context over time and garbage collection of – • Where is my data? What if I can't find it? Replace it? Or infer from a set of sensors Security / Privacy / Trust – • Feedback and feedforward Discovery / Replacement / Inference – – – • Fake collected sensor data Tracing sensor data, from source to transformations UI to context info – • Filters, aggregators, fusion Debugging and Simulation Support – – • People => Groups Architectural operators – • How much to save? Some is permanent, others transient Context hierarchies – • Allowing for statistical methods like HMMs and Bayesian Nets How am I seen by others? Performance – Fast enough for feedback

Context Data Model • Question #x – Refining context data? • Question #7 – Metadata for context data? – – – – Precision / Accuracy / Granularity Source / Trace chain Owner Age Volatility Time to Live Trust level of sensor / trust level of sensor data

Some definitions of context • "Any information that can be used to characterize the situation of an entity, where an entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and the application themselves. Context is typically the location, identity, and state of people, groups, and computational and physical objects. " – Abowd and Dey • "Context refers to the physical and social situation in which computational devices are embedded" – Moran and Dourish

Some definitions of context • Distributed cognition – Need to go beyond physical attributes – Look at “state of digital resources, people’s concepts, task state, social relations, local work culture” (Kirsh context essay) – Model underlying system of states, structures, and relations – Summary: model the key attributes of the whole system, the deep structure (instead of shallow surface structure) from individuals to offices to social structs and work practices

Some definitions of context • Situated action – actions are fluid, moment-by-moment, improvised, often unplanned, and highly context-dependent – “the context in which actions take place is what allows people to find it meaningful” (Dourish context essay) • Phenomenology – How people experience the world and find meaning within it – Shared meanings, common understandings – How does the meaning of the world reveal itself to us through our actions within it? – Thus, meaning (and hence context) arises from the ways in which we engage with and act within the world