Reliable Instrumented Cyber Physical Spaces Project Kickoff Meeting

Reliable Instrumented Cyber. Physical Spaces Project Kickoff Meeting Feb 7 th, 2011 – UC Irvine UCI: Nga Dang, Nikil Dutt, Leila Jalali, Xu Jie, Dmitri Kalashnikov, Zhijing Li, Sharad Mehrotra, Kazuyuki Tanimura, Ronen Vaisenberg, Nalini Venkatasubramanian, Xiujuan Yi, Liyan Zhang SRI: Grit Denker, Minyoung Kim, Carolyn Talcott 1

Agenda 10: 00 - 10: 15 -10: 45 - 12: 00 -1: 30 - 2: 45 -3: 00 3: 15 -4: 30 Welcome and Introduction Project Introduction and scope (Nalini) Presentations -UCI SAFire/Fire. Talk (Sharad/Dmitri) - SRI talks Lunch and poster presentations (by students) Scenario Discussion (surveillance, fire perspective) Break Project plans (research, system building) RCPS – Reliable Cyber. Physical Spaces 2

What are Cyber. Physical Systems (NSF) n n The term "cyber-physical systems" refers to the tight conjoining of and coordination between computational and physical resources. We envision that the cyber-physical systems of tomorrow will far exceed those of today in terms of adaptability, autonomy, efficiency, functionality, reliability, safety, and usability. Research advances in cyber-physical systems promise to transform our world with systems that respond more quickly (e. g. , autonomous collision avoidance), are more precise (e. g. , robotic surgery and nano-tolerance manufacturing), work in dangerous or inaccessible environments (e. g. , autonomous systems for search and rescue, firefighting, and exploration), provide large-scale, distributed coordination (e. g. , automated traffic control), are highly efficient (e. g. , zero-net energy buildings), augment human capabilities, and enhance societal wellbeing (e. g. , assistive technologies and ubiquitous healthcare monitoring and delivery). RCPS – Reliable Cyber. Physical Spaces 3

Instrumented Cyber. Physical Spaces Applications: surveillance & monitoring Observe nearby sensors Act Event: shooter on campus Analyze ! Shooter location: events UCI#out of doors/(3 interes 00, 506) t Shooter location: UCI#outdoors/(3 00, 506) RCPS – Reliable Cyber. Physical Spaces Applications: Situation Awareness

The Irvine Sensorium (Responsphere) Testbed Campus-Scale sensing, communication, storage, computing infrastructure - 200+ video cameras, Motes, sun spots, RFID, mobile cameras, gas sensors, -Mesh routers, Wi. Fi, power-line network, zigbee -storage & compute clusters RCPS – Reliable Cyber. Physical Spaces

ICPS applications Human as Sensor System Privacy Preserving Surveillance System ne ar by se ns ors ! SATWARE: semantic middleware for sentient spaces Occupancy Forecasting System Bren Hall Inauguration RF-ID tracking Calit 2 Recycling Monitor RCPS – Reliable Cyber. Physical Spaces Indoor Localization Framework SAFIRE- situational awareness System 6

Architecture SATWARE – Semantic Middleware for Sentient Spaces Powerful programming environment for sentient applications hides heterogeneity, errors, complexity Adaptive data collection Exploiting semantics RCPS – Reliable Cyber. Physical Spaces Native support for privacy

SATWARE Infrastructure Layer Video RCPS – Reliable Cyber. Physical Spaces 9

Challenges in Building ICPS n Sensors q q q Indoor localization still unsolved Sensor & platform diversity makes programming very complex. Robustness of sensors remains elusive n q Multimodal Sensors, e. g. , audio, video n q Calibration, sensitivity to ambient conditions, resilience in extreme situations both an opportunity and a challenge Semantic Sensing untapped potential despite benefits n E. g. , human speech, observations, blogs RCPS – Reliable Cyber. Physical Spaces 10

Challenges in Building ICPS n Infrastructure q Resource demands n q Application-driven constraints on quality, timeliness properties n q Increasing sensor complexity increasing network bandwidth Video, audio data with real-time constraints Battery based power supply impose significant restrictions n limits the transmission protocols that can be used by sensors RCPS – Reliable Cyber. Physical Spaces 11

Challenges in Building ICPS n Privacy in sentient spaces that monitor human activities ( E. g. , surveillance systems, smart buildings/infrastructure) q q Trust in infrastructure Policy languages to express privacy policies n q Policy enforcement requires inference control n q Who (can access), what (data), when and under which context , and for what (purpose). must ensure that inference made on data released does not violate privacy policies. Tradeoffs between loss of privacy and utility n Observer utility versus loss of privacy of target RCPS – Reliable Cyber. Physical Spaces 12

What can go wrong? n Infrastructure component errors/failures q q q n Device Failures Network Failures Congestion and Overloads Data Interpretation errors/failures q Uncertainty in Processing n q E. g. Speech/video/image processing Contextual errors n E. g. occlusions/obstructions to a light sensor RCPS – Reliable Cyber. Physical Spaces 13

Network Unreliability: Wi. Fi Mesh Commercial mesh routers not good enough 5 X improvement with new antenna technology Better signal coverage better building penetration RCPS – Reliable Cyber. Physical Spaces • Some Setup effort required • Not always feasible • Vulnerable to hardware failures

(Un) Reliability of Wi-Fi Networks • Varying traffic load • Varying level of contentions and congestions • Varying inter-device distance Ad-hoc 1 hop > Ad-hoc 2 hops > Private AP >>> Public AP • Increased bandwidth share • Reduced contentions/collisions RCPS – Reliable Cyber. Physical Spaces • Less interferences • Distributed Beaconing • No background traffic • Controllable configuration

SAFIRE Mote Sensor Deployment Heart Rate Proprietary EMF transmission Polar T 31 Heart rate strap transmitter Polar Heart Rate Module Crossbow MIB 510 Serial Gateway Crossbow MDA 300 CA Data Acquisition board on MICAz 2. 4 Ghz Mote Inertial positioning IMU (5 degrees of freedom) IEEE 802. 15. 4 (zigbee) Carbon monoxide Temperature, humidity RCPS – Reliable Cyber. Physical Spaces To SAFIRE Server Carboxyhaemoglobin, light

Sensing Unreliability ↑Mobility↓Reliability Network convergence, static gateway availability Calibration is essential Frequency matters!! mobile ↑Density↑Reliability matters!! Topology ↑Size↓Reliability SAFIRE: Situational Awareness for Firefighters

Approaches to enable reliable onsite networking n Exploit multiple networks that together provide connectivity (Mobiquitous 2005, WCNC 2007, INFOCOM 2009) n n Wi. Fi mesh – direct connectivity to a mesh router MANETS – hop by hop connectivity to gateway nodes Zigbee adhoc – connect to Wi. Fi backbone through gateway node Exploit mobility when disconnected (PERCOM 09, SECON 2010) n Store-and-forward networks (Delay Tolerant Networking) q n mobile nodes ferry data to gateway node Combine connected network clouds and disconnected networks (PWN 10) RCPS – Reliable Cyber. Physical Spaces

Note: Reliable infrastructure ≠Reliable Data n. Collection Sensing Errors Occur q Visibility Readings vary n q q n Occlusions etc. Spikes in Sp. CO readings due to FF movement Read errors due to misaligned sensor strip Reliability at application level is also needed n Sensor Calibration (MMCN 08) q n n Heart-rate, CO exposure Exploitation of Semantics, prediction Exploit application tolerance to errors RCPS – Reliable Cyber. Physical Spaces

Information Processing (Unreliability) Speech Recognition quality bottleneck – Poor recognition quality in noisy & realistic environments Speech Recognizer Output “This is a bad sentence” RCPS – Reliable Cyber. Physical Spaces This is a bed sun tan 20 20

RCPS Project Goals The goal of this project is to develop a semantic foundation, cross-layer system architecture and adaptation services to improve dependability in instrumented cyberphysical spaces (ICPS) based on the principles of “computation reflection”. RCPS – Reliable Cyber. Physical Spaces 21

Traditional Fault Tolerance Aspects n Failure Modeling q n n n Impact Analysis Failure Prevention/Avoidance Failure Detection Failure Recovery RCPS – Reliable Cyber. Physical Spaces 22

RCPS Main Contributions (Planned) n n Develop a digital state representation that guides a range of adaptations at different layers of the ICPS (i. e. networking, sensing, applications, cross-layer) to achieve end-to-end dependability at both the infrastructure and information levels. Techniques for reliable information delivery over multi-networks, quality aware data collection, semantic sensing and reconfiguration using overlapping capabilities of heterogeneous sensors. Adaptations driven by formal-methods based runtime analysis of system components, resource availability and application dependability needs. Responsphere, a real-world ICPS infrastructure on the University of California at Irvine campus, will serve as a testbed for development and validation of the overall “reflective” approach and the cross-layer adaptation techniques to achieve dependability. RCPS – Reliable Cyber. Physical Spaces 23

RCPS System Development RCPS – Reliable Cyber. Physical Spaces 24

Architecture RCPS System Plan – extend SATWARE Powerful programming environment for sentient applications hides heterogeneity, errors, complexity Adaptive data collection Exploiting semantics RCPS – Reliable Cyber. Physical Spaces Native support for privacy (ADD RELIABILITY)

Urgent Tasks n n Name Target Scenario(s) q q Surveillance Fire Response RCPS – Reliable Cyber. Physical Spaces 26

RCPS – Reliable Cyber. Physical Spaces 27

Related Project: SAFIRE 28

SAFIRE Project § Goal Improve the safety of firefighters by providing decision makers with greatly improved situational awareness during response activities. q n UC Irvine q q q q n Sharad Mehrotra Nalini Venkatasubramanian Chris Davison Dmitri Kalashnikov Jay Lickfett Jeffrey Xiu Ronen Vaisenberg Stefano Bonetti Imagecat Inc. q q q Paul Amyx Charlie Huyck Ron Eguchi RCPS – Reliable Cyber. Physical Spaces n Deltin Corporation q n Ron Cabrera Fire Fighter Forum q q County of LA Fire Dept. Newport Beach Fire Orange County Fire Authority City of Ontario

Main Deliverable – SAFIREStreams A software framework to create situational awareness from heterogeneous multimodal sensor streams n Captures/ingests data from heterogeneous SA sensors § § n Transforms raw sensor data to situational information § n Personal - Physiological (heart rate, blood CO, accelerometer) and location (Wi. Fi, Ultrasonic, RFID, GPS) Environmental– temperature, humidity, CO, light, sound Multimodal - Video cameras, Speech sensors (radio communication amongst responders) External data sources - (via Ebox technology) Declarative programming language for rapid application prototyping Provides core SA services § § Alerts, archival, replay functionalities Powerful UI for situation monitoring - Displays dynamic sensor data, overlay of contextual information RCPS – Reliable Cyber. Physical Spaces 30

Highlight: An End-to-end SA Tool for ICs SAFIREStreams Sensor Data Ingest Unit Sp. CO, light, inertial, RFID, heart rate, . . Temperature humidity, visibility Sensor Data Collection Sensor Stream Processing Module Virtual Sensors for Media Level events (Alerts, Queries, Replay, Triggers) Firefighter Status Dashboard Sensor Fusion Multisensor Event Extraction Multimedia Data Collection Visualization & Decision Support Services Available GIS layers Mapping and Localization Ambient CO Image/Video Audio/speech Receive /display alert messages. Weather Raw Data DB Event DB Raw Sensor data (sensors, speech, video) Semantically Enriched Event Data Sensor/Incident Storage& Archival Floor plans HAZMAT CAD Systems Demographics Occupancy Ebox External Data Access Programmed in Java/C++, executes on Windows based platforms RCPS – Reliable Cyber. Physical Spaces

SAFIREStreams Programming Execution RCPS – Reliable Cyber. Physical Spaces 5/27/09

Highlight –Drills & Experiments to Validate role of Sensors in Creating SA at Crisis Site q q q Haz. Mat, casualties, First Response drill and SAFIRE Deployment (16 SEP 08) Live Burn & CO Sensing Study: OCFA, LA County Fire (23 FEB 09) Haz. Mat drill (with multiple casualties) and SA Study (12 MAY 09) Tabletop exercise IC Usability Experiment (15 MAY 09) to determine role of sensor based awareness for decision making Analysis of in-field data collected to determine reliability of sensor data capture at crisis sites (IEEE PERNEMS 2010, IEEE IQ 2 S 2010) RCPS – Reliable Cyber. Physical Spaces 33 u

SAFIRE / FICB Usability Study q Goal n q Methodology n n q Test usability of SAFIRE technology for creating SA for ICs Table Top Exercise based on technology drill Drill Scenario stopped at 6 “freeze points” to assess Situational Awareness and impact on decision making Results n n Usability and decision-making impact significantly correlated with SAFIRE technology among ICs. Qualitative feedback overwhelmingly positive. Also, suggestions for improvement. RCPS – Reliable Cyber. Physical Spaces

Highlight – New Research Directions n SAFIRE Project led to new research directions q New Sensing Modalities: Radio communications between responders as a source of situational information n q Robust networking: crisis site communication requires “opportunistic networking” using multi-networks n q Initiated project with ICSI and SRI (two leading groups in speech processing and understanding) Initiated work on crisis site multi-networks with SRI researchers “On-the-fly” Information Integration: Ebox concept that provides dynamic access to site-specific information through web services interface and integrates it with SA tools for FF. n In discussion with DHS for proof of concept development RCPS – Reliable Cyber. Physical Spaces

Highlight – Technology Transfer n Formation of the HALO Consortium to explore q q q n Participants q q n Technology transfer opportunities within fire community Strategies to bring SAFIREStreams to real deployment Planning for continuity of research and development beyond AFG Grant Deltin Corporation Image. Cat Inc. University of California, Irvine Advisory Board of Fire Agencies Steps taken q q Transfer of code base out of UCI to Deltin and Imagecat Adding reliability to the code base (Summer 2010) Customizing SAFIRE streams to illustrate impact on specific FF casualty scenarios Demonstration of system to target FF groups. RCPS – Reliable Cyber. Physical Spaces

Lessons Learned n n Significant opportunities exist to improve FF safety through improved SA. Enabling technologies for creating SA systems exist / can be engineered q n Key challenge is transforming raw sensor feeds into actionable SA to support decision making q n Sensors, networking, data management, UI Heart rate versus danger to health SAFIRE stream system is a step in that direction q q Multi-sensor integration / fusion to improve quality/certainty of observations. Easily programmable interfaces to detect complex multi-sensor events. RCPS – Reliable Cyber. Physical Spaces 37

Related Project: Fire. Talk 38

RCPS – Reliable Cyber. Physical Spaces 39