Personal Assistant System Use of Computer Technologies to

Personal Assistant System: Use of Computer Technologies to Facilitate Elderly People with Assisted Living Presenter: Jennifer C. Hou‡ Collaboration with Linda Ball*, Stanley Birge*, Marco Caccamo‡, Carl Gunter‡, Karrie G. Karahalios‡, Narasimhan Nitya #, Lui Sha‡, Yang Yu# ‡ Dept of Computer Science University of Illinois at Urbana Champaign *College of Medicine, Washington University in Saint Louis #Pervasive Platforms and Architecture Lab, Motorola Labs jhou@cs. uiuc. edu, 217 -265 -6329

Population Aging n n n Aging of the baby boomer has become a social and economical issue. In the United States alone, the number of people over age 65 is expected to hit 70 million by 2030, almost doubling from 35 million in 2000. Table compiled by the U. S. Administration on Aging based on data from the U. S. Census Bureau.

Percentage of People of 65+ and 85+ n n n People over age 65 are expected to constitute 20% of the population in 2030. Similar increases are expected worldwide. Table compiled by the U. S. Administration on Aging based on data from the U. S. Census Bureau.

Similar Expenses Worldwide 2002 SOURCE: United Nations ▪ “Population Aging ▪ 2002”

Similar Expenses Worldwide 2030 SOURCE: United Nations ▪ “Population Aging ▪ 2002”

Consequences Are …. . n n Along with the increase of elderly people population, the expenditures of the United States for health care will project to rise to 15. 9% of the GDP ($2. 6 trillion) by 2010. -- Health care industry study, Digital Foresight Many elderly people will stay at home, rather than being consigned to expensive retirement homes. n n Even today, only 10% of elderly people of age 65 -85 and 25% of those of age >= 85 are institutionalized. Many elderly people choose to stay at home also for privacy/dignity issues.

Overview of the Talk n n n Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work n n n n Do Elderly People Accept This Technology? n n Systems Architecture Wireless communications Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Pilot studies Where Do We Go From Here?

Why Do We Care About Senior Care n n Because your parents are the next in line to be qualified as seniors, and you are next to the next in line. Can advances in sensing, object localization, wireless communications technologies n n enable elderly people to regain their capability of independent living? make possible unobtrusive supervision of basic needs of frail elderly and thereby replicate services of on-site health care providers? We believe the answers are Yes!

We Believe Technology Can Help n Time Driven Reminders of Daily Activities: Home PC n n Serves as the intelligence. Sends reminder messages to wireless-enabled appliances. Closes the loop with HCI/ localization techniques. Takes action in the lack of response: A reminder can be sent more times, after which a designated on-site personnel or a healthcare provider is notified. Jennifer, It is 8: 30 am. Time to take your Insulin injection before breakfast.

Overview of PAS Assisted Living Hub (ALH) ALSP Clinician Service Gateway Medical Device n A drop-box architecture in which n A security-enhanced, assisted living device called Authentication Manager for You (AMY) co-exists with a home PC, called the Assisted-Living Hub (ALH). n The ALH is equipped with multiple wireless interfaces and serves as the local intelligence. It also communicates with the Assisted Living Service Provider (ALSP). n Through web interfaces, ALSP allows healthcare providers and clinicians to retrieve/analyze data and give instructions.

A More Technical View of PAS Home Environment ALSP ALH IEEE 802. 11 WLAN . AMY Internet • Easy to deploy. • Few entities that are subject to security attacks. • Numerous applications can be built on top of it. Monitoring Service Clinicians

What Applications Are Most Critical Factors Contributing to Loss of Independence and Institutionalization* % of Residents as Need With Need Primary Cause ____________________________________ Needs prompting to take medications 95 42 Risk of injury due to falls 42 17 Unable to get up after a fall 20 17 Monitoring of vital signs too labor intensive 12 20 Needs physical assistance with Activities of Daily Living 90 67 Needs prompting to toilet on a schedule 67 17 Needs prompting to go to meals 33 10 Needs prompting to bathe 75 0 Gets lost in apartment 17 0 May wander out of facility 12 10 Needs monitoring of blood sugar frequently 20 8 Needs monitoring of weight daily/weekly 25 0 * Respondents include 8 geriatricians, 10 nurse administrators of assisted living facilities, and 6 home health clinicians at Washington University in Saint Louis

Application I: Time-based Reminder Services n PAS can help reminding residents of daily activities. n n n ALH obtains from the ALSP updated prescription and appointment records of a resident. When it is time for the resident to carry out their time driven routines, the ALH locates active wireless-enable devices and sends reminder messages to one or more devices that are in the proximity of the resident. Whether or not these routines are followed as advised is detected in a non-intrusive manner by exploiting sensor localization techniques. Assisted Living Hub (ALH) ALSP Gateway Medical Device Clinician Service

Application II: Monitoring of Physiological Functions n n n A number of physiological functions critical to maintaining homeostasis for different medical conditions can be measured by Bluetooth-enabled medical devices, transmitted to the ALH and then to the ALSP to be evalauted by healthcare providers. Measures will have a prescribed desired range and deviations from that range will generate an alert from the ALSP to the health care provider. This enables prompt intervention before the situation deterioates to a point requiring hospitalization and may simply be in the form of additional instructions to the resident. ALSP Home Environment IEEE 802. 11 WLAN . AMY. Internet Monitoring Service

Application III: Non-intrusive Monitoring of Daily Activities and locations n Detection of early warning signs for depression and/or other chronic diseases: n n The location of a resident and the duration in which he/she stays at that location are profiled by Motes-based (or Ubisense-based) sensing and tracking in a privacy-preserving manner. Movement profiles are transmitted to the ALSP and analyzed (if necessary) for n Early warning signs for severe depression (e. g. , not taking medicine, not eating, staying in bed for very long time) n Preventive measures (behavior changes) for chronic diseases commonly seen in elderly people (such as Parkinson’s disease and/or Alzheimer’s disease).

Application IV: Fall Detection n Residents wear sensors equipped with accelerometers (with fall detection algorithms that detect falls with the combination of speed and orientation changes). In the case of fall detection, the sensor device beeps and an alert message is sent to the ALSP and to a designated healthcare provider. In the case of false alarm, the resident can press a button and disable the message sending.

Overview of the Talk n n n Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work n n n n Do Elderly People Accept This Technology? n n Systems Architecture Wireless communications Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Pilot studies Where Do We Go From Here?

Software Infrastructure Needed n A software infrastructure that integrates sensing, communication, and event/information management. n Understand, analyze, structure and control the complex interactions across the layers of computing, communication and sensing along the dimensions of robustness, reliability, Qo. S, security and privacy. Reminders/social interaction Monitoring Emergency event processing Telemedicine Interface mgmt Events/data management Communication Sensing Dependability Evolvability Real-time Security & Privacy

Networked Software Architecture Socket s = new Socket(myaslp. com: 90) Active Application A ss. accept(); ALSP Server Under Normal Situation J 2 ME API for Accessing BT Nodes and Internet TCP/IP Stack Server. Socket ss = new Server. Socket(90); When the Gateway is Available Fixed ALH BT Stack OS TCP/IP Comm. API TCP/IP Stack Bluetooth Vital Sign Meters 802. 11 IP Routing Service 802. 11 MAC Wireline MAC Gateway Cellphone Network + Internet ALSP Server OS

System Architecture of the ALH

OS Layer in the ALH Architecture n The OS layer n n Is equipped with various communication stacks and corresponding platform-dependent APIs Depending on the family of peripherals to support, other stacks and APIs may include Bluetooth, Zigbee, and Infrared.

Middleware Layer in the ALH Architecture n Middleware n n n Device monitoring daemons: monitor the join/leave of peripheral devices in the environment and registers/de-registers the devices in the Device Registry Services. The register-de-register process includes creating/destroying proxy or singular proxy stub objects for the device. The proxy/proxy stub allows applications to be built upon well-known device APIs instead of vendor specific APIs/semantics. It also allows offthe-shelf devices to be integrated into the PAS system, as long as the vendor provides the semantics specifications on how to communicate with the device.

Middleware Layer in the ALH Architecture n n Device registry service: maintains a database of peripherals available in the environment. Each entry of the database is a proxy (or proxy stub) object created for a specific peripheral device. Applications will query this service for appropriate proxies. When an appropriate proxy is found, a clone of the proxy migrates to the application.

Middleware Layer in the ALH Architecture n The Unified Peripheral Communication APIs n n n Abstracts different network stack programming interfaces into a consistent paradigm, which basically follows java. net. * APIs. Allows users to specify (in XML) their Qo. S demands when establishing communication links, and the middleware returns approved Qo. S guarantees. This allows proxies to be developed independently of specific network stack programming interfaces.

Middleware Layer in the ALH Architecture n Internet Heartbeat Daemon: n n n periodically checks the availability of Internet access through the gateway router. When the gateway router fails/recovers, this daemon activates/deactivates the Bluetooth cell phone to access to the ALSP. ALH Main Daemon: is in charge of managing (start, suspend, stop, restart etc. ) all the application daemons and middleware daemons on the ALH.

Overview of the Talk n n n Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work n n n n Do Elderly People Accept This Technology? n n Systems Architecture Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Pilot studies Where Do We Go From Here?

Networked Software Architecture TV Peripheral Bluetooth Network earplug ALSP ALH Internet Gateway WLAN (e. g. IEEE 802. 11) Internet-capable Medical Meters

Networked Software Architecture Socket s = new Socket(myaslp. com: 90) Active Application A ss. accept(); ALSP Server Under Normal Situation J 2 ME API for Accessing BT Nodes and Internet TCP/IP Stack Server. Socket ss = new Server. Socket(90); When the Gateway is Available Fixed ALH BT Stack OS TCP/IP Comm. API TCP/IP Stack Bluetooth Internet Vital Sign Meters 802. 11 IP Routing Service 802. 11 MAC Wireline MAC Gateway ALSP Server OS

Server. Socket ss = new Server. Socket(90); ss. accept() s = new Networked Software Architecture Socket(128. 174. 11: 90) Passive Application A ALSP Server Under Normal Situation J 2 ME API for Accessing BT Nodes and Internet TCP/IP Stack When the Gateway is Available Fixed ALH BT Stack OS TCP/IP Comm. API TCP/IP Stack PC IP: 128. 174. 11 Bluetooth Dumb Dev 802. 11 IP Routing Service 802. 11 MAC Wireline MAC Gateway Cellphone Network + Internet ALSP Server OS

What If the Internet Gateway Is Down? n n n One major deficiency of PAS is that all the traffic is transported through the gateway AMY, which becomes unavailable when it fails or when the resident is away from home. To enhance robustness and ubiquity of PAS, we have used cell phones (transparent to users) as both a backup AMY and the local intelligence for data aggregation and acquisition. we have leveraged the programming capability of Motorola EZX platforms (mainly the A 780, E 680 and E 680 i phones) to incorporate the following novel features that are not currently present in any cell phones: n n Enabling cell phones to serve as local data storage/fusion intelligence. Enabling cell phones as a delivery endpoint for reminder messages

Networked Software Architecture mobile medical TV Peripheral devices Bluetooth Network earplug ALH WLAN (e. g. IEEE 802. 11) Internet-capable Medical Meters ALSP cellphone Internet GPRS network or Wi. Fi Base Station When the Gateway is Not Available

Socket s = new Socket(myaslp. com: 90) The cellphone modem is activated Cellphone DUN Modem (service provided by most BT enabled cellphones) Active Application A J 2 ME + Bluetooth Dial-Up Networking Profile (BT-DUN) for Internet Access Fixed ALH OS APIs (TAPI/NAPI, File system, etc. ) Service Layer (Network, Connectivity, Telephony, etc) BT Stack Cellphone OS BT Stack GPRS Server. Socket ss = new Server. Socket(90); ss. accept(); ALSP Server TCP/IP Comm. API TCP/IP Stack Bluetooth Medical Meter 802. 11 IP Routing Service 802. 11 MAC Wireline MAC A. L. Device Cellphone Network + Internet ALSP Server OS

Server. Socket ss = new Server. Socket(90); The cellphone modem is activated ss. accept() Cellphone DUN Modem (service provided by most BT enabled cellphones) Active Application A J 2 ME + Bluetooth Dial-Up Networking Profile (BT-DUN) for Internet Access Fixed Butler OS APIs (TAPI/NAPI, File system, etc. ) Service Layer (Network, Connectivity, Telephony, etc) BT Stack Cellphone OS BT Stack GPRS Socket s = new Socket(128. 174. 11. 12: 90) ALSP Server TCP/IP Comm. API TCP/IP Stack Bluetooth Medical Meter 802. 11 IP Routing Service 802. 11 MAC Wireline MAC Gateway Cellphone Network + Internet ALSP Server OS

Networked Software Architecture mobile medical Peripheral devices Bluetooth Network earplug Cellphone ALSP GPRS or Wi. Fi Internet Base Station WLAN (e. g. IEEE 802. 11) Internet-capable Medical Meters When cell phone takes the roll of ALH

Socket s = new Socket(myaslp. com: 90) The cellphone is used as a smart device Application 2 (Sending data to ALSP Server) APIs (TAPI/NAPI, File system, etc. ) J 2 ME Fixed Butler OS Qo. S based Application 1 (Listening to PC) Active Application A Service Layer (Network, Connectivity, Telephony, etc) BT Stack Cellphone OS BT Stack GPRS Or Wi. Fi Server. Socket ss = new Server. Socket(90); ss. accept(); ALSP Server TCP/IP Comm. API TCP/IP Stack Bluetooth Medical Meter 802. 11 IP Routing Service 802. 11 MAC Wireline MAC A. L. Device Cellphone Network + Internet ALSP Server OS

Networked Software Architecture Socket s = new Socket(myaslp. com: 90) Active Application A Server. Socket ss = new Server. Socket(90); ss. accept(); ALSP Server When cell phone takes the roll of ALH TCP/IP Comm. API J 2 ME API for Accessing BT Nodes and Internet GPRS Or Wi. Fi Fixed ALH BT Stack OS TCP/IP Stack Bluetooth Vital Sign Meters Internet GPRS network IP Routing Service or 802. 11 Base Station ALSP Server OS

Overview of the Talk n n n Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work n n n n Do Elderly People Accept This Technology? n n Systems Architecture Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Pilot studies Where Do We Go From Here?

Real-Time Tracking and Localization We have built a system that uses a combination of ultrasonic and RFID technologies as the underlying sensing mechanism for real-time tracking of both human and objects. Human Tracking: n n n Ultrasonic transmitters, called beacons and denoted by Bi, occasionally sends out an ultrasonic pulse and a short RF signal at the same time. Due to the speed difference between an RF signal and an ultrasonic signal, the listener denoted by Lj on the resident can infer its distance from the beacon using Time Difference Of Arrivals of the two signals. This distance measurement can be reported to the ALH for realtime tracking of the use (marked as (2) and (3)).

Real-Time Tracking and Localization n Object tracking n n n The RFID (radio frequency identification) reader attached on the wristband can read RFID tags of objects touched by the user. The objects can be tracked, whenever the user touches and/or carries them. The whereabouts of the objects are logged.

Overview of the Talk n n n Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work n n n n Do Elderly People Accept This Technology? n n Systems Architecture Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Pilot studies Where Do We Go From Here?

How Medical Data Can Be Securely Transmitted ? Pub. AS/Priv. AS Pub. C/Priv. C SOAP Envelope Header Medical Information Pub. C Pub KFAS Sign. on Msg. PW AP Sign. on Key PW AP PWAP [Michael et al. 06] KF

How Medical Data Can Be Securely Transmitted ? Pub. AS/Priv. AS SOAP Envelope Header Medical Information Pub. C KF Pub. AS Sign. on Msg. Sign. on Key PW Pub. C/Priv. C Sign. on Msg. PW AP AP PWAP KF

How Medical Data Can Be Securely Transmitted ? Pub. AS/Priv. AS Pub. C/Priv. C SOAP Envelope Header Medical Information Pub. C KF Pub. C Sign. on Msg. Priv AS Sign. on Key Priv AS PWAP KF

How Medical Data Can Be Securely Transmitted ? Pub. AS/Priv. AS Pub. C/Priv. C SOAP Envelope Header Medical Information Pub. C KF Pub. CPub KF C PWAP Sign. on Msg. Priv AS Sign. on Key Priv AS KF

Implementation

Implementation n H/W n n n Device: Digital Pulse Oximeter AL Hub: A labtop, Windows ALSP: A desktop, Linux Clinician’s computer: A labtop, Windows S/W n n Language: Java Database: My. SQL Web service: Apache web server, tomcat, axis 1. 4 Libraries: XML-SEC, SAAJ, avetana. Bluetooth, etc.

Security tokens n The following security tokens are set up n n n Doctor’s public key certificate The ALSP’s public key certificate A secret key shared among AP and AP’s family members or friends AP’s and family/friends’ password to ALSP Access control matrix (ALSP), Data. ID table (AL Hub) GDoc, GAS, KAP, U*; V*, PPA , PFA , ACLAP , ΔAP • GDoc: Doctors office’s public key certificate. • GAS: The ALSP's public key certificate. • U*; V*: URL pointers to the ALSP's public key certificate. U*; V* are the ones which AP and clinician have, respectively. • KAP: A secret key shared the AP and her family members. • PPA; PFA: AP's and Family's secret passwords (respectively) shared with the ALSP. • ACLAP: An access control list for the AP's records at the ALSP. • ΔAP: A mapping table from a serial number of a medical device to Data. ID

Bootstrapping 1) 2) 3) 4) 5) 6) 7) Clinician sets up relationship with ALSP / Exchange URL pointers Assisted person (AP) subscribes to an ISP / Set up a gateway for WPA network AP visits clinician. Clinician recommends a list of devices, hubs, and ALSPs. AP buys devices and hub and adds them to home network AP subscribes to ALSP sends a URL pointer and a password token. AP loads them into the hub. AP creates a shared “family key” for family members’ and friends’ access to data on the ALSP. Family members register passwords to ALSP. Hub generates ΔAP , a set of Data. ID. A data. ID is mapped to a device serial number and recipients’ URL pointers. Each entry is a tuple (serial number, data. ID, URL pointer) With the recipient information (clinicians and family members), ALSP creates access control list ACLAP. Gdoc, GAS, KAP, U*, V*, PPA, PFA, ACLAP, ΔAP Clinician AP ALSP Family

Overview of the Talk n n n Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work n n n n Do Elderly People Accept This Technology? n n Systems Architecture Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Pilot studies Where Do We Go From Here?

Pilot Study at Nazareth Living Center* n n n With the help of geriatricians at Washington University in Saint Louis, we have carried out a 3 -week pilot study at the Nazareth Living Center for Assisted Living in June-July 2006. This facility houses 110 well-educated, predominantly female residents, whose average was 88. Of 30 residents who attended a presentation, 14 agreed to participate. After administering a standard cognitive assessment, two residents were consented to test the PAS prototypes (application 1 and applicatoin 2) and ten residents to carry/wear a placeholder device for a period of two weeks. * Nazareth Living Center is a Catholic, nonprofit skilled nursing and assisted living community, sponsored by the Sisters of St. Joseph of Carondelet.

Pilot Study at Nazareth Living Center n Residents found PAS useful and were willing to wear the sensing apparatus: n n n The two residents using the PAS prototype found it to be quite useful. Residents not chosen to use the working prototype expressed their desire to use the working version as well. Residents lack in confidence in PAS when it did not work properly: n n n Wireless connectivity is made possible by a low -end Linksys WRT 54 G wireless router. This, coupled with concrete walls between residents' rooms and the nurse station, led to intermittent connectivity. Residents, as a result, were not confident in relying solely on PAS for medical monitoring. To address this problem for the time being, we will add a wireless repeater (i. e. , an additional Linksys WRT 54 G router configured in the client -mode and forwarding all packets to another, master router) between residents' rooms and the nurse station.

Pilot Study at Nazareth Living Center n Nurses/caretakers desire interfaces that provide security/privacy. n n With a high resident-to-nurse ratio, nurses were usually very busy and could not ensure that the information being displayed would not be viewed by unauthorized personnel. The need for privacy should be addressed by designing adequate access control to PAS. Nurses inquired whether or not medical data could be securely transmitted via wireless technology. This implies they also had concerns about PAS security.

Overview of the Talk n n n Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work n n n n Do Elderly People Accept This Technology? n n Systems Architecture Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface Pilot studies Where Do We Go From Here?

Where Do We Go From Here? n n Technology enhancement with respect to software robustness/reliability, security/privacy, and HCI. Extensive, hypothesis-driven clinical trials with Washington University in Saint Louis and Buehler Center on Aging, Health & Society, Northwest University n n The average cost of nursing home stay is about $8, 300/month*. Can PAS delay transfers of elderly people to skilled nursing facilities (SNFs) and improve the quality of their lives (by preserving independence)? Delaying institutionalization by simply one month would cover the cost of deploying PAS. *Phibbs CS, Holtz J-EC, Goldstein MK et al. The effect of geriatrics evaluation and management on nursing home use and health care costs: Results from a randomized trial. Med Care 2006; 44: 91 -5.

Concluding Remark n With most of the PIs being baby boomers, it is our sincere hope that we will be the first generation to live independently and with dignity with the help of PAS when we age, and the project will benefit the healthcare system and the society as a whole.