Automated camerabased fall detection of elderly persons http

Automated camera-based fall detection of elderly persons http: //www. examiner. com/article/fall-prevention Alex Edgcomb Department of Computer Science and Engineering University of California, Riverside Copyright © 2014 Alex Edgcomb, UC Riverside. 1 of 37

Outline 1. Elderly person falls and background work 2. Synch. SM and moving-region-based fall

Falls in the elderly population need to be detected • Leading cause of injury-related hospitalization 1 and death 2 • 34% have fallen in the last year 3 • 14% have fallen more than once 3 http: //www. examiner. com/article/fall-prevention • Post-fall long lie correlated with passing-away 4 • 50% who experience a long lie pass-away within 6 months 4 1 Baker, S. P. and A. H. Harvey. Fall injuries in the elderly. Clinics in geriatric medicine, 1985. http: //www. presstv. ir/detail/218170. html 2 Hoyert, D. L. , K. D. Kochanek, and S. L. Murphy. Deaths: ﬁnal data for 1997. National vital statistics reports, 1999. 3 Lord S. R. , J. A. Ward, P. Williams, and K. J. Anstey. An epidemiological study of falls in older community-dwelling women. Australian journal of public health, 1993. 4 Wild, D. , U. S. Nayak, and B. Isaacs. How dangerous are falls in old people at home? British medical journal (Clinical research ed. ), 1981. Copyright © 2014 Alex Edgcomb, UC Riverside. 3 of 37

Falls need to be automatically detected. False alarm rates must be low. In-home care Automated monitoring $3, 400 – 5, 800 / mo. 5 http: //www. chcb. org/servicesprograms/medical-care/elder-care Under $100 / mo. http: //www. mobilehelp now. com/products. php http: //www. lifelinesys. com/content/ lifeline-products/auto-alert Discontinued… Amber. Select and Alert 1 Discontinued b/c false alarms too high http: //www. primemedicalalert. com/fall-detection. html 5 Genworth 2013: Cost of care. https: //www. genworth. com/corporate/ about-genworth/industry-expertise/cost-of-care. html Copyright © 2014 Alex Edgcomb, UC Riverside. 4 of 37

Reasons for video-based assistive monitoring http: //www. mobilehelpnow. com/products. php Body-worn Pro: Anywhere Con: Not always worn 6 Detect many events and trends Privacy enhance-able 7, 8, 9 6 Bergmann, J. H. M. and A. H. Mc. Gregor. Body-Worn Sensor Design: What Do Patients and Clinicians Want? Annals of Biomedical Engineering. Volume 39, pgs. 2299 -2312, 2011. 7 Beach, S. , R. Schulz, K. Seelman, R. Cooper and E. Teodorski. Trade-Offs and Tipping Points in the Acceptance of Quality of Life Technologies: Results from a Survey of Manual and Power Wheelchair Users. Intl. Symposium on Quality of Life Technology, 2011. 8 Beach, S. , R. Schulz, J. Downs, J. Mathews, B Barron and K. Seelman. Disability, Age, and Informational Privacy Attitudes in Quality of Life Technology Applications: Results from a National Web Survey. ACM Transactions on Accessible Computing, 2009. 9 Demiris, G. , M. J. Rantz, M. A. Aud, K. D. Marek, H. W. Tyrer and M. Skubic, A. A. Hussam. Older adults’ attitudes towards and perceptions of ‘smart home’ technologies: a pilot study. Medical Informatics and The Internet in Medicine, 2004. 5 of 37 Copyright © 2014 Alex Edgcomb, UC Riverside.

Approaches to camera-based fall detection Minimum bounding rectangle (MBR) Head tracking 3 D projection 10 laying standing Image from paper by Auvinet 20 Increasing order of computational complexity 10 Auvinet, E. , F. Multon, A. Saint-Arnaud, J. Rousseau, and J. Meunier. Fall detection with multiple cameras: An occlusion-resistant method based on 3 -d silhouette vertical distribution. Information Technology in Biomedicine, IEEE Transactions on 15, no. 2 (2011): 290 -300. Copyright © 2014 Alex Edgcomb, UC Riverside. 6 of 37

MBR-based fall detection Hung – Occupied area and height 11 Miaou – Height-towidth threshold 12 Thome – Height and width probabilistic model 13 11 Hung, D. H. and H. Saito. The Estimation of Heights and Occupied Areas of Humans from Two Orthogonal Views for Fall Detection. IEEJ Trans. EIS 133, no. 1, 2013. 12 Miaou, S. -G. , P. -H. Sung and C. -Y. Huang. A Customized Human Fall Detection System Using Omni-Camera Images and Personal Information. Proceedings of the 1 st Distributed Diagnosis and Home Healthcare Conference, 2006. 13 Thome, N. , S. Miguet and S. Ambellouis. A Real-Time, Multiview Fall Detection System: A LHMM-Based Approach. IEEE Transactions on Circuits and Systems Copyright © 2014 Alex Edgcomb, UC Riverside. for Video Technology, Vol. 18, No. 11, November 2008. 7 of 37

Head tracking and 3 D projection Rougier – Head’s vertical velocity 14 Anderson – Combine two silhouettes 15 Auvinet – Person’s height volume 16 laying standing 14 Rougier, C. , J. Meunier, A. St-Arnaud, and J. Rousseau. Monocular 3 D head tracking to detect falls of elderly people. In Engineering in Medicine and Biology Society, 2006. EMBS'06. 28 th Annual International Conference of the IEEE, pp. 6384 -6387. IEEE, 2006. 15 Anderson, D. , et al. Linguistic summarization of video for fall detection using voxel person and fuzzy logic. Computer Vision and Image Understanding 113, 2009. 16 Auvinet, E. , F. Multon, A. Saint-Arnaud, J. Rousseau, and J. Meunier. Fall detection with multiple cameras: An occlusion-resistant method based on 3 -d silhouette vertical distribution. Information Technology in Biomedicine, IEEE Transactions on 15, no. 2, 2011. Copyright © 2014 Alex Edgcomb, UC Riverside. 8 of 37

Outline 1. Elderly person falls and background work 2. Moving-region and synch. SM-based fall

Person tracking with in-home video via GMM = Gaussian mixture model foregrounding Current frame Background model 17, 18 (Pixel-level GMM based on color) Minimum bounding rectangle (MBR) Foreground MBR builder (adjacent pixel groups merged) • Stop learning background if insignificant amount of motion • Learn a second frame with MBR area replaced by background MBR filters (dampen, smooth, & glitch-removal) • Person tracking may occur on the camera itself • Computer vision person trackers tend to be 10 x slower because of 3 D projections and additional modeling 17 Zivkovic, Z. Improved adaptive Gaussian mixture model for background subtraction. In Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17 th International Conference on, vol. 2, pp. 28 -31. IEEE, 2004. 18 Open. CV. http: //opencv. org/. November 2013. Copyright © 2014 Alex Edgcomb, UC Riverside. 10 of 37

Synchronous state machines: Good fit for fall detection Normal behavior Rapid descent Descent velocity Normal behavior Sitting, standing, or laying had. Fall = 0 Not laying Extended lay Fall suspected Fall detected had. Fall = 0 had. Fall = 1 Copyright © 2014 Alex Edgcomb, UC Riverside. had. Fall 11 of 37

Synchronous state machine (Synch. SM) fall detection (1 of 3) Synch. SMs promote capturing

Synch. SM fall detection (2 of 3) Copyright © 2014 Alex Edgcomb, UC Riverside.

Synch. SM fall detection (3 of 3) * Camera may only contribute single-camera fall

Synch. SM fall detection vs. state-of-the-art MBR 3 D proj. Head 3 D proj. MBR • 22 recordings # of cameras Synch. SM Hung Auvinet Rougier Anderson Miaou Thome from University of 1 0. 960 0. 955 0. 900 0. 820 Montreal data 2 0. 990 0. 958 1. 000 0. 980 set; each 3 0. 998 0. 806 recording has 4 1. 000 0. 997 multiple labels 5 1. 000 0. 999 • Trained on 1 6 1. 000 recording by 7 1. 000 selecting smallest 8 1. 000 threshold for sit# of cameras Synch. SM Hung Auvinet Rougier Anderson Miaou Thome lay that got 1 0. 995 0. 964 0. 860 0. 980 perfect accuracy 2 1. 000 0. 938 1. 000 • Tested with all 3 1. 000 combinations of 4 0. 995 0. 998 remaining 21 5 0. 993 1. 000 videos 6 1. 000 • Did not use OK-to 7 1. 000 -lay SMs. 8 - - 1. 000 - Copyright © 2014 Alex Edgcomb, UC Riverside. - - - Sensitivity A dash (-) means unreported or not applicable, such as Hung’s algorithm that uses exactly two cameras. Specificity 15 of 37

Fall behavior coverage MBR 3 D proj. Head Hung Auvinet Rougier Fall behavior Synch. SM Suspected fall event Y Person orientation Y Y Y Fall sense Y Y Y 3 D proj. Anderson Y Y Did not consider sudden downward movement Copyright © 2014 Alex Edgcomb, UC Riverside. MBR Miaou Thome Y Y Y Did not give time for person to get up 16 of 37

Trade-off: Accuracy and efficiency (1 of 2) Higher is better. Closest to top-right is best. 2 cameras 1 camera 35 30 25 20 Our method 15 Rougier Miaou 10 Thome 5 0 Efficiency (FPS per camera) 35 30 25 20 Our method 15 Hung 0. 2 0. 4 0. 6 Combined accuracy score 0. 8 1 Thome 5 0 0 Anderson 10 0 0. 2 0. 4 0. 6 Combined accuracy score 0. 8 1 Combined accuracy score = sensitivity * specificity Copyright © 2014 Alex Edgcomb, UC Riverside. 17 of 37

Trade-off: Accuracy and efficiency (2 of 2) Higher is better. Closest to top-right is best. 4+ cameras 35 30 25 20 Our method 15 Auvinet 10 5 0 Efficiency (FPS per camera) 35 30 25 20 0. 2 0. 4 0. 6 Combined accuracy score 0. 8 1 Auvinet 10 5 0 0 Our method 15 0 0. 2 0. 4 0. 6 Combined accuracy score 0. 8 1 Combined accuracy score = sensitivity * specificity Copyright © 2014 Alex Edgcomb, UC Riverside. 18 of 37

Synch. SM fall detection on other data sets • Ran synch. SM fall detection on 55 of my own recordings (26 falls, 29 non-falls) using 1 and 2 cameras. • Perfect accuracy • Ran synch. SM fall detection on 22. 5 hours of normal activity videos using 1 and 2 cameras. • Perfect accuracy Copyright © 2014 Alex Edgcomb, UC Riverside. 19 of 37

With more resources, can synch. SMs do better? Can head tracking improve fall detection?

Fall detection accuracy: 2 D/3 D head tracking vs MBR tracking • 87 video recordings, 1 min each 19 • 69 non-confounding recordings (35 fall, 34 non-fall) • 18 confounding recordings (5 fall, 13 non-fall) • Automated MBR: height, width, and top Head vertical position: 342 pixels • Manual 2 D head tracking by clicking on head • Manual 3 D head tracking by estimating head height from ground • Same synch. SMs. Suspected fall used feature. • Perfect accuracy with non-confounding scenarios 19 Edgcomb, A. and F. Vahid. Video-based fall detection dataset with 2 D and 3 D head tracking, and moving-region tracking. http: //www. cs. ucr. edu/~aedgcomb/3 D_2 D_head_an_MBR_videos. html, June 2014. Copyright © 2014 Alex Edgcomb, UC Riverside. Head height: 5. 5 feet 21 of 37

Non-falls Confounding recordings: Head tracking vs MBR tracking (1 of 2) Confounding recordings 3 D head 2 D head Crouch with box Y Kneel and move chair Y Sit quickly Y Sit then toss up item Y Sit then hands to side Y Hands up, down, then lay Y Y Hands up, down, then sit 1 Y Y Hands up, down, then sit 2 Y Sit then hands up/down MBR top MBR height MBR width Confused person orientation synch. SM Y Head tracking could tell that Y head not near ground Y Y Y Y Lay then toss up item Y Y Y Hands to side then sit Y Y Y Stand then toss up item Y Y Y Set cushion on couch Y Y Y Copyright © 2014 Alex Edgcomb, UC Riverside. 22 of 37

Confounding recordings: Head tracking vs MBR tracking (2 of 2) Falls Confounding 3 D recordings head Fall w/ vacuum 1 Fall w/ vacuum 2 Y Summary 2 D head MBR top MBR height MBR width Y Y Confused person orientation synch. SM Put book in shelf Y Y Y Look under couch Y Y Y Take picture off wall Y Y Y 3 D head 2 D head MBR top MBR height MBR width Sensitivity 0. 80 Specificity 1. 00 0. 85 0. 54 Head tracking allowed rule that head had to be near the ground. MBR suitable for a variety of scenarios. If confounding scenarios likely, then head tracking may be justified. Copyright © 2014 Alex Edgcomb, UC Riverside. 23 of 37

Assistive monitoring for the elderly Assistive monitoring analyzes data from cameras and sensors for

Commercial in-home assistive monitoring http: //www. grandcare. com/sensors/ http: //www. careinnovations. com/ Quiet. Care (Intel and GE) Motion sensor-based anomaly detection http: //beclose. com/ Be. Close Many sensor-based anomaly detection Copyright © 2014 Alex Edgcomb, UC Riverside. Grand. Care Many sensor-based anomaly detection and if-then user programmability 26 of 37

Master’s work – Monitoring and Notification Flow Language (MNFL)20, 21 Easy. Notify example • Spatial programming more intuitive than temporal. Under 8 mins to solve goal. • No compilation. Blocks are always executing, so users get instant feedback. 20 Edgcomb, A. , and F. Vahid. Feature extractors for integration of cameras and sensors during end-user programming of assistive monitoring systems. In Proceedings of the 2 nd Conference on Wireless Health, p. 13. ACM, 2011. 21 Edgcomb, A. , and F. Vahid. MNFL: the monitoring and notification flow language for assistive monitoring. In Proceedings of the 2 nd ACM SIGHIT International Health Informatics Symposium, pp. 191 -200. ACM, 2012. 27 of 37 Copyright © 2014 Alex Edgcomb, UC Riverside.

Estimating daily energy expenditure from video for assistive monitoring 22 Energy expenditure levels on Monday Feature F • We considered 12 features • Motion in video not correlated with energy expenditure (r = -0. 01, p = 0. 53) • Horizontal acceleration had highest correlation (r = 0. 80, p < 0. 01) • Power regression had best fit (R 2 -value = 0. 76) compared to linear, logarithmic, and exponential regressions. 22 Edgcomb, A. , and F. Vahid. Estimating Daily Energy Expenditure from Video for Assistive Monitoring, IEEE International Conference on Healthcare Copyright © 2014 Alex Edgcomb, UC Riverside. Informatics (ICHI), 2013. (to appear) Fidelity = correlation(video-based Calories, Body. Bugg) Actor 1 2 3 4 Combined Fidelity r = 0. 996 r = 1. 000 r = 0. 983 r = 0. 997 (Ideal is 1. 0) Accuracy = 1 - (|expected - observed|/expected) Average accuracy = 90. 9% (about same as body-worn device) Data set available: http: //www. cs. ucr. edu/~aedgcomb/28 of 37 video. Based. Energy. Estimate. html

Privacy perception and fall detection accuracy with privacy-enhanced video 23 Privacy critical for adoption but makes events harder to detect Does this style provide sufficient privacy for grandpa? Yes / No. Did a fall occur? If so, at about what second in the video? 376 participants Common privacy enhancements not providing sufficient privacy 23 Edgcomb, A. , and F. Vahid. Privacy Perception and Fall Detection Accuracy for In-Home Video Assistive Monitoring with Privacy Enhancements, ACM SIGHIT (Special Interest Group on Health Informatics) Record, 2012. Copyright © 2014 Alex Edgcomb, UC Riverside. 29 of 37

Falls have a characteristic shape that is nearly identical for raw and privacy-enhanced video

Privacy-enhanced fall detection 24 (1 of 2) Dynamic time warping Observed shape Similarity 0. 84 Binary tree classification 25 Observed shape Script to produce this image provided by Professor Keogh, 2012. 0. 46 Characteristic fall shape Non-fall DTW established time series technique 0. 88 24 Edgcomb, A. and F. Vahid. Automated Fall Detection on Privacy-Enhanced Non-fall Video. IEEE Engineering in Medicine and Biology Society, 2012. 25 Mueen, A. , E. Keogh and N. Young. Logical-shapelets: An Expressive Primitive for Time Series Classification. Proceedings of the 17 th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2011. Copyright © 2014 Alex Edgcomb, UC Riverside. Fall 31 of 37

Privacy-enhanced fall detection (2 of 2) • Binary tree classifier trained on raw video only • Evaluated using leave-oneout method • 23 videos, 1 min each • Each video labeled fall or not-fall Privacy enhancement Average sensitivity Average specificity Raw 0. 91 0. 92 Blur 1. 00 0. 67 Silhouette 0. 91 0. 75 Oval 0. 91 0. 92 Box 0. 82 0. 92 +More accurate fall detection than human observers. -This method does not consider the time a person spends on the ground post-fall. Copyright © 2014 Alex Edgcomb, UC Riverside. 32 of 37

Automated in-home assistive monitoring with 26 privacy-enhanced video Already discussed In room too long Enter to left Exit from left In region too long Fall detection Arisen in morning Person enters main living area Abnormally inactive Person home but inactive for extended period 26 Edgcomb, A. and F. Vahid. Automated In-Home Assistive Monitoring Energy trends Copyright © 2014 Alex Edgcomb, UC Riverside. with Privacy-Enhanced Video, IEEE International Conference on 33 of 37 Healthcare Informatics (ICHI), 2013. (to appear)

Most goals were achieved equally well even with privacy enhancements MNFL goals • Trained on raw video only • MNFL goals trained on different person than tested Raw Blur Silhouette Oval Box Data set available: http: //www. cs. ucr. edu/~aedgcomb/MNFLevents. html Energy estimation fid. /acc. Fall detection sens. /spec. 0. 997 / 90. 9% 0. 91 / 0. 92 0. 994 / 1. 00 / 0. 998 / 0. 91 / 0. 997 / 85. 6% 0. 91 / 0. 92 1. 000 / 0. 82 / 80. 5% 85. 0% 84. 3% 0. 67 0. 75 0. 92 In room too long sens. / spec. Arisen in morning sens. / spec. In region too long sens. / spec. Abnormally inactive during day sens. /spec. 1. 0 / 1. 0 0. 5 / 1. 0 1. 0 / 1. 0 Copyright © 2014 Alex Edgcomb, UC Riverside. 34 of 37

The background model tended to learn the blue much more than the actor Current

Accurate and efficient algorithms that adapt to privacy-enhanced video 27 Fall detection Specific-color hunter Energy estimation Edge-void filler 27 Edgcomb, A. and F. Vahid. Accurate and Efficient Algorithms that Adapt to Privacy-Enhanced Video for Improved Assistive Monitoring, ACM Transactions on Management Information Systems (TMIS): Special Issue on Informatics for Smart Health and Wellbeing, 2013. Copyright © 2014 Alex Edgcomb, UC Riverside. 36 of 37

Contributions • MBR and synch. SM-based fall detection is more accurate and efficient than all previous work. • MBR and head tracking are equally accurate, except in very specific cases. • Although monitoring goal accuracy degrades with privacy-enhanced video, adaptive algorithms can compensate without loosing computational efficiency. • The common privacy enhancements of silhouette and blur provide insufficient privacy, whereas a bounding oval or box were sufficient. Copyright © 2014 Alex Edgcomb, UC Riverside. 37 of 37

Summary of graduate research • Monitoring and notification flow language for assistive monitoring • Edgcomb, A. , and F. Vahid. Feature extractors for integration of cameras and sensors during end-user programming of assistive monitoring systems. In Proceedings of the 2 nd Conference on Wireless Health, p. 13. ACM, 2011. (2 pages) • Edgcomb, A. , and F. Vahid. MNFL: the monitoring and notification flow language for assistive monitoring. In Proceedings of the 2 nd ACM SIGHIT International Health Informatics Symposium, pp. 191 -200. ACM, 2012. • Estimating daily energy expenditure from video for assistive monitoring • Edgcomb, A. , and F. Vahid. Estimating Daily Energy Expenditure from Video for Assistive Monitoring, IEEE International Conference on Healthcare Informatics (ICHI), 2013. (to appear) • Participant privacy perceptions and fall detection accuracy with privacy enhancements • Edgcomb, A. , and F. Vahid. Privacy Perception and Fall Detection Accuracy for In-Home Video Assistive Monitoring with Privacy Enhancements, ACM SIGHIT (Special Interest Group on Health Informatics ) Record, 2012. • Automated fall detection on video • Edgcomb, A. and F. Vahid. Automated Fall Detection on Privacy-Enhanced Video. IEEE Engineering in Medicine and Biology Society, 2012. (4 pages) • Edgcomb, A. and F. Vahid. Accurate and Efficient Video-based Fall Detection using Moving-Region and State Machines. (To be submitted) • Automated in-home assistive monitoring with privacy-enhanced video • Edgcomb, A. and F. Vahid. Automated In-Home Assistive Monitoring with Privacy-Enhanced Video, IEEE International Conference on Healthcare Informatics (ICHI), 2013. (to appear) • Edgcomb, A. and F. Vahid. Accurate and Efficient Algorithms that Adapt to Privacy-Enhanced Video for Improved Assistive Monitoring, ACM Transactions on Management Information Systems (TMIS): Special Issue on Informatics for Smart Health and Wellbeing, 2013. • Efficacy of digitally-enhanced education • Edgcomb, A. and F. Vahid. Interactive Web Activities for Online STEM Learning Materials, American Society for Engineering Education Pacific Southwest Section Conference, 2013. • Edgcomb, A. and F. Vahid. Effectiveness of Online Textbooks vs. Interactive Web-Native Content, Proceedings of ASEE Annual Conference, 2014. (to appear) Copyright © 2014 Alex Edgcomb, UC Riverside. 38