Mood Scope Sensing mood from smartphone usage patterns

Mood. Scope: Sensing mood from smartphone usage patterns Robert Likamwa Lin Zhong Asia Yunxin Liu Nicholas D. Lane

Earlier today… 2

Some time in the future… Mood-Enhanced Apps Personal analytics Social ecosystems Media recommendation 3 of 30

Affective Computing (Mood and Emotion) Biometric-based (Skin conductivity, Temperature, Pulse rate) Highly temporal High cost of deployment Hassle Audio/Video-based (Affect. Aura, Emotion. Sense) Captures expressions Power hungry Slightly invasive 4 of 30

Can your mobile phone infer your mood?

Can your mobile phone infer your mood? From already-available, low-power information? * * No audio/video sensing, no body-instrumentation

Mood. Scope ∈ Affective Computing Usage Trace-based (Mood. Scope) Passive, Continuous How to model mood? Audio/Video-based Biometric-based Very direct, Fine-grained High cost of deployment Captures expressions Power hungry Slightly invasive 7 of 30

Mood is… • … a persistent long-lasting state o Lasts hours or days o Emotion lasts seconds or minutes • … a strong social signal o Drives communications o Drives interactions o Drives activity patterns 8 of 30

sad depressed stressed nervous bored calm attentive excited relaxed happy Circumplex model (Russell 1980) 9 of 30

Mood is… • … a persistent long-lasting state o Lasts hours or days o Emotion lasts seconds or minutes • … a strong social signal o Drives communications o Drives interactions o Drives activity patterns 10 of 32

How is the user communicating?

What apps is the user using?

f( usage ) = mood 13 of 30

i. Phone Livelab Logger • • • Web history Phone call history Sms history Email history Location history App usage Adapted From C. Shepard, A. Rahmati, C. Tossel, L. Zhong, And P. Kortum, "Livelab: Measuring Wireless Networks And Smartphone Users In The Field, " In Hotmetrics, 2010. 14 of 30

i. Phone Livelab Logger • • • Web history Phone call history Sms history Email history Location history App usage Runs as shell Hash private data Uploads logs to our server nightly How can we generate mood labels? 15 of 30

Mood Journaling App User-base 32 users aged between 18 and 29 11 females 16 of 30

• Detect a mood pattern Inference • Validate with only 60 days of data • Wide range of candidate usage data • Low computational resources 17 of 30

Daily Mood Averages • Separate pleasure, activeness dimension • Take the average over a day Σ( ________ 4 ) 18 of 30

Exploring Features • Communication o SMS o Email o Phone Calls • To whom? o # messages o Length/Duration Consider “Top 10” Histograms How many phone calls were ? made to #1? #2? … #10? ? How much time was spent on calls to #1? #2? … #10? 19 of 30

Exploring Features • Communication o SMS o Email o Phone Calls • Usage Activity o Applications o Websites visited o Location History • To whom? o # messages o Length/Duration • Which (app/site/location)? o # instances 20 of 30

Previous Mood • Use previous 2 pairs of mood labels 21 of 30

Data Type Email contacts SMS contacts Phone call contacts Website domains Location Clusters Apps Categories of Apps Histogram by: Dimensions # Messages # Characters # Calls Call Duration # Visits # App launches App Duration 10 10 10 # App launches 12 App Duration 12 Previous Pleasure and N/A Activeness Averages 4 22 of 30

? ? Model Design • Multi-Linear Regression o Minimize Mean Squared Error • Leave-One-Out Cross-Validation • Sequential Forward Feature Selection during training 23 of 30

Sequential Feature Selection Improvement of model as SFS adds more features 0. 8 Mean Squared Error 0. 7 (Each line is a different user) 0. 6 0. 5 0. 4 0. 3 0. 2 0. 1 0 1 6 11 16 21 Number of Features Used 26 31 24 of 30

Daily Mood Average Sample Prediction 4 3 Mood (Pleasure) Estimated Mood 2 0 10 20 30 Days 40 50 60 25 of 30

Error distributions • Error 2 of > 0. 25 will misclassify a mood label 93% < 0. 25 error 2 1 Squared Error 0. 1 0. 01 90%ile 75%ile 0. 001 0. 0001 Users 26 of 30

vs. Strawman Models using full-knowledge of a user’s data with LOOCV Model A: Assume User’s Average Mood 73% Accuracy Model B: Assume User’s Previous Mood 61% Accuracy Mood. Scope Training: 93% Accuracy. 27 of 30

Personalized Training 100% Model Accuracy 80% All-user model accuracy 60% 40% Incremental personalized model 20% 0% 10 20 30 40 Training Days 50 59 28 of 30

Personalized/All-user Hybrid Training 100% Model Accuracy 80% 60% 40% Incremental personalized model 20% 0% Hybrid mood model 10 20 30 40 Training Days 50 59 29 of 30

Resource-friendly Implementation Phone Cloud Inferred Mood Model Current Usage Mood Inputs/ Usage Logs Mood Model Training Mood and Usage History 30 of 30

Inferred Mood API

RICE UNIVERSITY TEXAS MEDICAL CENTER

Inferred Mood API

Mood. Scope: Sensing mood from smartphone usage patterns • Robustly (93%) detect each dimension of daily mood o On personalized models o Starts out with 66% on generalized models • Validate with 32 users x 2 months worth of data • Simple resource-friendly implementation

Discriminative Features Improvement of model as SFS adds more features 0. 8 Mean Squared Error 0. 7 0. 6 0. 5 Relevant features 0. 4 0. 3 0. 2 0. 1 0 1 6 11 16 21 Number of Features Used 26 31 35

Discriminative Features 120 Pleasure Activeness Number of Features 100 80 60 40 20 0 Calls Email SMS Web Apps Location Prev. Mood 36

TODO • Wider, longer-term evaluation o How does the model change over time? • In-use accuracy metrics o Not cross-validation • Social Factors/Impact o Study Mood-sharing o Provide assistance to psychologists 37