Online Bayesian Models for Personal Analytics in Social

Online Bayesian Models for Personal Analytics in Social Media Svitlana Volkova and Benjamin Van Durme [email protected] edu http: //www. cs. jhu. edu/~svitlana/ Center for Language and Speech Processing, Johns Hopkins University, Human Language Technology Center of Excellence

Social Media Predictive Analytics • Personalized, diverse and timely data • Can reveal user interests, preferences and opinions Social Network Prediction App - https: //apps. facebook. com/snpredictionapp/ Demographics. Pro – http: //www. demographicspro. com/ Wolphral. Alpha Analytics – http: //www. wolframalpha. com/facebook/

User Attribute Prediction Task Communications Political Preference … … …. . . Rao et al. , 2010; Conover et al. , 2011, Pennacchiotti and Popescu, 2011; Zamal et al. , 2012; Cohen and Ruths, 2013; Volkova et. al, 2014 Age Rao et al. , 2010; Zamal et al. , 2012; Cohen and Ruth, 2013; Nguyen et al. , 2011, 2013; Sap et al. , 2014 Gender … Garera and Yarowsky, 2009; Rao et al. , 2010; Burger et al. , 2011; Van Durme, 2012; Zamal et al. , 2012; Bergsma and Van Durme, 2013 AAAI 2015 Demo (joint work with Microsoft Research) Income, Education Level, Ethnicity, Life Satisfaction, Optimism, Personality, Showing Off, Self-Promoting

Outline I. Our Approach II. Dynamic (Streaming) Models III. Experimental Results IV. Practical Recommendations

Existing Approaches ~1 K Tweets* …. … Tweets as…. a … document …. … How long does it take for an average Twitter user …. to produce thousands of tweets? … …. What if we want to make reliable predictions … …. immediately after 10 tweets? … …. … *Rao et al. , 2010; Conover et al. , 2011; Pennacchiotti and Popescu, 2011 a; Burger et al. , 2011; Zamal et al. , 2012; Nguyen et al. , 2013

Attributed Social Networks *Conover et al. , 2011; Pennacchiotti and Popescu, 2011 a; Zamal et al. , 2012; Volkova et al. , 2014.

Our Approach Static (Batch) Predictions Streaming (Online) Inference Dynamic (Iterative) Learning and Prediction • Offline training • Offline predictions • No or limited network information • Offline training • Online predictions in time (ACL’ 14) • Exploring 6 types of neighborhoods • • + + + Online predictions Relying on neighbors Iterative re-training Active learning Interactive rationale annotation ① Streaming nature of SM: dynamic training and prediction ② Network structure: joint user-neighbour streams ③ Trade-off between prediction time vs. model quality

Online Predictions: Iterative Bayesian Updates ? ? Time …

Iterative Batch Learning ? t 2 t 1 … … R tm D ? t 1 t 2 … … tm Unlabeled ? tm Labeled t 1 Time § Iterative Batch Retraining (IB) § Iterative Batch with Rationale Filtering (IBR)

Rationales are explicitly highlighted ngrams in tweets that best justified why the annotators made their labeling decisions

Labeled Active Learning § Active Without Oracle (AWOO) … Unlabeled § Active With Rationale Filtering (AWR) … § Active With Oracle (AWO) 1 -Jan-2011 1 -Feb-2011 1 -Nov-2011 Time 1 -Dec-2011

Performance Metrics • Accuracy over time: • Find optimal models: – Data steam type (user, friend, user + friend) – Time (more correctly classified users faster) – Prediction quality (better accuracy over time)

Results: Iterative Batch Learning user + friend 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Correctly classified user Accuracy Correctly classified user IBR: higher precision user + friend 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Time: # correctly classified users increases over time IB faster, IBR slower Data stream selection: User + friend stream > user stream Accuracy IB: higher recall

Results: Active Learning user 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Correctly classified user + friend Accuracy Correctly classified user AWR: higher precision user + friend 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Time: Unlike IB/IBR models, AWOO/AWR models classify more users correctly faster (in Mar) but then plateaus Accuracy AWOO: higher recall

Results: Model Quality AWOO: user AWR: user 1. 0 0. 9 Accuracy 0. 5 Mar Jun Sep . . AWOO: user + friend AWR: user + friend _. 00 03 _x 00 03 _. . . _. 00 03 _J. . . 00 3 x 0 1. 0 0. 9 0. 8 0. 7 0. 6 0. 5. . Accuracy IB: user + friend IBR: user + friend _. 00 03 0. 5 Sep _x 1. 0 0. 9 0. 8 0. 7 0. 6 0. 5 Jun . . Accuracy Mar 0. 6 _x 0. 6 0. 7 J. . . 0. 7 0. 8 3_ 0. 8 x 0 00 Accuracy IB: user IBR: user _x user + friend > user batch < active

Summary • Active learning > iterative batch • N, UN > U: “neighbors give you away” • Higher confidence => higher precision, lower confidence => higher recall (as expected) • Rationales significantly improve results

Practical Recommendations • If you want to deliver ads fast but to be less confident in user attribute predictions: – use models with higher recall (AWOO, IB) – apply lower decision threshold e. g. , 0. 55 • If you want to deliver ads to a true target crowd but latter in time: – use models with higher precision (AWR, IBR) – apply higher decision threshold e. g. , 0. 95 – models with rational filtering (IBR, AWR) require less computation (lower-dimensional feature vectors), are more accurate but annotations cost money (Mechanical Turk) • For highly assortative attributes e. g. , political preference use a joint user-neighbor stream

I am on a job market. Hire me! Email: [email protected] edu Thank you! Labeled Twitter network data for gender, age, political preference prediction: http: //www. cs. jhu. edu/~svitlana/ Interested in using our models for your research or collaboration: code and pre-trained models for inferring demographic attributes, personality and 6 Ekman’s emotions available on request: [email protected] edu AAAI Technical Demo Inferring Latent User Properties from Texts Published in Social Media Wednesday, January 28 6: 30 – 8: 00 Zilker Ballroom