Online Bayesian Models for Personal Analytics in Social

  • Slides: 18
Download presentation
Online Bayesian Models for Personal Analytics in Social Media Svitlana Volkova and Benjamin Van

Online Bayesian Models for Personal Analytics in Social Media Svitlana Volkova and Benjamin Van Durme [email protected] edu http: //www. cs. jhu. edu/~svitlana/ Center for Language and Speech Processing, Johns Hopkins University, Human Language Technology Center of Excellence

Social Media Predictive Analytics • Personalized, diverse and timely data • Can reveal user

Social Media Predictive Analytics • Personalized, diverse and timely data • Can reveal user interests, preferences and opinions Social Network Prediction App - https: //apps. facebook. com/snpredictionapp/ Demographics. Pro – http: //www. demographicspro. com/ Wolphral. Alpha Analytics – http: //www. wolframalpha. com/facebook/

User Attribute Prediction Task Communications Political Preference … … …. . . Rao et

User Attribute Prediction Task Communications Political Preference … … …. . . Rao et al. , 2010; Conover et al. , 2011, Pennacchiotti and Popescu, 2011; Zamal et al. , 2012; Cohen and Ruths, 2013; Volkova et. al, 2014 Age Rao et al. , 2010; Zamal et al. , 2012; Cohen and Ruth, 2013; Nguyen et al. , 2011, 2013; Sap et al. , 2014 Gender … Garera and Yarowsky, 2009; Rao et al. , 2010; Burger et al. , 2011; Van Durme, 2012; Zamal et al. , 2012; Bergsma and Van Durme, 2013 AAAI 2015 Demo (joint work with Microsoft Research) Income, Education Level, Ethnicity, Life Satisfaction, Optimism, Personality, Showing Off, Self-Promoting

Outline I. Our Approach II. Dynamic (Streaming) Models III. Experimental Results IV. Practical Recommendations

Outline I. Our Approach II. Dynamic (Streaming) Models III. Experimental Results IV. Practical Recommendations

Existing Approaches ~1 K Tweets* …. … Tweets as…. a … document …. …

Existing Approaches ~1 K Tweets* …. … Tweets as…. a … document …. … How long does it take for an average Twitter user …. to produce thousands of tweets? … …. What if we want to make reliable predictions … …. immediately after 10 tweets? … …. … *Rao et al. , 2010; Conover et al. , 2011; Pennacchiotti and Popescu, 2011 a; Burger et al. , 2011; Zamal et al. , 2012; Nguyen et al. , 2013

Attributed Social Networks *Conover et al. , 2011; Pennacchiotti and Popescu, 2011 a; Zamal

Attributed Social Networks *Conover et al. , 2011; Pennacchiotti and Popescu, 2011 a; Zamal et al. , 2012; Volkova et al. , 2014.

Our Approach Static (Batch) Predictions Streaming (Online) Inference Dynamic (Iterative) Learning and Prediction •

Our Approach Static (Batch) Predictions Streaming (Online) Inference Dynamic (Iterative) Learning and Prediction • Offline training • Offline predictions • No or limited network information • Offline training • Online predictions in time (ACL’ 14) • Exploring 6 types of neighborhoods • • + + + Online predictions Relying on neighbors Iterative re-training Active learning Interactive rationale annotation ① Streaming nature of SM: dynamic training and prediction ② Network structure: joint user-neighbour streams ③ Trade-off between prediction time vs. model quality

Online Predictions: Iterative Bayesian Updates ? ? Time …

Online Predictions: Iterative Bayesian Updates ? ? Time …

Iterative Batch Learning ? t 2 t 1 … … R tm D ?

Iterative Batch Learning ? t 2 t 1 … … R tm D ? t 1 t 2 … … tm Unlabeled ? tm Labeled t 1 Time § Iterative Batch Retraining (IB) § Iterative Batch with Rationale Filtering (IBR)

Rationales are explicitly highlighted ngrams in tweets that best justified why the annotators made

Rationales are explicitly highlighted ngrams in tweets that best justified why the annotators made their labeling decisions

Labeled Active Learning § Active Without Oracle (AWOO) … Unlabeled § Active With Rationale

Labeled Active Learning § Active Without Oracle (AWOO) … Unlabeled § Active With Rationale Filtering (AWR) … § Active With Oracle (AWO) 1 -Jan-2011 1 -Feb-2011 1 -Nov-2011 Time 1 -Dec-2011

Performance Metrics • Accuracy over time: • Find optimal models: – Data steam type

Performance Metrics • Accuracy over time: • Find optimal models: – Data steam type (user, friend, user + friend) – Time (more correctly classified users faster) – Prediction quality (better accuracy over time)

Results: Iterative Batch Learning user + friend 300 1. 0 250 0. 8 200

Results: Iterative Batch Learning user + friend 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Correctly classified user Accuracy Correctly classified user IBR: higher precision user + friend 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Time: # correctly classified users increases over time IB faster, IBR slower Data stream selection: User + friend stream > user stream Accuracy IB: higher recall

Results: Active Learning user 300 1. 0 250 0. 8 200 0. 6 150

Results: Active Learning user 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Correctly classified user + friend Accuracy Correctly classified user AWR: higher precision user + friend 300 1. 0 250 0. 8 200 0. 6 150 0. 4 100 0. 2 50 0. 0 Mar Jun Sep Time: Unlike IB/IBR models, AWOO/AWR models classify more users correctly faster (in Mar) but then plateaus Accuracy AWOO: higher recall

Results: Model Quality AWOO: user AWR: user 1. 0 0. 9 Accuracy 0. 5

Results: Model Quality AWOO: user AWR: user 1. 0 0. 9 Accuracy 0. 5 Mar Jun Sep . . AWOO: user + friend AWR: user + friend _. 00 03 _x 00 03 _. . . _. 00 03 _J. . . 00 3 x 0 1. 0 0. 9 0. 8 0. 7 0. 6 0. 5. . Accuracy IB: user + friend IBR: user + friend _. 00 03 0. 5 Sep _x 1. 0 0. 9 0. 8 0. 7 0. 6 0. 5 Jun . . Accuracy Mar 0. 6 _x 0. 6 0. 7 J. . . 0. 7 0. 8 3_ 0. 8 x 0 00 Accuracy IB: user IBR: user _x user + friend > user batch < active

Summary • Active learning > iterative batch • N, UN > U: “neighbors give

Summary • Active learning > iterative batch • N, UN > U: “neighbors give you away” • Higher confidence => higher precision, lower confidence => higher recall (as expected) • Rationales significantly improve results

Practical Recommendations • If you want to deliver ads fast but to be less

Practical Recommendations • If you want to deliver ads fast but to be less confident in user attribute predictions: – use models with higher recall (AWOO, IB) – apply lower decision threshold e. g. , 0. 55 • If you want to deliver ads to a true target crowd but latter in time: – use models with higher precision (AWR, IBR) – apply higher decision threshold e. g. , 0. 95 – models with rational filtering (IBR, AWR) require less computation (lower-dimensional feature vectors), are more accurate but annotations cost money (Mechanical Turk) • For highly assortative attributes e. g. , political preference use a joint user-neighbor stream

I am on a job market. Hire me! Email: svitlana@jhu. edu Thank you! Labeled

I am on a job market. Hire me! Email: [email protected] edu Thank you! Labeled Twitter network data for gender, age, political preference prediction: http: //www. cs. jhu. edu/~svitlana/ Interested in using our models for your research or collaboration: code and pre-trained models for inferring demographic attributes, personality and 6 Ekman’s emotions available on request: [email protected] edu AAAI Technical Demo Inferring Latent User Properties from Texts Published in Social Media Wednesday, January 28 6: 30 – 8: 00 Zilker Ballroom