Where you Instagram Associating Your Instagram Photos with

Where you Instagram? Associating Your Instagram Photos with Points of Interest Xutao Li 1, Tuan-Anh Nguyen Pham 1, Gao Cong 1, Quan Yuan 1 Xiao-Li Li 2, Shonali Krishnaswamy 2 ¹Nanyang Technological University, Singapore ²Institute for Infocomm Research (I 2 R), A* STAR, Singapore

Outline • • • 1. Introduction 2. Related Work 3. Instagram Exploration 4. Proposed Method 5. Experimental Results 6. Conclusions 2

Motivation (1) • Instagram is mobile based photo-sharing system, which allows users to share their photos with friends. • Lunched in 2010, it has gained popularity rapidly over past several years. As of December 2014, it had over 300 million active users with more than 20 billion photos. 3

Motivation (2) • Instagram enables users to associate photos with Points of Interest (POI), which offers an exciting opportunity to study the association between photos and POIs. In Instagram, only less than 30% pictures are annotated to POIs by users. Knowing the association has important applications: (1) it provides both visual and textual annotations about POIs for users to explore; (2) it may be prerequisite for other mining tasks; (3) it will help to understand users’ preferences over POIs. 4

Challenges • Data scarcity problem: a great number of POIs have very few photos (power law distribution). We will utilize the user visiting preference as complementary information to alleviate the problem. • Heterogeneous information to utilize: we have not only visual and textual information, but also user visiting information to explore. (Users incline to take photos at new POIs, which makes the problem difficult). We will design a ranking objective function to effectively incorporate the heterogeneous information. 5

Outline • • • 1. Introduction 2. Related Work 3. Instagram Exploration 4. Proposed Method 5. Experimental Results 6. Conclusions 6

Geolocating Images • Grid Prediction: placing Flickr photos onto earth grids Authors Methods Publication Name Hays and Efros KNN (visual features) CVPR, 2008 Serdyukov et al. Language model(textual features) SIGIR, 2009 Liu et al. Language model(textual features) JASIST, 2014 Differences: (1) we aim to associate Instagram photos with POIs, instead of grids (2) these works do not consider the visiting preferences of users • Landmark Identification: identifying from a set of candidates the landmark that an image refers to Authors Methods Publication Name Crandall et al. Rank. SVM +Naïve Bayes (visual and textual features) WWW, 2009 Li et al. Multiclass SVM ICCV, 2009 Differences: (1) landmarks often refer to popular places and thus there exit sufficient photos (2) the number of landmark candidates considered in previous works is small 7

Geolocating Tweets or Users Authors Methods Publication Name Backstrom et al. Using social and spatial proximity WWW 2010 Cheng et al. A lattice-based neighborhood smoothing model CIKM 2010 Li et al. A unified discriminative influence model SIGKDD 2012 Differences: (1) these approaches focus on a city-level granularity NOT POI level (2) users’ visiting preferences over POIs and visual features of photos are not explored in the approaches 8

Outline • • • 1. Introduction 2. Related Work 3. Instagram Exploration 4. Proposed Method 5. Experimental Results 6. Conclusions 9

Datasets We crawled two sets of Instagram data from 13 Nov 2013 to 13 Nov 2014, for users of NYC and Singapore, respectively. Statistics NYC SG # of photos 21, 910, 375 13, 168, 666 # of POIs 602, 604 372, 104 # of users 126, 543 87, 281 % of photos annotated to POIs 29 26 % of photos with textual info. 92 94 10

Observation 1: Number of photos per POI • We plot the empirical Complementary Cumulative Distribution Function (CCDF) of number of photos per POI. We observe that both CCDFs follow the power law distribution. We find only 10% of POIs have more than 11 photos, and 90% of POIs have less than 11 photos. This is a challenge for us as a majority of POIs lack training samples. 11

Observation 2: Behaviors of users • We depict a histogram with the probabilities of all the users to take a photo at a new POI. We observe that a greater portion of users have high probability (say 0. 8 to 1. 0) to take photos at new POIs. This is a challenge for us to exploit users visiting preferences. 12

Outline • • • 1. Introduction 2. Related Work 3. Instagram Exploration 4. Proposed Method 5. Experimental Results 6. Conclusions 13

Approach Overview • Association problem as a ranking problem #shopping #discount #DFS Textual features Prediction scores of POIs 0. 9 0. 7 Visual features A learned ranking model 0. 6 0. 2 0. 1 User visiting profiles Next we will present how to learn the ranking model based on training data. 14

Combining Textual Context and Visual Content • A scoring function for POI l with textual and visual features xp, yp : textual interaction scores visual interaction scores • Following the idea of Rank. SVM, we have the following ranking objective: Photo p We expect the ground-truth POI is ranked than the other POIs ; otherwise, some loss is incurred. • To more effectively utilize the textual and visual features, we modify the objective function by incorporating two sub-objectives: textual sub-objective visual sub-objectives 15

Incorporating User Preference • The new scoring function to incorporate user information: User visiting preference scores • The objective function is thus modified as: User preference sub-objective: a weighted matrix factorization model 16

Learning Our Model • We optimize the objective function with Stochastic Gradient Descent (SGD) method Specifically, the gradients are calculated as follows: 17

Outline • • • 1. Introduction 2. Related Work 3. Instagram Exploration 4. Proposed Method 5. Experimental Results 6. Conclusions 18

Experimental Setup • Datasets (NYC and SG) v For each user, 70% photos are considered as training data, 10% as validation data and 20% as test data • Evaluation Metrics v Mean Reciprocal Rank v Accuracy@N • Baseline Algorithms Naive Bayes v Language Model v Rank. SVM We note that all the baseline methods can only utilize textual and visual features. v 19

Experimental Results (1) • Performance Comparison 1. The proposed Rank. TVU_P 2 P outperforms baselines methods substantially, thanks to the incorporation of user information. 2. When only considering textual and visual features, Rank. TVU_P 2 P still performs better than baseline methods. 20

Experimental Results (2) • Impact of different types of information 1. When incorporating the textual, visual and user information one by one, we find the performance gradually increases, indicating the usefulness of the three types of information. 2. Textual context and user information contribute more to the performance, compared to visual content. 21

Experimental Results (3) • Impact of sub-objective functions When removing sub-objective O 3, O 2 and O 1 one by one, the performance decreases gradually. This suggests the three sub-objective function are important for Rank. TVP_P 2 P to obtain good performance. 22

Conclusions • We investigate the Instagram data and make some interesting observations. • We define a new problem to association Instagram photos with POIs. A ranking model Rank. TVU_P 2 P is proposed to address the problem by leveraging textual, visual and user information. • Limitations of our work: v v We focus on how to associate photos with POIs. It is also important to determine whether a photo is really related to POIs. There are other types of information that could be investigated in Instagram, e. g. , time stamps of photos, category information of POIs, etc. 23

Thanks Q&A 24