Web Science Technologies University of Koblenz Landau Germany

Web Science & Technologies University of Koblenz ▪ Landau, Germany From Web 2. 0 to Web 3. 0 using Data Mining Steffen Staab & Rabeeh Abbasi

Web 2. 0 Folksonomies facilitate sharing resources • Pictures hibiscus flower • Bookmarks flickr. com/photos/ digitalart/3543019869/ 1942 penny • Publications flickr. com/photos/ fapeg/474807645/ Users annotate resources with tags (keywords) Tags can be used for searching and browsing resources We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 2 of 58

Why do you want to have a Semantic Web at all? 4 billion photos! Many of them tagged We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 3 of 58

Why do you want to have a Semantic Web at all? Solid growth Source: http: //www. kullin. net/2009/10/flickr-reaches-4 -billion-photos. html We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 4 of 58

Why I want to have a Semantic Web! We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 5 of 58

Why I want to have a Semantic Web! § Intelligent queries Thesauri Many photos are unsearchable! Requirement: Know that “ 40 ies” = “forties” We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 6 of 58

Why I want to have a Semantic Web! Faceted Search & Browsing: Semaplorer (btc. uni-koblenz. de) Requirement: Know what is a city, country, landmark We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 7 of 58

Why I want to have a Semantic Web! Is there a quantifiable benefit in Ontology Learning? We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 8 of 58

Problems and Features of Folksonomies Add Tags. . . Search Resources Browse Resources. . . Geotags 51° 30'03"N 0° 7'24"W Folksonomy Low-Level Features

Discovering and Exploiting Semantics 3 1 Recommend Tags. . . Improve Search Semantic Browsing. . . Geotags 2 51° 30'03"N 0° 7'24"W Folksonomy Low-Level Features

1 Improving Search in Folksonomies (Can we improve search, particularly for sparse results? )

Search in Folksonomies Billions of resources in Folksonomies But still many photos are unsearchable! We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 12 of 58

Still many photos are unsearchable, because • they have been tagged with other (semantically related) tags!

Sparseness in Folksonomies Most of the pictures in Flickr have only a few tags • Leads into extremely sparse data! We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 14 of 58

Simple Vector Space Model of a Folksonomy Consider the tagging system as a Vector Space Model (VSM) Each resource is represented as a vector of tags flickr. com/photos/ethankillian/364295181/ flickr. com/photos/fapeg/474807645/ flickr. com/photos/digitalart/3543019869/ flickr. com/photos/sondyaustin/3576959698/ We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 15 of 58

Discovering Semantically Related Tags Enrich the existing Vector Space Model using semantically related tags • Dimensions of Semantically Related Tags – Context of tags (Resource / User) – Distribution of tags (Similar / Generalized) Resource Context & Generalized Tags Resource Context & Similar Tags User Context & Similar Tags We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 16 of 58

Resource vs. User Context Resource Context • Narrow context, based on the resources associated to a tag User Context • Broader context, based on the vocabulary of a users We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 17 of 58

Semantically Related Tags – Tag Distribution • Similar Tags (Symmetric Similarity) – Using Cosine Similarity: We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 18 of 58

Semantically Related Tags – Tag Distribution • Generalized Tags (Asymmetric Similarity) – Using Modified Overlap Coefficient We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 19 of 58

en. Riched Vector Space Models for Folksonomies Use semantically related tags to enrich the existing vector space model Semantically Related Tags Original Vector Space We. ST Steffen Staab staab@uni-koblenz. de Enriched Vector Space IRMLe. S 2010 20 of 58

Examples – Context and Distribution of Tags We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 21 of 58

Evaluation Dataset • Image uploaded to Flickr between 2004/05 • Tags used by at least 10 different users • Dataset statistics – Photos: ~27 Million – Tags: ~92 Thousand – Users: ~0. 3 Million – Tag Assignments: ~94 Million • Queries – Subset of AOL query log We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 22 of 58

Evaluation User based evaluation • done by 18 users 150 queries split into 3 sets of 50 queries each • having exact matches: – 1 to 10 (sparse results) – 11 to 50 (normal) – more than 50 (too many results) Search results were presented to users for different vector space models Calculated Precisions @ 5, 10, 15, 20 We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 23 of 58

Results – Sparse Results Significant Improvement for sparse results We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 24 of 58

Results We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 25 of 58

Results – Large Result Set We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 26 of 58

2 a Browsing (landmarks) in Folksonomies

One prominent photo returned by Flickr when searching for „Koblenz“ » restroom Is this what you expect when searching for a place? We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 28 of 58

Usage context: Faceted Browsing • Semaplorer (btc. uni-koblenz. de) – Handles billions of triples, – Lacks semantics for individual photos – Cannot identify landmark photos We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 29 of 58

Finding Landmark Photos Which photos are (non-)representative for a given place? We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 30 of 58

Building a Classifier For a classifier we need • Features of the photos – Tags • Positive and Negative training examples – Exploit Flickr groups We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 31 of 58

Positive Flickr Groups Positive Training Examples Classification Model Classifier Negative Training Examples E -V +V E Normalization SVM Negative Flickr Groups flickr. com/photos/swamibu/2223726960/, flickr. com/photos/gunner 66/2219882643/, flickr. com/photos/mromega/2346732045/, flickr. com/photos/me_haridas/399049455/, flickr. com/photos/caribb/84655830/, flickr. com/photos/conner 395/1018557535/, flickr. com/photos/66164549@N 00/2508246015/, flickr. com/photos/kupkup/1356487670/, flickr. com/photos/asam/432194779/, flickr. com/photos/michaelfoleyphotography/392504158/

User Evaluation – – We. ST Datasets: Training 430282 photos, Test Dataset 232265 photos Comparison with state of the art World. Explorer 20 Users evaluated both methods for 50 different cities Improvement of 12% in precision for our method Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 33 of 58

2 b Semantic Browsing in Folksonomies We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 34 of 58

Social Tagging Systems – Browsing? § I want to “browse” vehicle images!!! how can I do it? • can I do it using a Tag Cloud? § Perhaps I need to structure the tags and resources! how can I do it? • Put them into categories (like Vehicles, People, etc)! – Do it Manually or with Training? » Might not be possible on a large scale! – Automatically and without any training! » Using T-ORG! We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 35 of 58

T-ORG – Classification § § Organize resources by putting their tags into categories depending upon their context Users can browse categories to retrieve required resources President Gerald Ford Nixon Pardon User B Eiffel tower Big Eyeful Paris France Miniatures Person Location Vehicle Categories We. ST Steffen Staab staab@uni-koblenz. de Group 1 Group 2 User A IRMLe. S 2010 36 of 58 Singen Cars Motors Ford 1955

T-ORG § Tag Organization using T-ORG Select ontologies related to the categories (e. g. Vehicle, People, etc. ) We. ST Prune and refine these ontologies according to the desired categories (add missing concepts, filter existing concepts) Steffen Staab staab@uni-koblenz. de Apply the classification algorithm T -KNOW to classify the tags and resources IRMLe. S 2010 37 of 58 Browse the categories to explore the tags and resources

Classifying the tags using T-KNOW Use well-known linguistic patterns to generate queries Select the concept which has the highest similarity with the context of the tag We. ST Search these patterns on Google and download search results Compare each Google search result with the context of the tag and extract the concept Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 38 of 58

T-KNOW – Computing Similarity § Compute similarity using cosine measure between Bag of Words (BOW) representation of “Tag Context” and “Search Result” Tag Context singen cars motors ford 1955 ĉ Search Result BOW â 1955 = 1 as = 0 cars = 1 ford = 1 foundation = 0 international = 0 motors = 1 organizations = 0 singen = 1 such = 0 § 1955 = 0 as = 1 cars = 0 ford = 1 foundation = 2 international = 1 motors = 0 organizations = 1 singen = 0 such = 1 cos(ĉ, â) = ĉ x â / |ĉ||â| = 0. 15 Only consider the results having similarity above a certain Threshold Result having the highest similarity is considered as final We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 39 of 58

Experimental Setup 4+1 Categories Person Location Vehicle 932 Concepts Author, Singer, Human, … Country, District, City, Village, … Vehicle, Car, Truck, Motorbike, Train, … Company, Organization, Firm, Foundation, … Organization Other 189 random Images from 9 Flickr groups We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 40 of 58 1754 Tags

Experimental Setup – Classifiers § Two human classifiers: K (gold standard) and S § T-KNOW We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 41 of 58

Experimental Setup – Evaluation § F-Measure A = set of correct classification by test (user S or T-KNOW) B = set of all classification by Gold Standard (user K) C = set of all classifications by test Precision = A / C Recall = A / B F-Measure = 2 * Precision * Recall / (Precision + Recall) We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 42 of 58

Results – F-Measure - Results not far away from Human Classification 79% 51% We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 43 of 58

3 Tag Recommendations in Folksonomies Simplifying the annotation process We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 44 of 58

Tag Recommendation Image Tags Clocktower Uhrturm Graz Austria Europe Herbst Autumn Night Nacht 47° 4′ 32″ N 15° 26′ 12″ E Steffen Staab staab@uni-koblenz. de http: //www. flickr. com/photos/64255998@N 00/283890351/ We. ST IRMLe. S 2010 45 of 58

Tag Recommendation Input Output Tag Recommender 47° 4′ 32″ N 15° 26′ 12″ E Steffen Staab staab@uni-koblenz. de http: //www. flickr. com/photos/64255998@N 00/283890351/ We. ST IRMLe. S 2010 46 of 58 Clocktower Uhrturm Graz Austria Europe Herbst Autumn Night Nacht

Tag Recommendation Input Output Tag Recommender 47° 4′ 32″ N 15° 26′ 12″ E Geographical Coordinates We. ST Tags Steffen Staab staab@uni-koblenz. de Low-level Features IRMLe. S 2010 47 of 58 Clocktower Uhrturm Graz Austria Europe Herbst Autumn Night Nacht

Tag Recommendation Input Output Tag Recommender 47° 4′ 32″ N 15° 26′ 12″ E Geographical Coordinates Tags Low-level Features Which image features to use? We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 48 of 58 Clocktower Uhrturm Graz Austria Europe Herbst Autumn Night Nacht

Features Available in Large Datasets Co. Ph. IR Dataset contains • 106 million images from Flickr with – Title, location, tag, comment, etc – Five standard MPEG-7 Features We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 49 of 58

Tag Recommender – System Overview Location 1 Features Location 1 Clusters Training Data We. ST Model Location N Features Steffen Staab staab@uni-koblenz. de Location N IRMLe. SClusters 2010 50 of 58

System Overview Input Output Tag Recommender Im ag 47° 4′ 32″ N 15° 26′ 12″ E e. F Ta gs ea tu res Location 1 Features Location 1 Clusters Training Data We. ST Clocktower Graz Austria Europe Autumn Night Uhrturm Nacht Herbst HDR Photomatix Model Location N Features Steffen Staab staab@uni-koblenz. de Location N IRMLe. SClusters 2010 51 of 58

Clustering Used Standard K-Means clustering We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 52 of 58

Identifying Representative Tags from Clusters Given a cluster containing homogeneous images • Find most representative tags – Tag Rank ~ Frequency of Users Cluster Tags User 1 User 2 We. ST Clocktower (4) Graz (4) Austria (3) Europe (3) Autumn (2) Night (2) Uhrturm (1) Nacht (1) Herbst (1) HDR (1) Photomatix (1) Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 53 of 58 User 3 User 4

Classification Map a new image to one of the existing clusters Find the nearest cluster centroid • Assign representative tags of the mapped cluster to the image Representative Tags Nearest Centroid We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 54 of 58

Evaluation Evaluated on tags of ~400, 000 Flickr images • 75% training data • 25% test data / gold standard – Evaluation based on the correctly identified tags of test data • Data is randomly split based on users (not resources) Ignored • 10 most frequent tags for each location (city) and • Tags used by less than three users Calculated • Precision • Recall • F-Measure We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 55 of 58

Micro Precision We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 56 of 58

Micro Recall We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 57 of 58

Micro F-Measure We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 58 of 58

Coverage Number of different tags correctly suggested We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 59 of 58

Summary We can discover semantics in Web 2. 0 and exploit them for Improved search Semantic Browsing Tag recommendation We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 63 of 58

Lessons Learned § There is a quantifiable value in ontology learning techniques § Improvement of user experience is about Semantics • A little bit of semantics goes a long way! – Who? – What? – When? – Where? Representativity / Prototypicality Do not ignore! Not represented in Sem. Web! We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 64 of 58

Most interesting result for „Koblenz“: A Castle half an hour away from Koblenz Much work left to be done…. We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 65 of 58

More details in: § Abbasi, Staab: Rich. VSM: en. Riched Vector Space Models for Folksonomies. (ACM Hypertext 2009) § Abbasi, Grzegorzek, Staab: Large Scale Tag Recommendation Using Different Image Representations. (SAMT 2009) § Abbasi, Chernov, Nejdl, Paiu, Staab: Exploiting Flickr Tags and Groups for Finding Landmark Photos. (ECIR 2009) § Abbasi, Staab, Cimiano: Organizing Resources on Tagging Systems using T-ORG. (ESWC 2007) We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 66 of 58

Thank you for your attention! We. ST Steffen Staab staab@uni-koblenz. de IRMLe. S 2010 67 of 58