Data Mining Mining Text and Web Data Han
Data Mining: — Mining Text and Web Data — Han & Kambr 9/17/2020 1
Mining Text and Web Data n Text mining, natural language processing and information extraction: An Introduction n Text categorization methods n Mining Web linkage structures n Summary 9/17/2020 2
Mining Text Data: An Introduction Data Mining / Knowledge Discovery Structured Data Home. Loan ( Loanee: Frank Rizzo Lender: MWF Agency: Lake View Amount: $200, 000 Term: 15 years ) 9/17/2020 Multimedia Free Text Loans($200 K, [map], . . . ) Frank Rizzo bought his home from Lake View Real Estate in 1992. He paid $200, 000 under a 15 -year loan from MW Financial. Hypertext <a href>Frank Rizzo </a> Bought <a hef>this home</a> from <a href>Lake View Real Estate</a> In <b>1992</b>. <p>. . . 3
Bag-of-Tokens Approaches Documents Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or … Token Sets Feature Extraction nation – 5 civil - 1 war – 2 men – 2 died – 4 people – 5 Liberty – 1 God – 1 … Loses all order-specific information! Severely limits context! 9/17/2020 4
Natural Language Processing A dog is chasing a boy on the playground Det Noun Aux Noun Phrase Verb Complex Verb Semantic analysis Dog(d 1). Boy(b 1). Playground(p 1). Chasing(d 1, b 1, p 1). + Det Noun Prep Det Noun Phrase Lexical analysis (part-of-speech tagging) Prep Phrase Verb Phrase Syntactic analysis (Parsing) Verb Phrase Sentence Scared(x) if Chasing(_, x, _). Scared(b 1) Inference (Taken from Cheng. Xiang Zhai, CS 397 cxz – Fall 2003) 9/17/2020 A person saying this may be reminding another person to get the dog back… Pragmatic analysis (speech act) 5
Word. Net An extensive lexical network for the English language • Contains over 138, 838 words. • Several graphs, one for each part-of-speech. • Synsets (synonym sets), each defining a semantic sense. • Relationship information (antonym, hyponym, meronym …) • Downloadable for free (UNIX, Windows) • Expanding to other languages (Global Word. Net Association) moist watery parched wet dry damp anhydrous arid synonym 9/17/2020 antonym 6
Text Databases and Information Retrieval (IR) n n Text databases (document databases) n Large collections of documents from various sources: news articles, research papers, books, digital libraries, e-mail messages, and Web pages, library database, etc. n Data stored is usually semi-structured n Traditional information retrieval techniques become inadequate for the increasingly vast amounts of text data Information retrieval n A field developed in parallel with database systems n Information is organized into (a large number of) documents n Information retrieval problem: locating relevant documents based on user input, such as keywords or example documents 9/17/2020 7
Information Retrieval n n Typical IR systems n Online library catalogs n Online document management systems Information retrieval vs. database systems n Some DB problems are not present in IR, e. g. , update, transaction management, complex objects n Some IR problems are not addressed well in DBMS, e. g. , unstructured documents, approximate search using keywords and relevance 9/17/2020 8
Basic Measures for Text Retrieval Relevant & Retrieved All Documents n n Precision: the percentage of retrieved documents that are in fact relevant to the query (i. e. , “correct” responses) Recall: the percentage of documents that are relevant to the query and were, in fact, retrieved 9/17/2020 9
Information Retrieval Techniques n n Basic Concepts n A document can be described by a set of representative keywords called index terms. n Different index terms have varying relevance when used to describe document contents. n This effect is captured through the assignment of numerical weights to each index term of a document. (e. g. : frequency, ) DBMS Analogy n Index Terms Attributes n Weights Attribute Values 9/17/2020 10
Keyword-Based Retrieval n n n A document is represented by a string, which can be identified by a set of keywords Queries may use expressions of keywords n E. g. , car and repair shop, tea or coffee, DBMS but not Oracle n Queries and retrieval should consider synonyms, e. g. , repair and maintenance Major difficulties of the model n Synonymy: A keyword T does not appear anywhere in the document, even though the document is closely related to T, e. g. , data mining n Polysemy: The same keyword may mean different things in different contexts, e. g. , mining 9/17/2020 12
Types of Text Data Mining n n n n Keyword-based association analysis Automatic document classification Similarity detection n Cluster documents by a common author n Cluster documents containing information from a common source Link analysis: unusual correlation between entities Sequence analysis: predicting a recurring event Anomaly detection: find information that violates usual patterns Hypertext analysis n Patterns in anchors/links n Anchor text correlations with linked objects 9/17/2020 13
Keyword-Based Association Analysis n Motivation n n Collect sets of keywords or terms that occur frequently together and then find the association or correlationships among them Association Analysis Process n n Preprocess the text data by parsing, stemming, removing stop words, etc. Evoke association mining algorithms n n n View a set of keywords in the document as a set of items in the transaction Term level association mining n n 9/17/2020 Consider each document as a transaction No need for human effort in tagging documents The number of meaningless results and the execution time is greatly reduced 14
Text Classification n Motivation n Automatic classification for the large number of on-line text documents (Web pages, e-mails, corporate intranets, etc. ) Classification Process n Data preprocessing n Definition of training set and test sets n Creation of the classification model using the selected classification algorithm n Classification model validation n Classification of new/unknown text documents Text document classification differs from the classification of relational data n Document databases are not structured according to attributevalue pairs 9/17/2020 15
Document Clustering n n Motivation n Automatically group related documents based on their contents n No predetermined training sets or taxonomies n Generate a taxonomy at runtime Clustering Process n Data preprocessing: remove stop words, stem, feature extraction, lexical analysis, etc. n Hierarchical clustering: compute similarities applying clustering algorithms. n Model-Based clustering (Neural Network Approach): clusters are represented by “exemplars”. (e. g. : SOM) 9/17/2020 16
Text Categorization n Pre-given categories and labeled document examples (Categories may form hierarchy) Classify new documents A standard classification (supervised learning ) problem Sports Categorization System Business Education Sports Business … … Science Education 9/17/2020 17
Applications n n n News article classification Automatic email filtering Webpage classification Word sense disambiguation …… 9/17/2020 18
Categorization Methods n n Manual: Typically rule-based n Does not scale up (labor-intensive, rule inconsistency) n May be appropriate for special data on a particular domain Automatic: Typically exploiting machine learning techniques n Vector space model based n n n Probabilistic or generative model based n 9/17/2020 K-nearest neighbor (KNN) Decision-tree (learn rules) Neural Networks (learn non-linear classifier) Support Vector Machines (SVM) Naïve Bayes classifier 19
Vector Space Model n n Represent a doc by a term vector n Term: basic concept, e. g. , word or phrase n Each term defines one dimension n N terms define a N-dimensional space n Element of vector corresponds to term weight n E. g. , d = (x 1, …, x. N), xi is “importance” of term i New document is assigned to the most likely category based on vector similarity. 9/17/2020 20
VS Model: Illustration Starbucks C 2 Category 3 C 3 new doc Microsoft 9/17/2020 Java C 1 Category 1 21
Categorization Methods n n Vector space model n K-NN n Decision tree n Neural network n Support vector machine Probabilistic model n n Naïve Bayes classifier Many, many others and variants exist [F. S. 02] n 9/17/2020 e. g. Bim, Nb, Ind, Swap-1, LLSF, Widrow-Hoff, Rocchio, Gis-W, … … 27
Evaluations n Effectiveness measure n Classic: Precision & Recall 9/17/2020 n Precision n Recall 28
Evaluation (con’t) n Benchmarks n Classic: Reuters collection n n A set of newswire stories classified under categories related to economics. Effectiveness n n n 9/17/2020 Difficulties of strict comparison n different parameter setting n different “split” (or selection) between training and testing n various optimizations … … However widely recognizable n Best: Boosting-based committee classifier & SVM n Worst: Naïve Bayes classifier Need to consider other factors, especially efficiency 29
Mining Text and Web Data n Text mining, natural language processing and information extraction: An Introduction n Text categorization methods n Mining Web linkage structures n n 9/17/2020 Based on the slides by Deng Cai Summary 31
Outline n Background on Web Search n VIPS (VIsion-based Page Segmentation) n Block-based Web Search n Block-based Link Analysis n Web Image Search & Clustering 9/17/2020 32
Search Engine – Two Rank Functions Ranking based on link structure analysis Search Rank Functions Similarity based on content or text Importance Ranking (Link Analysis) Relevance Ranking Backward Link (Anchor Text) Indexer Inverted Index Term Dictionary (Lexicon) Web Topology Graph Anchor Text Generator Meta Data Forward Index Forward Link Web Graph Constructor URL Dictioanry Web Page Parser Web Pages 9/17/2020 33
The Page. Rank Algorithm n Basic idea n n significance of a page is determined by the significance of the pages linking to it More precisely: n n 9/17/2020 Link graph: adjacency matrix A, Constructs a probability transition matrix M by renormalizing each row of A to sum to 1 Treat the web graph as a markov chain (random surfer) The vector of Page. Rank scores p is then defined to be the stationary distribution of this Markov chain. Equivalently, p is the principal right eigenvector of the transition matrix 34
Layout Structure n Compared to plain text, a web page is a 2 D presentation n Rich visual effects created by different term types, formats, separators, blank areas, colors, pictures, etc n Different parts of a page are not equally important Title: CNN. com International H 1: IAEA: Iran had secret nuke agenda H 3: EXPLOSIONS ROCK BAGHDAD … TEXT BODY (with position and font type): The International Atomic Energy Agency has concluded that Iran has secretly produced small amounts of nuclear materials including low enriched uranium and plutonium that could be used to develop nuclear weapons according to a confidential report obtained by CNN… Hyperlink: • URL: http: //www. cnn. com/. . . • Anchor Text: AI oaeda… Image: • URL: http: //www. cnn. com/image/. . . • Alt & Caption: Iran nuclear … Anchor Text: CNN Homepage News … 9/17/2020 35
Web Page Block—Better Information Unit Web Page Blocks Importance = Low Importance = Med Importance = High 9/17/2020 36
Motivation for VIPS (VIsion-based Page Segmentation) n Problems of treating a web page as an atomic unit n Web page usually contains not only pure content n Noise: navigation, decoration, interaction, … Multiple topics n Different parts of a page are not equally important Web page has internal structure n Two-dimension logical structure & Visual layout presentation n > Free text document n < Structured document Layout – the 3 rd dimension of Web page st n 1 dimension: content nd dimension: hyperlink n 2 n n n 9/17/2020 37
Is DOM a Good Representation of Page Structure? n Page segmentation using DOM n Extract structural tags such as P, TABLE, UL, TITLE, H 1~H 6, etc n n DOM is more related content display, does not necessarily reflect semantic structure How about XML? n A long way to go to replace the HTML 9/17/2020 38
Example of Web Page Segmentation (1) ( DOM Structure ) 9/17/2020 ( VIPS Structure ) 39
Example of Web Page Segmentation (2) ( DOM Structure ) n 9/17/2020 ( VIPS Structure ) Can be applied on web image retrieval n Surrounding text extraction 40
Web Page Block—Better Information Unit Page Segmentation Block Importance Modeling • Vision based approach • Statistical learning Web Page Blocks Importance = Low Importance = Med Importance = High 9/17/2020 41
A Sample of User Browsing Behavior 9/17/2020 42
Improving Page. Rank using Layout Structure n Z: block-to-page matrix (link structure) n X: page-to-block matrix (layout structure) n Block-level Page. Rank: n n Compute Page. Rank on the page-to-page graph Block. Rank: n 9/17/2020 Compute Page. Rank on the block-to-block graph 43
Mining Web Images Using Layout & Link Structure (ACMMM’ 04) 9/17/2020 44
Image Graph Model & Spectral Analysis n Block-to-block graph: n Block-to-image matrix (container relation): Y n Image-to-image graph: n Image. Rank n n Compute Page. Rank on the image graph Image clustering n 9/17/2020 Graphical partitioning on the image graph 45
Image. Rank n Relevance Ranking 9/17/2020 n Importance Ranking n Combined Ranking 46
Image. Rank vs. Page. Rank n n Dataset n 26. 5 millions web pages n 11. 6 millions images Query set n 45 hot queries in Google image search statistics Ground truth n Five volunteers were chosen to evaluate the top 100 results re-turned by the system (i. Find) Ranking method 9/17/2020 47
Image. Rank vs Page. Rank n Image search accuracy using Image. Rank and Page. Rank. Both of them achieved their best results at =0. 25. 9/17/2020 48
Example on Image Clustering & Embedding 1710 JPG images in 1287 pages are crawled within the website http: //www. yahooligans. com/content/animals/ Six Categories Fish Mammal Bird 9/17/2020 Amphibian Reptile Insect 49
9/17/2020 50
2 -D embedding of WWW images The image graph was constructed from block level link analysis 9/17/2020 The image graph was constructed from traditional page level link analysis 51
2 -D Embedding of Web Images n 9/17/2020 2 -D visualization of the mammal category using the second and third eigenvectors. 52
Web Image Search Result Presentation (a) (b) Figure 1. Top 8 returns of query “pluto” in Google’s image search engine (a) and Alta. Vista’s image search engine (b) n n 9/17/2020 Two different topics in the search result A possible solution: n Cluster search results into different semantic groups 53
Three kinds of WWW image representation n 9/17/2020 Visual Feature Based Representation n Traditional CBIR Textual Feature Based Representation n Surrounding text in image block Link Graph Based Representation n Image graph embedding 54
Clustering Using Visual Feature Figure 5. Five clusters of search results of query “pluto” using low level visual feature. Each row is a cluster. n 9/17/2020 From the perspectives of color and texture, the clustering results are quite good. Different clusters have different colors and textures. However, from semantic perspective, these clusters make little sense. 55
Clustering Using Textual Feature Figure 6. The Eigengap curve with k for the “pluto” case using textual representation Figure 7. Six clusters of search results of query “pluto” using textual feature. Each row is a cluster n 9/17/2020 Six semantic categories are correctly identified if we choose k = 6. 56
Clustering Using Graph Based Representation Figure 8. Five clusters of search results of query “pluto” using image link graph. Each row is a cluster n n n Each cluster is semantically aggregated. Too many clusters. In “pluto” case, the top 500 results are clustered into 167 clusters. The max cluster number is 87, and there are 112 clusters with only one image. 9/17/2020 57
Combining Textual Feature and Link Graph Figure 10. The Eigengap curve with k for the “pluto” case using textual and link combination Figure 9. Six clusters of search results of query “pluto” using combination of textual feature and image link graph. Each row is a cluster n Combine two affinity matrix 9/17/2020 58
Final Presentation of Our System n n 9/17/2020 Using textual and link information to get some semantic clusters Use low level visual feature to cluster (re-organize) each semantic cluster to facilitate user’s browsing 59
Summary n n n 9/17/2020 More improvement on web search can be made by mining webpage Layout structure Leverage visual cues for web information analysis & information extraction Demos: n http: //www. ews. uiuc. edu/~dengcai 2 n Papers n VIPS demo & dll 60
www. cs. uiuc. edu/~hanj Thank you !!! 9/17/2020 61
- Slides: 54