Text and Web Search Text Databases and IR

Text and Web Search

Text Databases and IR n Text databases (document databases) n n Large collections of documents from various sources: news articles, research papers, books, digital libraries, e -mail messages, and Web pages, library database, etc. Information retrieval n n n A field developed in parallel with database systems Information is organized into (a large number of) documents Information retrieval problem: locating relevant documents based on user input, such as keywords or example documents

Information Retrieval n n Typical IR systems n Online library catalogs n Online document management systems Information retrieval vs. database systems n Some DB problems are not present in IR, e. g. , update, transaction management, complex objects n Some IR problems are not addressed well in DBMS, e. g. , unstructured documents, approximate search using keywords and relevance

Basic Measures for Text Retrieval Relevant & Retrieved All Documents n n Precision: the percentage of retrieved documents that are in fact relevant to the query (i. e. , “correct” responses) Recall: the percentage of documents that are relevant to the query and were, in fact, retrieved

Information Retrieval Techniques n Index Terms (Attribute) Selection: n n n Stop list Word stem Index terms weighting methods Terms Documents Frequency Matrices Information Retrieval Models: n n n Boolean Model Vector Model Probabilistic Model

Problem - Motivation n n Given a database of documents, find documents containing “data”, “retrieval” Applications: n Web n law + patent offices n digital libraries n information filtering

Problem - Motivation n n Types of queries: n boolean (‘data’ AND ‘retrieval’ AND NOT. . . ) n additional features (‘data’ ADJACENT ‘retrieval’) n keyword queries (‘data’, ‘retrieval’) How to search a large collection of documents?

Full-text scanning n for single term: n (naive: O(N*M)) ABRACADABRA text CAB pattern

Full-text scanning n for single term: n (naive: O(N*M)) n Knuth, Morris and Pratt (‘ 77) n build a small FSA; visit every text letter once only, by carefully shifting more than one step ABRACADABRA text CAB pattern

Full-text scanning ABRACADABRA text CAB pattern CAB. . . CAB

Full-text scanning n for single term: n (naive: O(N*M)) n Knuth Morris and Pratt (‘ 77) n Boyer and Moore (‘ 77) n preprocess pattern; start from right to left & skip! ABRACADABRA text CAB pattern

Text - Detailed outline n text n n n problem full text scanning inversion signature files clustering information filtering and LSI

Text – Inverted Files

Text – Inverted Files Q: space overhead? A: mainly, the postings lists

Text – Inverted Files n n n how to organize dictionary? stemming – Y/N? n Keep only the root of each word ex. inverted, inversion invert insertions?

Text – Inverted Files n n n how to organize dictionary? n B-tree, hashing, TRIEs, PATRICIA trees, . . . stemming – Y/N? insertions?

Text – Inverted Files n postings list – more Zipf distr. : eg. , rank-frequency plot of ‘Bible’ log(freq) freq ~ 1/rank / ln(1. 78 V) log(rank)

Text – Inverted Files n postings lists n Cutting+Pedersen n (keep first 4 in B-tree leaves) n how to allocate space: [Faloutsos+92] n geometric progression n compression (Elias codes) [Zobel+] – down to 2% overhead! n Conclusions: needs space overhead (2%-300%), but it is the fastest

Vector Space Model and Clustering n n Keyword (free-text) queries (vs Boolean) each document: -> vector (HOW? ) each query: -> vector search for ‘similar’ vectors

Vector Space Model and Clustering n main idea: each document is a vector of size d: d is the number of different terms in the database document. . . data. . . ‘indexing’ aaron data zoo d (= vocabulary size)

Document Vectors n n Documents are represented as “bags of words” Represented as vectors when used computationally n n A vector is like an array of floating points Has direction and magnitude Each vector holds a place for every term in the collection Therefore, most vectors are sparse

Document Vectors One location for each word. nova A 10 B 5 C D E F G 5 H I galaxy heat 5 3 10 h’wood film 10 occurs 910 8 10 times role diet 7 5 text “Nova” in A “Galaxy” occurs 5 times in text A 10 “Heat” occurs 3 times in text A 9 7 means 0 occurrences. ) 9 (Blank 6 10 7 2 5 fur 10 10 8 1 3

Document Vectors One location for each word. nova A 10 B 5 C D E F G 5 H I galaxy heat 5 3 10 h’wood film role diet “Hollywood” occurs 78 times 7 in text I 10 “Film” occurs 9 5 times 10 in 5 text I “Diet” occurs 1 time in text I 10 “Fur” occurs 3 times in text I 9 6 7 10 7 2 5 fur 10 10 9 8 1 3

Document Vectors Document ids nova A 10 B 5 C D E F G 5 H I galaxy heat 5 3 10 h’wood 10 9 film 8 10 role diet 7 5 10 9 6 7 10 7 2 5 fur 10 10 9 8 1 3

We Can Plot the Vectors Star Doc about astronomy Doc about movie stars Doc about mammal behavior Diet

Vector Space Model and Clustering Then, group nearby vectors together n Q 1: cluster search? n Q 2: cluster generation? Two significant contributions n ranked output n relevance feedback

Vector Space Model and Clustering n cluster search: visit the (k) closest superclusters; continue recursively MD TRs CS TRs

Vector Space Model and Clustering n ranked output: easy! MD TRs CS TRs

Vector Space Model and Clustering n relevance feedback (brilliant idea) [Roccio’ 73] MD TRs CS TRs

Vector Space Model and Clustering n n relevance feedback (brilliant idea) [Roccio’ 73] How? MD TRs CS TRs

Vector Space Model and Clustering n How? A: by adding the ‘good’ vectors and subtracting the ‘bad’ ones MD TRs CS TRs

Cluster generation n Problem: n given N points in V dimensions, n group them

Cluster generation n Problem: n given N points in V dimensions, n group them (typically a k-means or AGNES is used)

Assigning Weights to Terms n n n Binary Weights Raw term frequency tf x idf n Recall the Zipf distribution n Want to weight terms highly if they are n frequent in relevant documents … BUT n infrequent in the collection as a whole

Binary Weights n Only the presence (1) or absence (0) of a term is included in the vector

Raw Term Weights n The frequency of occurrence for the term in each document is included in the vector

Assigning Weights n n tf x idf measure: n term frequency (tf) n inverse document frequency (idf) -- a way to deal with the problems of the Zipf distribution Goal: assign a tf * idf weight to each term in each document

tf x idf

Inverse Document Frequency n IDF provides high values for rare words and low values for common words For a collection of 10000 documents

Similarity Measures for document vectors Simple matching (coordination level match) Dice’s Coefficient Jaccard’s Coefficient Cosine Coefficient Overlap Coefficient

tf x idf normalization n Normalize the term weights (so longer documents are not unfairly given more weight) n normalize usually means force all values to fall within a certain range, usually between 0 and 1, inclusive.

Vector space similarity (use the weights to compare the documents)

Computing Similarity Scores 1. 0 0. 8 0. 6 0. 4 0. 2 0. 4 0. 6 0. 8 1. 0

Vector Space with Term Weights and Cosine Matching Term B 1. 0 0. 8 D 2 Q Q = (0. 4, 0. 8) D 1=(0. 8, 0. 3) D 2=(0. 2, 0. 7) 0. 6 0. 4 D 1 0. 2 0. 4 0. 6 Term A 0. 8 1. 0 Di=(di 1, wdi 1; di 2, wdi 2; …; dit, wdit) Q =(qi 1, wqi 1; qi 2, wqi 2; …; qit, wqit)

Text - Detailed outline n Text databases n n n problem full text scanning inversion signature files (a. k. a. Bloom Filters) Vector model and clustering information filtering and LSI

Information Filtering + LSI n [Foltz+, ’ 92] Goal: n n n users specify interests (= keywords) system alerts them, on suitable newsdocuments Major contribution: LSI = Latent Semantic Indexing n latent (‘hidden’) concepts

Information Filtering + LSI Main idea n map each document into some ‘concepts’ n map each term into some ‘concepts’ ‘Concept’: ~ a set of terms, with weights, e. g. n “data” (0. 8), “system” (0. 5), “retrieval” (0. 6) -> DBMS_concept

Information Filtering + LSI Pictorially: term-document matrix (BEFORE)

Information Filtering + LSI Pictorially: concept-document matrix and. . .

Information Filtering + LSI. . . and concept-term matrix

Information Filtering + LSI Q: How to search, eg. , for ‘system’?

Information Filtering + LSI A: find the corresponding concept(s); and the corresponding documents

Information Filtering + LSI A: find the corresponding concept(s); and the corresponding documents

Information Filtering + LSI Thus it works like an (automatically constructed) thesaurus: we may retrieve documents that DON’T have the term ‘system’, but they contain almost everything else (‘data’, ‘retrieval’)

SVD n LSI: find ‘concepts’

SVD - Definition A[n x m] = U[n x r] L [ r x r] (V[m x r])T n n A: n x m matrix (eg. , n documents, m terms) U: n x r matrix (n documents, r concepts) L: r x r diagonal matrix (strength of each ‘concept’) (r : rank of the matrix) V: m x r matrix (m terms, r concepts)

SVD - Example n A = U L VT - example: retrieval inf. lung brain data CS = MD x x

SVD - Example n A = U L VT - example: retrieval CS-concept inf. lung MD-concept brain data CS = MD x x

SVD - Example n A = U L VT - example: doc-to-concept similarity matrix retrieval CS-concept inf. lung MD-concept brain data CS = MD x x

SVD - Example n A = U L VT - example: retrieval inf. lung brain data CS = MD ‘strength’ of CS-concept x x

SVD - Example n A = U L VT - example: retrieval inf. lung brain data term-to-concept similarity matrix CS-concept CS = MD x x

SVD - Example n A = U L VT - example: retrieval inf. lung brain data term-to-concept similarity matrix CS-concept CS = MD x x

SVD for LSI ‘documents’, ‘terms’ and ‘concepts’: n U: document-to-concept similarity matrix n V: term-to-concept sim. matrix n L: its diagonal elements: ‘strength’ of each concept

SVD for LSI n Need to keep all the eigenvectors? n NO, just keep the first k (concepts)

Web Search n What about web search? n n n First you need to get all the documents of the web…. Crawlers. Then you have to index them (inverted files, etc) Find the web pages that are relevant to the query Report the pages with their links in a sorted order Main difference with IR: web pages have links n may be possible to exploit the link structure for sorting the relevant documents…

Kleinberg’s Algorithm (HITS) n Main idea: In many cases, when you search the web using some terms, the most relevant pages may not contain this term (or contain the term only a few times) n n n Harvard : www. harvard. edu Search Engines: yahoo, google, altavista Authorities and hubs

Kleinberg’s algorithm n n Problem dfn: given the web and a query find the most ‘authoritative’ web pages for this query Step 0: find all pages containing the query terms (root set) Step 1: expand by one move forward and backward (base set)

Kleinberg’s algorithm n Step 1: expand by one move forward and backward

Kleinberg’s algorithm n n on the resulting graph, give high score (= ‘authorities’) to nodes that many important nodes point to give high importance score (‘hubs’) to nodes that point to good ‘authorities’) hubs authorities

Kleinberg’s algorithm observations n recursive definition! n each node (say, ‘i’-th node) has both an authoritativeness score ai and a hubness score hi

Kleinberg’s algorithm Let E be the set of edges and A be the adjacency matrix: the (i, j) is 1 if the edge from i to j exists Let h and a be [n x 1] vectors with the ‘hubness’ and ‘authoritativiness’ scores. Then:

Kleinberg’s algorithm Then: k l i m a i = hk + h l + hm that is ai = Sum (hj) over all j that (j, i) edge exists or a = AT h

Kleinberg’s algorithm i n p q symmetrically, for the ‘hubness’: hi = a n + a p + a q that is hi = Sum (qj) over all j that (i, j) edge exists or h=Aa

Kleinberg’s algorithm In conclusion, we want vectors h and a such that: h=Aa a = AT h Start from a and h to all 1. Then apply the following trick: h=Aa=A(ATh)=(AAT)h =. . =(AAT)2 h. . = (AAT)k h a = (ATA)ka

Kleinberg’s algorithm In short, the solutions to h=Aa a = AT h are the left- and right- eigenvectors of the adjacency matrix A. Starting from random a’ and iterating, we’ll eventually converge (Q: to which of all the eigenvectors? why? )

Kleinberg’s algorithm (Q: to which of all the eigenvectors? why? ) A: to the ones of the strongest eigenvalue, because of property : (AT A ) k v’ ~ (constant) v 1 So, we can find the a and h vectors and the page with the highest a values are reported!

Kleinberg’s algorithm - results Eg. , for the query ‘java’: 0. 328 www. gamelan. com 0. 251 java. sun. com 0. 190 www. digitalfocus. com (“the java developer”)

Kleinberg’s algorithm - discussion n n ‘authority’ score can be used to find ‘similar pages’ to page p closely related to ‘citation analysis’, social networs / ‘small world’ phenomena

google/page-rank algorithm n n n closely related: The Web is a directed graph of connected nodes imagine a particle randomly moving along the edges (*) compute its steady-state probabilities. That gives the Page. Rank of each pages (the importance of this page) (*) with occasional random jumps

Page. Rank Definition n Assume a page A and pages T 1, T 2, …, Tm that point to A. Let d is a damping factor. PR(A) the Pagerank of A. C(A) the outdegree of A. Then:

google/page-rank algorithm n Compute the PR of each page~identical problem: given a Markov Chain, compute the steady state probabilities p 1. . . p 5 2 1 4 5 3

Computing Page. Rank n n Iterative procedure Also, … navigate the web by randomly follow links or with prob p jump to a random page. Let A the adjacency matrix (n x n), ci out-degree of page i Prob(Ai->Aj) = dn-1+(1 -d)ci– 1 Aij A’[i, j] = Prob(Ai->Aj)

google/page-rank algorithm n Let A’ be the transition matrix (= adjacency matrix, row-normalized : sum of each row = 1) 2 1 3 = 4 5

google/page-rank algorithm n Ap=p A 2 1 p = 3 = 4 5 p

google/page-rank algorithm n n Ap=p thus, p is the eigenvector that corresponds to the highest eigenvalue (=1, since the matrix is row-normalized)

Kleinberg/google - conclusions SVD helps in graph analysis: hub/authority scores: strongest left- and right- eigenvectors of the adjacency matrix random walk on a graph: steady state probabilities are given by the strongest eigenvector of the transition matrix

References n Brin, S. and L. Page (1998). Anatomy of a Large-Scale Hypertextual Web Search Engine. 7 th Intl World Wide Web Conf.