5 Vector Space and Probabilistic Retrieval Models Many

5. Vector Space and Probabilistic Retrieval Models Many slides in this section are adapted from Prof. Joydeep Ghosh (UT ECE) who in turn adapted them from Prof. Dik Lee (Univ. of Science and Tech, Hong Kong) These notes are based, in part, on notes by Dr. Raymond J. Mooney at the University of Texas at Austin. 1

Keyword Discrimination Model • The Vector representation of documents can be used as the source of another approach to term weighting – Question: what happens if we removed one of the words used as dimensions in the vector space? – If the average similarity among documents changes significantly, then the word was a good discriminator – If there is little change, the word is not as helpful and should be weighted less • Note that the goal is to have a representation that makes it easier for a queries to discriminate among documents • Average similarity can be measured after removing each word from the matrix – Any of the similarity measures can be used (we will look at a variety of other similarity measures later). 2

Keyword Discrimination i Measuring average similarity (assume there are N documents) sim(D 1, D 2) = similarity score for pair of documents D 1 and D 2 Computationally Expensive i Better way to calculate AVG-SIM 4 Calculate centroid D* (avg. document vector = Sum vectors / N) 4 Then: 3

Keyword Discrimination i Discrimination value (discriminant) and term weights disck > 0 ==> termk is a good discriminant disck < 0 ==> termk is a poor discriminant disck = 0 ==> termk is indifferent i Computing Term Weights 4 New weight for a term k in a document i is the original term frequency of k in i time the discriminant value: 4

Keyword Discrimination - Example Using Normalized Cosine Note: D* for each of the SIMk is now computed with only two terms 5

Keyword Discrimination - Example This shows that t 1 tends to be a poor discriminator, while t 3 is a good discriminator. The new term weight will now reflect the discrimination value for these terms. Note that further normalization can be done to make all term weights positive. 6

Signal-To-Noise Ratio • Based on work of Shannon in 1940’s on Information Theory – Developed a model of communication of messages across a noisy channel – Goal is to devise an encoding of messages that is most robust in the face of channel noise • In IR, messages describe the content of documents – Amount of information about the document from a word is inversely proportional to its probability of occurrence – The least informative words are those that occur approximately uniformly across the corpus of documents • a word that occurs with the similar frequency across many documents (e. g. , “the”, “and”, etc. ) is less informative than one that occurs with high frequency in one or two documents • Shannon used entropy (a logarithmic measure) to measure average information gain with noise defined as its inverse 7

Signal-To-Noise Ratio pk = Prob(term k occurs in document i) = tfik / tfk Infok = - pk log pk Note: here we always take Noisek = - pk log (1/pk) logs to be base 2. Note: NOISE is the negation of AVG-INFO, so only one of these needs to be computed in practice. The weight of term k in document i 8

Signal-To-Noise Ratio - Example pk = tfik / tfk Note: By definition, if the term k does not appear in the document, we assume Info(k) = 0 for that doc. This is the “entropy” of term k in the collection 9

Signal-To-Noise Ratio - Example The weight of term k in document i Additional normalization can be performed to have values in the range [0, 1] 10

Probabilistic Term Weights • Probabilistic model makes explicit distinctions between occurrences of terms in relevant and non-relevant documents • If we know pi: probability of term xi appears in relevant doc. qi: probability of term xi appears in non-relevant doc. with binary and independence assumption, the weight of term xi in document Dk is: • Estimates of pi and qi requires relevance information: – – using test queries and test collections to “train” the values of pi and qi other AI/learning technique? Intelligent Information Retrieval 11

Other Vector Space Similarity Measures Simple Matching: Cosine Coefficient: Dice’s Coefficient: Jaccard’s Coefficient: 12

Vector Space Similarity Measures i Consider the following two document and the query vectors: D 1 = (0. 8, 0. 3) D 2 = (0. 2, 0. 7) Q = (0. 4, 0. 8) i Computing similarity using Jaccard’s Coefficient: i Computing similarity using Dice’s Coefficient: sim(Q, D 1) = 0. 73 sim(Q, D 2) = 0. 96 13

Vector Space Similarity Measures -- Example 14

Naïve Implementation Convert all documents in collection D to tf-idf weighted vectors, dj, for keyword vocabulary V. Convert query to a tf-idf-weighted vector q. For each dj in D do Compute score sj = cos. Sim(dj, q) Sort documents by decreasing score. Present top ranked documents to the user. Time complexity: O(|V|·|D|) Bad for large V & D ! |V| = 10, 000; |D| = 100, 000; |V|·|D| = 1, 000, 000 15

Comments on Vector Space Models • Simple, mathematically based approach. • Considers both local (tf) and global (idf) word occurrence frequencies. • Provides partial matching and ranked results. • Tends to work quite well in practice despite obvious weaknesses. • Allows efficient implementation for large document collections. 16

Problems with Vector Space Model • Missing semantic information (e. g. word sense). • Missing syntactic information (e. g. phrase structure, word order, proximity information). • Assumption of term independence (e. g. ignores synonomy). • Lacks the control of a Boolean model (e. g. , requiring a term to appear in a document). – Given a two-term query “A B”, may prefer a document containing A frequently but not B, over a document that contains both A and B, but both less frequently. 17

3. Probabilistic Model • Attempts to be theoretically sound – try to predict the probability of a document’s being relevant, given the query – there are many variations – usually more complicated to compute than v. s. – usually many approximations are required • Relevance information is required from a random sample of documents and queries (training examples) • Works about the same (sometimes better) than vector space approaches 18

Basic Probabilistic Retrieval • Retrieval is modeled as a classification process • Two classes for each query: the relevant and non-relevant documents (with respect to a given query) – could easily be extended to three classes (i. e. add a don’t care) • Given a particular document D, calculate the probability of belonging to the relevant class – retrieve if greater than probability of belonging to non-relevant class – i. e. retrieve if P(R|D) > P(NR|D) • Equivalently, rank by a discriminant value (also called likelihood ratio) P(R|D) / P(NR|D) • Different ways of estimating these probabilities lead to different models 19

Basic Probabilistic Retrieval • A given query divides the document collection into two sets: relevant and non-relevant 4 If a document set D has been selected in response to a query, retrieve the document if P(R|D) Relevant Documents dis(D) > 1 where dis(D) = P(R|D) / P(NR|D) 4 is the discriminant of D P(NR|D) Document Non-Relevant Documents 4 This criteria can be modified by weighting the two probabilities 20

Estimating Probabilities • Bayes’ Rule can be used to “invert” conditional probabilities: • Applying that to discriminant function: • Note that P(R) is the probability that a random document is relevant to the query, and P(NR) = 1 - P(R) = n / N and P(NR) = 1 - P(R) = (N - n) / N where n = number of relevant documents, and N = total number of documents in the collection 21

Estimating Probabilities • Now we need to estimate P(D|R) and P(D|NR) – If we assume that a document is represented by terms t 1, . . . , tn, and that these terms are statistically independent, then – and similarly we can compute P(D|NR) – Note that P(ti|R) is the probability that a term ti occurs in a relevant document, and it can be estimated based on previously available sample (e. g. , through relevance feedback) – So, based on the probability of the distribution of terms in relevant and non-relevant documents we can estimate whether the document should be retrieved (i. e, if dis(D) > 1) – Note that documents that are retrieved can be ranked based on the value of the discriminant 22

Probabilistic Retrieval - Example Since the discriminant is less than one, document D should not be retrieved 23

Probabilistic Retrieval (cont. ) i In practice, can’t build a model for each query 4 Instead a general model is built based on query-document pairs in the historical (training) data 4 Then for a given query Q, the discriminant is computed only based on the conditional probabilities of the query terms 4 If query term t occurs in D, take P(t|R) and P(t|NR) 4 If query term t does not appear in D, take 1 -P(t|R) and 1 - P(t|NR) Q = t 1, t 3, t 4 D = t 1, t 4, t 5 24

Probabilistic Models Advantages • Strong theoretical basis • In principle should supply the best predictions of relevance given available information • Can be implemented similarly to Vector Disadvantages • Relevance information is required -- or is “guestimated” • Important indicators of relevance may not be term -though terms only are usually used • Optimally requires on-going collection of relevance information 25

Vector and Probabilistic Models • • • Support “natural language” queries Treat documents and queries the same Support relevance feedback searching Support ranked retrieval Differ primarily in theoretical basis and in how the ranking is calculated – Vector assumes relevance – Probabilistic relies on relevance judgments or estimates 26