Information Retrieval Models Vector Space Models Cheng Xiang

  • Slides: 30
Download presentation
Information Retrieval Models: Vector Space Models Cheng. Xiang Zhai Department of Computer Science University

Information Retrieval Models: Vector Space Models Cheng. Xiang Zhai Department of Computer Science University of Illinois, Urbana-Champaign

Empirical IR vs. Model-based IR • Empirical IR: – heuristic approaches – solely rely

Empirical IR vs. Model-based IR • Empirical IR: – heuristic approaches – solely rely on empirical evaluation – assumptions not always clearly stated – findings: empirical observations; may or may not generalize well • Model-based IR: – theoretical approaches – rely more on mathematics – assumptions are explicitly stated – findings: principles, models that may work well or not work well; generalize better • Boundary may not be clear and a combination is generally necessary 2

History of Research on IR Models • 1960: First probabilistic model • 1970 s:

History of Research on IR Models • 1960: First probabilistic model • 1970 s: Active research on retrieval models [Maron & Kuhns 60] started – Vector-space model [Salton et al. 75] – Classic probabilistic model [Robertson & Sparck Jones 76] – Probability Ranking Principle [Robertson 77] • 1980 s: Further development of different models – Non-classic logic model [Rijsbergen 86] – Extended Boolean [Salton et al. 83] – Early work on learning to rank [Fuhr 89] 3

History of Research on IR Models (cont. ) • 1990 s: retrieval model research

History of Research on IR Models (cont. ) • 1990 s: retrieval model research driven by TREC – Inference network [Turtle & Croft 91] – BM 25/Okapi [Robertson et al. 94] – Pivoted length normalization [Singhal et al. 96] – Language model [Ponte & Croft 98] • 2000 s-present: retrieval model influenced by machine learning and Web search – Further development of language models [Zhai & Lafferty 01, Lavrenko & Croft 01] – Divergence from randomness [Amati et al. 02] – Axiomatic model [Fang et al. 04] – Markov Random Field [Metzler & Croft 05] – Further development of Learning to rank [Joachimes 02, Burges et al. 05] 4

Modeling Relevance: Raodmap for Retrieval Models Relevance constraints Relevance (Rep(q), Rep(d)) Similarity P(r=1|q, d)

Modeling Relevance: Raodmap for Retrieval Models Relevance constraints Relevance (Rep(q), Rep(d)) Similarity P(r=1|q, d) r {0, 1} Probability of Relevance Regression Model (Fuhr 89) [Fang et al. 04] Div. from Randomness (Amati & Rijsbergen 02) P(d q) or P(q d) Probabilistic inference Generative Model Different inference system rep & similarity Learn. To Rank. Doc Query (Joachims 02, generation … Prob. concept Inference Berges et al. 05) space model network Vector space Prob. distr. model Classical LM (Wong & Yao, 95) model (Turtle & Croft, 91) model prob. Model approach (Salton et al. , 75) (Wong & Yao, 89) (Robertson & (Ponte & Croft, 98) Sparck Jones, 76) (Lafferty & Zhai, 01 a) 5

1. Vector Space Models

1. Vector Space Models

The Basic Question Given a query, how do we know if document A is

The Basic Question Given a query, how do we know if document A is more relevant than B? One Possible Answer If document A uses more query words than document B (Word usage in document A is more similar to that in query)

Relevance = Similarity • Assumptions – Query and document are represented similarly – A

Relevance = Similarity • Assumptions – Query and document are represented similarly – A query can be regarded as a “document” – Relevance(d, q) similarity(d, q) • R(q) = {d C|f(d, q)> }, f(q, d)= (Rep(q), Rep(d)) • Key issues – How to represent query/document? – How to define the similarity measure ? 8

Vector Space Model • Represent a doc/query by a term vector – Term: basic

Vector Space Model • Represent a doc/query by a term vector – Term: basic concept, e. g. , word or phrase – Each term defines one dimension – N terms define a high-dimensional space – Element of vector corresponds to term weight – E. g. , d=(x 1, …, x. N), xi is “importance” of term i • Measure relevance by the distance between the query vector and document vector in the vector space 9

VS Model: illustration Starbucks D 2 D 9 D 11 ? ? D 5

VS Model: illustration Starbucks D 2 D 9 D 11 ? ? D 5 D 3 D 10 D 4 D 6 Query D 7 D 8 Microsoft Java D 1 ? ? 10

What the VS model doesn’t say • How to define/select the “basic concept” –

What the VS model doesn’t say • How to define/select the “basic concept” – Concepts are assumed to be orthogonal • How to assign weights – Weight in query indicates importance of term – Weight in doc indicates how well the term characterizes the doc • How to define the similarity/distance measure 11

What’s a good “basic concept”? • Orthogonal – Linearly independent basis vectors – “Non-overlapping”

What’s a good “basic concept”? • Orthogonal – Linearly independent basis vectors – “Non-overlapping” in meaning • No ambiguity • Weights can be assigned automatically and hopefully accurately • Many possibilities: Words, stemmed words, phrases, “latent concept”, … • “Bag of words” representation works “surprisingly” well! 12

How to Assign Weights? • Very very important! • Why weighting – Query side:

How to Assign Weights? • Very very important! • Why weighting – Query side: Not all terms are equally important – Doc side: Some terms carry more information about contents • How? – Two basic heuristics • TF (Term Frequency) = Within-doc-frequency • IDF (Inverse Document Frequency) – Document length normalization 13

TF Weighting • Idea: A term is more important if it occurs more frequently

TF Weighting • Idea: A term is more important if it occurs more frequently in a document • Formulas: Let f(t, d) be the frequency count of term t in doc d – Raw TF: TF(t, d) = f(t, d) – Log TF: TF(t, d)=log ( f(t, d) +1) – Maximum frequency normalization: TF(t, d) = 0. 5 +0. 5*f(t, d)/Max. Freq(d) – “Okapi/BM 25 TF”: TF(t, d) = k f(t, d)/(f(t, d)+k(1 -b+b*doclen/avgdoclen)) • Normalization of TF is very important! 14

TF Normalization • Why? – Document length variation – “Repeated occurrences” are less informative

TF Normalization • Why? – Document length variation – “Repeated occurrences” are less informative than the “first occurrence” • Two views of document length – A doc is long because it uses more words – A doc is long because it has more contents • Generally penalize long doc, but avoid overpenalizing (e. g. , pivoted normalization) 15

TF Normalization (cont. ) Norm. TF Raw TF “Pivoted normalization”: Using avg. doc length

TF Normalization (cont. ) Norm. TF Raw TF “Pivoted normalization”: Using avg. doc length to regularize normalization 1 -b+b*doclen/avgdoclen b varies from 0 to 1 Normalization interacts with the similarity measure 16

IDF Weighting • Idea: A term is more discriminative/important if it occurs only in

IDF Weighting • Idea: A term is more discriminative/important if it occurs only in fewer documents • Formula: IDF(t) = 1+ log(n/k) n – total number of docs k -- # docs with term t (doc freq) • Other variants: – IDF(t) = log((n+1)/k) – IDF(t)=log ((n+1)/(k+0. 5)) • What are the maximum and minimum values of IDF? 17

Non-Linear Transformation in IDF(t) = 1+ log(n/k) 1+log(n) Linear penalization 1 k (doc freq)

Non-Linear Transformation in IDF(t) = 1+ log(n/k) 1+log(n) Linear penalization 1 k (doc freq) N =totoal number of docs in collection Is this transformation optimal? 18

TF-IDF Weighting • TF-IDF weighting : weight(t, d)=TF(t, d)*IDF(t) – Common in doc high

TF-IDF Weighting • TF-IDF weighting : weight(t, d)=TF(t, d)*IDF(t) – Common in doc high tf high weight – Rare in collection high idf high weight • Imagine a word count profile, what kind of terms would have high weights? 19

Empirical distribution of words • There are stable language-independent patterns in how people use

Empirical distribution of words • There are stable language-independent patterns in how people use natural languages • A few words occur very frequently; most occur rarely. E. g. , in news articles, – Top 4 words: 10~15% word occurrences – Top 50 words: 35~40% word occurrences • The most frequent word in one corpus may be rare in another 20

Zipf’s Law • rank * frequency constant Word Freq. High entropy words Word Rank

Zipf’s Law • rank * frequency constant Word Freq. High entropy words Word Rank (by Freq) Generalized Zipf’s law: Applicable in many domains 21

How to Measure Similarity? How about Euclidean? 22

How to Measure Similarity? How about Euclidean? 22

What Works the Best? Error [ ] • Use single words • Use stat.

What Works the Best? Error [ ] • Use single words • Use stat. phrases • Remove stop words • Stemming • Others(? ) (Singhal 2001; Singhal et al. 1996) 23

Relevance Feedback in VS • Basic setting: Learn from examples – Positive examples: docs

Relevance Feedback in VS • Basic setting: Learn from examples – Positive examples: docs known to be relevant – Negative examples: docs known to be non-relevant • – How do you learn from this to improve performance? General method: Query modification – Adding new (weighted) terms – Adjusting weights of old terms • – Doing both The most well-known and effective approach is Rocchio [Rocchio 1971] 24

Rocchio Feedback: Illustration Centroid of relevant documents Centroid of non-relevant documents -- --+ +

Rocchio Feedback: Illustration Centroid of relevant documents Centroid of non-relevant documents -- --+ + ++ + --+ q q m ++ + - - - + + ++ -+ + + - -- -- 25

Rocchio Feedback: Formula New query Origial query Parameters Rel docs Non-rel docs 26

Rocchio Feedback: Formula New query Origial query Parameters Rel docs Non-rel docs 26

Rocchio in Practice • • • How can we optimize the parameters? Can it

Rocchio in Practice • • • How can we optimize the parameters? Can it be used for both relevance feedback and pseudo feedback? How does Rocchio feedback affect the efficiency of scoring documents? How can we improve the efficiency? 27

Advantages of VS Model • Empirically effective! (Top TREC performance) • Intuitive • Easy

Advantages of VS Model • Empirically effective! (Top TREC performance) • Intuitive • Easy to implement • Well-studied/Most evaluated • The Smart system – Developed at Cornell: 1960 -1999 – Still widely used • Warning: Many variants of TF-IDF! 28

Disadvantages of VS Model • Assume term independence • Assume query and document to

Disadvantages of VS Model • Assume term independence • Assume query and document to be the same • Lack of “predictive adequacy” – Arbitrary term weighting – Arbitrary similarity measure • Lots of parameter tuning! 29

What You Should Know • Basic idea of the vector space model • TF-IDF

What You Should Know • Basic idea of the vector space model • TF-IDF weighting • Pivoted length normalization (read [Singhal et al. 1996] to know more) • BM 25/Okapi retrieval function (particularly TF weighting) • How Rocchio feedback works 30