A Formal Study of Information Retrieval Heuristics Hui

A Formal Study of Information Retrieval Heuristics Hui Fang , Tao , Cheng. Xiang Zhai University of Illinois at Urbana Champaign Urbana SIGIR 2004 Presented by CHU Huei-Ming 2004/01/17

Outline • Formal Definitions of Heuristic Retrieval Constraints • Analysis of Three Representative Retrieval Formulas – Pivoted Normalization Method – Okapi Method – Dirichlet Prior Method • Experiments • Conclusion and Future Work 2

Formal Definitions of Heuristic Retrieval Constraints • Six intuitive and desirable constraints • Any reasonable retrieval formula should satisfy – – Term Frequency Constraints (TFCs) Term Discrimination Constraints (TDC) Length Normalization Constraints (LNCs) TF-Length Constraints (TF-LNC) 3

Formal Definitions of Heuristic Retrieval Constraints • Term Frequency Constraints (TFCs) – TFC 1: q={w} , Assume |d 1|=|d 2|. If c(w, d 1) > c(w, d 2), then f(d 1, q) > f(d 2, q) – TFC 2: q={w} , Assume |d 1|=|d 2|=|d 3| , c(w, d 1)>0, If c(w, d 2) - c(w, d 1) =1 , c(w, d 3) - c(w, d 2) =1 then f(d 2, q) - f(d 1, q) > f(d 3, q) - f(d 2, q) 4

Formal Definitions of Heuristic Retrieval Constraints • Term Discrimination Constraints (TDC) – Let q be a query , and w 1, w 2 q be two query term – Assume |d 1|=|d 2| , c(w 1, d 1) + c(w 2, d 1)= c(w 1, d 2) + c(w 2, d 2) – If idf(w 1) ≥ idf(w 2) and c(w 1, d 1) > c(w 2, d 2) , then f(d 1, q) ≥ f(d 2, q) 5

Formal Definitions of Heuristic Retrieval Constraints • Length Normalization Constraints (LNCs) – LNC 1 • Let q be a query , d 1 and d 2 are two documents • If some word w’ q , c(w’, d 2) = c(w’, d 1) +1 but for any query term w, c(w, d 2) = c(w, d 1) then f(d 1, q) ≥ f(d 2, q) – LNC 2 • Let q be a query , ∀ k >1 , d 1 and d 2 are two documents • If |d 1| = k · |d 2| and for all terms w , c(w, d 1) = k · c(w, d 2), • then f(d 1, q) ≥ f(d 2, q). 6

Formal Definitions of Heuristic Retrieval Constraints • TF-Length Constraints (TF-LNC) – q={w}, d 1 and d 2 are two documents – If c(w, d 1) > c(w, d 2) and |d 1|=|d 2| + c(w, d 1) - c(w, d 2) – then f(d 1, q) > f(d 2, q) 7

Formal Definitions of Heuristic Retrieval Constraints 8

Analysis of Three Representative Retrieval Formulas • Pivoted Normalization Method • Okapi Method • Dirichlet Prior Method 9

Analysis of Three Representative Retrieval Formulas Pivoted Normalization Method • Retrieval function • Analyzing 10

Analysis of Three Representative Retrieval Formulas Pivoted Normalization Method • Check TF-LNC constraint when |d 1|=avdl , it is equivalent to the • TF-LNC is satisfied only if s is blow a certain upper bound 11

Analysis of Three Representative Retrieval Formulas Pivoted Normalization Method • Check the LNC 2 constraint 12

Analysis of Three Representative Retrieval Formulas Pivoted Normalization Method • Consider common case when |d 2|=avdl • Performance can be bad for a large s 13

Analysis of Three Representative Retrieval Formulas Pivoted Normalization Method • Check TDC constraint – It is equivalent to c(w 2, d 1) ≥ c(w 1, d 2) this is conditional satisfied 14

Analysis of Three Representative Retrieval Formulas Okapi Method • Retrieval function • k 1 (between 1. 0~2. 0 ) b (usually 0. 75) and k 3 (between 0 ~1000) 15

Analysis of Three Representative Retrieval Formulas Okapi Method • Analysis – When df(w)> N/2 , the IDF part in the formula will be a negative value – When the IDF part is positive (mostly true for keyword query) – TFC and LNCs are satisfied – TF-LNC constraint : considering a common case when |d 2|=avdl the constraint is equivalent to b ≤ avdl / c(w, d 2) – TDC is equivalent to c(w 2, d 1) ≥ c(w 1, d 2) same as the formula above 16

Analysis of Three Representative Retrieval Formulas Okapi Method • Modify Okapi Method – Solve the problem of negative IDF – Replace the original IDF in Okapi with the regular IDF in the pivoted normalization formula – The performance is better on the verbose queries • Analysis result 17

Analysis of Three Representative Retrieval Formulas Dirichlet Prior Method • Retrieval function • Use Dirichlet prior smoothing method to smooth a document language model • Rank the documents according to the likelihood of the query according to the estimated language model of each document 18

Analysis of Three Representative Retrieval Formulas Dirichlet Prior Method • Analysis – LNC 2 constraint is equivalent to c(w , d 2) ≥ |d 2| p(w|C) • Which is usually satisfied for content-carrying words – TDC constraint led to some lower bound for parameter 19

Analysis of Three Representative Retrieval Formulas Dirichlet Prior Method • Analysis – TDC : consider a common case of w 2 , p(w 2|C)=1/avdl – Means for discriminative words with a high term frequency in a document , needs to be sufficiently large – In order to balance the TF and IDF appropriately 20

Experiments Setup • Document set – AP: news article , DOE: technical report, FR: government documents, – ADF : combination of AP, DOE, FR – Web: web data used in the TREC 8 – Trec 7: ad hoc data used in the TREC 7 – Trec 8: ad hoc data used in the TREC 8 21

Experiments Setup • Query combination – – Short-keyword (SK, keyword title) Shot-verbose (SV, one sentence description) Long-keyword (LK, keyword list) Long-verbose (LV, multiple sentences) • Preprocessing – Only stemming with the Porter’s stemmer – No stop words have been removed 22

Experiments Parameter Sensitivity • Pivoted normalization method • The analysis of LNC 2 constraint for the pivoted normalization methods suggests the s should be smaller than 0. 4 23

Experiments Parameter Sensitivity • Okapi method k 1 =1. 2, k 3 =1000, b changes from 0. 1 to 1. 0 24

Experiments Parameter Sensitivity • Dirichlet prior method 25

Experiments Parameter Sensitivity • Dirichlet prior method 26

Experiments Performance Comparison 27

Experiments Performance Comparison • For any query type, the performance of Dirichlet prior method is comparable to pivoted normalization method • For keyword queries, the performance of Okapi is comparable to the other two retrieval formulas • For verbose queries, the performance of Okapi may be worse than others due to the possible negative IDF part in the formula 28

Experiments Performance Comparison • Average precision comparison 29

Conclusion and Future Work • Define six basic constraints that any reasonable retrieval function should satisfy • When the constraints is not satisfied, it often indicates nonoptimality of the method 30