Personalizing Web Search Jaime Teevan MIT with Susan

Personalizing Web Search Jaime Teevan, MIT with Susan T. Dumais and Eric Horvitz, MSR

Demo

Personalizing Web Search l Motivation l Algorithms l Results l Future Work

Study of Personal Relevancy l 15 SIS users x ~10 queries l Evaluate 50 results ¡ Highly l Query relevant / Relevant / Irrelevant selection ¡ Previously issued query ¡ Chose from 10 pre-selected queries l Collected ¡ 53 evaluations for 137 queries of pre-selected queries (2 -9/query)

Relevant Results Have Low Rank Highly Relevant Irrelevant

Same Query, Different Intent l Different meanings ¡ “Information about the astronomical/astrological sign of cancer” ¡ “information about cancer treatments” l Different ¡ “is intents there any new tests for cancer? ” ¡ “information about cancer treatments”

Same Intent, Different Evaluation l Query: Microsoft ¡ “information about microsoft, the company” ¡ “Things related to the Microsoft corporation” ¡ “Information on Microsoft Corp” l 31/50 ¡ Only rated as not irrelevant 6/31 do more than one agree ¡ All three agree only for www. microsoft. com

More to Understand l Do people cluster? ¡ Even if they can’t state their intention l How are the differences reflected? ¡ Can they be seen from the information on a person’s computer? l Can we do better than the ranking that would make everyone the most happy? ¡ Best common ranking: +38% ¡ Best personalized ranking: +55%

Personalizing Web Search l Motivation l Algorithms l Results l Future Work

Personalization Algorithms l Standard IR Query Document Server Client User l Related to relevance feedback l Query expansion v. Result re-ranking

Result Re-Ranking l Takes full advantage of SIS l Ensures privacy l Good evaluation framework l Look at light weight user models ¡ Collected on server side ¡ Sent as query expansion

BM 25 with Relevance Feedback Score = Σ tfi * wi N ni ri R wi = log N ni

BM 25 with Relevance Feedback Score = Σ tfi * wi N ni ri R wi = log (ri+0. 5)(N-ni-R+ri+0. 5) (ni-ri+0. 5)(R-ri+0. 5)

User Model as Relevance Feedback Score = Σ tfi * wi N R ni ri wi = log N’ = N+R ni’ = ni+ri (r(ri+0. 5)(N’-n -R+ri+0. 5)(N-ni’-R+r i+0. 5) (n (nii’-- rrii+0. 5)(R-rii+0. 5)

User Model as Relevance Feedback World Score = Σ tfi * wi N User R ni ri

User Model as Relevance Feedback World Score = Σ tfi * wi N User World related to query R ri ni ni N

User Model as Relevance Feedback World Score = Σ tfi * wi N User World related to query R ri ni ni N ri R User related to query Query Focused Matching

User Model as Relevance Feedback World Focused Matching Score = Σ tfi * wi N User Web related to query R ri ni ni N ri R User related to query Query Focused Matching

Parameters l Matching l User representation l World representation l Query expansion

Parameters l Matching l User Query focused World focused representation l World representation l Query expansion

User Representation l Stuff I’ve Seen (SIS) index l Recently indexed documents l Web documents in SIS index l Query history l Relevance judgments l None

Parameters l Matching l User Query focused World focused representation l World representation l Query expansion All SIS Recent SIS Web SIS Query history Relevance feedback None

Parameters l Matching l User Query Focused World Focused representation l World representation l Query expansion All SIS Recent SIS Web SIS Query History Relevance Feedback None

World Representation l Document Representation ¡ Full text ¡ Title and snippet l Corpus Representation ¡ Web ¡ Result set – title and snippet ¡ Result set – full text

Parameters l Matching l User Query focused World focused representation l World l Query representation expansion All SIS Recent SIS Web SIS Query history Relevance feedback None Full text Title and snippet Web Result set – full text Result set – title and snippet

Query Expansion l All words in document The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through. . . l Query focused The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through. . .

Parameters l Matching l User Query focused World focused All SIS Recent SIS Web SIS Query history Relevance feedback None representation l World l Query Full text Title and snippet representation expansion Web Result set – full text Result set – title and snippet All words Query focused

Personalizing Web Search l Motivation l Algorithms l Results l Future Work

Baselines l Best possible l Random l Text based ranking l Web ranking l URL Boost http: //mail. yahoo. com/inbox/msg 10 +1

Best Parameter Settings l Richer user representation better ¡ SIS > Recent > Web > Query History > None Suggests rich client important l Efficiency hacks don’t hurt ¡ Snippets query focused ¡ Length normalization not an issue l Query focus good

Text Alone Not Enough l Better than some baselines ¡ Better than random ¡ Better than no user representation ¡ Better than relevance feedback l Worse than Web results l Blend in other features ¡ Web ranking ¡ URL boost

Good, but Lots of Room to Grow l Best combination: 9. 1% improvement l Best possible: 51. 5% improvement l Assumes best Web combination selected l Only improves results 2/3 of the time

Personalizing Web Search l Motivation l Algorithms l Results l Future Work

Finding the Best Parameter Setting l Almost always some parameter setting that improves results l Use learning to select parameters ¡ Based on individual ¡ Based on query ¡ Based on results l Give user control?

Further Exploration of Algorithms l Larger parameter space to explore ¡ More complex user model subsets ¡ Different parsing (e. g. , phrases) ¡ Tune BM 25 parameters l What is really helping? ¡ Generic user model or personal model ¡ Use different indices for the queries l Deploy system

Practical Issues l Efficiency ¡ Can interfaces mitigate some of the issues? l Merging ¡ Query l issues server and client expansion Get more relevant results in the set to be re-ranked ¡ Design snippets for personalization

Thank you!