The Re Search Engine Jaime Teevan MIT CSAIL

The Re: Search Engine Jaime Teevan MIT, CSAIL

“Pick a card, any card. ”

Case 1 Case 2 Case 3 Case 4 Case 5 Case 6

Your Card is GONE!

People Forget a Lot

Change Blindness http: //www. usd. edu/psyc 301/Change. Blindness. htm

Change Blindness http: //www. usd. edu/psyc 301/Change. Blindness. htm

Re: Search Engine ?

Merge Old and New Results Old Merged New

We still need magic!

Overview ♠ Memorability study ♠ Recognition study ♠ Assumptions ♠ Implementation issues ♠ Evaluation issues ♠ Choose your own adventure

Memorability Study ♠ Participants issued self-selected query ♠ After an hour, asked to fill out a survey ♠ 129 people remembered something

Data Analysis ♠ Probability of being remembered ♥ Anything? # of words? # of fields? ♥ Features ♣ Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc. ♣ Query features: query type, query length, # of search in session, elapsed time, etc. ♠ Remembered rank v. real rank ♥ Map remembered rank to real rank

“Memorability”

Remembered Results Ranked High

Recognition Study ♠ Same set-up as Memorability Study ♠ Follow-up survey: Results the same? ♥ Case 1: New results 19% ♥ Case 2: Random 4 same 38% ♥ Case 3: Clicked to top 41% ♥ Case 4: Same results 66% ♥ Case 5: Intelligent merging 81% ♠ 165 people completed both steps

Assumptions ♠ Re-search v. search ♠ Memorable v. relevant ♠ Results change v. stay the same ♠ Hide change v. show change ♠ Forget v. remember as forgettable ♠ Merge v. identify old or new Why? How to test? What if I’m wrong?

Implementation Issues ♠ Page of cached result may disappear ♠ Multiple result pages ♠ Identifying repeat queries ♥ Exact query may be forgotten ♥ User identified ♥ Search sessions are not repeat queries

Evaluation Issues ♠ Various goals to test ♥ Does a merged list look like the original? ♥ Does merging make re-finding easier? ♥ Is search improved overall? ♠ Lab study ♥ How to set up re-finding task? ♥ Timing differences significant enough? ♠ Longitudinal study – What to measure? ♠ What are good baselines?

Choose Your Own Adventure ♠ Re-search v. search ♠ Memorable v. relevant ♠ Results change v. stay the same ♠ Hide change v. show change ♠ Forget v. remember as forgettable ♠ Merge v. identify old or new ♠ Implementation issues ♠ Evaluation issues

Choose Your Own Adventure ♠ Re-search v. search ♠ Memorable v. relevant ♠ Results change v. stay the same ♠ Hide change v. show change ♠ Forget v. remember as forgettable ♠ Merge v. identify old or new ♠ Implementation issues ♠ Evaluation issues (Done)

Hide Change v. Show Change ♠ Why I think change should be hidden ♥ Example: dynamic menus ♠ How to prove ♥ New results better, called the same or worse ♥ Baseline for testing – 2 lists, change explicit ♠ What if we should show change? ♥ Memorability suggests changes to highlight ♥ Other applications where want to hide change (Done)

Memorable v. Relevant ♠ Why I think memorability is important ♥ Relevance at a future date is what matters ♥ Necessary to hide change ♠ How to prove ♥ Baseline for lab study with target first ♠ What if relevance is what’s important? ♥ Mapping between memorable and relevant ♥ Useful related work on implicit feedback (Done)

Re-search v. Search ♠ Why I think people repeat searches ♥ Information seeking literature ♥ Re-finding consistently reported as a problem ♠ How to prove ♥ Study shows prefer to follow known paths ♥ Search log analysis ♠ What if people just want to search? ♥ Memorable results ranked first ♥ Other domains where list consistency matters (Done)

Merge v. Identify Old and New ♠ Why I think results should be merged ♥ Information need not necessarily one or other ♥ People don’t like to do extra work ♠ How to prove ♥ Search log analysis ♥ Look at what people do in longitudinal study ♥ Lab study – timing becomes an issue ♠ What if people want to identify query type? ♥ Other applications where merging is useful (Done)

Results Change v. Stay the Same ♠ Why I think results change ♥ How search engines work ♥ Personalization and dynamic content ♠ How to prove ♥ Track query results ♠ What if results don’t change? ♥ Probably will in future applications ♥ Existing applications where lists change (Done)

Forget v. Remember as Forgettable ♠ Why I think people forget ♥ Visual analogy ♠ How to prove ♥ Lab study – Do people find new information? ♥ Longitudinal study – Ever click on new result? ♠ What if remember as forgettable? ♥ Build better model of memorability ♥ Highlight important changes (Done)

Implementation Issues ♠ Page of cached result may disappear ♠ Multiple result pages ♠ Identifying repeat queries ♥ User identified ♥ Search sessions are not repeat queries ♥ Exact query may be forgotten (Done)

Evaluation Issues ♠ Various goals to test ♥ Does a merged list look like the original? ♥ Does merging make re-finding easier? ♥ Is search improved overall? ♠ Lab study ♥ How to set up re-finding task? ♥ Timing differences significant enough? ♠ Longitudinal study – What to measure? ♠ What are good baselines? (Done)

Jaime Teevan teevan@mit. edu

Strategies for Finding Teleporting Orienteering

Why Do People Orienteer? ♠ The tools don’t work ♠ Easier than saying what you want ♠ You know where you are ♠ You know what you find

Structural Consistency Important All must be the same to re-find the information! New name

Absolute Consistency Unnecessary New name Focus on search result lists

Query Changes ♠ Most changes are simple ♥ Capitalization ♥ Phrasing ♥ Word ordering ♥ Word form ♥ New queries shorter ♠ What about longer time horizons? ♠ Recognition v. recall

Result List Changes ♠ Tracked 10 queries on Google for a year+ ♠ 1. 18 of top 10 disappear each week ♠ Rate of change likely to increase, e. g. : ♥ Raw personalization ♥ Relevance feedback ♠ People forget their queries ♥ 28% of queries forgotten within an hour

Example: “neon signs”