The TREC Conferences http trec nist gov Ellen

The TREC Conferences http: //trec. nist. gov Ellen Voorhees Text REtrieval Conference (TREC)

TREC Philosophy • TREC is a modern example of the Cranfield tradition – system evaluation based on test collections • Emphasis on advancing the state of the art from evaluation results – TREC’s primary purpose is not competitive benchmarking – experimental workshop: sometimes experiments fail! Text REtrieval Conference (TREC)

Cranfield at Fifty • Evaluation methodology is still valuable… • carefully calibrated level of abstraction – has sufficient fidelity to real user tasks to be informative – general enough to be broadly applicable, feasible, relatively inexpensive • …but is showing some signs of age • size is overwhelming our ability to evaluate • new abstractions need to carefully accommodate variability to maintain power Text REtrieval Conference (TREC)

Evaluation Difficulties • Variability • despite stark abstraction, user effect still dominates Cranfield results • Size matters • effective pooling has corpus size dependency • test collection construction costs depend on number of judgments • Model coarseness • even slightly different tasks may not be good fit – e. g. , legal discovery, video features Text REtrieval Conference (TREC)

TREC 2009 • All tracks used some new, large document set • Different trade-offs in adapting evaluation strategy • tension between evaluating current participants’ ability to do the task and building reusable test collections • variety of tasks that are not simple ranked-list retrieval Text REtrieval Conference (TREC)

Clue. Web 09 Document Set • Snapshot of the WWW in early 2009 • crawled by CMU with support from NSF • distributed through CMU • used in four TREC 2009 tracks: Web, Relevance Feedback, Million Query, and Entity • Full corpus • about one billion pages and 25 terabytes of text • about half is in English • Category B • English-only subset of about 50 million pages (including Wikipedia) to permit wider participation Text REtrieval Conference (TREC)

TREC 2009 Participants Applied Discovery Beijing Institute of Technology Beijing U. of Posts and Telecommunications Cairo Microsoft Innovation Center Carnegie Mellon University Chinese Academy of Sciences (2) Logik Systems, Inc. Microsoft Research Asia Microsoft Research Cambridge University of Applied Science Geneva University of Arkansas, Little Rock University of California, Santa Cruz Milwaukee School of Engineering Mugla University National Institute of Information and Communications Technology University of Delaware (2) University of Glasgow University of Illinois, Urbana-Champaign Clearwell Systems, Inc. Clearly Gottlieb Steen & Hamilton, with Backstop LLC Dalian University of Technology Delft University of Technology EMC - CMA - R&D Equivio Fondazione Ugo Bordoni Fraunhofer SCAI Fudan University H 5 Heilongjiang Inst. of Technology Integreon International Inst. of Information Technology, Hyderabad Northeastern University Open Text Corporation University of Iowa University of Lugano Peking University Pohang U. of Science & Technology Purdue University Queensland University of Technology RMIT University Sabir Research South China University of Technology SUNY Buffalo Tsinghua University Universidade do Porto University College Dublin University of Maryland, College Park University of Massachusetts, Amherst The University of Melbourne University of Padova University of Paris University of Pittsburgh University of Twente University of Waterloo (2) Ursinus College Yahoo! Research York University (2) Know-Center Lehigh University of Alaska, Fairbanks University of Amsterdam (2) ZL Technologies, Inc. Text REtrieval Conference (TREC)

The TREC Tracks Personal documents Blog Spam Retrieval in a domain Chemical IR Genomics Answers, not documents Novelty QA, Entity Searching corporate repositories Legal Enterprise Size, efficiency, & web search Terabyte, Million Query Web VLC Beyond text Video Speech OCR Cross-language Chinese Spanish Beyond just English Interactive, HARD, Feedback Human-in-the-loop Filtering Routing Static text Ad Hoc, Robust 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Streamed text Text REtrieval Conference (TREC)

TREC 2010 • Blog, Chemical IR, Entity, Legal, Relevance Feedback, Web continuing • Million Query merged with Web • New “Sessions” track: investigate search behavior over a series of queries (series of length 2 for first running in 2010) Text REtrieval Conference (TREC)

TREC 2011 • Track proposals due Monday (Sept 27) • New track on searching free text fields of medical records likely Text REtrieval Conference (TREC)