Modern Information Retrieval Chapter 1 Introduction Ricardo BaezaYates

Modern Information Retrieval Chapter 1: Introduction Ricardo Baeza-Yates Berthier Ribeiro-Neto 1

Motivation n Example of the user information need Ø Ø Ø Topic: NCAA college tennis team Description: Find all the pages (documents) containing information on college tennis teams which (1) are maintained by an university in the USA and (2) participate in the NCAA tennis tournament. Narrative: To be relevant, the page must include information on the national ranking of the team in the last three years and the email or phone number of the team coach. 2

IR Research n Information retrieval vs Data retrieval n Research Ø Ø Ø information search information filtering (routing) document classification and categorization user interfaces and data visualization cross-language retrieval 3

IR History n 1970 n 1990, WWW 4

The User Task n Retrieval (Searching) Ø n classic information search process where clear objectives are defined Browsing Ø a process where one’s main objectives are not clearly defined and might change during the interaction with the system 5

Logical View of the Documents n Text Operations Ø Ø n reduce the complexity of the document representation a full text a set of index terms Steps 1. 2. 3. 4. Stopwords removing Stemming Noun groups. . . 6

Past, Present, and Future n Early Development Ø n Library Ø n Index Author name, title, subject headings, keywords The Web and Digital Libraries Ø Hyperlinks 7

Conventional Text-Retrieval Systems Automatic Text Processing G. Salton, Addison-Wesley, 1989. (Chapter 9) 8

Data Retrieval n A specified set of attributes is used to characterize each record. EMPLOYEE(NAME, SSN, BDATE, ADDR, SEX, SALARY, DNO) n Exact match between the attributes used in query formulations and those attached to the document. SELECT BDATE, ADDR FROM EMPLOYEE WHERE NAME = ‘John Smith’ 9

Text-Retrieval Systems n n Content identifiers (keywords, index terms, descriptors) characterize the stored texts. Degrees of coincidence between the sets of identifiers attached to queries and documents query formulation content analysis 10

Possible Representation n Document representation (Text operation) Ø Ø Ø n Query (Query operation) Ø Ø Ø n unweighted index terms (term vectors) weighted index terms … unweighted or weighted index terms Boolean combinations (or, and, not) … Search operation must be effective Ø (Indexing) 11

File Structures n Main requirements Ø Ø n fast-access for various kinds of searches large number of indices Alternatives Ø Ø Ø Inverted Files Signature Files PAT trees 12

Inverted Files n File is represented as an array of indexed documents. 13

Inverted-file process n The document-term array is inverted (transposed). 14

Inverted-file process (Continued) n Take two or more rows of an inverted term-document array, and produce a single combined list of document identifiers. n Ex: Query= (term 2 and term 3) term 2 1 1 0 0 term 3 0 1 1 1 ---------------------------1 <-- D 2 15

List-merging for two ordered lists n n The inverted-index operations to obtain answers are based on list-merging process. Example T 1: {D 1, D 3} T 2: {D 1, D 2} Merged(T 1, T 2): {D 1, D 2, D 3} 16

Extensions of Inverted Index Operations (Distance Constraints) n Distance Constraints Ø Ø (A within sentence B) terms A and B must co-occur in a common sentence (A adjacent B) terms A and B must occur adjacently in the text 17

Extensions of Inverted Index Operations (Distance Constraints) n Implementation Ø Ø include term-location in the inverted indexes information: {P 345, P 348, P 350, …} retrieval: {P 123, P 128, P 345, …} include sentence-location in the indexes information: {P 345, 25; P 345, 37; P 348, 10; P 350, 8; …} retrieval: {P 123, 5; P 128, 25; P 345, 37; P 345, 40; …} 18

Extensions of Inverted Index Operations (Distance Constraints) Ø Ø Ø Include paragraph numbers in the indexes sentence numbers within paragraphs word numbers within sentences information: {P 345, 2, 3, 5; …} retrieval: {P 345, 2, 3, 6; …} Query examples (information adjacent retrieval) (information within five words retrieval) Cost: the size of indexes 19

Retrieval models Classic Models Boolean Vector Probabilistic Set Theoretic Fuzzy Extended Boolean Algebraic Generalized Vector Latent Semantic Index Neural Networks Probabilistic Inference Network Belief Network 20

Classic IR Model n Basic concepts : Each document is described by a set of representative keywords called index terms. n Assign a numerical weights to distinct relevance between index terms. 21

Boolean model n n n Binary decision criterion Data retrieval model Advantage Ø n clean formalism, simplicity Disadvantage Ø Ø It is not simple to translate an information need into a Boolean expression. exact matching may lead to retrieval of too few or too many documents 22

Vector model n Assign non-binary weights to index terms in queries and in documents. => TFx. IDF n Compute the similarity between documents and query. => Sim(Dj, Q) n More precise than Boolean model. 23

Term Weights n Term Weights Di={Ti 1, 0. 2; Ti 2, 0. 5; Ti 3, 0. 6} n Issues Ø Ø How to generate the term weights? How to apply the term weights? • Sum the weights of all document terms that match the given query. • Rank the output documents in the descending order of term weight. 24

Boolean Query with Term Weights n n n Transform a Boolean expression into disjunctive normal form. T 1 and (T 2 or T 3) = (T 1 and T 2) or (T 1 and T 3) For each conjunct, compute the minimum term weight of any document term in that conjunct. The document weight is the maximum of all the conjunct weights. 25

Boolean Query with Term Weights n Example: Q=(T 1 and T 2) or T 3 Document Vectors Conjunct Weights (T 1 and T 2) D 1=(T 1, 0. 2; T 2, 0. 5; T 3, 0. 6) (T 1 and T 2) or T 3 0. 2 0. 6 0. 2 0. 1 0. 2 D 2=(T 1, 0. 7; T 2, 0. 2; T 3, 0. 1) D 1 is preferred. (T 3) Query Weight 26

Summary n n n Conventional IR systems Evaluation Text operations (Term selection) Query operations (Pattern matching, Relevance feedback) Indexing (File structure) Modeling 27

Resources n Journals Ø Ø Ø n Journal of American Society of Information Sciences ACM Transactions on Information Systems Information Processing and Management Information Systems (Elsevier) Knowledge and Information Systems (Springer) Conferences Ø Ø ACM SIGIR, DL, CIKM, CHI, etc. Text Retrieval Conference (TREC) 28