Relevance in information science Tefko Saracevic Ph D

Relevance in information science Tefko Saracevic, Ph. D tefko@scils. rutgers. edu http: //www. scils. rutgers. edu/~tefko/ © 2008 Tefko Saracevic 1

Preface 1975 & 1976 n Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science 2007 n Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II & part III © 2008 Tefko Saracevic Relevance 2

Purpose trace the evolution of thinking on relevance in information science for the past three decades provide an updated framework within which the still widely dissonant ideas on relevance might be interpreted and related to one another © 2008 Tefko Saracevic Relevance 3

… in information science Information retrieval (IR) systems offer their version of what may be relevant People go their way & asses relevance The two worlds interact Concern here: n n human world of relevance how IR deals with relevance NOT covered © 2008 Tefko Saracevic Relevance 4

Historical note Relevance came into IR unannounced From very start after WWII IR was about retrieval of relevant information First concerns: about “false drops” – non relevant retrievals First direct recognition in 1955 with proposal for precision & recall as evaluation measures based on relevance © 2008 Tefko Saracevic Relevance 5

Organization of presentation 1. What is the NATURE of relevance? n 1. 1 Meaning? n 1. 2 Theories? n 1. 3 Models? 2. What are MANIFESTATIONS of relevance? 3. What is the BEHAVIOR of relevance? 4. What are the EFFECTS of relevance? © 2008 Tefko Saracevic Relevance 6

1. 1 Meaning of relevance Intuitively well understood n n same perception globally – “y’know” a “to” and context always present Relevance: n n a relation between objects P & Q along property R may also include a measure S of the strength of connection © 2008 Tefko Saracevic Relevance 7

… in information science Relation between inf. or inf. objects (Ps) & contexts (Qs) based on some property R & measure of intensity S But: relevance is not given – it is established Big questions & challenges n n How does relevance happen? Who does it, under what circumstances, and how? © 2008 Tefko Saracevic Relevance 8

Summary: Meaning of relevance Relevance attributes involve: n n n relation intention context w internal; external n n inference selection interaction measurement © 2008 Tefko Saracevic Relevance 9

1. 2 Theories of relevance Theories suggested in several fields n n logic – in deduction of inferences, to reject fallacies – relevance logics philosophy – in phenomenology structure & functioning of the “life-world” (Alfred Schutz) w it is stratified & relevance is the principle for stratification w no single relevance but an interdependent system of relevances (plural) w thematic (topical), interpretational, & motivational relevance © 2008 Tefko Saracevic Relevance 10

… theories in communication Sperber & Wilson: Relevance Theory n n based on inferential model of communication what must be relevant and why to an individual with a single cognitive intention? posited a cognitive & a communicative principle of relevance assessed in terms of cognitive effects & processing effort w individuals pick up the most relevant stimuli & process them to maximize their relevance © 2008 Tefko Saracevic Relevance 11

Summary: Theories of relevance IS did not develop indigenous theory but a few “theories-on-loan” attempts n n n Logic theories not applied, yet Schutz’s life-world theory used to some extend in specifying manifestations & models Sperber & Wilson’s theory used few times as explanation & guide w not tested © 2008 Tefko Saracevic Relevance 12

1. 3 Models of relevance Reviews – also produced models Syracuse school of relevance n dynamic & situational model & Nilan, 1990) (Schamber, Eisenberg w connection with human information behavior “Whole history of relevance” n (Mizzaro, 1997) duality in modeling & studying w documents & queries, 1959 -1976 w dynamics & multidimensionality, 1977 -present © 2008 Tefko Saracevic Relevance 13

Split between system & user models Opposing views of IR: systems & users w IR traditional model does not deal with users Battle royal started by Dervin & Nilan (1986) w criticism of system viewpoint w call for user orientation Several user & interaction models proposed – to reconcile, bridge n still relevance has two basic models & cultures & they map like Australia © 2008 Tefko Saracevic Relevance 14

User vs system models “Informing systems design” n became mantra of all relevance studies “Tell us what to do and we will do it. ” n response from systems side But “telling” is not that simple Issue is not a conflict but: n how can we make user & system side work together for benefit of both? © 2008 Tefko Saracevic Relevance 15

Summary: relevance models All IR & inf. seeking models have relevance at their base Traditional IR model has most simplified –“weak”- version of relevance n but with the weak model IR is successful Variety of integrative models have been proposed n more complex models = increased challenge to incorporate in practice © 2008 Tefko Saracevic Relevance 16

© 2008 Tefko Saracevic Relevance 17

2. Manifestations of relevance “How many relevances in IR? ” (Mizzaro, 1998) w Several manifestations recognized since 1950 s w Issue: What given objects (Ps & Qs) are related by what given property (Rs) as relation? w [adjective] relevance or different name Duality strikes again n n subject (topic, system) relevance vs user (psychological, cognitive) relevance objective vs subjective relevance © 2008 Tefko Saracevic Relevance 18

Issue of primacy: weak and strong relevance Does topical relevance underlie all others? w predictably two answers: yes and no n n in a strict correspondence between query & answer topical is basic – weak relevance if derivation is involved topical may not be basic – strong relevance w weak relevance is more associated with systems w strong more with people © 2008 Tefko Saracevic Relevance 19

Beyond duality Numerous other kinds of relevance were identified n for user relevances: w psychological, cognitive, affective, situational, socio-cognitive, pertinence, utility … n for topical relevances: w logical, systems, algorithmic, documentary, bibliographic … n each indicates different relations © 2008 Tefko Saracevic Relevance 20

Summary: Manifestations of relevance There a limited number of manifestations – could be grouped: n n n System or algorithmic relevance Topical or subject relevance Cognitive relevance or pertinence Situational relevance or utility Affective relevance These are interdependent – they feed on each other interactively © 2008 Tefko Saracevic Relevance 21

3. Behavior of relevance Relevance does not behave, people do n n how humans determine relevance of information or information objects? reviewed only studies that have data w some 30 studies – started in 1991 w related to information seeking & use studies & implicit relevance studies – not reviewed Pattern for review: w [author] used [subjects] to do [tasks] in order to study [object of research] © 2008 Tefko Saracevic Relevance 22

Relevance clues What makes information or information objects relevant? What do people look for in order to infer relevance? w two approaches: topic & clues analysis Clues research: w uncover & classify attributes or criteria that users concentrate on while making relevance inferences w usually on documents but also other objects © 2008 Tefko Saracevic Relevance 23

Relevance dynamics Do relevance inferences and criteria change over time for the same user and task, and if so, how? As a user progresses through various stages of a task n n n the user’s cognitive state changes the task changes as well thus, something about relevance also is changing. © 2008 Tefko Saracevic Relevance 24

Relevance feedback What factors affect the process of relevance feedback? n What types of feedback? How much is feedback used? Dealing here with manual not automatic feedback n behavior of people when involved in manual feedback © 2008 Tefko Saracevic Relevance 25

Summary on behavior Caveats abound – nothing standardized n still refreshing to see data Some generalizations on clues: n criteria finite in number, similar, but different weights assigned w Different users, tasks, progress in tasks, classes of users = similar criteria = different weights w Different ratings of relevance = similar criteria = different weights. © 2008 Tefko Saracevic Relevance 26

…summary clues Clues criteria: n n n n content object validity use or situational match cognitive match affective match belief match © 2008 Tefko Saracevic Relevance 27

… summary clues Criteria are not independent; people apply multiple criteria; they interact n n content (topic) criteria very important but not sole – interact with others for search outputs value of results as a whole critical Visual information = faster inference than textual information © 2008 Tefko Saracevic Relevance 28

… summary dynamics Inferences dependent on task stage n n criteria stable, selection changes Different stages = differing selections but different stages = similar criteria = different weights Increased focus = increased discrimination = more stringent relevance inferences What is topical changes with progress in time and task © 2008 Tefko Saracevic Relevance 29

… summary feedback Several kinds n search term, content, magnitude, tactics Use of relevance feedback = increase in performance n however, used rarely in practice Searching behavior different when using feedback © 2008 Tefko Saracevic Relevance 30

5. Effects of relevance Works both ways: relevance affected by and affects host of factors Relevance judges n n What factors inherent in relevance judges make a difference in relevance inferences? How large are and what affects individual differences in relevance inferences? w similar question asked for a number of information activities – indexing, searching … w most often studied: domain knowledge © 2008 Tefko Saracevic Relevance 31

Relevance judgments What factors affect relevance judgments? n n short answer: a lot of them approach: classify into tables e. g. w Schamber (1994) 80 factors, 6 categories w Harter (1996) 24 factors, 4 categories n different approach here: w classify studies along basic assumptions in IR evaluations © 2008 Tefko Saracevic Relevance 32

Central assumptions Relevance is: n n n topical binary independent stable consistent if pooling: complete Not to prove or disprove these assumptions but to organize studies along questions © 2008 Tefko Saracevic Relevance 33

Beyond topicality Do people infer relevance based on topicality only? Other factors enter & interact Only a few studies directly addressed w Wang & Soergel (1998) 11 criteria for selection, with topicality being top w Xu & Chen (2006) in web searching: topicality & novelty most significant, then reliability & understandability © 2008 Tefko Saracevic Relevance 34

Beyond binary Are relevance inferences binary i. e. relevant – not relevant? If not, what gradation do people use in inferences about relevance of information or information objects? Number of studies addressed this n n studied distributions of relevance inferences regions of relevance © 2008 Tefko Saracevic Relevance 35

Beyond independence Are information objects assessed independently of each other? Does the order or size of the presentation affect relevance judgments? n n Only a few studies on the questions Includes presentation of different representations © 2008 Tefko Saracevic Relevance 36

Beyond stability Are relevance judgments stable as tasks and other aspects change? Do relevance inferences and criteria change over time for the same user and task, and if so how? n mentioned already under dynamics w judgments not completely stable, criteria are w Plato: “Everything is flux. ” © 2008 Tefko Saracevic Relevance 37

Beyond consistency Are relevance judgments consistent among judges or group of judges? w human judgments about anything informational are not consistent, relevance included Gull (1956) opened the Pandora box w classic example of law of unintended consequences Some 6 studies addressed consistency n subjects: experts, students © 2008 Tefko Saracevic Relevance 38

But does it matter? How does inconsistency in human relevance judgments affect results of IR evaluation? n main contention by critics Five studies 1968 – 2000 addressed the question n four also showed magnitude of agreement © 2008 Tefko Saracevic Relevance 39

Summary: judges Subject expertise accounts strongly w higher expertise = higher agreement, less differences w lesser expertise = more leniency in judgment n large variability in relevance inferences by individuals w same range as in other cognitive processes © 2008 Tefko Saracevic Relevance 40

Summary: judgments Relevance is measurable! None of the 5 postulates hold n but by simplifying relevance for labs IR made significant advances Relevance judgments are not binary but are bimodal n regions of low, middle and high relevance w high peaks at both end Order affect relevance judgment © 2008 Tefko Saracevic Relevance 41

… summary: judgments Consistency: w Higher expertise = higher consistency = more stringent. Lower expertise = lower consistency= more encompassing w overlap using different populations hovers around 30% w higher expertise up to 80% w when 3 rd, 4 th … judge added overlap falls w Higher expertise =larger overlap. Lower expertise =smaller overlap. More judges = less overlap. © 2008 Tefko Saracevic Relevance 42

Summary: does it matter? In lab conditions disagreement among judges does not affect evaluation n rank order of different IR systems changes minimally w Different judges = same relative performance (on the average) n n swaps in ranking do occur = low probability but performance for individual topics differs significantly w law of averages kicks in © 2008 Tefko Saracevic Relevance 43

Summary: measures Users can use a variety of scales n n there is no “best” scale magnitude scales very appropriate but hard to explain & analyze © 2008 Tefko Saracevic Relevance 44

Epilogue Many things changed in IR & information science but goals the same As to nature of relevance: w marked progress in understanding w little in theory w diversification in models As to manifestations: w consensus there are several kinds of relevance, grouped in a half dozen or so well distinguished classes - interdependent © 2008 Tefko Saracevic Relevance 45

… epilogue As to behavior & effects: n seen a number of experimental & observational studies w lifted the discourse beyond debate, anecdotes to data interpretation w but generalizations difficult – findings should be treated as hypotheses © 2008 Tefko Saracevic Relevance 46

… epilogue: reflections Relevance is poor n no funding for relevance research w of studies with data less than 17% mentioned outside funding, half from outside the US w scholarship progressed sporadically & all over the place Globalization of IR – globalization of relevance n as relevance went global & to the masses many & different research questions emerged © 2008 Tefko Saracevic Relevance 47

… epilogue: reflections… Proprietary IR – proprietary relevance n major search engines proprietary = relevance proprietary w for innovation must study users & use, but findings kept private w relevance research into public & private branch w paradox of the internet © 2008 Tefko Saracevic Relevance 48

… epilogue: research agenda beyonds Beyond behaviorism & black box n n many studies stimulus-response & no diagnostics need to go use/adapt other approaches © 2008 Tefko Saracevic Relevance 49

… epilogue: research agenda beyonds Beyond mantra n “implication for system design” w incorporating user concerns & characteristics not that simple w integration between user/cognitive & systems approaches needed w relevance research and IR research should at least get engaged, if not married w interactive research on the right track © 2008 Tefko Saracevic Relevance 50

… epilogue: research agenda beyonds Beyond students n ~70% of behavior & effect studies used students as population w not surprising – they are affordable w we know a lot about student relevance w does it generalize to other populations? © 2008 Tefko Saracevic Relevance 51

In conclusion Information technology & systems will change dramatically n n even in the short run and in unforeseeable directions But relevance is here to stay! © 2008 Tefko Saracevic Relevance 52

© 2008 Tefko Saracevic Relevance 53

© 2008 Tefko Saracevic Relevance 54