Statement Map Assisting Information Credibility Analysis by Visualizing

Statement Map: Assisting Information Credibility Analysis by Visualizing Arguments Koji Murakami†, Eric Nichols†, Suguru Matsuyoshi†, Asuka Sumida†, Shouko Masuda†‡, Kentaro Inui† and Yuji Matsumoto† Nara Institute of Science and Technology† Osaka Prefecture University‡

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Outline 1. Introduction 2. What is Statement Map? 3. The Project 4. Generating Statement Maps 5. Conclusion & Future work 2

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Background The importance of the internet as a source information • overtaking newspapers as a news outlet • rivaling television as a news source for young people Internet: a source of information for important decisions • Health care / Medical information / Large purchases Is Web information reliable enough? ! • Information is incorrect / bad / wrong in many cases People are at risk of believing incorrect information 3

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Supporting Credibility Evaluation Ex) “Do vaccines cause autism? ” Ranking in IR does not reflect any measure of credibility Query : “autism” Google shows us links to the page of … Wiki. Pedia page Legitimate medical organizations Anti-vaccinations movement Query : “vaccine autism” Pro-vaccine: There is no link between vaccines and autism Anti-vaccine: There is a strong link between vaccines and autism • Need to survey a wide range of opinions • Need background knowledge of problem • Too much information to handle 4

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Evaluating Information Credibility 1) Manually factcheck popular question or rumors - snopes. com, factcheck. org, etc. ) -> Coverage problems, users don’t know about sites 2) User Education Campaigns - Sense about Science: education on the importance of the Scientific Method - Credibility Commons: educate the user on how to spot bad information -> Hard for users to remember and apply evaluation methods 3) Direct evaluation of website credibility - Stanford’s Web Credibility Project: investigate impact of website appearance, source verification, etc. on credibility 5

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Evaluating Information Credibility 4) Re-ranking results produced by Search Engine - Reference Extract: rank search results giving preference to sites favored by user-trusted librarians → Trust by proxy, coverage issues We need to consider the CONTENTS too! • Automatically gather opinions and summarize them • Organize into pros vs. cons • Show users evidence supporting each position Mapping various opinions on the Web!!! 6

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Organizing various opinions 7 Query : Do vaccines cause autism? Mercury-based vaccine preservatives actually have caused autism in children. . There is no valid scientific evidence that vaccines cause autism. ”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. The weight of the evidence indicates that vaccines are not associated with autism. NO EVIDENCE exists to link the controversial MMR vaccine and autism …. Retrieve!!

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Organizing various opinions 8 Query : Do vaccines cause autism? Mercury-based vaccine preservatives actually have caused autism in children. . There is no valid scientific evidence that vaccines cause autism. ”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. The weight of the evidence indicates that vaccines are not associated with autism. NO EVIDENCE exists to link the controversial MMR vaccine and autism …. Check contents!!

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Organizing various opinions Query : Do vaccines cause autism? Mercury-based vaccine preservatives actually have caused autism in children. ”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. …. There is no valid scientific evidence that vaccines cause autism. The weight of the evidence indicates that vaccines are not associated with autism. NO EVIDENCE exists to link the controversial MMR vaccine and autism …. Classify!! 9

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Organizing various opinions Query : Do vaccines cause autism? YES! NO! There is no valid scientific evidence that vaccines cause autism. Mercury-based vaccine preservatives actually have caused autism in children. ”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. …. The weight of the evidence indicates that vaccines are not associated with autism. NO EVIDENCE exists to link the controversial MMR vaccine and autism CONFLICT …. Recognize semantic relation!! 10

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion An example of Statement Map Query: Do vaccines cause autism? VACCINES CAUSE AUTISM [FOCUS] ・Mercury-based vaccine preservatives actually have caused autism in children. ・”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. [EVIDENCE] MY CHILD WAS DIAGNOSED WITH AUTISM RIGHT AFTER THE VACCINE ・My son then had the MMR, and then when he was three he was diagnosed with autism. VACCINES DON’T CAUSE AUTISM [CONFLICT] ・There is no valid scientific evidence that vaccines cause autism. ・The weight of the evidence indicates that vaccines are not associated with autism. [EVIDENCE] [CONFLICT] ANECDOTES ARE NOT EVIDENCE ・Vaccinations are given around the same time children can be first diagnosed Mapping arguments : Statement Map 1. a visualization of the semantic relations between statements 2. as a graph which consists of nodes and edges nodes: consist of nearly equivalent statements edges: Semantic relations between nodes 11

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion An example of Statement Map Query: Do vaccines cause autism? VACCINES CAUSE AUTISM [FOCUS] ・Mercury-based vaccine preservatives actually have caused autism in children. ・”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. [EVIDENCE] MY CHILD WAS DIAGNOSED WITH AUTISM RIGHT AFTER THE VACCINE ・My son then had the MMR, and then when he was three he was diagnosed with autism. VACCINES DON’T CAUSE AUTISM [CONFLICT] ・There is no valid scientific evidence that vaccines cause autism. ・The weight of the evidence indicates that vaccines are not associated with autism. [EVIDENCE] [CONFLICT] ANECDOTES ARE NOT EVIDENCE ・Vaccinations are given around the same time children can be first diagnosed A group [FOCUS]: • contains statements closest to the user’s query • statements include opinions that support a causal link between vaccines and autism 12

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion An example of Statement Map Query: Do vaccines cause autism? VACCINES CAUSE AUTISM [FOCUS] ・Mercury-based vaccine preservatives actually have caused autism in children. ・”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. [EVIDENCE] MY CHILD WAS DIAGNOSED WITH AUTISM RIGHT AFTER THE VACCINE ・My son then had the MMR, and then when he was three he was diagnosed with autism. VACCINES DON’T CAUSE AUTISM [CONFLICT] ・There is no valid scientific evidence that vaccines cause autism. ・The weight of the evidence indicates that vaccines are not associated with autism. [EVIDENCE] [CONFLICT] ANECDOTES ARE NOT EVIDENCE ・Vaccinations are given around the same time children can be first diagnosed A relation [CONFLICT]: • contains statements that are in opposition to the statements of [FOCUS] • statements include opinions that opposite a causal link between vaccines and autism 13

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion An example of Statement Map Query: Do vaccines cause autism? VACCINES CAUSE AUTISM [FOCUS] ・Mercury-based vaccine preservatives actually have caused autism in children. ・”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. [EVIDENCE] MY CHILD WAS DIAGNOSED WITH AUTISM RIGHT AFTER THE VACCINE ・My son then had the MMR, and then when he was three he was diagnosed with autism. VACCINES DON’T CAUSE AUTISM [CONFLICT] ・There is no valid scientific evidence that vaccines cause autism. ・The weight of the evidence indicates that vaccines are not associated with autism. [EVIDENCE] [CONFLICT] ANECDOTES ARE NOT EVIDENCE ・Vaccinations are given around the same time children can be first diagnosed A relation [EVIDENCE]: • contains supporting evidence for [FOCUS] statements and [CONFLICT] statements • have mono-directional arrows • may have opposition relation between groups • several clusters may have [EVIDENCE] 14

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Supporting Credibility Evaluation Query: Do vaccines cause autism? VACCINES CAUSE AUTISM ・Mercury-based vaccine preservatives actually have caused autism in children. ・”It’s biologically plausible that the MMR vaccine causes autism. " said Dr. Wakefield. [EVIDENCE] MY CHILD WAS DIAGNOSED WITH AUTISM RIGHT AFTER THE VACCINE ・My son then had the MMR, and then when he was three he was diagnosed with autism. VACCINES DON’T CAUSE AUTISM [CONFLICT] ・There is no valid scientific evidence that vaccines cause autism. ・The weight of the evidence indicates that vaccines are not associated with autism. [EVIDENCE] [CONFLICT] ANECDOTES ARE NOT EVIDENCE ・Vaccinations are given around the same time children can be first diagnosed Enables users to visually: • survey various viewpoints on a topic of interest • compare conflicting opinions with their evidence • see the support or lack thereof for each viewpoint Statement Map helps user come to an informed conclusion 15

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion What is our goal? <Statement Map Generation> • Capturing various implicit information from descriptions on the Web, such as opinions • Sentiment Polarities of expressions • Factuality Modal status of event mentions • Recognizing three important semantic relations between statements on the Web • AGREEMENT to cluster similar statements • CONFLICT to capture differences of opinion • EVIDENCE to support other statements 16

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Capturing opinions 17 <Surveying statements on the Web> • Dealing with opinions is very important • Recognizing attitudinal relations: - Factuality analysis A typology of semantic relations in statement map generation - Attribution analysis Relation Type Relation Label - Sentiment analysis • Defining Semantic relations to be captured for Statement Map Generation Logical Relations (for Facts) Agreement Attitudinal Relations (For Opinions) Agreeing Opinion Contradiction Confinement Conflicting Opinion Agreeing Evaluative Polarity Conflicting Evaluative Polarity Evaluation (+/-/Other) Evidential Relation Evidence

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Related works : Relation recognition 18 （Dagan+ 2005） １．RTE （Recognizing Textual Entailment Challenge） • A hot research topic in the NLP community • Identifying relations between text (t) and hypothesis (h) • Target Relation: <ENTAILMENT>, <CONTRADICTION>, <UNKNOWN> <ENTAILMENT> (t) The fifth World Social Forum (WSF) has kicked off in Porto Alegre, Rio Grande do Sul state, Brazil. (h) The WSF takes place in Brazil. <CONTRADICTION> (t) Councilman Pierce had built the home for his daughter on E. 28 th Street off Rossville Blvd, as he lives nearby. (h) Pierce lives on E. 28 th Street. Three relations are not enough to handle Web arguments! What type of semantic relation can be recognized between (A) and (B)? (A) Vaccines can trigger autism in a vulnerable subset of children. (B) Mercury-based vaccine actually have caused autism in children.

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Related works : Relation recognition （Radev+ 2001） 19 ２．CST （Cross-document Structure Theory） • Expanded rhetorical structure analysis to cross-documents • Target Relations: 18 kinds including <SUBSUMPTION>, <FOLLOW-UP> <SUBSUMPTION> (A) With 3 wins this year, Green Bay has the best record in the NFL. (B) Green Bay has 3 wins this year. <FOLLOW-UP> (A) 102 casualties have been reported in the earthquake region. (B) So far, no casualties from the quake have been confirmed. • Classification models have not been developed well - Performance is low for several relations - There is still room for improvement • We are updating classification methods with recent technological advances

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Project Credibility Analysis Project (Statement Map Project) - A period : Apr. 2008 ｰ Mar. 2011 - Budget : $1. 2 M - People : 2 full-time postdocs and 2 part-time linguists - Multi-lingualization: Japanese, English, Spanish, and Portuguese 20

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Project NLP Subtasks to generate Statement Map: • Grasp various information of statements u Sentiment Analysis u Factuality Analysis u Applying Knowledge Base (Entities, Events) • Recognize semantic relation between statements u Alignment and Reasoning u Rhetorical Parsing (Evidence Detection) • Discuss necessary semantic relations u Corpus construction • Multilingualization u Preparation of tools for English u Corpus construction 21

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure How to generate “Statement Maps” ? ? ? 22

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) 23

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Passage Retrieval 24 -Tokenization -Parsing … Statement S 135 (There is no valid scientific evidence that vaccines cause autism. ) dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (Arg 1) (Arg 2) (cause) (valid) (scientific)(evidence)(there is no) (Arg 1)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Passage Retrieval 25 - Tokenization - Parsing … Statement S 135 (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. <wr, Affirm, Positive> dep. <wr, Affirm. , Positive> Sentiment Analysis Factuality Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (Arg 1) hypernym: [medicine] (Arg 2) hypernym: [disease] (cause) synonym: [bring on] (valid) (scientific)(evidence)(there is no) (Arg 1) hypernym: [grounds] antonym: [exist] Applying Knowledge Bases

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Passage Retrieval 26 - Tokenization - Parsing … Statement S 135 (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. <wr, Affirm, Positive> dep. <wr, Affirm. , Positive> Sentiment Analysis Factuality Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (Arg 1) hypernym: [medicine] (Arg 2) hypernym: [disease] (cause) synonym: [bring on] (valid) (scientific)(evidence)(there is no) (Arg 1) hypernym: [grounds] antonym: [exist] Sentiment polarity dictionary: - 5, 500 predicates - 133, 12 compound nouns Applying Knowledge Bases

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Passage Retrieval 27 - Tokenization - Parsing … Statement S 135 (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. <wr, Affirm, Positive> dep. <wr, Affirm. , Positive> Sentiment Analysis Factuality Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (Arg 1) hypernym: [medicine] (Arg 2) hypernym: [disease] (cause) synonym: [bring on] (valid) (scientific)(evidence)(there is no) (Arg 1) hypernym: [grounds] antonym: [exist] Applying Knowledge Bases • Annotating each event mention in a given text with a triplet : <Source, Modality, Polarity> • Constructing a corpus from blog posts : 24, 858 event mentions manually annotated with <Modality, Polarity> pairs.

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Passage Retrieval 28 - Tokenization - Parsing … Statement S 135 (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. <wr, Affirm, Positive> dep. <wr, Affirm. , Positive> Sentiment Analysis Factuality Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (Arg 1) hypernym: [medicine] (Arg 2) hypernym: [disease] (cause) synonym: [bring on] (valid) (scientific)(evidence)(there is no) (Arg 1) hypernym: [grounds] antonym: [exist] Applying Knowledge Bases • Knowledge Base for Entities: - 101, 946 synsets composed of 328, 534 nouns - 3, 534, 357 hyponym relations • Knowledge Base for Events: - a thesaurus that includes 9, 582 predicates - 52, 722 binary relations between predicate-argument structures

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Passage Retrieval 29 - Tokenization - Parsing … Statement S 135 (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. <wr, Affirm, Positive> dep. <wr, Affirm. , Positive> Sentiment Analysis Factuality Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (Arg 1) hypernym: [medicine] S 1, S 2 S 1, S 3 … (Arg 2) hypernym: [disease] S 134, S 135 Alignment and Reasoning (cause) synonym: [bring on] (valid) (scientific)(evidence)(there is no) (Arg 1) hypernym: [grounds] antonym: [exist] Applying Knowledge Bases … Rhetorical Parsing

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 30 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. : : synonym: : maneku (cause) S 135: There is no valid scientific evidence that vaccines cause autism. nai (there are no) hikiokosu (bring on) <Wr, Affirm, Positive> ga ga : : hypernym: : wakuchin (vaccines) MMR (MMR) byouki (disease) no Suigin be-su (Mercury-based) jiheisyou (autism) [Negative] no kodomo (children) aru (exist) : : synonym: : konkyo syouko (evidence) (grounds) wo : : hypernym: : antonym: : toiu na na : : synonym: : Maneku hikiokosu kagakuteki (cause) (bring on) (scientific) <Wr, Affirm, Negative> ga : : hypernym: : wo Wakuchin iyakuhin jiheisyou (vaccines) (medicine) (autism) [Negative] datou (valid) : : hypernym: : byouki (disease)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 31 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. : : synonym: : maneku (cause) ga : : hypernym: : wakuchin (vaccines) MMR (MMR) S 135: There is no valid scientific evidence that vaccines cause autism. nai (there are no) hikiokosu (bring on) <Wr, Affirm, Positive> Synonym + opposite : : hypernym: : jiheisyou Polarity byouki (disease) (autism) toiu wo [Negative] no Suigin be-su (Mercury-based) : : antonym: : no kodomo (children) Identity ga aru (exist) : : synonym: : konkyo syouko (evidence) (grounds) na na : : synonym: : Maneku hikiokosu kagakuteki (cause) (bring on) (scientific) <Wr, Affirm, Negative> ga : : hypernym: : wo Wakuchin iyakuhin jiheisyou (vaccines) (medicine) (autism) Hypernym Contradiction [Negative] datou (valid) : : hypernym: : byouki (disease)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Passage Retrieval 32 - Tokenization - Parsing … Statement S 135 (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. <wr, Affirm, Positive> dep. <wr, Affirm. , Positive> Sentiment Analysis Factuality Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (Arg 1) (Arg 2) hypernym: [medicine] S 1, S 2 S 1, S 3 … hypernym: [disease] (cause) synonym: [bring on] (valid) (scientific)(evidence)(there is no) (Arg 1) hypernym: [grounds] Applying Knowledge Bases antonym: [exist] S 134, S 135, S 64, contradiction Draw a Statement Map S 1, S 7, grounds S 14, S 47, grounds … S 1, S 2, synonymous S 1, S 3, contradiction … Alignment and Reasoning … S 134, S 128, grounds Rhetorical Parsing

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Constructing a Japanese Corpus Semantic relations between statements in this corpus Category Relations Definition Logical Relations Agreement Both A and B is TURE at the same time Contradiction Both A and B can not be TRUE at the same time Confinement B confines the situations in which A applies Agreeing Opinion Different sources are in agreement or their opinions entail one another Conflicting Opinion Different sources disagree or their opinions are contradictory Agreeing Evaluative Polarity A and B evaluate something from different perspectives and their opinions have the same polarity Conflicting Evaluative Polarity A and B evaluate something from different perspectives and their opinions have opposite polarities Evaluation A has an evaluation B of the form (P)ositive, (N)egative or (O)ther Attitudinal Relations （Positive/Negative/Other） Negative Relations Negative There is no relation between A and B, or the relation cannot be classified into a known category 33

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Constructing a Japanese Corpus 34 • Extracting statements from sentences in Web documents • Annotating semantic relations between two statements • (Examples of [EVIDENCE] are not collected in this corpus) An entry in the corpus has (A)-(G) (A) Real sentence (1) in a Web document (B) Real sentence (2) in a Web document There is no link between the MMR vaccine and autism, according to the largest ever published study about this controversial issue. The weight of the evidence indicates that vaccines are not associated with autism. (C) Statement (1) (E) <Citation> The is no link between the MMR vaccine and autism. (D) Statement (2) (F) <Entailment> vaccines are not associated with autism. (G) <Agreement> Currently, about 1, 600 statement pairs have been annotated

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map : 1. a visualization of the semantic relations between statements 2. as a graph which consists of nodes and edges nodes: statements edges: semantic relations between nodes To generate Statement Map: • Grasp various information of statements u Sentiment Analysis u Factuality Analysis u Applying Knowledge Base (Entity, Event) • Recognize semantic relation between statements u Alignment and Reasoning u Rhetorical Parsing (Evidence Detection) • Discuss necessary semantic relations u Corpus construction • Multilingualization u Preparation tools for English u Corpus construction 35

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Future work • Developing each subtask: - To expand dictionary or database (Sentiment Analysis, Knowledge Bases) - To construct corpora (Factuality Analysis, Relation Recognition) • Developing Statement Map Generation system • Constructing a corpus of [EVIDENCE] relation based on RST 36

37

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) dep. Sentiment Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) (Arg 2) (Arg 1) -Tokenization -Parsing 42

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Sentiment Analysis • Detect strings with implicit sentiment polarity in a given statement Sentiment polarity dictionary: - 5, 500 predicates - 133, 12 compound nouns (example) (S 64: Mercury-based MMRs actually bring on autism in children. ) Negative dep. Suiginbe-su no / MMR ha / kodomo no / jiheisyou wo / hikiokosu. (Mercury-based) (MMRs) (children) (autism) (bring on) (Arg 1) (Arg 2) 43

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. Sentiment Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) (Arg 2) (Arg 1) -Tokenization -Parsing 44

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) Negative dep. Sentiment Analysis dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) (Arg 2) (Arg 1) -Tokenization -Parsing Factuality Analysis 45

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 46 Factuality Analysis • For each event mention, we identify the modal status of the event entity referred to in the event mention, namely, whether the event actually took place, will take place, or is just hypothetical. • We have created a corpus from blog posts and other Web documents. It has 24, 858 event mentions manually annotated with <Modality, Polarity> pairs. (example) (S 64: Mercury-based MMRs actually bring on autism in children. ) Event mention! dep. Suiginbe-su no / MMR ha / kodomo no / jiheisyou wo / hikiokosu. (Mercury-based) (MMRs) (children) (austism) (bring on) (Arg 1) (Arg 2)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Factuality Analysis Annotate each event mention in a given text with a triplet <Source, Modality, Polarity> • Source [The person or entity that is expressing the private state in a given text. ] • Modality [24 classes: Affirmation, Inference, Counterfactual, Conditional Affirmative, Deserving, Schedule, Intention, Wish, Request, Hypothesis, etc. ] • Polarity [3 classes: Positive, Negative and Unknown] (example) (S 64: Mercury-based MMRs actually bring on autism in children. ) <writer, Affirmation, Positive> dep. Suiginbe-su no / MMR ha / kodomo no / jiheisyou wo / hikiokosu. (Mercury-based) (MMRs) (children) (autism) (bring on) (Arg 1) (Arg 2) 47

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) Negative <Wr, Affirm, Negative> <Wr, Affirm, Positive> dep. Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) (Arg 2) (Arg 1) -Tokenization -Parsing Sentiment Analysis Factuality Analysis 48

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) Negative <Wr, Affirm, Negative> <Wr, Affirm, Positive> dep. -Tokenization -Parsing Sentiment Analysis Factuality Analysis Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) (Arg 2) (Arg 1) Applying Knowledge Bases 49

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Applying Knowledge Bases • We look up synonyms, antonyms and hypernyms of the words in the given sentence in two knowledge bases. (S 64: Mercury-based MMRs actually bring on autism in children. ) dep. Suigin be-su no / MMR ha / kodomo no / jiheisyou wo / hikiokosu. (Mercury-based) (MMR) (children) (autism) (bring on) (Arg 2) (Arg 1) KB for Entities hypernym: [metal? ? ] hypernym: [medicine] KB for Events hypernym: [disease] synonym: [cause] [Knowledge Base for Entities]: [Knowledge Base for Events]: u 101, 946 synsets composed of u a thesaurus that includes 9, 582 328, 534 nouns predicates u 3, 534, 357 hyponym relations u 52, 722 binary relations 50 between

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) Negative <Wr, Affirm, Negative> <Wr, Affirm, Positive> dep. -Tokenization -Parsing Sentiment Analysis Factuality Analysis Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) hypernym: [medicine] (Arg 2) hypernym: [disease] (Arg 1) synonym: [bring on] hypernym: [grounds] antonym: [exist] Applying Knowledge Bases 51

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) Negative <Wr, Affirm, Negative> <Wr, Affirm, Positive> dep. -Tokenization -Parsing Sentiment Analysis Factuality Analysis Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) hypernym: [medicine] S 1, S 2 S 1, S 3 (Arg 2) hypernym: [disease] … S 134, S 135 Alignment and Reasoning (Arg 1) synonym: [bring on] hypernym: [grounds] antonym: [exist] Applying Knowledge Bases … Rhetorical Parsing 52

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval … (There is no valid scientific evidence that vaccines cause autism. ) Negative <Wr, Affirm, Negative> <Wr, Affirm, Positive> dep. -Tokenization -Parsing Sentiment Analysis Factuality Analysis Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) hypernym: [medicine] S 1, S 2 S 1, S 3 (Arg 2) hypernym: [disease] … S 134, S 135 Alignment and Reasoning (Arg 1) synonym: [bring on] hypernym: [grounds] antonym: [exist] Applying Knowledge Bases … Rhetorical Parsing 53

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 54 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. : : synonym: : maneku (cause) S 135: There is no valid scientific evidence that vaccines cause autism. nai (there are no) hikiokosu (bring on) <Wr, Affirm, Positive> ga ga : : hypernym: : wakuchin (vaccines) MMR (MMR) byouki (disease) no Suigin be-su (Mercury-based) jiheisyou (autism) [Negative] no kodomo (children) aru (exist) : : synonym: : konkyo syouko (evidence) (grounds) wo : : hypernym: : antonym: : toiu na na : : synonym: : Maneku hikiokosu kagakuteki (cause) (bring on) (scientific) <Wr, Affirm, Negative> ga : : hypernym: : wo Wakuchin iyakuhin jiheisyou (vaccines) (medicine) (autism) [Negative] datou (valid) : : hypernym: : byouki (disease)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 55 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. : : synonym: : maneku (cause) S 135: There is no valid scientific evidence that vaccines cause autism. nai (there are no) hikiokosu (bring on) <Wr, Affirm, Positive> ga ga : : hypernym: : wakuchin (vaccines) MMR (MMR) byouki (disease) jiheisyou (autism) [Negative] no Suigin be-su (Mercury-based) no kodomo (children) Identity Hypernym aru (exist) : : synonym: : konkyo syouko (evidence) (grounds) wo : : hypernym: : antonym: : toiu na na : : synonym: : Maneku hikiokosu kagakuteki (cause) (bring on) (scientific) <Wr, Affirm, Negative> ga : : hypernym: : wo Wakuchin iyakuhin jiheisyou (vaccines) (medicine) (autism) [Negative] datou (valid) : : hypernym: : byouki (disease)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 56 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. : : synonym: : maneku (cause) S 135: There is no valid scientific evidence that vaccines cause autism. nai (there are no) hikiokosu (bring on) <Wr, Affirm, Positive> ga ga : : hypernym: : wakuchin (vaccines) MMR (MMR) wo : : hypernym: : byouki (disease) Synonym jiheisyou (autism) [Negative] no Suigin be-su (Mercury-based) : : antonym: : no kodomo (children) Identity Hypernym toiu aru (exist) : : synonym: : konkyo syouko (evidence) (grounds) na na : : synonym: : Maneku hikiokosu kagakuteki (cause) (bring on) (scientific) <Wr, Affirm, Negative> ga : : hypernym: : wo Wakuchin iyakuhin jiheisyou (vaccines) (medicine) (autism) [Negative] datou (valid) : : hypernym: : byouki (disease)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 57 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. S 135: There is no valid scientific evidence that vaccines cause autism. nai (there are no) : : antonym: : <Wr, Affirm, Positive> : : hypernym: : wakuchin (vaccines) : : synonym: : hikiokosu maneku : : synonym: : hikiokosu Synonym maneku (bring on) (cause) (bring on) <Wr, Affirm, Positive> (cause) <Wr, Affirm, Positive> aru (exist) : : synonym: : Maneku hikiokosu (cause) (bring on) ga <Wr, Affirm, Negative> konkyo syouko kagakuteki ga (evidence) : : hypernym: : (grounds) MMR jiheisyou Hypernym Wakuchin iyakuhin(scientific) wakuchin byouki (MMR) (autism) (vaccines)(disease) (medicine) na toiu ga ga wo : : hypernym: : [Negative] wo no : : hypernym: : no : : synonym: : Maneku hikiokosu (bring on) (cause) jiheisyou : : hypernym: : jiheisyou Identyty byouki Negative> byouki kodomo <Wr, Affirm, iyakuhin Suigin be-su (autism) (disease) (autism) (children) (Mercury-based) (medicine) (disease) [Negative] ga wo wo na datou (valid)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 58 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. S 135: There is no valid scientific evidence that vaccines cause autism. Contradiction nai (there are no) : : antonym: : <Wr, Affirm, Positive> : : hypernym: : wakuchin (vaccines) : : synonym: : hikiokosu maneku : : synonym: : hikiokosu Synonym maneku (bring on) (cause) (bring on) <Wr, Affirm, Positive> (cause) <Wr, Affirm, Positive> aru (exist) : : synonym: : Maneku hikiokosu (cause) (bring on) ga <Wr, Affirm, Negative> konkyo syouko kagakuteki ga (evidence) : : hypernym: : (grounds) MMR jiheisyou Hypernym Wakuchin iyakuhin(scientific) wakuchin byouki (MMR) (autism) (vaccines)(disease) (medicine) na toiu ga ga wo : : hypernym: : [Negative] wo no : : hypernym: : no : : synonym: : Maneku hikiokosu (bring on) (cause) jiheisyou : : hypernym: : jiheisyou Identyty byouki Negative> byouki kodomo <Wr, Affirm, iyakuhin Suigin be-su (autism) (disease) (autism) (children) (Mercury-based) (medicine) (disease) [Negative] ga wo wo na datou (valid)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion 59 Alignment and Reasoning S 64: Mercury-based MMRs actually bring on autism in children. : : synonym: : maneku (cause) S 135: There is no valid scientific evidence that vaccines cause autism. nai (there are no) hikiokosu (bring on) <Wr, Affirm, Positive> ga ga : : hypernym: : wakuchin (vaccines) MMR (MMR) wo : : hypernym: : byouki (disease) Synonym jiheisyou (autism) [Negative] no Suigin be-su (Mercury-based) : : antonym: : no kodomo (children) Identity toiu aru (exist) : : synonym: : konkyo syouko (evidence) (grounds) na na : : synonym: : Maneku hikiokosu kagakuteki (cause) (bring on) (scientific) <Wr, Affirm, Negative> ga : : hypernym: : wo Wakuchin iyakuhin jiheisyou (vaccines) (medicine) (autism) Hypernym Contradiction [Negative] datou (valid) : : hypernym: : byouki (disease)

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation Procedure Query: Yobou-sesshu de jiheishou ni naru? (Do vaccines cause autism? ) Statement S 1 … Statement S 135 Passage Retrieval -Tokenization -Parsing … (There is no valid scientific evidence that vaccines cause autism. ) Negative <Wr, Affirm, Negative> <Wr, Affirm, Positive> dep. Sentiment Analysis Factuality Analysis Wakuchin ga / jiheishou wo / maneku to iu / datou na / kagakuteki shouko ha / nai (vaccines) (autism) (cause) (valid) (scientific) (evidence) (there is no) (Arg 1) hypernym: [medicine] (Arg 2) (Arg 1) hypernym: [disease] synonym: [bring on] hypernym: [grounds] Applying Knowledge Bases antonym: [exist] S 1, S 2 S 1, S 3 … S 134, S 135 S 1, S 2, synonymous S 1, S 3, contradiction S 135, S 64, contradiction Draw a Statement Map S 1, S 7, grounds S 14, S 47, grounds … … Alignment and Reasoning … S 134, S 128, grounds Rhetorical Parsing 60

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Constructing a Japanese Corpus Semantic relations between statements in this corpus Category Relations Definition Logical Relations Agreement Both A and B is TURE at the same time Contradiction Both A and B can not be TRUE at the same time Confinement Aの成立をBが条件、範囲の限定をする Agreeing Opinion 異なる判断主体が同義、もしくは含意の言明を表する Conflicting Opinion Other sources 異なる判断主体が矛盾する言明を表する Agreeing Evaluative Polarity ある側面に対する何らかの評価をA, Bが行い、それら の極性が一致 Conflicting Evaluative Polarity ある側面に対する何らかの評価をA, Bが行い、それら の極性が対立 Evaluation Aに対しBが（P）ポジティブ、（N）ネガティブ、（O） その他の評価、を行う Attitudinal Relations （Positive/Negative/Other） Negative Relations Negative No relation between A and B, or some relation between A and B does not fit into any other relations above 62

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Constructing a Japanese Corpus • Extracting statements from sentences in Web documents • Annotating semantic relations between two statements • (Examples of [EVIDENCE] are not collected in this corpus) An entry in the corpus has (A)-(G) (A) Real sentence (1) in a Web document (B) Real sentence (2) in a Web document There is no link between the MMR vaccine and autism, according to the largest ever published study about this controversial issue. The weight of the evidence indicates that vaccines are not associated with autism. (C) Statement (1) (E) <Citation> The is no link between the MMR vaccine and autism. (D) Statement (2) (F) <Entailment> vaccines are not associated with autism. (G) <Agreement> Currently, about 1600 statement pairs have been annotated

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map Generation in English System Corpus in English 64

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Statement Map : 1. a visualization of the semantic relations between statements 2. as a graph which consists of nodes and edges nodes: statements edges: semantic relations between nodes To generate Statement Map: • Grasp various information of statements u Sentiment Analysis u Factuality Analysis u Applying Knowledge Base (Entity, Event) • Recognize semantic relation between statements u Alignment and Reasoning u Rhetorical Parsing (Evidence Detection) • Discuss necessary semantic relations u Corpus construction • Multilingualization u Preparation tools for English u Corpus construction 66

Introduction • Statement Map • The Project • Generating Statement Map • Conclusion Future work • Developing each subtasks: - To expand dictionary or database (Sentiment Analysis, Knowledge Bases) - To construct corpora (Factuality Analysis, Relation Recognition) • Developing Statement Map Generation system • Constructing a corpus of [EVIDENCE] relation based on RST 67