Joint Inference and Disambiguation of Implicit Sentiments via
Joint Inference and Disambiguation of Implicit Sentiments via Implicature Constraints Lingjia Deng*, Janyce Wiebe*^, Yoonjung Choi^ * Intelligent Systems Program, University of Pittsburgh ^ Department of Computer Science, University of Pittsburgh
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Experimental Result Conclusions Intelligent Systems Program 2 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Experiment Result Conclusions Intelligent Systems Program 3 2/24/2021
Introduction Scenario: The government proposes the bill of Affordable Care Act. We want to get everyone’s opinion of it. We can collect opinions by doing a survey, a questionnaire, etc. We can also collect the writers’ stances by analyzing their posts online. Intelligent Systems Program 4 2/24/2021
Introduction “The bill will lower skyrocketing healthcare costs. ” Explicit (Direct) Sentiment: writer negative toward skyrocketing healthcare costs The healthcare cost is too high. I cannot afford it. Implicit (Inferred) Sentiment: writer positive toward the bill will lower costs There is a chance that the costs could be decreased! I love it! writer positive toward the bill The bill is able to do this! I’ll vote for it! WHAT ABOUT BILL? Intelligent Systems Program 5 2/24/2021
Good. For/Bad. For Event “The bill will lower skyrocketing healthcare costs. ” <bill, lower, healthcare costs> Good. For/Bad. For Event (Deng et al. , ACL 2013 short) triple <agent, good. For/bad. For event, theme> good. For event: positive effect on theme help, increase, etc bad. For event: negative effect on theme harm, decrease, etc Reverser (Deng et al. , ACL 2013 short) A reverser flips the polarity of a good. For/bad. For event. e. g. The bill will not lower the healthcare costs. Intelligent Systems Program 6 2/24/2021
Good. For/Bad. For Event Good. For/Bad. For Corpus (Deng et al. , ACL 2013 short): 134 political editorials e. g. <bill, lower, healthcare costs> e. g. <positive, bad. For, negative> writer’s sentiments toward agent and theme almost 20% sentences have clear good. For/bad. For events available at mpqa. cs. pitt. edu Intelligent Systems Program 7 2/24/2021
Utilizing Good. For/Bad. For Event The ultimate goal of this work is utilizing good. For/bad. For information a triple <agent, good. For/bad. For, theme> to detect writer’s sentiments toward entities: agents, themes Explicit sentiment Implicit sentiment Intelligent Systems Program 8 2/24/2021
Utilizing Good. For/Bad. For Event Given a document, (Q 1) (Q 2) (Q 3) (Q 4) (Q 5) which spans are good. For/bad. For events? what is the polarity of the event: good. For or bad. For? does this event has a reverser? which spans are agents and themes? what are the writer’s sentiments toward the agents & themes? Graph-based Sentiment Propagation (Deng and Wiebe, EACL 2014) Build a graph using manual annotations of (Q 1)-(Q 4) Apply loopy belief propagation to infer sentiment Evaluate only on the final part (Q 5) Intelligent Systems Program 9 2/24/2021
Utilizing Good. For/Bad. For Event Given a document, (Q 1) (Q 2) (Q 3) (Q 4) (Q 5) which spans are good. For/bad. For events? what is the polarity of the event: good. For or bad. For? does this event has a reverser? which spans are agents and themes? what are the writer’s sentiments toward the agents & themes? In this work, Utilize the manual annotations of (Q 1) Automatically extract local results of (Q 2)-(Q 5) Optimize the local results using Integer Linear Programming Evaluate on (Q 2)-(Q 5) Intelligent Systems Program 10 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Experiment Result Conclusions Intelligent Systems Program 11 2/24/2021
Related Work Sentiment Anlysis Classifying explicit sentiment. (Wiebe et al. , 2005; Johansson and Moschitti, 2013; Yang and Cardie, 2013) Investigate features directly implying sentiment. (Zhang and Liu, 2011; Feng et al. , 2013) OURS Bridge between explicit sentiment and implicit sentiment Good. For/Bad. For Previous work do not cover all inferences related to good. For/bad. For events. Choi and Cardie, 2008; Moilanen et al. , 2010; Anand Reschke 2010; 2011; Goyal et al. , 2012) OURS Define a set of rules revealing inferences among agents, themes and good. For/bad. For events. (Deng and Wiebe, EACL 2014) Call for less manual annotations (this work) Intelligent Systems Program 12 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Experiment Result Conclusions Intelligent Systems Program 13 2/24/2021
Good. For/Bad. For Implicature “The bill will lower the skyrocketing healthcare costs. ” <bill, lower, healthcare costs> sentiment(healthcare costs) = negative & lower = bad. For sentiment(lower) = positive & lower = bad. For sentiment(bill) = positive sentiment(theme) = negative & bad. For sentiment(event) = positive & bad. For sentiment(agent) = positive Intelligent Systems Program 14 2/24/2021
Good. For/Bad. For Implicature sentiment(event) good. For/bad. For sentiment(agent) sentiment(theme) positive good. For positive negative gooo. For negative positive bad. For positive negative bad. For negative positive (Deng and Wiebe, EACL 2014) <agent, good. For, theme> sentiment(agent) = sentiment(theme) <agent, bad. For, theme> sentiment(agent) ≠ sentiment(theme) the sentiments are opposite Intelligent Systems Program 15 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Local Detectors Optimization Framework Overview Objective Function and Constraints Experiment Result Conclusions Intelligent Systems Program 16 2/24/2021
Analyzing Good. For/Bad. For Event Given a good. For/bad. For span in a document, (Q 2) what is the polarity of the event: good. For or bad. For? (Q 3) does this event has a reverser? (Q 4) which spans are agent & theme? (Q 5) what are the writer’s sentiments toward agent & theme? Each local detector is run to answer each question above. Intelligent Systems Program 17 2/24/2021
Local Detectors (Q 2) what is the polarity of the event: good. For or bad. For? A sense-level good. For/bad. For lexicon (Choi et al. , WASSA 2014) A good. For/bad. For span has m good. For senses and n bad. For senses: good. For score = m / (m+n) bad. For score = n / (m+n) Intelligent Systems Program 18 2/24/2021
Local Detectors (Q 3) does this event has a reverser? A word-level shifter lexicon (Wilson, 2008). Three categories of reversers: negation They will not lower your coverage verb reversers …new rules to prevent companies from overcharging patients others bureaucracy will cut costs without hurting the old Intelligent Systems Program 19 2/24/2021
Local Detectors (Q 3) does this event has a reverser? Find a reverser word in the sentence. Stanford dependency parser: reverser word dependency path good. For/bad. For span. negation: neg verb reverser: xcomp, pcomp, obj others: advmod, pcomp, cc, xcomp, nsubj, neg The shorter the path is, the more likely there is a reverser. d = length of the path, σ = thredhold reversed score = 1/d- σ Intelligent Systems Program 20 2/24/2021
Local Detectors (Q 4) which spans are agents & themes? Two agent candidates and two theme candidates. Semantic agent: semantic role labeling (A 0>A 1>A 2) Syntactic agent: Stanford dependency parser Semantic theme: semantic role labeling (A 1>A 2>A 0) Syntactic theme: Stanford dependency parser Intelligent Systems Program 21 2/24/2021
Local Detectors (Q 5) what are the writer’s sentiments toward agents & themes? The same local sentiment detector from (Deng and Wiebe, EACL 2014) majority voting using: Opinion Finder (Wilson et al. , 2005) Opinion Extractor (Johansson and Moschitti, 2013) MPQA subjectivity lexicon (Wilson et al. , 2005) General Inquirer (Stone et al. , 1966) connotation lexicon (Feng et al. , 2013) positive score, negative score (0. 5~1) Intelligent Systems Program 22 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Local Detectors Optimization Framework Overview Objective Function and Constraints Experiment Result Conclusions Intelligent Systems Program 23 2/24/2021
Optimization Framework Overview (Q 3) is the polarity reversed? Agent 1 Agent 2 pos: 0. 7 neg: 0. 5 (Q 2) is it good. For or bad. For? reversed good. For bad. For Theme 2 pos: 0. 5 reverser: 0. 9 good. For: 0. 8 neg: 0. 6 0. 2 bad. For: Theme 1 pos: 0. 5 neg: 0. 5 pos: 0. 7 neg: 0. 5 (Q 4) which spans are agents and themes? (Q 5) what are the writer’s sentiments? Intelligent Systems Program 24 2/24/2021
Optimization Framework Overview Agent 1 Agent 2 pos: 0. 7 neg: 0. 5 reversed good. For bad. For Theme 2 pos: 0. 5 reverser: 0. 9 good. For: 0. 8 neg: 0. 6 0. 2 Intelligent Systems Program 25 bad. For: Theme 1 pos: 0. 5 neg: 0. 5 pos: 0. 7 neg: 0. 5 2/24/2021
Optimization Framework Overview Agent 1 Agent 2 pos: 0. 7 neg: 0. 5 reversed good. For bad. For Theme 2 pos: 0. 5 reverser: 0. 9 good. For: 0. 8 neg: 0. 6 0. 2 bad. For: Theme 1 pos: 0. 5 neg: 0. 5 pos: 0. 7 neg: 0. 5 The framework selects a subset of labels containing: Fortunately, the implicature rules in (Deng and Wiebe, one label from the four agent sentiment labels, EACL 2014) define dependencies among these Agent 1 -pos, Agent 1 -neg, Agent 2 -pos, Agent 2 -neg ambiguities: one/no label from the reversed labels, good. For: sentiment(agent) = sentiment(theme) one labelsentiment(agent) from the gfbf polarity labels, bad. For: ≠ sentiment(theme) one label from the four sentiment labels. in Theses dependencies aretheme encoded as constraints the framework. Intelligent Systems Program 26 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Local Detectors Optimization Framework Overview Objective Function and Constraints Experiment Result Conclusions Intelligent Systems Program 27 2/24/2021
Objective Function variable i could be: i in Entity Set agent candidate theme candidate i in GFBF Set good. For/bad. For event Intelligent Systems Program c is oneisof the lables of i <i, k, j> a <agent, good. For/bad. For, theme> triple i in GFBF Set c inin{good. For, reversed} i, j Entity Set; bad. For, k in GFBF Set i in Entity Set c in {positive, negative} 28 2/24/2021
Objective Function ic: variable i assigned label c u: binary indicator of i choosing c ξ, δ: slack variables of triple <I, k, j> representing this triple is an exception to good. For, bad. For rule (exception: 1) p: score of local detector The framework assigns values (0 or 1) to u maximizing the scores given by the local detectors, and assigns values (0 or 1) to ξ, δ minimizing the cases where good. For/bad. For implicature rules are violated. Integer Linear Programming (ILP) is used. Intelligent Systems Program 29 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Local Detectors Optimization Framework Overview Objective Function and Constraints Experiment Result Conclusions Intelligent Systems Program 30 2/24/2021
Basic Constraints For a triple <agent, good. For/bad. For, theme>, for the good. For/bad. For, the framework chooses one from the two labels: {good. For, bad. For} for the agent, the framework chooses one from the four labels: {agent 1 -pos, agent 1 -neg, agent 2 -pos, agent 2 -neg} for theme, it is the same Intelligent Systems Program 31 2/24/2021
Good. For Implicature Constraints In a good. For event, sentiments are the same 0 1 0 0 1 good. For: 1 bad. For: 0 0 ANY VALUE Intelligent Systems Program 32 OR exception: 1 not exception: 0 0 1 2/24/2021
Bad. For Implicature Constraints In a bad. For event, sentiments are opposite 0 1 1 0 good. For: 0 bad. For: 1 0 0 ANY VALUE Intelligent Systems Program 33 OR exception: 1 not exception: 0 0 1 2/24/2021
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Experiment Result The good. For/bad. For corpus (Deng et al. , ACL 2013 short) Conclusions Intelligent Systems Program 34 2/24/2021
Evaluation Metrics (Q 5) what are the writers’evaluation: sentiments toward agents and themes? For sentiment detection Precision, Recall and F-measure on non-neutral agents/themes only 8 agents/themes are neural in the corpus Auto: Agent 2 -pos Gold: Agent 1 -pos what is “auto=gold” ? we’re extracting agent/theme span and detecting sentiment strict evaluation: wrong relaxed evaluation: correct Agent 1 -pos, Agent 1 -neg, Agent 2 -pos, Agent 2 -neg simultaneously: strict evaluation: both the chosen span and the sentiment are correct relaxed evaluation: the sentiment is correct, regardless of the span Intelligent Systems Program 35 2/24/2021
Performances of Sentiment Detection Local Baseline For (Agent 1 -pos, Agent 1 -neg, Agent 2 -pos, Agent 2 -neg), choose the one with maximal local score; If the local detector fails to detect any sentiment, the local baseline is wrong. Majority Baseline Always chooses Agent 1 -pos (semantic agent). strict evaluation relaxed evaluation Precision Recall F-measure Precision ILP+coref 0. 4660 ILP 0. 4401 Local 0. 4956 Majority 0. 3862 Intelligent Systems Program Recall F-measure 0. 6471 0. 4401 0. 5939 0. 2891 0. 3652 0. 5983 0. 3490 0. 4408 0. 3862 0. 5462 36 2/24/2021
Performances (Q 2) (Q 3) (Q 4) what is the polarity of the event: good. For or bad. For? does event has a reverser? Forthis agent/theme span detector, good. For/bad. For polarity which spans are agents detector: and themes? detector, reverser Accuracy = (# auto = gold) / (# all events in the corpus) Local Baseline: Local detector good. For/bad. For being reversed agent/theme polarity span ILP 0. 7725 0. 8900 0. 6854 Local 0. 7068 0. 8807 0. 6667 The lexicon doesn’t cover all good. For/bad. For words. But by the framework we could infer the polarity of the word.
Outline Introduction Related Work Good. For/Bad. For Implicature Optimization Framework Experiment Result Conclusions Intelligent Systems Program 38 2/24/2021
Conclusions The ultimate goal of this work is utilizing good. For/bad. For information to detect writer’s sentiments toward entities. The global optimization framework jointly infers the polarity of gfbf events whether or not they are reversed, which candidate NPs are the agent and theme the writer’s sentiments toward them. Compared to the baselines, the framework improves 10 points in F-measure of sentiment detection improves 7 points in accuracy of good. For/bad. For polarity disambiguation Intelligent Systems Program 39 2/24/2021
Questions? The good. For/bad. For corpus is available at mpqa. cs. pitt. edu. Intelligent Systems Program 40 2/24/2021
References Lingjia Deng and Janyce Wiebe. 2014. Sentiment propagation via implicature constraints. In Meeting of the European Chapter of the Association for Computational Linguistics (EACL-2014). Lingjia Deng, Yoonjung Choi, and Janyce Wiebe. 2013. Benefactive/malefactive event and writer attitude anno- tation. In Proceedings of the 51 st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 120– 125, Sofia, Bulgaria, August. Association for Computational Linguistics. Theresa Wilson, Janyce Wiebe, , and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase -level sentiment analysis. In HLP/EMNLP, pages 347– 354. Richard Johansson and Alessandro Moschitti. 2013. Relational features in fine-grained opinion analysis. Compu- tational Linguistics, 39(3). P. J. Stone, D. C. Dunphy, M. S. Smith, and D. M. Ogilvie. 1966. The General Inquirer: A Computer Approach to Content Analysis. MIT Press, Cambridge. Song Feng, Jun Sak Kang, Polina Kuznetsova, and Yejin Choi. 2013. Connotation lexicon: A dash of sentiment beneath the surface meaning. In Proceedings of the 51 th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Sofia, Bulgaria, Angust. Association for Computational Linguistics. Intelligent Systems Program 41 2/24/2021
Related Work Classify explicit sentiments and extracting explicit opinion expressions, holders and targets (Wiebe et al. , 2005; Johansson and Moschitti, 2013; Yang and Cardie, 2013). Identify words/Phrases directly implicit opinions. (Zhang and Liu, 2011; Feng et al. , 2013) e. g. “sunshine” has positive connotation OURS we focus on how we can bridge between explicit and implicit sentiments via inference Intelligent Systems Program 42 2/24/2021
Related Work Infer an overall polarity of a sentence by compositional semantics. (Choi and Cardie, 2008; Moilanen et al. , 2010) Identify classes of good. For/bad. For terms, and carry out studies involving artificially constructed good. For/bad. For triples and corpus examples matching fixed linguistic templates. (Anand Reschke 2010; 2011) Generate a lexicon of patient polarity verbs, which correspond to good. For/bad. For events whose spans are verbs. (Goyal et al. , 2012) do not cover all cases relevant to good. For/bad. For events OURS (Deng and Wiebe, 2014) defines a generalized set of implicature rules and proposes a graph-based model to achieve sentiment propagation between the agents and themes of gfbf events Intelligent Systems Program 43 2/24/2021
Local Detectors (Q 5) what are the writer’s sentiments toward agents & themes? The same local sentiment detector from (Deng and Wiebe, EACL 2014) majority voting using: Opinion Finder (Wilson et al. , 2005) Opinion Extractor (Johansson and Moschitti, 2013) MPQA subjectivity lexicon (Wilson et al. , 2005) General Inquirer (Stone et al. , 1966) connotation lexicon (Feng et al. , 2013) positive score, negative score (0. 5~1) Intelligent Systems Program 44 2/24/2021
Local Detectors (Q 5) what are the writer’s sentiments? <agent, good. For/bad. For, theme> 1. sentiment toward agent/theme 2. sentiment toward good. For/bad. For event to increase coverage: sentiment toward theme positive score, negative score (0. 5~1) Intelligent Systems Program 45 2/24/2021
Local Detectors Why not train a system on the good. For/bad. For corpus? Only the writer’s sentiments toward the agents and themes of gfbf events are annotated in the corpus. There are many false negatives of sentiments toward entities. e. g. the writer is positive toward X, but X is not part of any good. For/bad. For event, so the positive sentiment is not annotated. The corpus does not support training a classifier. Intelligent Systems Program 46 2/24/2021
Co-reference In the Framework If two agents/themes co-refer, they should the assigned with the same sentiment label. If two good. For/bad. For events have the same agent, the two agents of the two events should be assigned with the same sentiment label. The reform will decrease the healthcare costs and improve the medical qualify as expected. If two agents/themes satisfy the criterions above, Coref(i, j) = 1 Intelligent Systems Program 47 2/24/2021
Co-reference In the Framework New constraints (Similar to good. For constraints) New objective function Intelligent Systems Program 48 2/24/2021
Adding Co-reference Performances (Q 5) what’re the writer’s sentiments toward agents & themes? Local + Coref Baseline Following the baseline Local If two agents/themes co-ref, and one of the two is assigned sentiment, then the other will be assigned with the same sentiment. strict evaluation relaxed evaluation Precision Recall F-measure Precision ILP 0. 4401 0. 5939 ILP+coref 0. 4660 0. 6471 Local+coref 0. 5025 0. 3103 0. 3836 0. 6210 0. 3834 0. 4741 Intelligent Systems Program 49 Recall F-measure 0. 6471 2/24/2021
Adding Co-reference Performances (Q 2) what is the polarity of the event: good. For or bad. For? (Q 3) does this event has a reverser? (Q 4) which spans are agents and themes? Local Baseline: Local detector good. For/bad. For being reversed agent/theme polarity span ILP 0. 7725 0. 8900 0. 6854 ILP + coref 0. 7747 0. 8807 0. 6710 Local 0. 7068 0. 8807 0. 6667 Intelligent Systems Program 50 2/24/2021
- Slides: 50