Stancebased Argument Mining Modeling Implicit Argumentation Using Stance

Stance-based Argument Mining – Modeling Implicit Argumentation Using Stance Michael Wojatzki & Torsten Zesch

Argumentation in Social Media “I am against Atheism!” “I am a Christian” “I believe in hell” #Jesus. Or. Hell 27 November 2020 Michael Wojatzki & Torsten Zesch 2

Application-Driven Argument Mining • • Atheism? 27 November 2020 • No evidence for religion • there is no god • there was no Christ religious freedom liberal beliefs religious beliefs • life after death • belief in God • Christianity • Islam conservative beliefs Michael Wojatzki & Torsten Zesch 3

Application-Driven Argument Mining • • (200) • • Atheism? 27 November 2020 (300) • No evidence for religion (100) • there is no god (20) • there was no Christ (80) religious freedom (50) liberal beliefs (50) religious beliefs (250) • life after death (50) • belief in God (200) • Christianity (150) • Islam (50) conservative beliefs (50) Michael Wojatzki & Torsten Zesch 4

Argumentation • Argument : = claim + optional supporting structures e. g. premises (Palau and Moens, 2009; Peldszus and Stede, 2013, Green et al. 2014) As the Bible says that infidels are going to hell, I am against atheism! Premise 27 November 2020 Michael Wojatzki & Torsten Zesch Claim 5

Argumentation in Social Media Implicitness implicit 27 November 2020 Bible: infidels are going to hell! I am against atheism Premise Claim Michael Wojatzki & Torsten Zesch 6

Argumentation in Social Media Noise & Implicitness implicit #Jesus. Or. Hell I am against atheism 27 November 2020 ? Claim Michael Wojatzki & Torsten Zesch 7

Stance-based Model • Stance: • in favor or against a target (=topic/subject) • w. r. t. a given debate • Debate Stance: • stance towards the target of the debate • often implicit, has to be inferred Atheism ⊖ • Explicit Stance: • explicit stance-taking towards some target • aids inference of debate stance 27 November 2020 Michael Wojatzki & Torsten Zesch Christianity � 8

Inspecting Stances Existence of Hell � Atheism ⊖ Christianity � As the Bible says that infidels are going to hell, I am against atheism! Christianity � ⊖ Atheism Existence of Hell � implicit Bible: infidels are going to hell! Christianity � ⊖ Atheism I am against atheism Existence of Hell � implicit #Jesus. Or. Hell I am against atheism 27 November 2020 Michael Wojatzki & Torsten Zesch 9

Iceberg Metaphor Atheism ⊖ Debate Stance observable Explicitness Supernatural Power � Explicit Stance unobservable ⊖ Atheism 27 November 2020 Michael Wojatzki & Torsten Zesch Implicitness God will judge those infidels! 10

Evaluation of the Model • How reliable can we annotate it? • What does it tell us about the debate? • How well can we automate it? Explicit Stance Debate Stance 27 November 2020 Michael Wojatzki & Torsten Zesch 11

Data • Sem. Eval task 6 2016 (Mohammad et al. , 2016) • subset on Atheism (Train + Test-Set) • 733 tweets • classes: � vs. NONE ⊖ • Annotation (three annotators) • Debate Stance (Reannotation) • Explicit Stances 27 November 2020 Michael Wojatzki & Torsten Zesch 12

Reannotation of Debate Stance • original annotation by turkers from USA • different cultural background • same questionnaire • remove Tweets if stance not inferable (18 tweets) • @bdutt @lalitkmodi @mohdasim 1 ohh. so i think why r u seculr. . . nice friend • you're doing the work of ending domination. " (Bell Hooks) (2/2) #feminism #civilrights 27 November 2020 Michael Wojatzki & Torsten Zesch 13

Annotation of Explicit Targets • Our use-case: overview on argumentation • degree of abstraction from surface forms needed Christianity � I believe in Jesus my Savior Christ is my Lord • define (less) explicit targets • choose granularity with respect to data 27 November 2020 Michael Wojatzki & Torsten Zesch Explicit Stance Debate Stance 14

Data-driven Selection 1. Candidates • 50 most frequent concepts (Nouns + Named Entities) • 25 concepts most strongly associated (Dice ) with Atheism� and Atheism ⊖ (Smadja et al. , 1996) 2. manually grouped and filtered • at this stage of our work, we can not evaluate whether the set is best possible choice 27 November 2020 Michael Wojatzki & Torsten Zesch 15

Explicit Targets Supernatural Power Secularism Islam Christianity 27 November 2020 Conservatism Religious Freedom USA No Evidence Freethinking Same-Sex Marriage Michael Wojatzki & Torsten Zesch Life after Death 16

Explicit Targets – Annotation • annotate targets only if there is some evidence in the text • do not infer a target if there is no textual hint on it <Then they will go away to eternal punishment, but the righteous to eternal life. > Matthew 25: 46 Christianity � 27 November 2020 Michael Wojatzki & Torsten Zesch ⊖ Islam 17

Inter-Annotator-Agreement complete model (joint decision): κ = 0. 63 27 November 2020 Michael Wojatzki & Torsten Zesch 18

Stance Pattern Analysis • What explicit stances appear together? • What is the relation between explicit and debate stances? Atheism ⊖ Christianity � 27 November 2020 Existence of Hell � Michael Wojatzki & Torsten Zesch 19

Patterns 1 st Order Atheism � 27 November 2020 Atheism Michael Wojatzki & Torsten Zesch ⊖ 20

Patterns 2 nd Order Atheism � 27 November 2020 Michael Wojatzki & Torsten Zesch ⊖ 21

Supervised Classification • Can we detect the scheme automatically? • explicit stances? • debate stances? • Experiments with state-of-the-art stance-detection system (Mohammad et al. , 2016) ⊖ • three-way classification (�, , NONE) • SVM (linear Kernel)* 27 November 2020 Michael Wojatzki & Torsten Zesch 22

Features – Explicit Stance Detection ⊖ • for each of target classify: � vs. NONE • word n-grams (1, 2, 3) • character n-grams (2, 3, 4, 5) ⊖ � vs. NONE word n-grams character n-grams Bible: infidels are going to hell! 27 November 2020 Michael Wojatzki & Torsten Zesch 23

Explicit Stances 0. 94 micro-averaged F 1 0. 78 0. 95 0. 79 0. 69 0. 53 majority class baseline our approach Supernatural Power (335) Christianity (223) Islam (43) • only top 2 explicit targets significant gains over baseline • other targets in <5% of the instances à not enough data à specialized features possible 27 November 2020 Michael Wojatzki & Torsten Zesch 24

Features ⊖ • for the debate stance classify: � vs. NONE • n-grams • explicit stances ⊖ � vs. NONE word n-grams character n-grams explicit stance Bible: infidels are going to hell! 27 November 2020 Michael Wojatzki & Torsten Zesch 25

Debate Stance micro averaged F 1 0. 88 0. 66 0. 65 0. 67 n-grams explicit stance (predicted) explicit stance (predicted + ngrams) 0. 49 majority class baseline explicit stance (oracle) • models for explicit stance similar to state-of-the-art • significant gain for oracle condition 27 November 2020 Michael Wojatzki & Torsten Zesch 26

Future Work • fully automated creation of explicit targets • apply more sophisticated machine learning • unlabeled data (distant supervision, word embeddings) (Mohammad et al. 2016) • sequential classification, deep learning • transfer explicit classifiers between domains 27 November 2020 Michael Wojatzki & Torsten Zesch 27

Stance-based Argument Mining • applicable to noisy and implicit argumentation • reliable to annotate • interesting insights into nature of debates • potential to boost automated stance detection 27 November 2020 Michael Wojatzki & Torsten Zesch 28

References Stab, C. , & Gurevych, I. (2014). Annotating Argument Components and Relations in Persuasive Essays. In COLING (pp. 1501 -1510). Eckle-Kohler, J. , Kluge, R. , & Gurevych, I. (2015). On the Role of Discourse Markers for Discriminating Claims and Premises in Argumentative Discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Lisbon, Portugal, to appear. Peldszus, A. , & Stede, M. (2013). From argument diagrams to argumentation mining in texts: A survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 7(1), 1 -31. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. (2016). Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation, Sem. Eval ’ 16, San Diego, California, June. Green, N. , Ashley, K. , Litman, D. , Reed C. , & Walker V. (2014) Proceedings of the First Workshop on Argumentation Mining, ACL Mohammad, S. M. , Sobhani, P. , & Kiritchenko, S. (2016). Stance and sentiment in tweets. ar. Xiv preprint ar. Xiv: 1605. 01655. 27 November 2020 Michael Wojatzki & Torsten Zesch 29

Backup Slides 27 November 2020 Michael Wojatzki & Torsten Zesch 30

Sem. Eval Stance Questionnaire 27 November 2020 31

Explicit Targets 27 November 2020 Michael Wojatzki & Torsten Zesch 32

Granularity of Targets • frequent vs. as explicit as possible ATHEISM every utterance • data-driven approach tires to find optimum 27 November 2020 Michael Wojatzki & Torsten Zesch 33

Annotating Explicit Stance • as we need textual evidence, look for key words • Matthew 25: 46, Christ, American Values, … • incorporate semantics • Do you have the feeling that the author is explicitly addressing the issue vs. continuing his logic Life after Death � ⊖ No Evidence for Religion Then they will go away to eternal punishment 27 November 2020 Michael Wojatzki & Torsten Zesch 34