Documentlevel Semantic Orientation and Argumentation Presented by Marta

Document-level Semantic Orientation and Argumentation Presented by Marta Tatu CS 7301 March 15, 2005

or ? Semantic Orientation Applied to Unsupervised Classification of Reviews Peter D. Turney ACL-2002

Overview Unsupervised learning algorithm for classifying reviews as recommended or not recommended p The classification is based on the semantic orientation of the phrases in the review which contain adjectives and adverbs p 3

Algorithm p p p Input: review Identify phrases that contain adjectives or adverbs by using a part-of-speech tagger Estimate the semantic orientation of each phrase Assign a class to the given review based on the average semantic orientation of its phrases Output: classification ( or ) 4

Step 1 p p Apply Brill’s part-of-speech tagger on the review Adjective are good indicators of subjective sentences. In isolation: n p unpredictable steering ( ) / plot ( ) Extract two consecutive words: one is an adjective or adverb, the other provides the context First Word Second Word Third Word (not extracted) 1. JJ NN or NNS Anything 2. RB, RBR, or RBS JJ Not NN nor NNS 3. JJ JJ Not NN nor NNS 4. NN or NNS JJ Not NN nor NNS 5. RB, RBR, or RBS VB, VBD, VBN, or VBG Anything 5

Step 2 p p Estimate the semantic orientation of the extracted phrases using PMI-IR (Turney, 2001) Pointwise Mutual Information (Church and Hanks, 1989): p Semantic Orientation: p PMI-IR estimates PMI by issuing queries to a search engine (Altavista, ~350 million pages) 6

Step 2 – continued Added 0. 01 to hits to avoid division by zero p If hits(phrase NEAR “excellent”) and hits(phrase NEAR “poor”)≤ 4, then eliminate phrase p Added “AND (NOT host: epinions)” to the queries not to include the Epinions website p 7

Step 3 p p p Calculate the average semantic orientation of the phrases in the given review If the average is positive, then If the average is negative, then Phrase POS tags SO direct deposit JJ NN 1. 288 local branch JJ NN 0. 421 small part JJ NN 0. 053 online service JJ NN 2. 780 well other RB JJ 0. 237 low fees JJ NNS 0. 333 … true service JJ NN -0. 732 other bank JJ NN -0. 850 inconveniently located RB VBN -1. 541 Average Semantic Orientation 0. 322 8

Experiments p 410 reviews from Epinions n n n p 170 (41%) ( ) 240 (59%) ( ) Average phrases per review: 26 Baseline accuracy: 59% Domain Accuracy Correlation Automobiles 84. 00% 0. 4618 Banks 80. 00% 0. 6167 Movies 65. 83% 0. 3608 Travel Destinations 70. 53% 0. 4155 All 74. 39% 0. 5174 9

Discussion p What makes the movies hard to classify? n n n The average SO tends to classify a recommended movies as not recommended Evil characters make good movies The whole is not necessarily the sum of the parts Good beaches do not necessarily add up to a good vacation p But good automobile parts usually add up to a good automobile p 10

Applications Summary statistics for search engines p Summarization of reviews p n p Pick out the sentence with the highest positive/negative semantic orientation given a positive/negative review Filtering “flames” for newsgroups n When the semantic orientation drops below a threshold, the message might be a potential flame 11

Questions ? Comments ? p Observations ? p 12

? Sentiment Classification using Machine Learning Techniques Bo Pang, Lillian Lee and Shivakumar Vaithyanathan EMNLP-2002

Overview Consider the problem of classifying documents by overall sentiment p Three machine learning methods besides the human-generated lists of words p n n n Naïve Bayes Maximum Entropy Support Vector Machines 14

Experimental Data Movie-review domain p Source: Internet Movie Database (IMDb) p Stars or numerical value ratings converted into positive, negative, or neutral » no need to hand label the data for training or testing p Maximum of 20 reviews/author/sentiment category p n n n 752 negative reviews 1301 positive reviews 144 reviewers 15

List of Words Baseline p p p Maybe there are certain words that people tend to use to express strong sentiments Classification done by counting the number of positive and negative words in the document Random-choice baseline: 50% 16

Machine Learning Methods p Bag-of-features framework: n n {f 1, …, fm} predefined set of m features ni(d) = number of times fi occurs in document d n p (Naïve Bayes) 17

Machine Learning Methods – continued (Maximum Entropy) p where Fi, c is a feature/class function: p Support vector machines: Find hyperplane that maximizes the margin. The constraint optimization problem: n cj is the correct class of document dj 18

Evaluation 700 positive-sentiment and 700 negativesentiment documents p 3 equal-sized folds p The tag “NOT_” was added to every word between a negation word (“not”, “isn’t”, “didn’t”) and the first punctuation mark p n p “good” is opposite to “not very good” Features: n n 16, 165 unigrams appearing at least 4 times in the 1400 -document corpus 16, 165 most often occurring bigrams in the same data 19

Results p POS information added to differentiate between: “I love this movie” and “This is a love story” 20

Conclusion p Results produced by the machine learning techniques are better than the humangenerated baselines n n SVMs tend to do the best Unigram presence information is the most effective Frequency vs. presence: “thwarted expectation”, many words indicative of the opposite sentiment to that of the entire review p Some form of discourse analysis is necessary p 21

Questions ? Comments ? p Observations ? p 22

Summarizing Scientific Articles: Experiments with Relevance and Rhetorical Status Simone Teufel and Marc Moens CL-2002

Overview Summarization of scientific articles: restore the discourse context of extracted material by adding the rhetorical status of each sentence in the document p Gold standard data for summaries consisting of computational linguistics articles annotated with the rhetorical status and relevance for each sentence p Supervised learning algorithm which classifies sentences into 7 rhetorical categories p 24

Why? p Knowledge about the rhetorical status of the sentence enables the tailoring of the summaries according to user’s expertise and task n n p Nonexpert summary: background information and the general purpose of the paper Expert summary: no background, instead differences between this approach and similar ones Contrasts or complementarity among articles can be expressed 25

Rhetorical Status p p p Generalizations about the nature of scientific texts + information to enable the construction of better summaries Problem structure: problems (research goals), solutions (methods), and results Intellectual attribution: what the new contribution is, as opposed to previous work and background (generally accepted statements) Scientific argumentation Attitude toward other people’s work: rival approach, prior approach with a fault, or an approach contributing parts of the authors’ own solution 26

Metadiscourse and Agentivity p Metadiscourse is an aspect of scientific argumentation and a way of expressing attitude toward previous work n p “we argue that”, “in contrast to common belief, we” Agent roles in argumentation: rivals, contributors of part of the solution (they), the entire research community, or the authors of the paper (we) 27

Citations and Relatedness Just knowing that an article cites another is often not enough p One needs to read the context of the citation to understand the relation between the articles p n n Article cited negatively or contrastively Article cited positively or in which the authors state that their own work originates from the cited work 28

Rhetorical Annotation Scheme p p p Only one category assigned to each full sentence Nonoverlapping, nonhierarchical scheme The rhetorical status is determined on the basis of the global context of the paper 29

Relevance Select important content from text p Highly subjective » low human agreement p Sentence is considered relevant if it describes the research goal or states a difference with a rival approach p Other definitions: relevant sentence if it shows a high level of similarity with a sentence in the abstract p 30

Corpus p 80 conference articles n n n p Association for Computational Linguistics (ACL) European Chapter of the Association for Computational Linguistics (EACL) Applied Natural Language Processing (ANLP) International Joint Conference on Artificial Intelligence (IJCAI) International Conference on Computational Linguistics (COLING). XML markups added 31

The Gold Standard 3 tasked-trained annotators p 17 pages of guidelines p 20 hours of training p No communication between annotators p Evaluation measures of the annotation: p n n Stability Reproducibility 32

Results of Annotation p Kappa coefficient K (Siegel and Castellan, 1988) where P(A)= pairwise agreement and P(E)= random agreement p Stability: K=. 82, . 81, . 76 (N=1, 220 and k=2) p Reproducibility: K=. 71 33

The System p Supervised machine learning Naïve Bayes 34

Features p Absolute location of a sentence n Limitations of the author’s own method can be expected to be found toward the end, while limitations of other researchers’ work are discussed in the introduction 35

Features – continued p Section structure: relative and absolute position of sentence within section: n p First, last, second or third, second-last or thirdlast, or either somewhere in the first, second, or last third of the section Paragraph structure: relative position of sentence within a paragraph n Initial, medial, or final 36

Features – continued p Headlines: type of headline of current section n p Introduction, Implementation, Example, Conclusion, Result, Evaluation, Solution, Experiment, Discussion, Method, Problems, Related Work, Data, Further Work, Problem Statement, or Non-Prototypical Sentence length n Longer or shorter than 12 words (threshold) 37

Features – continued Title word contents: does the sentence contain words also occurring in the title? p TF*IDF word contents p n n p High values to words that occur frequently in one document, but rarely in the overall collection of documents Do the 18 highest-scoring TF*IDF words belong to the sentence? Verb syntax: voice, tense, and modal linguistic features 38

Features – continued p Citation n p Citation (self), citation (other), author name, or none + location of the citation in the sentence (beginning, middle, or end) History: most probable previous category n n AIM tends to follow CONTRAST Calculated as a second pass process during training 39

Features – continued p Formulaic expressions: list of phrases described by regular expressions, divided into 18 classes, comprising a total of 644 patterns n Clustering prevents data sparseness 40

Features – continued p Agent: 13 types, 167 patterns n n The placeholder WORK_NOUN can be replaced by a set of 37 nouns including theory, method, prototype, algorithm Agent classes with a distribution very similar with the overall distribution of target categories were excluded 41

Features – continued p Action: 365 verbs clustered into 20 classes based on semantic concepts such as similarity, contrast n n n PRESENTATION_ACTIONs: present, report, state RESEARCH_ACTIONs: analyze, conduct, define, and observe Negation is considered 42