Sentiment Lexicon Creation from Lexical Resources Bas Heerschop

Sentiment Lexicon Creation from Lexical Resources Bas Heerschop Alexander Hogenboom Flavius Frasincar Erasmus School of Economics Erasmus University Rotterdam basheerschop@gmail. com Erasmus School of Economics Erasmus University Rotterdam hogenboom@ese. eur. nl Erasmus School of Economics Erasmus University Rotterdam frasincar@ese. eur. nl June 15, 2011 BIS 2011

2 Outline • Introduction • Sentiment Lexicon Creation • Framework • Performance • Conclusions • Future Work BIS 2011

3 Introduction (1) • The Web offers an overwhelming amount of textual data, containing traces of sentiment • Insight into sentiment is crucial for, e. g. , financial markets, reputation management, and marketing • The challenge of automatically extracting sentiment from an ever-growing amount of data can be addressed by sentiment mining techniques • Sentiment mining is typically focused on determining the polarity of natural language texts BIS 2011

4 Introduction (2) • Existing sentiment mining approaches are typically based on word frequencies, yet there is a tendency of involving various other aspects of content • Most approaches rely on lists of words and their sentiment scores: sentiment lexicons • Existing lexicon creation methods have been assessed with respect to a manually created lexicon and have not been properly compared yet • Which sentiment lexicon creation method performs well in the actual sentiment mining process? BIS 2011

5 Sentiment Lexicon Creation (1) • Manual creation is cumbersome • Alternative: exploiting (vast) lexical resources • A popular lexical resource is Word. Net: – – Freely available, on-line semantic lexical resource Designed to be used under program control Organized into sets of synonyms (synsets) Synsets are linked to one another through several relations (e. g. , synonymy, antonymy, hyponymy, or meronymy) BIS 2011

6 Sentiment Lexicon Creation (2) • Possible method: traversing relations in lexical resource (Kim and Hovy 2004, Hu and Liu 2004, Lerman et al. 2009) • Start with manually created set with score 1 for positive synsets and score -1 for negative synsets • Iteratively propagate sentiment to related synsets (using Word. Net relations) • Weaken propagated score each iteration • Resulting scores range from -1 (very negative) to 1 (very positive) BIS 2011

7 Sentiment Lexicon Creation (3) BIS 2011

8 Sentiment Lexicon Creation (4) • Alternative: Page. Rank-based propagation to similar synsets (Esuli and Sebastiani 2007) • Synsets are linked by means of the words (references to synsets) used in their glosses (descriptions) • Iteratively update sentiment of each synset with a weighted average of a constant and the sentiment of its related synsets, proportionally to the total number of associations of these related synsets (using Extended Word. Net synset relations based on glosses) • Execute for manually created positive and negative seed set and combine obtained scores into scores ranging from -1 (very negative) to 1 (very positive) BIS 2011

9 Sentiment Lexicon Creation (5) BIS 2011

10 Sentiment Lexicon Creation (6) BIS 2011

11 Sentiment Lexicon Creation (7) • Alternatively, glosses can be analyzed by means of classifiers: Senti. Word. Net (Esuli and Sebastiani 2006) • Synsets are classified as objective, positive, or negative by eight ternary classifiers • Scores are calculated as proportion of classifiers assigning the three respective labels • Sentiment scores are calculated by subtracting negativity from positivity scores, yielding scores ranging from -1 (very negative) to 1 (very positive) • Classifiers differ in training data (expansion of seed set using Word. Net relations) and learning approaches (Support Vector Machines and Rocchio classifiers) BIS 2011

12 Sentiment Lexicon Creation (8) BIS 2011

13 Framework • Sentiment lexicon creation and subsequent lexiconbased document scoring • Document scoring involves initial per-sentence wordlevel Part-of-Speech (POS) tagging, lemmatizing, and Word Sense Disambiguation (WSD) • Words are then assigned scores in the range [-1, 1], retrieved from the sentiment lexicon • The sum of word scores is used to classify a document as positive (1) or negative (-1) BIS 2011

14 Performance (1) • Implementation in C#, Microsoft SQL Server database, Open. NLP-based POS tagger, Word. Net API for lemmatization and WSD • Evaluation on 1, 000 positive and 1, 000 negative English movie reviews (Pang and Lee 2004): – Traversing Word. Net relations (WN) – Page. Rank-based propagation of seed set (PRS) and bootstrapped with Senti. Word. Net scores (PRSWN) – Senti. Word. Net (SWN) • Evaluation measures: precision, recall, and F 1, as well as overall accuracy and macro-level F 1 BIS 2011

15 Performance (2) Positive Negative Overall Method Prec. Rec. F 1 Prec. Rec. WN 51. 0% 94. 3% 66. 2% 62. 3% 9. 4% 16. 3% 51. 9% 41. 3% PRS 49. 8% 86. 6% 63. 3% 48. 6% 12. 5% 19. 9% 49. 7% 41. 6% PRSWN 49. 6% 43. 0% 46. 1% 49. 7% 56. 3% 52. 8% 49. 7% 49. 4% 84. 3% 67. 5% 68. 8% 34. 6% 46. 0% 57. 5% 58. 8% SWN 56. 3% BIS 2011 F 1 Acc. F 1

16 Conclusions • Many existing sentiment mining approaches rely on lexical resources, which can be created in various ways • We have evaluated exploiting semantic relations, Page. Rank-based algorithms, and machine learning (Senti. Word. Net) for sentiment lexicon creation • Overall, Senti. Word. Net outperforms the other methods on our corpus, yet Page. Rank-based propagation yields the least biased sentiment classifier BIS 2011

17 Future Work • Investigate sentiment lexicon creation methods yielding less biased classifiers • Develop and assess other sentiment lexicon creation methods, e. g. , by propagating document scores to word scores • Compare the performance of different methods on a manually created lexicon such as Micro-WN(Op) BIS 2011

18 Questions? • Feel free to contact: Alexander Hogenboom Erasmus School of Economics Erasmus University Rotterdam P. O. Box 1738, 3000 DR, The Netherlands hogenboom@ese. eur. nl BIS 2011

19 References • Esuli, A. , Sebastiani, F. : SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining. In: 5 th Conference on Language Resources and Evaluation (LREC 2006), European Language Resources Association (ELRA) (2006) 417— 422 • Esuli, A. , Sebastiani, F. : Page. Ranking Word. Net Synsets: An Application to Opinion Mining. In: 45 th Annual Meeting of the Association of Computational Linguistics (ACL 2007), ACL (2007) 424— 431 • Hu, M. , Liu, B. : Mining and Summarizing Customer Reviews. In: 10 th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2004), ACM (2004) 168— 177 • Kim, S. , Hovy, E. : Determining the Sentiment of Opinions. In: 20 th International Conference on Computational Linguistics (COLING 2004), ACL (2004) 1367 • Lerman, K. , Blair-Goldensohn, S. , Mc. Donald, R. : Sentiment Summarization: Evaluating and Learning User Preferences. In: 12 th Conference of the European Chapter of the ACL (EACL 2009), ACL (2009) 514— 522 • Pang, B. , Lee, L. : A Sentimental Education: Sentiment Analysis using Subjectivity Summarization based on Minimum Cuts. In: 42 nd Annual Meeting of the Association for Computational Linguistics (ACL 2004), ACL (2004) 271— 280 BIS 2011