CSCE 771 Natural Language Processing NLTK Cookbook Text
CSCE 771 Natural Language Processing NLTK Cookbook Text Topics n Smoothing again: Readings: Chapters January 16, 2013
Python Text Processing with NLTK 2. 0 Cookbook 1. Tokenizing Text and Word. Net Basics 2. Replacing and Correcting Words 3. Creating Custom Corpora 4. Part-of-Speech Tagging 5. Extracting Chunks 6. Transforming Chunks and Trees 7. Text Classification 8. Distributed Processing and Handling Large Datasets 9. Parsing Specific Data – 2– CSCE 771 Spring 2013
Chapter 1. Tokenizing Text and Word. Net Basics In this chapter, we will cover: Tokenizing text into sentences Tokenizing sentences into words Tokenizing sentences using regular expressions Filtering stopwords in a tokenized sentence Looking up synsets for a word in Word. Net Looking up lemmas and synonyms in Word. Net Calculating Word. Net synset similarity Discovering word collocations – 3– CSCE 771 Spring 2013
Chapter 2. Replacing and Correcting Words In this chapter, we will cover: Stemming words Lemmatizing words with Word. Net Translating text with Babelfish Replacing words matching regular expressions Removing repeating characters Spelling correction with Enchant Replacing synonyms Replacing negations with antonyms Perkins, Jacob (2010 -11 -09). Python Text Processing with NLTK 2. 0 Cookbook (p. 25). Packt Publishing. Kindle Edition. – 4– CSCE 771 Spring 2013
Chapter 3. Creating Custom Corpora In this chapter, we will cover: Setting up a custom corpus Creating a word list corpus Creating a part-ofspeech tagged word corpus Creating a chunked phrase corpus Creating a categorized text corpus Creating a categorized chunk corpus reader Lazy corpus loading Creating a custom corpus view Creating a Mongo. DB backed corpus reader Corpus editing with file locking Perkins, Jacob (2010 -11 -09). Python Text Processing with NLTK 2. 0 Cookbook (p. 45). Packt Publishing. Kindle Edition. – 5– CSCE 771 Spring 2013
Chapter 4. Part-of-Speech Tagging 1. Default tagging 2. Training a unigram part-of-speech tagger 3. Combining taggers with backoff tagging 4. Training and combining 5. Ngram taggers 6. Creating a model of likely word tags 7. Tagging with regular expressions 8. Affix tagging 9. Training a Brill tagger 10. Training the Tn. T tagger 11. Using Word. Net for tagging Tagging proper names – 6– CSCE 771 Spring 2013
Chapter 5. Extracting Chunks In this chapter, we will cover: Chunking and chinking with regular expressions Merging and splitting chunks with regular expressions Expanding and removing chunks with regular expressions Partial parsing with regular expressions Training a tagger-based chunker Classification-based chunking Extracting named entities Extracting proper noun chunks Extracting location chunks Training a named entity chunker Perkins, Jacob (2010 -11 -09). Python Text Processing with NLTK 2. 0 Cookbook (p. 111). Packt Publishing. Kindle Edition. – 7– CSCE 771 Spring 2013
Chapter 6. Transforming Chunks and Trees In this chapter, we will cover: Filtering insignificant words Correcting verb forms Swapping verb phrases Swapping noun cardinals Swapping infinitive phrases Singularizing plural nouns Chaining chunk transformations Converting a chunk tree to text Flattening a deep tree Creating a shallow tree Converting tree nodes Perkins, Jacob (2010 -11 -09). Python Text Processing with NLTK 2. 0 Cookbook (p. 143). Packt Publishing. Kindle Edition. – 8– CSCE 771 Spring 2013
Chapter 7. Text Classification In this chapter, we will cover: Bag of Words feature extraction Training a naive Bayes classifier Training a decision tree classifier Training a maximum entropy classifier Measuring precision and recall of a classifier Calculating high information words Combining classifiers with voting Classifying with multiple binary classifiers Perkins, Jacob (2010 -11 -09). Python Text Processing with NLTK 2. 0 Cookbook (p. 167). Packt Publishing. Kindle Edition. – 9– CSCE 771 Spring 2013
Chapter 8. Distributed Processing and Handling Large Datasets In this chapter, we will cover: Distributed tagging with execnet Distributed chunking with execnet Parallel list processing with execnet Storing a frequency distribution in Redis Storing a conditional frequency distribution in Redis Storing an ordered dictionary in Redis Distributed word scoring with Redis and execnet Perkins, Jacob (2010 -11 -09). Python Text Processing with NLTK 2. 0 Cookbook (p. 201). Packt Publishing. Kindle Edition. – 10 – CSCE 771 Spring 2013
Chapter 9. Parsing Specific Data In this chapter, we will cover: Parsing dates and times with Dateutil Time zone lookup and conversion Tagging temporal expressions with Timex Extracting URLs from HTML with lxml Cleaning and stripping HTML Converting HTML entities with Beautiful. Soup Detecting and converting character encodings Perkins, Jacob (2010 -11 -09). Python Text Processing with NLTK 2. 0 Cookbook (p. 227). Packt Publishing. Kindle Edition. – 11 – CSCE 771 Spring 2013
– 12 – CSCE 771 Spring 2013
– 13 – CSCE 771 Spring 2013
– 14 – CSCE 771 Spring 2013
– 15 – CSCE 771 Spring 2013
– 16 – CSCE 771 Spring 2013
– 17 – CSCE 771 Spring 2013
- Slides: 17