Introduction to Natural Language Processing Speaker Email jllunchu

Introduction to Natural Language Processing Speaker: 呂瑞麟 國立中興大學資管系教授 Email: jllu@nchu. edu. tw URL: http: //web. nchu. edu. tw/~jlu

Outline n What and Why n Bag-of-Words Model n n Tokenization Stop-words Removal Stemming and Lemmatization Vectorizing

What and Why n 一般相信，自然語言處理（Natural Language Processing; NLP）大致是從 1950 年代開始 n n n 自從 Siri 問世，又再次爆紅 NLP 的挑戰主要在 speech recognition (speech-to-text), natural language understanding (NLU), and natural language generation (transforms structured data into plain-English content). AI-complete problem. (很有挑戰的意思)

What and Why n n 本章的範圍： allows working with text documents, considering their internal structure and the distribution of words Applications: n n 分類：國安、假新聞等 分群

What and Why n In NLP, a very common pipeline can be subdivided into the following steps: n n n Collecting a document (or sentences, twitter) into a corpus. Tokenizing, stopword (articles, prepositions and so on) removal, and stemming (reduction to radix-form). Building a common vocabulary. Vectorizing the documents. Classifying or clustering the documents.

What and Why n Python NLP 的重要套件 n n NLTK (Natural Language Toolkits) Before starting to work with NLTK, it's normally necessary to download all the additional elements (corpora, dictionaries, and so on) using a specific graphical interface. This can be done in the following way: n n n import nltk. download() 雖然課本建議完整下載，我覺得是用到的時候， 再下載。

Bag-of-Words Model n A text (such as a sentence or a document) is represented as the bag of its words, disregarding grammar and even word order.

Bag-of-Words Model n n Commonly used in methods of document classification where the (frequency of) occurrence of each word is used as a feature for training a classifier. Features 一般用來代表這個字在一份文件 的重要性。如何代表重要性很重要： n n 出現次數（可是 the, of 的次數很多，所以要 去除 stop words） 還有許多代表重要性的 measures, ex. tf-idf

Tokenization n splitting a text into atoms (sentences, words, or parts of words), normally defined as tokens. n sentence tokenizing: split large text into sentences, which are normally delimited by a full stop or another equivalent mark.

Example: Sentence Tokenizing n n from nltk. tokenize import sent_tokenize english_text = 'Where is the closest train station? I need to reach London. '

Example: Word Tokenizing n 利用 string 的 split n n text 1 = 'This is a simple text. ' text 2 = 'This isn't a simple text. Count 1, 2, 3 and then go!'

Example: Word Tokenizing n 利用 Treebank. Word. Tokenizer n n from nltk. tokenize import Treebank. Word. Tokenizer tbwt = Treebank. Word. Tokenizer() This tokenizer performs the following steps: 1. split standard contractions, e. g. don't -> do n't and they'll -> they ‘ll 2. treat most punctuation characters as separate tokens 3. split off commas and single quotes, when followed by whitespace 4. separate periods that appear at the end of line

Example: Word Tokenizing n 利用 Regexp. Tokenizer n n from nltk. tokenize import Regexp. Tokenizer rwt = Regexp. Tokenizer('[a-z. A-Z 0 -9']+’) n 稱之為 pattern，只保留字元是 a-z, A-Z, 0 -9, 和單引號

Regular Expression n Patterns are defined as meta characters, and there are three types of meta characters: n n Class Quantifier, and Position Misc.

Regular Expression n Class meta characters:

Regular Expression n Quantifiers:

Regular Expression n Position meta characters:

Regular Expression n Misc. meta characters:

Regular Expression n In python, meta characters are defined using string or raw string. n n n 以 'Hellon. World' 而言，raw string 會將包含 n 的內容 傳出去，而 string 則不會將 n 傳出去。 為了避免可能的困擾，在設定 regular expressions 時， 大多設定為 raw string 以手機號碼的範例來說： n n n import re msg = 'Her phone number is 2025 -0180. I also exchange my number 2025 -0195 with a recruiter. ' pattern = re. compile(r'd{4}-d{4}') results = pattern. findall(msg) print(results)

Stop-word removal n n 目前我個人的建議是：考量應用，再決定是 否執行 Stopwords are part of a normal speech (articles, conjunctions, and so on), but their occurrence frequency is very high and they don't provide any useful semantic information. n n For these reasons, it's a good practice to filter sentences and corpora by removing them all. NLTK provides lists of stopwords for the most common languages

Example: Stopword removal n n from nltk. corpus import stopwords sw = set(stopwords('english')) n n sw 包含所有 NLTK 的 stopwords rwt = Regexp. Tokenizer('[a-z. A-Z 0 -9']+') tok 1 = rwt. tokenize(text 1) clean_tok 1 = [t for t in tok 1 if t not in sw]

Example: Language detection n from langdetect import detect n n n 若有下列訊息，須執行 conda install langdetect(text 2) detect('機器學習')

Example: Stopword removal n 其他套件：langid、langua (zh-tw) 等

Stemming and Lemmatization n Both techniques tackle the same idea: Reduce a word to its root or base unit. n n n It’s often a data pre-processing step both techniques are sometimes called text (or word) normalization. 為了降低一份文件可能產生的 token 數

Stemming and Lemmatization n n stemming（字幹搜尋）：有一種說法叫做 “去字尾”。running 和 runs run lemmatization（詞形還原）：running, runs, ran run 如果只處理英文，NLTK 有 Porter. Stemmer 和 Lancaster. Stemmer 若有其他語系，則有 Snowball. Stemmer

Example: Stemming n n n from nltk. stem. snowball import Porter. Stemmer from nltk. stem. lancaster import Lancaster. Stemmer from nltk. stem. snowball import Snowball. Stemmer ps = Porter. Stemmer() ls = Lancaster. Stemmer() ss = Snowball. Stemmer(language='english') n n 必須加上 language 的參數 右表是支援的語系

Example: Stemming n n n w 1 = 'flies' w 2 = 'teen' w 3 = 'teenager' ps. stem(w 1) ls. stem(w 1) ss. stem(w 1) ps. stem(w 2) ls. stem(w 2) ss. stem(w 2) ps. stem(w 3) ls. stem(w 3) ss. stem(w 3)

Example: Lemmatization n n from nltk. stem import Word. Net. Lemmatizer wt = Word. Net. Lemmatizer() wt. lemmatize(w 1) wt. lemmatize(w 2) wt. lemmatize(w 3)

Text Vectorization n transforming text tokens into numerical vectors. n n The most common techniques are based on a count or frequency computation the whole corpus must be processed in order to determine how many unique tokens are present and their frequencies

Example: Count vectorizing n n n n n from sklearn. feature_extraction. text import Count. Vectorizer corpus = [ 'This is a simple test corpus', 'A corpus is a set of text documents', 'We want to analyze the corpus and the documents', 'Documents can be automatically tokenized' ] cv = Count. Vectorizer() vectorized_corpus = cv. fit_transform(corpus) n vectorized_corpus is a term-document matrix

Example: Count vectorizing n cv. vocabulary_ n 包含 terms 以及其相對應的 index

Example: Count vectorizing n Exercise: n n 請將 corpus 的內容先依據之前說的 preprocessing（含 stopword removal 和 stemming/lemmatization 等），再建立 termdocument matrix. 然後比較各種的 size （vocabulary, termdocument matrix）

N-Grams n So far, we have considered only single tokens (also called unigrams), but in many contexts, it's useful to consider short sequences of words (bigrams or trigrams) as atoms for our classifiers, just like all the other tokens. n n For example, if we are analyzing the sentiment of some texts, it could be a good idea to consider bigrams such as pretty good, very bad, and so on. In addition to words, a gram can also be a character. Given a sequence of letters (for example, the sequence "for ex"), what is the likelihood of the next letter?

N-Grams n n n You can think of an N-gram as the sequence of N words Take ‘please turn your homework’ as an example, if N = 2, a 2 -gram (or bigram) is a two-word sequence of words like “please turn”, “turn your”, or ”your homework” If N = 3, then “please turn your”, or “turn your homework”.

N-Grams n n Intuitive formula is P(w|h), the probability of word w, given some history, h. If it is assumed that the probability of a word depends only on the previous word is also known as Markov assumption. That is,

N-Grams n If the corpus is n n n To find the probability of the word “you” following the word “thank”, we can write this as P (you | thank). n n n 'He said thank you', 'He said bye as he walked through the door', 'He went to San Diego', 'San Diego has nice weather', 'It is raining in San Francisco’ =(No. of times “Thank You” occurs) / (No. of times “Thank” occurs) = 1/1 = 1 How about the probability of the word ‘Diego’ following the word ‘San’?

N-Grams: Example n n n from sklearn. feature_extraction. text import Count. Vectorizer corpus = [ 'He said thank you', 'He said bye as he walked through the door', 'He went to San Diego', 'San Diego has nice weather', 'It is raining in San Francisco’ ] cv = Count. Vectorizer(ngram_range=(1, 2)) vectorized_corpus = cv. fit_transform(corpus) cv. vocabulary_ ngram_range=(min_n, max_m)

N-Grams: Example he, 10

Tf-idf vectorizing n The goal of vectorizing is normally preparing the data for a classifier; therefore it's necessary to avoid features that are present very often, because their information decreases when the number of global occurrences increases. n n For example, in a corpus about a sport, the word match could be present in a huge number of documents; therefore it's almost useless as a classification feature. Count vectorizing does not consider the number of global occurrences.

Tf-idf vectorizing n n If we have a corpus C with n documents, we define termfrequency, the number of times a token occurs in a document, as: We define inverse-document-frequency, as the following measure: In other words, idf(t, C) measures how much information is provided by every single term. In fact, if count(D, t) = n, it means that a token is always present and idf(t, C) comes close to 0, and vice-versa. The term 1 in the denominator is a correction factor, which avoids null idf for count(D, t) = n.

Tf-idf vectorizing n Therefore, tf-idf is defined as follows:

Example: Tf-idf vectorizing n n n from sklearn. feature_extraction. text import Tfidf. Vectorizer tfidfv = Tfidf. Vectorizer() vectorized_corpus = tfidfv. fit_transform(corpus) n ‘the’ (第三份文件，第 14 個字) 的 tf-idf 還是接近 0. 64

A sample text classifier based on the Reuters corpus n 先下載 Reuters corpus n n n import nltk. download('reuters') Reuters 總共分成 90 個 categories n 為了簡化，我們只取出其中兩類：rubber and cotton

A sample text classifier based on the Reuters corpus n n from nltk. corpus import reuters 和其他的 datasets 比較難找到好用的 API 文件： n n http: //www. nltk. org/howto/corpus. html#corpusreader-classes 的 Data Access Methods 或者以圖示的方法

A sample text classifier based on the Reuters corpus n 取出屬於 rubber 和 cotton categories 的文件 n n n Xr = np. array(reuters. sents(categories='rubber')) Xc = np. array(reuters. sents(categories='cotton')) Xw = np. concatenate((Xr, Xc))

A sample text classifier based on the Reuters corpus n As each document is already split into tokens, we need to rebuild the full sentences: n n n X = [] for document in Xw: X. append(' '. join(document). strip(). lower())

A sample text classifier based on the Reuters corpus n 對每一份文件內的 tokens 進行處理（包含 stopword removal 等）: n n n from nltk. tokenize import Regexp. Tokenizer from nltk. stem. snowball import Snowball. Stemmer from nltk. corpus import stopwords sw = set(stopwords('english')) rwt = Regexp. Tokenizer('[a-z. A-Z 0 -9']+') ess = Snowball. Stemmer('english', ignore_stopwords=True) ignore_stopwords: If set to True, stopwords are not stemmed and returned unchanged.

A sample text classifier based on the Reuters corpus n 將 token 內的 stop-word 移除，並且仍然 維持 list 的資料結構 n [t for t in rwt. tokenize(X[0]) if t not in sw] n 留意：in 不見了

A sample text classifier based on the Reuters corpus n 我們還需要把前面處理後的 t 進行 stemming/lemmatization，因此要改成 n [ess. stem(t) for t in rwt. tokenize(X[0]) if t not in sw] n 留意：widens widen

A sample text classifier based on the Reuters corpus n 由於我們之前利用 Tfidf. Vectorizer 處理的 corpus 是字串，而我們目前有的卻是 tokens， 這時我們需要利用 join() 把 tokens 組成字串 n ' '. join([ess. stem(t) for t in rwt. tokenize(X[0]) if t not in sw])

A sample text classifier based on the Reuters corpus n 最後，我們利用迴圈把 X 處理完，並將結果儲存到 Xnew n n Xnew = [] for document in X: toks = rwt. tokenize(document) Xnew. append(' '. join([ess. stem(t) for t in toks if t not in sw]))

A sample text classifier based on the Reuters corpus n 利用 Tfidf. Vectorizer 產生 term-document matrix n n n from sklearn. feature_extraction. text import Tfidf. Vectorizer tfidfv = Tfidf. Vectorizer() Xv = tfidfv. fit_transform(Xnew) (1016, 2806): 共 1016 分文件，每份文件以 2806 的 vector 呈現（移除 stopword 之後，共有 2806 個字） 建議看一下課本的程式碼，尤其是 tokenizer 的定義與 使用方式，這比我們的寫法還精減

A sample text classifier based on the Reuters corpus n 資料前處理結束了，我們可以開始跑分類 了。先定義 Y，我們假設 rubber 是 0，而 cotton 是 1 n n n Yr = np. zeros(shape=Xr. shape) Yc = np. ones(shape=Xc. shape) Y = np. concatenate((Yr, Yc))

A sample text classifier based on the Reuters corpus n 利用 Logistic. Regression 來分類 n n n from sklearn. model_selection import train_test_split from sklearn. linear_model import Logistic. Regression X_train, X_test, Y_train, Y_test = train_test_split(Xv, Y, test_size=0. 25, random_state=100) rf = Logistic. Regression(max_iter=500) rf. fit(X_train, Y_train) rf. score(X_test, Y_test)

中文處理 n n n n n i� mport jieba. set_dictionary('python/data/dict. txt. big’) sen = "我們在野生動物園玩" � ## segmentation/tokenizing words = jieba. cut(sen, cut_all=False) # 兩種方式列印結果 � print(" ". join(words)) ## for word in words: print(word)

中文處理 n n n POS Tagging i� mport jieba. posseg as pseg words = pseg. cut(sen) for word, flag in words: print('%s %s' % (word, flag))

References n n Language Models: N-Gram An Introduction to N-grams

References n n Stemming and Lemmatization in Python jieba: python 的中文斷詞系統 n n NLP 中文斷詞最方便的開源 具之一 — Jieba 中研院的 https: //github. com/ckiplab/ckiptagger