Natural Language Processing Lecture Notes 2 6122021 1

Natural Language Processing Lecture Notes 2 6/12/2021 1

Assignment 1 • Assignment 1 6/12/2021 2

Cosine similarity • Cosine of angle between two vectors • The denominator involves the lengths of the vectors. Normalization 6/12/2021 3

Next Lectures: Words • The next lectures will begin our study of “words” and will cover three related topics (Ch 2 -3): – Finite state automata – Finite state transducers – English morphology • We’ll start by reviewing regular expressions and finite state automata, and see where they can get us in NLP 6/12/2021 4

Chapter 2: Regular Expressions and Text Searching • Everybody does it – Emacs, vi, perl, python, awk, grep, sed, Word, netscape, etc. • Views of Regular Expressions: – Practical way to specify textual search strings (bells and whistles to make things easier) – Formal specifications of regular languages (only three operations are strictly needed: U, o, and *) 6/12/2021 5

Example • /$[0 -9]+(. [0 -9])? / • Matches: – It costs $765. 03 – It costs $871 • Let D = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} • $(DD* U DD*. DD) 6/12/2021 6

Use of Regular Expressions in NLP • Simple but powerful tools support large corpus analysis and “shallow processing” – What word is most likely to begin a declarative sentence vs. a question? – Which adjectives are conjoined most often with “excellent” and “poor”? – In your own email, are you more or less polite than the people you correspond with? (and with labeled data, how can a system learn this? ) – Which candidate for President is mentioned most often in the news (and with labeled data, which is mentioned most favorably)? – Word frequency and co-occurrence statistics – Simple interactive applications like Eliza 6/12/2021 7

Example • Find me all instances of the word “the” in a text. – /the/ – /[t. T]he/ – /b[t. T]heb/ • The second RE catches “the” when it begins a sentence (increasing Recall), while third RE does not match “the” when embedded in another word such as “other” (increasing Precision). 6/12/2021 8

Errors • The process we just went through was based on fixing two kinds of errors – Not matching things that we should have matched (The) • False negatives (decreasing Recall) – Matching strings that we should not have matched (there, then, other) • False positives (decreasing Precision) 6/12/2021 9

Errors cont. • We’ll be telling the same story for may tasks, all semester. Reducing the error rate for an application often involves two antagonistic efforts: – Increasing Precision (minimizing false positives) – Increasing Recall/Coverage (minimizing false negatives). 6/12/2021 10

Text Searches using REs • General: Ch 2 of the text • If regular expressions aren’t in your programming repertoire: decide which language you’ll use, and find a tutorial • Assignment 1 6/12/2021 11

Finite State Automata • Remember: Regular expressions and finite-state automata are equivalent (given any RE, an equivalent FSA can be constructed, and vice-versa) • FSAs and their close relatives are a theoretical foundation of much of the field of NLP 6/12/2021 12

FSAs as Graphs • Let’s start with the sheep language from the text – /baa+!/ 6/12/2021 13

Sheep FSA • We can say the following things about this machine – It has 5 states – At least b, a, and ! are in its alphabet – q 0 is the start state – q 4 is an accept state – It has 5 transitions (in this short-hand version) 6/12/2021 14

But note • There are other machines that recognize this language 6/12/2021 15

And Multiple REs… • /baaa*!/ • /baa+!/ • We’ll return to this point later… 6/12/2021 16

More Formally • You can specify an FSA by enumerating the following things (precisely: a DFSA). – The set of states: Q – A finite alphabet: Σ – A start state – A set of accept/final states – A transition function that maps QxΣ to Q 6/12/2021 17

About Alphabets • Don’t take that word to narrowly; it just means we need a finite set of symbols in the input. • These symbols can and will stand for bigger objects that can have internal structure. 6/12/2021 18

Dollars and Cents 6/12/2021 19

Another View • FSAs are also representable as tables 3 3 6/12/2021 20

Recognition • Recognition is the process of determining if a string should be accepted by a machine • Or… it’s the process of determining if a string is in the language we’re defining with the machine • Or… it’s the process of determining if a regular expression matches a string 6/12/2021 21

Recognition • Traditionally, (Turing’s idea) this process is depicted with a tape. 6/12/2021 22

Recognition • • • Simply a process of starting in the start state Examining the current input Consulting the table Going to a new state and updating the tape pointer. Until you run out of tape. 6/12/2021 23

D-Recognize 6/12/2021 24

D-Recognize (with technically correct DFSA) 6/12/2021 25

Key Points • Deterministic means that at each point in processing there is always one unique thing to do (no choices). • D-recognize is a simple table-driven interpreter • The algorithm is universal for all DFSAs. To recognize a new regular language, you merely change the alphabet and table. • So, searching for a string using a RE involves compiling the RE into a table, and passing the table to an interpreter. 6/12/2021 26

Example • What does the following regular expression do to the input “DOES SHE LIKE CLASS? ” • S/([A-Z]+) +([A-Z]+)/2 1/ 6/12/2021 27

Generative Formalisms • A Formal Language is a set of strings composed of symbols from a finite set of symbols (the alphabet). • Finite-state automata define formal languages (without having to enumerate all the strings in the language) • The term Generative is based on the view that you can run the machine as a generator to get strings from the language. 6/12/2021 28

Generative Formalisms • FSAs can be viewed from two perspectives: – Acceptors that can tell you if a string is in the language – Generators to produce all and only the strings in the language 6/12/2021 29

Non-Determinism A deterministic machine for talking sheep: A non-deterministic machine for talking sheep: In q 2, input a: move to q 3 or stay in q 2 6/12/2021 30

Non-Determinism cont. – Epsilon transitions – Key point: these transitions do not examine or advance the tape during recognition ε 6/12/2021 31

Equivalence • Non-deterministic machines can be converted to deterministic ones with a fairly simple construction • That means that they have the same power; non-deterministic machines are not more powerful than deterministic ones • It also means that one way to do recognition with a non-deterministic machine is to turn it into a deterministic one. 6/12/2021 32

Non-Deterministic Recognition • In a ND FSA there exists at least one path through the machine for a string that is in the language defined by the machine. • But not all paths directed through the machine for an accept string lead to an accept state. • No paths through the machine lead to an accept state for a string not in the language. 6/12/2021 33

Non-Deterministic Recognition • So success in a non-deterministic recognition occurs when a path is found through the machine that ends in an accept. • Failure occurs when none of the possible paths lead to an accept state. 6/12/2021 34

Example b q 0 6/12/2021 a q 1 a a q 2 ! q 3 q 4 35

Example 6/12/2021 36

Example 6/12/2021 37

Example 6/12/2021 38

Example 6/12/2021 39

Example 6/12/2021 40

Example 6/12/2021 41

Example 6/12/2021 42

Example 6/12/2021 43

Key Points • States in the search space are pairings of tape positions and states in the machine. • By keeping track of as yet unexplored states, a recognizer can systematically explore all the paths through the machine given an input. 6/12/2021 44

ND-Recognize Code 6/12/2021 45

Search Strategies • • If you’re not careful such searches can go into an infinite loop. Recall: Depth-first-search – Most recently created states considered first – Agenda is a stack – LIFO – Can get stuck in an infinite loop – More efficient in memory Recall: Breadth-first-search – States considered in creation order – Agenda is a queue – FIFO – Cannot get stuck in an infinite loop – Very inefficient in memory Dynamic programming/A*: later in the course 6/12/2021 46

ND-Recognize Code First(agenda) DFS: GEN-NEW-STATES(current-search-state) + agenda BFS: agenda + GEN-NEW-STATES(current-search-state) 6/12/2021 47

Why Bother? • Non-determinism doesn’t get us more formal power and it requires search, so why bother? – Often more natural solutions – Often much smaller 6/12/2021 48

Summing Up • Regular expressions, NDFSAs, and DFSAs can represent a subset of natural languages (regular languages) – Relatively easy to use – Can be difficult to scale up – Cannot represent all NL phenomena 6/12/2021 49