The Dream Itd be great if machines could

  • Slides: 60
Download presentation

The Dream • It’d be great if machines could – Process our email (usefully)

The Dream • It’d be great if machines could – Process our email (usefully) – Translate languages accurately – Help us manage, summarize, and aggregate information – Use speech as a UI (when needed) – Talk to us / listen to us • But they can’t: – Language is complex, ambiguous, flexible, and subtle – Good solutions need linguistics and machine learning knowledge • So:

What is NLP? • • Fundamental goal: deep understand of broad language – Not

What is NLP? • • Fundamental goal: deep understand of broad language – Not just string processing or keyword matching! End systems that we want to build: – Ambitious: speech recognition, machine translation, information extraction, dialog interfaces, question answering, trend finding … – Modest: spelling correction, text categorization…

Speech Systems • Automatic Speech Recognition (ASR) – Audio in, text out – SOTA:

Speech Systems • Automatic Speech Recognition (ASR) – Audio in, text out – SOTA: 0. 3% for digit strings, 5% dictation, 50%+ TV • Text to Speech (TTS) – Text in, audio out – SOTA: totally intelligible (if sometimes unnatural) • Speech systems currently: – Model the speech signal – Model language

Machine Translation • Translation systems encode: – Something about fluent language – Something about

Machine Translation • Translation systems encode: – Something about fluent language – Something about how two languages correspond (middle of term) • SOTA: for easy language pairs, better than nothing, but more an understanding aid than a replacement for human translators

Information Extraction • Information Extraction (IE) – Unstructured text to database entries – SOTA:

Information Extraction • Information Extraction (IE) – Unstructured text to database entries – SOTA: perhaps 70% accuracy for multi-sentence temples, 90%+ for single easy fields

Question Answering • Question Answering: – More than search – Ask general comprehension questions

Question Answering • Question Answering: – More than search – Ask general comprehension questions of a document collection – Can be really easy: “What’s the capital of Wyoming? ” – Can be harder: “How many US states’ capitals are also their largest cities? ” – Can be open ended: “What are the main issues in the global warming debate? ” • SOTA: Can do factoids, even when text isn’t a perfect match

What is nearby NLP? • Computational Linguistics – Using computational methods to learn more

What is nearby NLP? • Computational Linguistics – Using computational methods to learn more about how language works – We end up doing this and using it • Cognitive Science – Figuring out how the human brain works – Includes the bits that do language – Humans: the only working NLP prototype! • Speech? – Mapping audio signals to text – Traditionally separate from NLP, converging? – Two components: acoustic models and language models – Language models in the domain of stat NLP

What is this Class? • Three aspects to the course: – Linguistic Issues •

What is this Class? • Three aspects to the course: – Linguistic Issues • What are the range of language phenomena? • What are the knowledge sources that let us disambiguate? • What representations are appropriate? – Technical Methods • Learning and parameter estimation • Increasingly complex model structures • Efficient algorithms: dynamic programming, search – Engineering Methods • Issues of scale • Sometimes, very ugly hacks • We’ll focus on what makes the problems hard, and what works in practice…

Class Requirements and Goals • Class requirements – Uses a variety of skills /

Class Requirements and Goals • Class requirements – Uses a variety of skills / knowledge: • Basic probability and statistics • Basic linguistics background • Decent coding skills (Java) – Most people are probably missing one of the above – We’ll address some review concepts with sections, TBD • Class goals – Learn the issues and techniques of statistical NLP – Build the real tools used in NLP (language models, taggers, parsers, translation systems) – Be able to read current research papers in the field – See where the gaping holes in the field are!

Rational versus Empiricist Approaches to Language (I) • Question: What prior knowledge should be

Rational versus Empiricist Approaches to Language (I) • Question: What prior knowledge should be built into our models of NLP? • Rationalist Answer: A significant part of the knowledge in the human mind is not derived by the senses but is fixed in advance, presumably by genetic inheritance (Chomsky: poverty of the stimulus). • Empiricist Answer: The brain is able to perform association, pattern recognition, and generalization and, thus, the structures of Natural Language can be learned.

Rational versus Empiricist Approaches to Language (II) • Chomskyan/generative linguists seek to describe the

Rational versus Empiricist Approaches to Language (II) • Chomskyan/generative linguists seek to describe the language module of the human mind (the Ilanguage) for which data such as text (the Elanguage) provide only indirect evidence, which can be supplemented by native speakers intuitions. • Empiricists approaches are interested in describing the E-language as it actually occurs. • Chomskyans make a distinction between linguistic competence and linguistic performance. They believe that linguistic competence can be described in isolation while Empiricists reject this notion.

Empiricist • Seeks methods that can work on raw text as it exists –

Empiricist • Seeks methods that can work on raw text as it exists – Knowledge induction (automatic learning), not by disambiguation • American structuralism – The work of Shannon – Assign probabilities on linguistic events compared to concentrating on categorical judgments about rare types of sentences

Some Early NLP History • 1950’s: – Foundational work: automata, information theory, etc. –

Some Early NLP History • 1950’s: – Foundational work: automata, information theory, etc. – First speech systems – Machine translation (MT) hugely funded by military (imagine that) • Toy models: MT using basically word-substitution – Optimism! • 1960’s and 1970’s: NLP Winter – Bar-Hillel (FAHQT) and ALPAC reports kills MT – Work shifts to deeper models, syntax – … but toy domains / grammars (SHRDLU, LUNAR) • 1980’s: The Empirical Revolution – – Expectations get reset Corpus-based methods become central Deep analysis often traded for robust and simple approximations Evaluate everything

Today’s Approach to NLP • From ~1970 -1989, people were concerned with the science

Today’s Approach to NLP • From ~1970 -1989, people were concerned with the science of the mind and built small (toy) systems that attempted to behave intelligently. • Recently, there has been more interest on engineering practical solutions using automatic learning (knowledge induction). • While Chomskyans tend to concentrate on categorical judgements about very rare types of sentences, statistical NLP practitioners concentrate on common types of sentences.

Why is NLP Difficult? • NLP is difficult because Natural Language is highly ambiguous.

Why is NLP Difficult? • NLP is difficult because Natural Language is highly ambiguous. • Example: “The company is training workers” has 2 or more parse trees (i. e. , syntactic analyses). • “List the sales of the products produced in 1973 with the products produced in 1972” has 455 parses. • Therefore, a practical NLP system must be good at making disambiguation decisions of word sense, word category, syntactic structure, and semantic scope.

Methods that don’t work well • Maximizing coverage while minimizing ambiguity is inconsistent with

Methods that don’t work well • Maximizing coverage while minimizing ambiguity is inconsistent with symbolic NLP. • Furthermore, hand-coded syntactic constraints and preference rules are time consuming to build, do not scale up well and are brittle in the face of the extensive use of metaphor in language. • Example: if we code animate being --> swallow --> physical object I swallowed his story, hook, line, and sinker. The supernova swallowed the planet.

Classical NLP: Parsing • Write symbolic or logical rules: • Use deduction systems to

Classical NLP: Parsing • Write symbolic or logical rules: • Use deduction systems to prove parses from words – Minimal grammar on “Fed raises” sentence: 36 parses – Simple 10 -rule grammar: 592 parses – Real-size grammar: many millions of parses • This scaled very badly, didn’t yield broad-coverage tools

NLP: Annotation • Much of NLP is annotating text with structure which specifies how

NLP: Annotation • Much of NLP is annotating text with structure which specifies how it’s assembled. – Syntax: grammatical structure – Semantics: “meaning, ” either lexical or compositional

What Made NLP Hard? • The core problems: – Ambiguity – Sparsity – Scale

What Made NLP Hard? • The core problems: – Ambiguity – Sparsity – Scale – Unmodeled Variables

Problem: Ambiguities • Headlines: – – – – Iraqi Head Seeks Arms Ban on

Problem: Ambiguities • Headlines: – – – – Iraqi Head Seeks Arms Ban on Nude Dancing on Governor’s Desk Juvenile Court to Try Shooting Defendant Teacher Strikes Idle Kids Stolen Painting Found by Tree Kids Make Nutritious Snacks Local HS Dropouts Cut in Half Hospitals Are Sued by 7 Foot Doctors • Why are these funny?

Syntactic Ambiguities • Maybe we’re sunk on funny headlines, but normal, boring sentences are

Syntactic Ambiguities • Maybe we’re sunk on funny headlines, but normal, boring sentences are unambiguous? • Our company is training workers. • Fed raises interest rates 0. 5 % in a measure against inflation

Dark Ambiguities • Dark ambiguities: most analyses are shockingly bad (meaning, they don’t have

Dark Ambiguities • Dark ambiguities: most analyses are shockingly bad (meaning, they don’t have an interpretation you can get your mind around) • Unknown words and new usages • Solution: We need mechanisms to focus attention on the best ones, probabilistic techniques do this

Semantic Ambiguities • Even correct tree-structured syntactic analyses don’t always nail down the meaning

Semantic Ambiguities • Even correct tree-structured syntactic analyses don’t always nail down the meaning Every morning someone’s alarm clock wakes me up John’s boss said he was doing better

Other Levels of Language • Tokenization/morphology: – What are the words, what is the

Other Levels of Language • Tokenization/morphology: – What are the words, what is the sub-word structure? – Often simple rules work (period after “Mr. ” isn’t sentence break) – Relatively easy in English, other languages are harder: • Segmentation • Morphology • Discourse: how do sentences relate to each other? • Pragmatics: what intent is expressed by the literal meaning, how to react to an utterance? • Phonetics: acoustics and physical production of sounds • Phonology: how sounds pattern in a language

Disambiguation for Applications • Sometimes life is easy – Can do text classification pretty

Disambiguation for Applications • Sometimes life is easy – Can do text classification pretty well just knowing the set of words used in the document, same for authorship attribution – Word-sense disambiguation not usually needed for web search because of majority effects or intersection effects (“jaguar habitat” isn’t the car) • Sometimes only certain ambiguities are relevant he hoped to record a world record • Other times, all levels can be relevant (e. g. , translation)

Problem: Scale • People did know that language was ambiguous! – …but they hoped

Problem: Scale • People did know that language was ambiguous! – …but they hoped that all interpretations would be “good” ones (or ruled out pragmatically) – …they didn’t realize how bad it would be

Corpora • A corpus is a collection of text – Often annotated in some

Corpora • A corpus is a collection of text – Often annotated in some way – Sometimes just lots of text – Balanced vs. uniform corpora • Examples – Newswire collections: 500 M+ words – Brown corpus: 1 M words of tagged “balanced” text – Penn Treebank: 1 M words of parsed WSJ – Canadian Hansards: 10 M+ words of aligned French / English sentences – The Web: billions of words of who knows what

Corpus-Based Methods • A corpus like a treebank gives us three important tools: –

Corpus-Based Methods • A corpus like a treebank gives us three important tools: – It gives us broad coverage

Corpus-Based Methods – It gives us statistical information This is a very different kind

Corpus-Based Methods – It gives us statistical information This is a very different kind of subject/object asymmetry than what many linguists are interested in.

Corpus-Based Methods – It lets us check our answers!

Corpus-Based Methods – It lets us check our answers!

Problem: Sparsity • However: sparsity is always a problem – New unigram (word), bigram

Problem: Sparsity • However: sparsity is always a problem – New unigram (word), bigram (word pair), and rule rates in newswire

The (Effective) NLP Cycle • Pick a problem (usually some disambiguation) • Get a

The (Effective) NLP Cycle • Pick a problem (usually some disambiguation) • Get a lot of data (usually a labeled corpus) • Build the simplest thing that could possibly work • Repeat: – See what the most common errors are – Figure out what information a human would use – Modify the system to exploit that information • Feature engineering • Representation design • Machine learning methods • We’re going to do this over and over again

Language isn’t Adversarial • One nice thing: we know NLP can be done! •

Language isn’t Adversarial • One nice thing: we know NLP can be done! • Language isn’t adversarial: – It’s produced with the intent of being understood – With some understanding of language, you can often tell what knowledge sources are relevant • But most variables go unmodeled – Some knowledge sources aren’t easily available (realworld knowledge, complex models of other people’s plans) – Some kinds of features are beyond our technical ability to model (especially cross-sentence correlations)

What Statistical NLP can do for us • Disambiguation strategies that rely on hand-coding

What Statistical NLP can do for us • Disambiguation strategies that rely on hand-coding produce a knowledge acquisition bottleneck and perform poorly on naturally occurring text. • A Statistical NLP approach seeks to solve these problems by automatically learning lexical and structural preferences from corpora. In particular, Statistical NLP recognizes that there is a lot of information in the relationships between words. • The use of statistics offers a good solution to the ambiguity problem: statistical models are robust, generalize well, and behave gracefully in the presence of errors and new data

Corpora • • • Brown Corpus – 1 million words British National Corpus –

Corpora • • • Brown Corpus – 1 million words British National Corpus – 100 mil. Words American National Corpus – 10 mil. words -> 100 Penn Tree. Bank - parsed WSJ text Canadian Hansard – parallel corpus (bilingual) Dictionaries: • Longman Dictionary of Contemporary English • Word. Net (hierarchy of synsets)

Things that can be done with Text Corpora (I) Word Counts • Word Counts

Things that can be done with Text Corpora (I) Word Counts • Word Counts to find out: – What are the most common words in the text. – How many words are in the text (word tokens and word types). – What the average frequency of each word in the text is. • Limitation of word counts: Most words appear very infrequently and it is hard to predict much about the behavior of words that do not occur often in a corpus. ==> Zipf’s Law.

Things that can be done with Text Corpora (II) Zipf’s Law • If we

Things that can be done with Text Corpora (II) Zipf’s Law • If we count up how often each word type of a language occurs in a large corpus and then list the words in order of their frequency of occurrence, we can explore the relationship between the frequency of a word, f, and its position in the list, known as its rank, r. • Zipf’s Law says that: f ∝ 1/r • Significance of Zipf’s Law: For most words, our data about their use will be exceedingly sparse. Only for a few words will we have a lot of examples.

Common words in Tom Sawyer

Common words in Tom Sawyer

Frequencies of frequencies in Tom Sawyer

Frequencies of frequencies in Tom Sawyer

Zipf's law in Tom Sawyer

Zipf's law in Tom Sawyer

Zipf's law in Tom Sawyer

Zipf's law in Tom Sawyer

Zipf’s Law

Zipf’s Law

Zipf's law for the Brown corpus

Zipf's law for the Brown corpus

Mandelbrot's formula for the Brown corpus

Mandelbrot's formula for the Brown corpus

Things that can be done with Text Corpora (III) Collocations • A collocation is

Things that can be done with Text Corpora (III) Collocations • A collocation is any turn of phrase or accepted usage where somehow the whole is perceived as having an existence beyond the sum of its parts (e. g. , disk drive, make up, bacon and eggs). • Collocations are important for machine translation. • Collocations can be extracted from a text (example, the most common bigrams can be extracted). However, since these bigrams are often insignificant (e. g. , “at the”, “of a”), they can be filtered.

순서 • bigram: of the, in the, to the, on the …. New York,

순서 • bigram: of the, in the, to the, on the …. New York, he said, as a • Filtering : : adjective+noun, noun+noun – last year, next year : : : 예외적

Commonest bigrams in the NYT

Commonest bigrams in the NYT

Filtered common bigrams in the NYT

Filtered common bigrams in the NYT

Things that can be done with Text Corpora (IV) Concordances • Finding concordances corresponds

Things that can be done with Text Corpora (IV) Concordances • Finding concordances corresponds to finding the different contexts in which a given word occurs. • One can use a Key Word In Context (KWIC) concordancing program. • Concordances are useful both for building dictionaries for learners of foreign languages and for guiding statistical parsers.

KWIC display

KWIC display

Syntactic frames for showed in Tom Sawyer

Syntactic frames for showed in Tom Sawyer

Why study NLP Statistically? • Up until the late 1980’s, NLP was mainly investigated

Why study NLP Statistically? • Up until the late 1980’s, NLP was mainly investigated using a rule-based approach. • However, rules appear too strict to characterize people’s use of language. • This is because people tend to stretch and bend rules in order to meet their communicative needs. • Methods for making the modeling of language more accurate are needed and statistical methods appear to provide the necessary flexibility.

Subdivisions of NLP • Parts of Speech and Morphology (words, their syntactic function in

Subdivisions of NLP • Parts of Speech and Morphology (words, their syntactic function in sentences, and the various forms they can take). • Phrase Structure and Syntax (regularities and constraints of word order and phrase structure). • Semantics (the study of the meaning of words (lexical semantics) and of how word meanings are combined into the meaning of sentences, etc. ) • Pragmatics (the study of how knowledge about the world and language conventions interact with literal meaning).

Topics Covered in this course

Topics Covered in this course

Tools and Resources Used • Probability/Statistical Theory: Statistical Distributions, Bayesian Decision Theory. • Linguistics

Tools and Resources Used • Probability/Statistical Theory: Statistical Distributions, Bayesian Decision Theory. • Linguistics Knowledge: Morphology, Syntax, Semantics and Pragmatics. • Corpora: Bodies of marked or unmarked text to which statistical methods and current linguistic knowledge can be applied in order to discover novel linguistic theories or interesting and useful knowledge organization.

Textbook and other useful information • Foundations of Statistical Natural Language Processing, by Chris

Textbook and other useful information • Foundations of Statistical Natural Language Processing, by Chris Manning and Hinrich Schütze, MIT Press, 1999. • Course Website: borame. cs. pusan. ac. kr