11 761 Language and Statistics Spring 2016 Roni

11 -761 Language and Statistics Spring 2016 Roni Rosenfeld http: //www. cs. cmu. edu/~roni/11761 -s 16/

Course Goals and Style u u Carnegie Mellon Teaching statistical foundation and techniques for language technologies Plugging gaping holes in LTI/CS grad student education in probability, statistics and information theory. 2

Course philosophy u Socratic Method l l u u based on how fast we move Lots of Probability, Statistics, Information theory l u Will try to maintain in spite of large class size participation strongly encouraged (pls state your name) Highly interactive Highly adaptable l u Carnegie Mellon not in the abstract, but rather as the need arises Lectures emphasize intuition, not rigor or detail l background reading will have rigor & detail 3

Course Prerequisites & Mechanics u You need to be able to program, from scratch. l u u u Largest program is O(100) lines You need to be comfortable with probabilities l u Carnegie Mellon Can you derive Bayes equation in your sleep? 11661 (masters level): no final project Hand in assignments via Blackboard Vigorous enforcement of collaboration & disclosure policy 4

Background Material Carnegie Mellon No single book exists which covers the course material. u “Foundations of Statistical NLP”, Manning & Schutze l u u “Statistical Methods in Speech Recognition”, Jelinek “Text Compression”, Bell, Cleary & Witten l u u u Computational Linguistics perspective first 4 chapters; rest is mostly text compression “Probability and Statistics”, De. Groot “All of Statistics” & “All of nonparametric Statistics”, Wasserman Lots of individual articles 5

High Level Syllabus (subject to change) Carnegie Mellon u Language Technology formalisms l l u Words, Words l u [estimators, ML, zero frequency, smoothing, shrinkage, G-T] N-grams: l u type vs, token, Zipf, Mandlebrot, heterogeneity of langauge Modeling Word distributions - the unigram: l u source-channel formulation Bayes classifier Deleted Interpolation Model, backoff, toolkit Measuring Success: perplexity l [entropy, KL-div, MI], the entropy of English, alternatives 6

Syllabus (continued) u Clustering: l l u l Hidden Markov Models, revisiting interpolated and class ngrams Part-Of-Speech tagging, Word Sense Disambiguation Decision & Regression Trees l u class-based N-grams, hierarchical clustering hard and soft clustering Latent Variable Models, EM l u Carnegie Mellon Particularly as applied to language Stochastic Grammars l (SCFG, inside-outside alg. , Link grammar) 7

Syllabus (continued) u Maximum Entropy Modeling l u caches, backoff Dimensionality reduction l u exponential models, ME principle, feature induction. . . Language Model Adaptation l u Carnegie Mellon latent semantic analysis, word 2 vec Syntactic Language Models 8