Machine Learning Introduction Nonparametric Classifiers Eric Xing Lecture

Machine Learning l Where does it come from: l http: //www. cs. cmu. edu/~epxing/Class/10701/

Logistics l l Text book l Chris Bishop, Pattern Recognition and Machine Learning (required)

Logistics l Class mailing list l Adv. ml@gmail. com l Home work l Exam

What is Learning is about seeking a predictive and/or executable understanding of natural/artificial subjects,

Machine Learning © Eric Xing @ CMU, 2006 -2010

Fetching a stapler from inside an office --- the Stanford STAIR robot © Eric

What is Machine Learning? Machine Learning seeks to develop theories and computer systems for

Where Machine Learning is being used or can be useful? Information retrieval Speech recognition

Natural language processing and speech recognition l Now most pocket Speech Recognizers or Translators

Object Recognition l Behind a security camera, most likely there is a computer that

Robotic Control I l The best helicopter pilot is now a computer! l it

Text Mining l l We want: Reading, digesting, and categorizing a vast text database

cacatcgctgcgtttcggcagctaattgccttttagaaattattttcccatttcgagaaactcgtgtgggatgccggatgcggctttcaatcacttctggcccgggatcggattgggtcacattgtctgcgggctctattgtctcgatccg ggcgcagttcgcgtgcttagcggtcagaaaggcagagattcggattgatgcgctggcagcagggcacaaagatctaatgactggcaaatcgctacaaattaaagtccggcggctaatgagcggactgaagccactttg attaaccaaaaaacagcagataaacaaaaacggcaaagaaaattgccacagagttgtcacgctttgttgcacaaacatttgtgcagaaaagtgaaaagcttttagccattattaagtttttcctcagctcgctggcagcacttgcgaatgt ctgatgttcctcataaatgaaaattaatgtttgctctacgctccaccgaactcgcttgtttgggggattggctaatcgcggctagatcccaggcggtataaccttttcgcttcatcagttgtgaaaccagatggctggtgttttggc cagcggactcccctcgaacgctctcgaaatcaagtggctttccagccggcccgctgggccgctcgcccactggaccggtattcccaggccacactgtaccgcataatcctcgccagactcggcgctgataaggcccaatgt actccgcaggcgtctatttatgccaaggaccgttcttcttcagctttcggctcgagtatttgttgtgccatgttggttacgatgccaatcgcggtacagttatgcaaatgagcagcgaataccgctcactgacaatgaacggcgtcttgtc tattcatgctgacattcattcctttggttttttgtcttcgacggactgaaaagtgcggagagaaacccaaaaacagaagcgcgcaaagcgccgttaatatgcgaactcattgaagttatcacaacaccatatccat catatcaatatcgctattattaacgatcatgctctgctgatcaagtattcagcgctagattcgacagattgaatcgagctcaatagactcaacagactccactcgacagatgcgcaatgccaaggacaattgcc tggagtaaacgaggcgtatgcgcaacctgcacctggcggacgcggcgtatgcgcaatgtgcaattcgcttaccttctcgttgcgggtcaggaactcccagatgggaatggccgatgacgagctgatctgaatgtggaaggcgcccagcagg aagattactttcgccgcagtcgtcatggtgtcgttgctgcttttatgttgcgtactccgcactacacggagagttcaggggattcgtgctccgtgatctgtgatccgtgttccgtgggtcaattgcacggttgtgtaaccttcgtg tctttttagggcccaataaaagcgcttttgtggcggcttgatagattatcacttggtttcggtggctagccaagtggctttcttctgtccgacgcacttaattgaattaaccaaacaacgagcgtggccaattcgtattatcgctgt tacgtgtgtctcagcttgaaacgcaaaagcttgtttcacacatcggtttctcggcaagatgggggagtcggtctagggagaggggcgcccaccagtcgatcacgaaaacggcgaattccaagcgaaacggagcacta agtactatgtcgaacaaccgatcgcggcgatgtcagtgagtcgtcttcggacagcgctggcgctccacacgtatttaagctctgagatcggctttgggagagcgcagagagcgccatcgcacggcagagcgaaagcggcagtgagcgaaag gagcggcagcgggtgggggatcgggagccccccgaaaaaaacagaggcgcacgtcgatgccatcggggaattggaacctcaatgtgtgggaatgtttaaatattctgtgttaggtagtttcatagactatagattctcatacagat

Paradigms of Machine Learning l Supervised Learning l l , learn Given , learn

Elements of Learning l Here are some important elements to consider before you start:

Theories of Learning For the learned F(; q ) l l l l l

Classification l Representing data: l Hypothesis (classifier) © Eric Xing @ CMU, 2006 -2010

Decision-making as dividing a high-dimensional space l Classification-specific Dist. : P(X|Y) l Class prior

The Bayes Rule l What we have just did leads to the following general

The Bayes Decision Rule for Minimum Error l The a posteriori probability of a

Example of Decision Rules l When each class is a normal … l We

Bayes Error l We must calculate the probability of error l the probability that

More on Bayes Error l Bayes error is the lower bound of probability of

Learning Classifier l The decision rule: l Learning strategies l Generative Learning l Discriminative

Supervised Learning l K-Nearest-Neighbor Classifier: where the h(X) is represented by all the data,

Recall: Vector Space Representation l Each document is a vector, one component for each

Test Document = ? Sports Science Arts © Eric Xing @ CMU, 2006 -2010

1 -Nearest Neighbor (k. NN) classifier Sports Science Arts © Eric Xing @ CMU,

2 -Nearest Neighbor (k. NN) classifier Sports Science Arts © Eric Xing @ CMU,

3 -Nearest Neighbor (k. NN) classifier Sports Science Arts © Eric Xing @ CMU,

K-Nearest Neighbor (k. NN) classifier Voting k. NN Sports Science Arts © Eric Xing

Classes in a Vector Space Sports Science Arts © Eric Xing @ CMU, 2006

k. NN Is Close to Optimal l Cover and Hart 1967 l Asymptotically, the

Nearest-Neighbor Learning Algorithm l Learning is just storing the representations of the training examples

k. NN is an instance of Instance-Based Learning l What makes an Instance-Based Learner?

Euclidean Distance Metric l Or equivalently, l Other metrics: l L 1 norm: |x-x'|

Case Study: k. NN for Web Classification l Dataset l 20 News Groups (20

Results: Binary Classes Accuracy alt. atheism vs. comp. graphics rec. autos vs. rec. sport.

Results: Multiple Classes Random select 5 -out-of-20 classes, repeat 10 runs and average Accuracy

Is k. NN ideal? © Eric Xing @ CMU, 2006 -2010

Is k. NN ideal? … more later © Eric Xing @ CMU, 2006 -2010

Effect of Parameters l Sample size l l l Dimensionality l l The relative

Summary l l Machine Learning is Cool and Useful!! l Paradigms of Machine Learning.

Slides: 44

Download presentation

Machine Learning Introduction & Nonparametric Classifiers Eric Xing Lecture 1, August 12, 2010 Reading: © Eric Xing @ CMU, 2006 -2010

Machine Learning l Where does it come from: l http: //www. cs. cmu. edu/~epxing/Class/10701/ l http: //www. cs. cmu. edu/~epxing/Class/10708/ © Eric Xing @ CMU, 2006 -2010

Logistics l l Text book l Chris Bishop, Pattern Recognition and Machine Learning (required) l Tom Mitchell, Machine Learning l David Mackay, Information Theory, Inference, and Learning Algorithms l Daphnie Koller and Nir Friedman, Probabilistic Graphical Models Class resource l http: //www. cis. pku. edu. cn/faculty/vision/wangliwei/DS. html l http: //www. cs. cmu. edu/~epxing/Class/DS TA: l Dr. Jun Zhu l Gang Chen, Zheng Wang, Kun Gai, Jianwen Zhang, Shouchun Chen Host: l Liwei Wang, Peking University l Changshui Zhang, Tsinghua University © Eric Xing @ CMU, 2006 -2010

Logistics l Class mailing list l Adv. ml@gmail. com l Home work l Exam l Project © Eric Xing @ CMU, 2006 -2010

What is Learning is about seeking a predictive and/or executable understanding of natural/artificial subjects, phenomena, or activities from … Apoptosis + Medicine Grammatical rules Manufacturing procedures Natural laws … © Eric Xing @ CMU, 2006 -2010 Inference

Machine Learning © Eric Xing @ CMU, 2006 -2010

Fetching a stapler from inside an office --- the Stanford STAIR robot © Eric Xing @ CMU, 2006 -2010

What is Machine Learning? Machine Learning seeks to develop theories and computer systems for l representing; l classifying, clustering and recognizing; l reasoning under uncertainty; l predicting; l and reacting to l … complex, real world data, based on the system's own experience with data, and (hopefully) under a unified model or mathematical framework, that l can be formally characterized analyzed l can take into account human prior knowledge l can generalize and adapt across data and domains l can operate automatically and autonomously l and can be interpreted and perceived by human. © Eric Xing @ CMU, 2006 -2010

Where Machine Learning is being used or can be useful? Information retrieval Speech recognition Computer vision Games Robotic control Pedigree Evolution © Eric Xing @ CMU, 2006 -2010 Planning

Natural language processing and speech recognition l Now most pocket Speech Recognizers or Translators are running on some sort of learning device --- the more you play/use them, the smarter they become! © Eric Xing @ CMU, 2006 -2010

Object Recognition l Behind a security camera, most likely there is a computer that is learning and/or checking! © Eric Xing @ CMU, 2006 -2010

Robotic Control I l The best helicopter pilot is now a computer! l it runs a program that learns how to fly and make acrobatic maneuvers by itself! l no taped instructions, joysticks, or things like … A. Ng 2005 © Eric Xing @ CMU, 2006 -2010

Text Mining l l We want: Reading, digesting, and categorizing a vast text database is too much for human! © Eric Xing @ CMU, 2006 -2010

cacatcgctgcgtttcggcagctaattgccttttagaaattattttcccatttcgagaaactcgtgtgggatgccggatgcggctttcaatcacttctggcccgggatcggattgggtcacattgtctgcgggctctattgtctcgatccg ggcgcagttcgcgtgcttagcggtcagaaaggcagagattcggattgatgcgctggcagcagggcacaaagatctaatgactggcaaatcgctacaaattaaagtccggcggctaatgagcggactgaagccactttg attaaccaaaaaacagcagataaacaaaaacggcaaagaaaattgccacagagttgtcacgctttgttgcacaaacatttgtgcagaaaagtgaaaagcttttagccattattaagtttttcctcagctcgctggcagcacttgcgaatgt ctgatgttcctcataaatgaaaattaatgtttgctctacgctccaccgaactcgcttgtttgggggattggctaatcgcggctagatcccaggcggtataaccttttcgcttcatcagttgtgaaaccagatggctggtgttttggc cagcggactcccctcgaacgctctcgaaatcaagtggctttccagccggcccgctgggccgctcgcccactggaccggtattcccaggccacactgtaccgcataatcctcgccagactcggcgctgataaggcccaatgt actccgcaggcgtctatttatgccaaggaccgttcttcttcagctttcggctcgagtatttgttgtgccatgttggttacgatgccaatcgcggtacagttatgcaaatgagcagcgaataccgctcactgacaatgaacggcgtcttgtc tattcatgctgacattcattcctttggttttttgtcttcgacggactgaaaagtgcggagagaaacccaaaaacagaagcgcgcaaagcgccgttaatatgcgaactcattgaagttatcacaacaccatatccat catatcaatatcgctattattaacgatcatgctctgctgatcaagtattcagcgctagattcgacagattgaatcgagctcaatagactcaacagactccactcgacagatgcgcaatgccaaggacaattgcc tggagtaaacgaggcgtatgcgcaacctgcacctggcggacgcggcgtatgcgcaatgtgcaattcgcttaccttctcgttgcgggtcaggaactcccagatgggaatggccgatgacgagctgatctgaatgtggaaggcgcccagcagg aagattactttcgccgcagtcgtcatggtgtcgttgctgcttttatgttgcgtactccgcactacacggagagttcaggggattcgtgctccgtgatctgtgatccgtgttccgtgggtcaattgcacggttgtgtaaccttcgtg tctttttagggcccaataaaagcgcttttgtggcggcttgatagattatcacttggtttcggtggctagccaagtggctttcttctgtccgacgcacttaattgaattaaccaaacaacgagcgtggccaattcgtattatcgctgt tacgtgtgtctcagcttgaaacgcaaaagcttgtttcacacatcggtttctcggcaagatgggggagtcggtctagggagaggggcgcccaccagtcgatcacgaaaacggcgaattccaagcgaaacggagcacta agtactatgtcgaacaaccgatcgcggcgatgtcagtgagtcgtcttcggacagcgctggcgctccacacgtatttaagctctgagatcggctttgggagagcgcagagagcgccatcgcacggcagagcgaaagcggcagtgagcgaaag gagcggcagcgggtgggggatcgggagccccccgaaaaaaacagaggcgcacgtcgatgccatcggggaattggaacctcaatgtgtgggaatgtttaaatattctgtgttaggtagtttcatagactatagattctcatacagat gagtccttcgagccgattatacacgacagcaaaatatttcagtcgcgcttgggcaaaaggcttaagcacgactcccagtccccccttacatttgtcttcctaagcccctggagccactatcaaacttgttctacgcttgcactgaaaatag accaaagtaaacaatcaaaaagaccaaaaacaataacaaccagcaccgagtcgaacatcagtgaggcattgcaaaaatttcaaagtcaagtttgcgtcgtcatcgcgtctgagtccgatcaagccgggcttgtaattgaagttgttgatga ttactggattgtggcgaattctggtcagcatacttaacagcagcccgctaattaagcaaaataaacatatcaaattccagaatgcgacggcgccatcatcctgtttgggaattcgcgggcagatcgtttaattcaattaaaaggta aaaagggagcagaagaatgcgatcgctggaatttcctaacatcacggaccccataaatttgataagcccgagctcgctgcgttgagtcagccaccccacatccccaaatccccgccaaaagaagacagctgggttgttgactcgccagatt attgcagtggacctggtcaaagaagcaccgttaatgtgctgattccattcgattccatccgggaatgcgataaaggctctgatccaagcaactgcaatccggatttcgattttctctttccatttggttttgtatttacgta aagcattctaatgaagacttggagaagacttacgttatattcagaccatcgtgcgatagaggatgagtcatttccatatggccgaaatttattatgtttactatcgtttttagaggtgttttttggacttaccaaaagaggcatttgtttt ttcaactgaaaagatatttaaattttttcttggaccattttcaaggttccggatatatttgaaacacactagcagtgttggtaagttacatgtatttctataatgtcatattcctttgtccgtattcaaatcgaatactccacatct ttgtacttgaggaattggcgatcgtagcgatttcccccgccgtaaagttcctgatcctcgttgtttttgtacatcataaagtccggattctgctcgtcgccgaagatgggaacgaagctgccaaagctgagagtctgcttgaggtgctggt gtcccagctggataaccttgctgtacagatcggcatctgcctggagggcacgatcgaaatccttccagtggacgaacttcacctgctcgctgggaatagcgttgttgtcaagcagctcaaggagcgtattcgagttgacgggctgcaccac ctgctccttcgctggggattcccctgcgggtaagcgccgcttggactcgtttccaaatcccatagccacgccagcagaggagtaacagagctcwhereisthegenetgattaaaaatatcctttaagaaagcccatgggtataact actgcgtcctatgcgaggaatggtctttaggttctttatggcaaagttctcgcttgcccagccgcggtacgttcttggtgatctttaggaagaatcctggactactgtcgtctgcctggcttatggccacaagacccaccaagagc aggactgttatgattctcatgctgatgcgactgaagcttcacctgactcctgctccacaattggtggcctttatatagcgagatccacccgcatcttgcgtggaatagaaatgcgggtgactccaggaattagcattatcggaaagt ataaaactgaactaacctgacctaaatgcctggccataattaagtgcatacacattacttacatttgtataagaactaaattttatagtacataccacttgcgtatgtaaatgcttgtcttttctcttatatacgttttata cccagcatattttacgtaaaaacggtaatgcgaacataacttattggggcccggaccgcaaaccggccaaacgcgtttgcacccataaaaacataagggcaacaaaaaaattgttaagctgttgtttatttttgcaatcgaa cgctcaaatagctgcgatcactcgggagcagggtaaagtcgcctcgaaacaggaagctgaagcatcttctataaatacactcaaagcgatcattccgaggcgagtctggttagaaatttacatggactgcaaaaaggtatagccccacaaa cacatcgctgcgtttcggcagctaattgccttttagaaattattttcccatttcgagaaactcgtgtgggatgccggatgcggctttcaatcacttctggcccgggatcggattgggtcacattgtctgcgggctctattgtctcgatccg ggcgcagttcgcgtgcttagcggtcagaaaggcagagattcggattgatgcgctggcagcagggcacaaagatctaatgactggcaaatcgctacaaattaaagtccggcggctaatgagcggactgaagccactttg attaaccaaaaaacagcagataaacaaaaacggcaaagaaaattgccacagagttgtcacgctttgttgcacaaacatttgtgcagaaaagtgaaaagcttttagccattattaagtttttcctcagctcgctggcagcacttgcgaatgt ctgatgttcctcataaatgaaaattaatgtttgctctacgctccaccgaactcgcttgtttgggggattggctaatcgcggctagatcccaggcggtataaccttttcgcttcatcagttgtgaaaccagatggctggtgttttggc cagcggactcccctcgaacgctctcgaaatcaagtggctttccagccggcccgctgggccgctcgcccactggaccggtattcccaggccacactgtaccgcataatcctcgccagactcggcgctgataaggcccaatgt actccgcaggcgtctatttatgccaaggaccgttcttcttcagctttcggctcgagtatttgttgtgccatgttggttacgatgccaatcgcggtacagttatgcaaatgagcagcgaataccgctcactgacaatgaacggcgtcttgtc tattcatgctgacattcattcctttggttttttgtcttcgacggactgaaaagtgcggagagaaacccaaaaacagaagcgcgcaaagcgccgttaatatgcgaactcattgaagttatcacaacaccatatccat catatcaatatcgctattattaacgatcatgctctgctgatcaagtattcagcgctagattcgacagattgaatcgagctcaatagactcaacagactccactcgacagatgcgcaatgccaaggacaattgcc tggagtaaacgaggcgtatgcgcaacctgcacctggcggacgcggcgtatgcgcaatgtgcaattcgcttaccttctcgttgcgggtcaggaactcccagatgggaatggccgatgacgagctgatctgaatgtggaaggcgcccagcagg aagattactttcgccgcagtcgtcatggtgtcgttgctgcttttatgttgcgtactccgcactacacggagagttcaggggattcgtgctccgtgatctgtgatccgtgttccgtgggtcaattgcacggttgtgtaaccttcgtg tctttttagggcccaataaaagcgcttttgtggcggcttgatagattatcacttggtttcggtggctagccaagtggctttcttctgtccgacgcacttaattgaattaaccaaacaacgagcgtggccaattcgtattatcgctgt tacgtgtgtctcagcttgaaacgcaaaagcttgtttcacacatcggtttctcggcaagatgggggagtcggtctagggagaggggcgcccaccagtcgatcacgaaaacggcgaattccaagcgaaacggagcacta agtactatgtcgaacaaccgatcgcggcgatgtcagtgagtcgtcttcggacagcgctggcgctccacacgtatttaagctctgagatcggctttgggagagcgcagagagcgccatcgcacggcagagcgaaagcggcagtgagcgaaag gagcggcagcgggtgggggatcgggagccccccgaaaaaaacagaggcgcacgtcgatgccatcggggaattggaacctcaatgtgtgggaatgtttaaatattctgtgttaggtagtttcatagactatagattctcatacagat gagtccttcgagccgattatacacgacagcaaaatatttcagtcgcgcttgggcaaaaggcttaagcacgactcccagtccccccttacatttgtcttcctaagcccctggagccactatcaaacttgttctacgcttgcactgaaaatag accaaagtaaacaatcaaaaagaccaaaaacaataacaaccagcaccgagtcgaacatcagtgaggcattgcaaaaatttcaaagtcaagtttgcgtcgtcatcgcgtctgagtccgatcaagccgggcttgtaattgaagttgttgatga ttactggattgtggcgaattctggtcagcatacttaacagcagcccgctaattaagcaaaataaacatatcaaattccagaatgcgacggcgccatcatcctgtttgggaattcgcgggcagatcgtttaattcaattaaaaggta aaaagggagcagaagaatgcgatcgctggaatttcctaacatcacggaccccataaatttgataagcccgagctcgctgcgttgagtcagccaccccacatccccaaatccccgccaaaagaagacagctgggttgttgactcgccagatt attgcagtggacctggtcaaagaagcaccgttaatgtgctgattccattcgattccatccgggaatgcgataaaggctctgatccaagcaactgcaatccggatttcgattttctctttccatttggttttgtatttacgta aagcattctaatgaagacttggagaagacttacgttatattcagaccatcgtgcgatagaggatgagtcatttccatatggccgaaatttattatgtttactatcgtttttagaggtgttttttggacttaccaaaagaggcatttgtttt ttcaactgaaaagatatttaaattttttcttggaccattttcaaggttccggatatatttgaaacacactagcagtgttggtaagttacatgtatttctataatgtcatattcctttgtccgtattcaaatcgaatactccacatct ttgtacttgaggaattggcgatcgtagcgatttcccccgccgtaaagttcctgatcctcgttgtttttgtacatcataaagtccggattctgctcgtcgccgaagatgggaacgaagctgccaaagctgagagtctgcttgaggtgctggt gtcccagctggataaccttgctgtacagatcggcatctgcctggagggcacgatcgaaatccttccagtggacgaacttcacctgctcgctgggaatagcgttgttgtcaagcagctcaaggagcgtattcgagttgacgggctgcaccac ctgctccttcgctggggattcccctgcgggtaagcgccgcttggactcgtttccaaatcccatagccacgccagcagaggagtaacagagctctgaaaacagttcatggtttaaaaatatcctttaagaaagcccatgggtataact actgcgtcctatgcgaggaatggtctttaggttctttatggcaaagttctcgcttgcccagccgcggtacgttcttggtgatctttaggaagaatcctggactactgtcgtctgcctggcttatggccacaagacccaccaagagc aggactgttatgattctcatgctgatgcgactgaagcttcacctgactcctgctccacaattggtggcctttatatagcgagatccacccgcatcttgcgtggaatagaaatgcgggtgactccaggaattagcattatcggaaagt ataaaactgaactaacctgacctaaatgcctggccataattaagtgcatacacattacttacatttgtataagaactaaattttatagtacataccacttgcgtatgtaaatgcttgtcttttctcttatatacgttttata © Eric Xing @ CMU, 2006 -2010 Bioinformatics Where is the gene?

Paradigms of Machine Learning l Supervised Learning l l , learn Given , learn , s. t. Reinforcement Learning l Given learn l , s. t. Unsupervised Learning l l Given , s. t. Active Learning l Given , learn , s. t. © Eric Xing @ CMU, 2006 -2010

Elements of Learning l Here are some important elements to consider before you start: l Task: l l Data and other info: l l l l Exact inference? MCMC? Variational? Gradient? Greedy search? Online? Batch? Distributed? Evaluation: l l MLE? MCLE? Max margin? Log loss, hinge loss, square loss? … Tractability and exactness trade off: l l BN? MRF? Regression? SVM? Bayesian/Frequents ? Parametric/Nonparametric? Objective/Loss function: l l Input and output (e. g. , continuous, binary, counts, …) Supervised or unsupervised, of a blend of everything? Prior knowledge? Bias? Models and paradigms: l l Embedding? Classification? Clustering? Topic extraction? … Visualization? Human interpretability? Perperlexity? Predictive accuracy? It is better to consider one element at a time! © Eric Xing @ CMU, 2006 -2010

Theories of Learning For the learned F(; q ) l l l l l Consistency (value, pattern, …) Bias versus variance Sample complexity Learning rate Convergence Error bound Confidence Stability … © Eric Xing @ CMU, 2006 -2010

Classification l Representing data: l Hypothesis (classifier) © Eric Xing @ CMU, 2006 -2010

Decision-making as dividing a high-dimensional space l Classification-specific Dist. : P(X|Y) l Class prior (i. e. , "weight"): P(Y) © Eric Xing @ CMU, 2006 -2010

The Bayes Rule l What we have just did leads to the following general expression: This is Bayes Rule © Eric Xing @ CMU, 2006 -2010

The Bayes Decision Rule for Minimum Error l The a posteriori probability of a sample l Bayes Test: l Likelihood Ratio: l Discriminant function: © Eric Xing @ CMU, 2006 -2010

Example of Decision Rules l When each class is a normal … l We can write the decision boundary analytically in some cases … homework!! © Eric Xing @ CMU, 2006 -2010

Bayes Error l We must calculate the probability of error l the probability that a sample is assigned to the wrong class l Given a datum X, what is the risk? l The Bayes error (the expected risk): © Eric Xing @ CMU, 2006 -2010

More on Bayes Error l Bayes error is the lower bound of probability of classification error l Bayes classifier is theoretically best classifier that minimize probability of classification error l Computing Bayes error is in general a very complex problem. Why? l Density estimation: l Integrating density function: © Eric Xing @ CMU, 2006 -2010

Learning Classifier l The decision rule: l Learning strategies l Generative Learning l Discriminative Learning l Instance-based Learning (Store all past experience in memory) l A special case of nonparametric classifier © Eric Xing @ CMU, 2006 -2010

Supervised Learning l K-Nearest-Neighbor Classifier: where the h(X) is represented by all the data, and by an algorithm © Eric Xing @ CMU, 2006 -2010

Recall: Vector Space Representation l Each document is a vector, one component for each term (= word). Doc 1 Doc 2 Doc 3 . . . Word 1 3 0 0 . . . Word 2 0 8 1 . . . Word 3 12 1 10 . . . 0 1 3 . . . 0 0 0 . . . l Normalize to unit length. l High-dimensional vector space: l Terms are axes, 10, 000+ dimensions, or even 100, 000+ l Docs are vectors in this space © Eric Xing @ CMU, 2006 -2010

k. NN Is Close to Optimal l Cover and Hart 1967 l Asymptotically, the error rate of 1 -nearest-neighbor classification is less than twice the Bayes rate [error rate of classifier knowing model that generated data] l In particular, asymptotic error rate is 0 if Bayes rate is 0. l Where does k. NN come from? l Nonparametric density estimation © Eric Xing @ CMU, 2006 -2010

Nearest-Neighbor Learning Algorithm l Learning is just storing the representations of the training examples in D. l Testing instance x: l l Compute similarity between x and all examples in D. Assign x the category of the most similar example in D. l Does not explicitly compute a generalization or category prototypes. l Also called: l l l Case-based learning Memory-based learning Lazy learning © Eric Xing @ CMU, 2006 -2010

k. NN is an instance of Instance-Based Learning l What makes an Instance-Based Learner? l A distance metric l How many nearby neighbors to look at? l A weighting function (optional) l How to relate to the local points? © Eric Xing @ CMU, 2006 -2010

Euclidean Distance Metric l Or equivalently, l Other metrics: l L 1 norm: |x-x'| l L∞ norm: max |x-x'| (elementwise …) l Mahalanobis: where S is full, and symmetric l Correlation l Angle l Hamming distance, Manhattan distance l … © Eric Xing @ CMU, 2006 -2010

Case Study: k. NN for Web Classification l Dataset l 20 News Groups (20 classes) l Download : (http: //people. csail. mit. edu/jrennie/20 Newsgroups/) l 61, 118 words, 18, 774 documents l Class labels descriptions © Eric Xing 2006 -2008 © Eric Xing@ @ CMU, 2006 -2010

Results: Binary Classes Accuracy alt. atheism vs. comp. graphics rec. autos vs. rec. sport. baseball comp. windows. x vs. rec. motorcycles k © Eric Xing @ CMU, 2006 -2010 2006 -2008

Results: Multiple Classes Random select 5 -out-of-20 classes, repeat 10 runs and average Accuracy All 20 classes k © Eric Xing @ CMU, 2006 -2010 2006 -2008

Effect of Parameters l Sample size l l l Dimensionality l l The relative scalings in the distance metric affect region shapes. Weight l l How smooth? Metric l l Curse of dimensionality Density l l The more the better Need efficient search algorithm for NN Spurious or less relevant points need to be downweighted K © Eric Xing @ CMU, 2006 -2010

Summary l l Machine Learning is Cool and Useful!! l Paradigms of Machine Learning. l Design elements learning l Theories on learning Fundamental theory of classification l Bayes optimal classifier l Instance-based learning: k. NN – a Nonparametric classifier l A nonparametric method does not rely on any assumption concerning the structure of the underlying density function. l l l Very little “learning” is involved in these methods Good news: l Simple and powerful methods; Flexible and easy to apply to many problems. l k. NN classifier asymptotically approaches the Bayes classifier, which is theoretically the best classifier that minimizes the probability of classification error. Bad news: l High memory requirements l Very dependant on the scale factor for a specific problem. © Eric Xing @ CMU, 2006 -2010