CMSC 471 Machine Learning Nave Bayes Neural Networks

CMSC 471 Machine Learning: Naïve Bayes, Neural Networks, Clustering Skim 20. 5 1

The Naïve Bayes Classifier Some material adapted from slides by Tom Mitchell, CMU. 2

The Naïve Bayes Classifier n Recall Bayes rule: n Which is short for: n We can re-write this as: 3

Deriving Naïve Bayes n Idea: use the training data to directly estimate: and n Then, we can use these values to estimate using Bayes rule. n Recall that representing the full joint probability is not practical. 4

Deriving Naïve Bayes n However, if we make the assumption that the attributes are independent, estimation is easy! n In other words, we assume all attributes are conditionally independent given Y. n Often this assumption is violated in practice, but more on that later… 5

Deriving Naïve Bayes n Let n Then, we can estimate and directly from the training data by counting! Sky sunny rainy sunny and label Y be discrete. Temp warm cold warm Humid normal high P(Sky = sunny | Play = yes) = ? Wind strong Water warm cool Forecast same change Play? yes no yes P(Humid = high | Play = yes) = ? 6

The Naïve Bayes Classifier n Now we have: which is just a one-level Bayesian Network Yj X 1 n … Xi Labels (hypotheses) … Xn Attributes (evidence) To classify a new point Xnew: 7

The Naïve Bayes Algorithm n For each value yk Estimate P(Y = yk) from the data. n For each value xij of each attribute Xi n n Estimate P(Xi=xij | Y = yk) n Classify a new point via: n In practice, the independence assumption doesn’t often hold true, but Naïve Bayes performs very well despite it. 8

Naïve Bayes Applications n Text classification Which e-mails are spam? n Which e-mails are meeting notices? n Which author wrote a document? n n Classifying mental states Learning P(Brain. Activity | Word. Category) Pairwise Classification Accuracy: 85% People Words Animal Words 9

Neural Networks Adapted from slides by Tim Finin and Marie des. Jardins. Some material adapted from lecture notes by Lise Getoor and Ron Parr 10

Neural function n n Brain function (thought) occurs as the result of the firing of neurons Neurons connect to each other through synapses, which propagate action potential (electrical impulses) by releasing neurotransmitters Synapses can be excitatory (potential-increasing) or inhibitory (potential-decreasing), and have varying activation thresholds Learning occurs as a result of the synapses’ plasticicity: They exhibit long-term changes in connection strength There about 1011 neurons and about 1014 synapses in the human brain(!) 11

Biology of a neuron 12

Brain structure n Different areas of the brain have different functions n n n We don’t know how different functions are “assigned” or acquired n n Some areas seem to have the same function in all humans (e. g. , Broca’s region for motor speech); the overall layout is generally consistent Some areas are more plastic, and vary in their function; also, the lower-level structure and function vary greatly Partly the result of the physical layout / connection to inputs (sensors) and outputs (effectors) Partly the result of experience (learning) We really don’t understand how this neural structure leads to what we perceive as “consciousness” or “thought” Artificial neural networks are not nearly as complex or intricate as the actual brain structure 13

Comparison of computing power INFORMATION CIRCA 1995 Computer Human Brain Computation Units Storage Units Cycle time Bandwidth Updates / sec 1 CPU, 105 Gates 1011 Neurons 104 bits RAM, 1010 bits disk 1011 neurons, 1014 synapses 10 -8 sec 10 -3 sec 104 bits/sec 1014 bits/sec 105 1014 n n Computers are way faster than neurons… But there a lot more neurons than we can reasonably model in modern digital computers, and they all fire in parallel Neural networks are designed to be massively parallel The brain is effectively a billion times faster 14

Neural networks Output units Hidden units Input units Layered feed-forward network n n n Neural networks are made up of nodes or units, connected by links Each link has an associated weight and activation level Each node has an input function (typically summing over weighted inputs), an activation function, and an output 15

Model of a neuron n Neuron modeled as a unit i weights on input unit j to i, wji net input to unit i is: n Activation function g() determines the neuron’s output n n g() is typically a sigmoid output is either 0 or 1 (no partial activation) 16

“Executing” neural networks n n Input units are set by some exterior function (think of these as sensors), which causes their output links to be activated at the specified level Working forward through the network, the input function of each unit is applied to compute the input value n n Usually this is just the weighted sum of the activation on the links feeding into this node The activation function transforms this input function into a final value n Typically this is a nonlinear function, often a sigmoid function corresponding to the “threshold” of that node 17

Learning rules n Rosenblatt (1959) suggested that if a target output value is provided for a single neuron with fixed inputs, can incrementally change weights to learn to produce these outputs using the perceptron learning rule n assumes binary valued input/outputs n assumes a single linear threshold unit 18

Perceptron learning rule n If the target output for unit i is ti n Equivalent to the intuitive rules: n If output is correct, don’t change the weights n If output is low (oi=0, ti=1), increment weights for all the inputs which are 1 n If output is high (oi=1, ti=0), decrement weights for all inputs which are 1 n Must also adjust threshold. Or equivalently assume there is a weight w 0 i for an extra input unit that has an output of 1. 19

Perceptron learning algorithm n n n Repeatedly iterate through examples adjusting weights according to the perceptron learning rule until all outputs are correct n Initialize the weights to all zero (or random) n Until outputs for all training examples are correct n for each training example e do n compute the current output oj n compare it to the target tj and update weights Each execution of outer loop is called an epoch For multiple category problems, learn a separate perceptron for each category and assign to the 20

Representation limitations of a perceptron n Perceptrons can only represent linear threshold functions and can therefore only learn functions which linearly separate the data. n i. e. , the positive and negative examples are separable by a hyperplane in n-dimensional space <W, X> - = 0 > 0 on this side < 0 on this side 21

Perceptron learnability Perceptron Convergence Theorem: If there is a set of weights that is consistent with the training data (i. e. , the data is linearly separable), the perceptron learning algorithm will converge (Minicksy & Papert, 1969) n Unfortunately, many functions (like parity) cannot be represented by LTU n 22

Learning: Backpropagation Similar to perceptron learning algorithm, we cycle through our examples n if the output of the network is correct, no changes are made n if there is an error, the weights are adjusted to reduce the error n The trick is to assess the blame for the error and divide it among the contributing weights n 23

Output layer n As in perceptron learning algorithm, we want to minimize difference between target output and the output actually computed activation of hidden unit j (Ti – Oi) derivative of activation function 24

Hidden layers n n n Need to define error; we do error backpropagation. Intuition: Each hidden node j is “responsible” for some fraction of the error I in each of the output nodes to which it connects. I divided according to the strength of the connection between hidden node and the output node and propagated back to provide the j values for the hidden layer: update rule: 25

Backprogation algorithm Compute the values for the output units using the observed error n Starting with output layer, repeat the following for each layer in the network, until earliest hidden layer is reached: n propagate the values back to the previous layer n update the weights between the two layers n 26

Backprop issues “Backprop is the cockroach of machine learning. It’s ugly, and annoying, but you just can’t get rid of it. ” Geoff Hinton n Problems: n black box n local minima n 27

Unsupervised Learning: Clustering Some material adapted from slides by Andrew Moore, CMU. Visit http: //www. autonlab. org/tutorials/ for Andrew’s repository of Data Mining tutorials. 28

Unsupervised Learning n n Supervised learning used labeled data pairs (x, y) to learn a function f : X→Y. But, what if we don’t have labels? No labels = unsupervised learning Only some points are labeled = semi-supervised learning n n Labels may be expensive to obtain, so we only get a few. Clustering is the unsupervised grouping of data points. It can be used for knowledge discovery. 29

Clustering Data 30

K-Means Clustering K-Means ( k , data ) • Randomly choose k cluster center locations (centroids). • Loop until convergence 1. Assign each point to the cluster of the closest centroid. 2. Reestimate the cluster centroids based on the data assigned to each. 31

K-Means Clustering K-Means ( k , data ) • Randomly choose k cluster center locations (centroids). • Loop until convergence 1. Assign each point to the cluster of the closest centroid. 2. Reestimate the cluster centroids based on the data assigned to each. 32

K-Means Clustering K-Means ( k , data ) • Randomly choose k cluster center locations (centroids). • Loop until convergence 1. Assign each point to the cluster of the closest centroid. 2. Reestimate the cluster centroids based on the data assigned to each. 33

K-Means Animation Example generated by Andrew Moore using Dan Pelleg’s superduper fast K-means system: Dan Pelleg and Andrew Moore. Accelerating Exact k-means Algorithms with Geometric Reasoning. Proc. Conference on Knowledge Discovery in Databases 1999. 34

Problems with K-Means n Very sensitive to the initial points. n Do many runs of k-Means, each with different initial centroids. n Seed the centroids using a better method than random. (e. g. Farthest-first sampling) n Must manually choose k. n Learn the optimal k for the clustering. (Note that this requires a performance measure. ) 35

Problems with K-Means n How do you tell it which clustering you want? n Constrained clustering techniques Same-cluster constraint (must-link) Different-cluster constraint (cannot-link) 36