Outline Parameter estimation Maximum likelihood estimation 1152020 Visual

Outline • Parameter estimation – Maximum likelihood estimation 11/5/2020 Visual Perception Modeling 1

Bayes Decision Theory • Assumptions – Suppose that there are c categories • { 1, 2, . . . , c} – The prior probability and class conditional density are known – There a possible actions • { 1, 2, . . . , a} – Loss function ( i | j} describe the loss incurred for taking action i when the state of nature is j 11/5/2020 Visual Perception Modeling 2

Bayes Decision Rule • To minimize the overall risk, compute the conditional risk and select the action for which the conditional risk is minimum – The resulting minimum overall risk is called the Bayes risk, which is the best performance 11/5/2020 Visual Perception Modeling 3

Discriminant Functions for Normal Density • Minimum error rate classification for normal density • Three different cases 11/5/2020 Visual Perception Modeling 4

Parameter Estimation • We could design an optimal classifier if we knew the prior probabilities and the classconditional densities – Unfortunately, in pattern recognition applications we rarely have this kind of complete knowledge about the probabilistic structure of the problem • Training data – Some vague, general knowledge about the problem – A number of design samples 11/5/2020 Visual Perception Modeling 5

Parameter Estimation – cont. • Two approaches – Parameter estimation • Estimate the parameters of the unknown probabilities and probability densities – Non-parametric procedures • Multi-layer perceptrons and in general neural networks • Fisher linear discriminant function • Work in the feature space directly 11/5/2020 Visual Perception Modeling 6

Parameter Estimation – cont. • Parameter estimation – Maximum-likelihood approach • Parameters as quantities whose values are fixed but unknown • The best estimate of their value is the one that maximizes the probability of obtaining the samples – Bayesian learning • Parameters are random variables with known prior distribution • Observations convert the prior into posteriori 11/5/2020 Visual Perception Modeling 7

Maximum-Likelihood Estimation • Assumptions – We separate a collection of samples according to class • D 1, D 2, . . . , Dc – Samples in Dj are drawn independently according to the probability p(x| j) – We assume that p(x| j) has a known parametric form and is uniquely determined by the value of a parameter vector j – To simplify further, we assume that samples in Di give no information about j if i j 11/5/2020 Visual Perception Modeling 8

Maximum-Likelihood Estimation – cont. • Suppose that D contains n samples – x 1, . . . , xn – By assumption that samples were drawn independently, we have – The maximum-likelihood estimate of is the value of * that maximizes p(D| ) 11/5/2020 Visual Perception Modeling 9

Maximum-Likelihood Estimation – cont. • Log-likelihood 11/5/2020 Visual Perception Modeling 10

Maximum-Likelihood Estimation – cont. • The maximum likelihood solution is – A solution * can be a true global maximum, a local maximum, or a minimum, or an inflection point of l( ) • We need to check each solution individually • Or calculate the second derivatives to identify the global optimum 11/5/2020 Visual Perception Modeling 11

Maximum-Likelihood Estimation – cont. • Gaussian case - Unknown 11/5/2020 Visual Perception Modeling 12

Maximum-Likelihood Estimation – cont. • Gaussian case - Unknown and – Univariate case 11/5/2020 Visual Perception Modeling 13

Maximum-Likelihood Estimation – cont. • Gaussian case - Unknown and continued 11/5/2020 Visual Perception Modeling 14

Maximum-Likelihood Estimation – cont. • Bias – For a large number of samples, 11/5/2020 Visual Perception Modeling 15