Outline Classification continued Parameter estimation 1232020 Visual Perception

Outline • Classification – continued • Parameter estimation 12/3/2020 Visual Perception Modeling 1

Assumptions • Suppose that there are c categories – { 1, 2, . . . , c} • The prior probability and class conditional density are known • There a possible actions – { 1, 2, . . . , a} • Loss function ( i | j} describe the loss incurred for taking action i when the state of nature is j 12/3/2020 Visual Perception Modeling 2

Loss Function • The expected loss function given a particular observation x • The overall risk 12/3/2020 Visual Perception Modeling 3

Bayes Decision Rule • To minimize the overall risk, compute the conditional risk and select the action for which the conditional risk is minimum – The resulting minimum overall risk is called the Bayes risk, which is the best performance 12/3/2020 Visual Perception Modeling 4

Minimum-Error-Rate Classification • Zero-one loss • For minimum error rate, – Decide 1 if P( 1 | x) > P( 2 | x) – This is the Bayes decision rule 12/3/2020 Visual Perception Modeling 5

Discriminant Functions • The classifier is said to assign a feature vector x to class i if – gi(x) > gj(x) for all j i – This can be viewed as a network – If f(. ) is a monotonically increasing function, f(g(x)) and g(x) as discriminant function will give the same classification result 12/3/2020 Visual Perception Modeling 6

Decision Regions • The effect of decision rule is to divide the feature space into c decision regions – R 1, R 2, . . , Rc – The regions are separated by decision boundaries 12/3/2020 Visual Perception Modeling 7

Normal Density • Gaussian density – Properties • • 12/3/2020 Mean Variance Entropy Central limit theorem Visual Perception Modeling 8

Discriminant Functions for Normal Density • Minimum error rate classification for normal density 12/3/2020 Visual Perception Modeling 9

Normal Density – Cont. • Case I: – i = 2 I – Linear discriminant function – Minimum distance classifier as a special case • Template matching 12/3/2020 Visual Perception Modeling 10

Normal Density – Cont. • Case II – i = – Mahalanobis distance – The resulting discriminant function is also linear 12/3/2020 Visual Perception Modeling 11

Normal Density – Cont. • Case III – i arbitrary – Hyper-quadric 12/3/2020 Visual Perception Modeling 12

Bayes Decision Theory – Discrete Features • In this case, a feature can only take one of the m discrete values v 1, . . . , vm • To minimize the overall risk, select the action that minimizes 12/3/2020 Visual Perception Modeling 13

Parameter Estimation • We could design an optimal classifier if we knew the prior probabilities and the classconditional densities – Unfortunately, in pattern recognition applications we rarely have this kind of complete knowledge about the probabilistic structure of the problem • Training data – Some vague, general knowledge about the problem – A number of design samples 12/3/2020 Visual Perception Modeling 14

Parameter Estimation – cont. • Two approaches – Parameter estimation • Estimate the parameters of the unknown probabilities and probability densities – Non-parametric procedures • Multi-layer perceptrons and in general neural networks • Fisher linear discriminant function 12/3/2020 Visual Perception Modeling 15

Parameter Estimation – cont. • Parameter estimation – Maximum-likelihood approach • Parameters as quantities whose values are fixed but unknown • The best estimate of their value is the one that maximizes the probability of obtaining the samples – Bayesian learning • Parameters are random variables with known prior distribution • Observations convert the prior into posteriori 12/3/2020 Visual Perception Modeling 16

Maximum-Likelihood Estimation • The general principle – Log-likelihood • Gaussian cases – Unknown and 12/3/2020 Visual Perception Modeling 17

Bayesian Estimation • Class-conditional densities • Parameter Distribution • Gaussian case – Univariate case – Multivariate case 12/3/2020 Visual Perception Modeling 18

Bayesian Estimation – cont. • General theory 12/3/2020 Visual Perception Modeling 19