Pattern Recognition Mathematic Review Hamid R Rabiee Jafar

Pattern Recognition Mathematic Review Hamid R. Rabiee Jafar Muhammadi Ali Jalali

Probability Space ² A triple of (Ω, F, P) ² Ω represents a nonempty set, whose elements are sometimes known as outcomes or states of nature ² F represents a set, whose elements are called events. The events are subsets of Ω. F should be a “Borel Field”. ² P represents the probability measure. ² Fact: P(Ω) = 1 Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 2 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Random Variables ² Random variable is a “function” (“mapping”) from a set of possible outcomes of the experiment to an interval of real (complex) numbers. ² In other words: ² Examples: ² Mapping faces of a dice to the first six natural numbers. ² Mapping height of a man to the real interval (0, 3] (meter or something else). ² Mapping success in an exam to the discrete interval [0, 20] by quantum 0. 1 Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 3 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Random Variables (Cont’d) ² Random Variables ² Discrete ² Dice, Coin, Grade of a course, etc. ² Continuous ² Temperature, Humidity, Length, etc. ² Random Variables ² Real ² Complex Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 4 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Density/Distribution Functions ² Probability Mass Function (PMF) ² Discrete random variables ² Summation of impulses ² The magnitude of each impulse represents the probability of occurrence of the outcome ² PMF values are probabilities. ² Example I: ² Rolling a fair dice Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 5 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Density/Distribution Functions (Cont’d) ² Example II: ² Summation of two fair dices ² Note : Summation of all probabilities should be equal to ONE. (Why? ) Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 6 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Density/Distribution Functions (Cont’d) ² Probability Density Function (PDF) ² Continuous random variables ² The probability of occurrence of Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 7 http: //ce. sharif. edu/courses/85 -86/1/ce 725/ will be

Famous Density Functions ² Some famous masses and densities ² Uniform Density ² Gaussian (Normal) Density ² Exponential Density Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 8 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Distribution Measures ² Most basic type of descriptor for spatial distributions, include: ² Mean Center ² Variance & Standard Deviation ² Covariance ² Correlation Coefficient Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 9 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Expected Value ² Expected value ( population mean value) ² Properties of Expected Value ² The expected value of a constant is the constant itself. E[b]= b ² If a and b are constants, then E[a. X+ b]= a E[X]+ b ² If X and Y are independent RVs, then E[XY]= E[X ]* E[Y] ² If X is RV with PDF f(X), g(X) is any function of X, then, ² if X is discrete ² if X is continuous Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 10 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Variance & Standard Deviation ² Let X be a RV with E(X)=u, the distribution, or spread, of the X values around the expected value can be measured by the variance ( is the standard deviation of X). ² The Variance Properties ² The variance of a constant is zero. ² If a and b are constants, then ² If X and Y are independent RVs, then Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 11 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Covariance ² Covariance of two RVs, X and Y: Measurement of the nature of the association between the two. ² Properties of Covariance: ² If X, Y are two independent RVs, then ² If a, b, c and d are constants, then ² If X is a RV, then ² Covariance Value ² Cov(X, Y) is positively big = Positively strong relationship between the two. ² Cov(X, Y) is negatively big = Negatively strong relationship between the two. ² Cov(X, Y)=0 = No relationship between the two. Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 12 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Covariance Matrices ² If X is a n-Dim RV, then the covariance defined as: whose ijth element sij is the covariance of xi and xj: then Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 13 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Covariance Matrices (cont’d) ² Properties of Covariance Matrix: ² If the variables are statistically independent, the covariances are zero, and the covariance matrix is diagonal. ² noting that the determinant of S is just the product of the variances, then we can write ² This is the general form of a multivariate normal density function, where the covariance matrix is no longer required to be diagonal. Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 14 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Correlation Coefficient ² Correlation: Knowing about a random variable “X”, how much information will we gain about the other random variable “Y” ? ² The population correlation coefficient is defined as ² The Correlation Coefficient is a measure of linear association between two variables and lies between -1 and +1 ² -1 indicating perfect negative association ² +1 indicating perfect positive association Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 15 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Moments ² Moments n ² nth order moment of a RV X is the expected value of X : ² Normalized form (Central Moment) ² Mean is first moment ² Variance is second moment added by the square of the mean. Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 16 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Moments (cont’d) ² Third Moment ² Measure of asymmetry ² Often referred to as skewness If symmetric s= 0 ² Fourth Moment ² Measure of flatness ² Often referred to as Kurtosis small kurtosis large kurtosis Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 17 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Sample Measures ² Sample: a random selection of items from a lot or population in order to evaluate the characteristics of the lot or population ² Sample Mean: ² Sample Variance: ² Sample Covariance: ² Sample Correlation: ² 3 th center moment ² 4 th center moment Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 18 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

More on Gaussian Distribution ² The Gaussian Distribution Function ² Sometimes called “Normal” or “bell shaped” ² Perhaps the most used distribution in all of science ² Is fully defined by 2 parameters ² 95% of area is within 2σ ² Normal distributions range from minus infinity to plus infinity Gaussian with m=0 and s=1 Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 19 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

More on Gaussian Distribution (Cont’d) ² Normal Distributions with the Same Variance but Different Means ² Normal Distributions with the Same Means but Different Variances Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 20 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

More on Gaussian Distribution (Cont’d) ² Standard Normal Distribution ² A normal distribution whose mean is zero and standard deviation is one is called the standard normal distribution. ² any normal distribution can be converted to a standard normal distribution with simple algebra. This makes calculations much easier. Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 21 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Central Limit Theorem ² Why is The Gaussian PDF is so applicable? Central Limit Theorem ² Illustrating CLT ² a) 5000 Random Numbers ² b) 5000 Pairs (r 1+r 2) of Random Numbers ² c) 5000 Triplets (r 1+r 2+r 3) of Random Numbers ² d) 5000 12 -plets (r 1+r 2+…+r 12) of Random Numbers Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 22 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Central Limit Theorem ² Central Limit Theorem says that ² we can use the standard normal approximation of the sampling distribution regardless of our data. We don’t need to know the underlying probability distribution of the data to make use of sample statistics. ² This only holds in samples which are large enough to use the CLT’s “large sample properties. ” So, how large is large enough? ² Some books will tell you 30 is large enough. ² Note: The CLT does not say: “in large samples the data is distributed normally. ” Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 23 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Two Normal-like distributions ² T-Student Distribution ² Chi-Squared Distribution Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 24 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Distances ² Each clustering problem is based on some kind of “distance” between points. ² A Euclidean space has some number of real-valued dimensions and “dense” points. ² There is a notion of “average” of two points. ² A Euclidean distance is based on the locations of points in such a space. ² A Non-Euclidean distance is based on properties of points, but not their “location” in a space. ² Distance Matrices ² Once a distance measure is defined, we can calculate the distance between objects. These objects could be individual observations, groups of observations (samples) or populations of observations. ² For N objects, we then have a symmetric distance matrix D whose elements are the distances between objects i and j. Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 25 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Axioms of a Distance Measure ² d is a distance measure if it is a function from pairs of points to reals such that: 1. d(x, y) > 0. 2. d(x, y) = 0 iff x = y. 3. d(x, y) = d(y, x). 4. d(x, y) < d(x, z) + d(z, y) (triangle inequality ). ² For the purpose of clustering, sometimes the distance (similarity) is not required to be a metric ² No Triangle Inequality ² No Symmetry Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 26 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Distance Measures ² Euclidean Distance ² L 2 Norm ² L 1, L∞ Norm ² Mahalanobis Distance Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 27 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Euclidean Distance (L 2 Norm) ² A possible distance measure for spaces equipped with a Euclidean metric ² Multivariate distances between populations ² Calculates the Euclidean distance between two “points” defined by the multivariate means of two populations based on p variables. ² Does not take into account differences among populations in within-population variability nor correlations among variables. Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 28 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Euclidean Distance (L 2 Norm) (Cont’d) ² Multivariate distances between populations X 2 Sample 1 X 1 Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 29 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Euclidean Distance (L 1, L∞ Norm) ² L 1 Norm: sum of the differences in each dimension. ² Manhattan distance = distance if you had to travel along coordinates only. ² L∞ norm : d(x, y) = the maximum of the differences between x and y in any dimension. y = (9, 8) 32 2+ 5 )= : 4 rm = ( o n ) 5 L 2 t(x, y dis 4 x = (5, 5) 3 L 1 -norm: dist(x, y) = 4+3 = 7 L∞ -norm: dist(x, y) = Max(4, 3) = 4 Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 30 http: //ce. sharif. edu/courses/85 -86/1/ce 725/

Mahalanobis Distance ² Distances are computed based on means, variances and covariances for each of g samples (populations) based on p variables. ² Mahalanobis distance “weights” the contribution of each pair of variables by the inverse of their covariance. ² ² are samples means. is the Covariance between samples. Sharif University of Technology, Department of Computer Engineering, Pattern Recognition Course 31 http: //ce. sharif. edu/courses/85 -86/1/ce 725/