 # Chapter 5 Ordinary Least Square Regression We will

• Slides: 32 Chapter 5: Ordinary Least Square Regression We will be discussing q The Linear Regression Model q Estimation of the Unknowns in the Regression Model q Some Special Cases of the Model Mathematical Marketing 1 The Basic Regression Model The model expressed in scalars The model elaborated in matrix terms A succinct matrix expression for it y = X + e Mathematical Marketing 2 Prediction Based on a Linear Combination The E(y) is given by Together with the previous slide, we can say that Key Question: How Do We Get Values for the vector? Mathematical Marketing 3 Parameter Estimation of We will cover two philosophies of parameter estimation, the least squares principle and maximum likelihood. Each of these has the following steps: q Pick an objective function to optimize q Find the derivative of f with respect to the unknown parameters q Find values of the unknowns where that derivative is zero The two differ in the first step. The least squares principle would have us pick f = e′e as our objective function which we will minimize. Mathematical Marketing 4 We Wish to Minimize e′e We want to minimize f = e′e over all possible values of elements in the vector The function f depends on Mathematical Marketing 5 Minimizing e e Cont'd These two terms are the same so f = y y – 2 y X + X X. Mathematical Marketing 6 What Is the Derivative? Our objective function is the sum f = y y – 2 y X + X X Now we need to determine the derivative, and set it to a column of k zeroes. The derivative of a sum is equal to the sum of the derivatives, so we can handle it in pieces. Mathematical Marketing 7 A Quickie Review of Some Derivative Rules The derivative of a constant: The derivative of a linear combination: The derivative of a transpose: The derivative of a quadratic form: Mathematical Marketing 8 The Derivative of the Sum Is the Sum of the Derivatives f = y y – 2 y X + X X (The derivative of a constant) (The derivative of a quadratic form) (The derivative of a linear combination and the derivative of a transpose) Mathematical Marketing 9 Beta Gets a Hat Add these all together and set equal to zero And with some algebra (This one has a name) (This one has a hat) Mathematical Marketing 10 Is Our Formula Any Good? Mathematical Marketing 11 What Do We Really Mean by “Good”? Unbiasedness Consistency Sufficiency as n . does not depend on Efficiency is smaller than other estimators Mathematical Marketing 12 Two Key Assumptions The behavior of the estimator is driven by the error input According to the Gauss-Markov Assumption, V(e) = looks like Mathematical Marketing 13 The Likelihood Principle Consider a sample of 3: 10, 11 and 12. What is ? 5 10 11 12 Mathematical Marketing 10 11 12 14 Maximum Likelihood According to ML, we should pick values for the parameters that maximize the probability of the sample. To do this we need to follow these steps: q Derive the probability of an observation q Assuming independent observations, calculate the likelihood of the sample using multiplication q Take the log of the sample likelihood q Derive the derivative of the log likelihood with respect to the parameters q Figure out what the parameters must be so that the derivative is equal to a vector of zeroes • With linear models we can do this analytically using algebra • With non-linear models we sometimes have to use brute force hill “climbing” routines Mathematical Marketing 15 The Probability of Observation yi Mathematical Marketing 16 Multiply Out the Probability of the Whole Sample Mathematical Marketing 17 Take the Log of the Sample Likelihood ln exp(a) = a ln 1 = 0 ln ab = b ln a Mathematical Marketing 18 Figure Out the Derivative and Set Equal to Zero From here we are just a couple of easy algebraic steps away from the normal equations, and the least squares formula, If ML estimators exist for a model, that estimator is guaranteed to be consistent, asymptotically normally distributed and asymptotically efficient. Mathematical Marketing 19 Sums of Squares SSError = y y - y X(X X)-1 X y SSError = SSTotal - SSPredictable Mathematical Marketing 20 Using the Covariance Matrix Instead of the Raw SSCP Matrix The covariance matrix of the y and x variables looks like Mathematical Marketing 21 Using Z Scores We can calculate a standard version of using Z scores Or use the correlation matrix of all the variables Mathematical Marketing 22 The Concept of Partialing Imagine that we divided up the x variables into two sets: so that the Beta's were also divided the same way: The model becomes Mathematical Marketing 23 The Normal Equations The normal equations would then be or Subtract Mathematical Marketing from the first equation gives us 24 The Estimator for the First Set Solving for the Estimator for the first set yields The usual formula Factoring What is this? Mathematical Marketing 25 The P and M Matrices Define P = X(X X)-1 X and define M = I – P, = I - X(X X)-1 X. Mathematical Marketing 26 An Intercept Only Model Mathematical Marketing 27 The Intercept Only Model 2 The model becomes Mathematical Marketing 28 The P Matrix in This Case Mathematical Marketing 29 The M Matrix Mathematical Marketing 30 Response Surface Models: Linear vs Quadratic Mathematical Marketing 31 Response Surface Models: The Sign of the Betas Mathematical Marketing 32