Chapter 5 Ordinary Least Square Regression We will

  • Slides: 32
Download presentation
Chapter 5: Ordinary Least Square Regression We will be discussing q The Linear Regression

Chapter 5: Ordinary Least Square Regression We will be discussing q The Linear Regression Model q Estimation of the Unknowns in the Regression Model q Some Special Cases of the Model Mathematical Marketing 1

The Basic Regression Model The model expressed in scalars The model elaborated in matrix

The Basic Regression Model The model expressed in scalars The model elaborated in matrix terms A succinct matrix expression for it y = X + e Mathematical Marketing 2

Prediction Based on a Linear Combination The E(y) is given by Together with the

Prediction Based on a Linear Combination The E(y) is given by Together with the previous slide, we can say that Key Question: How Do We Get Values for the vector? Mathematical Marketing 3

Parameter Estimation of We will cover two philosophies of parameter estimation, the least squares

Parameter Estimation of We will cover two philosophies of parameter estimation, the least squares principle and maximum likelihood. Each of these has the following steps: q Pick an objective function to optimize q Find the derivative of f with respect to the unknown parameters q Find values of the unknowns where that derivative is zero The two differ in the first step. The least squares principle would have us pick f = e′e as our objective function which we will minimize. Mathematical Marketing 4

We Wish to Minimize e′e We want to minimize f = e′e over all

We Wish to Minimize e′e We want to minimize f = e′e over all possible values of elements in the vector The function f depends on Mathematical Marketing 5

Minimizing e e Cont'd These two terms are the same so f = y

Minimizing e e Cont'd These two terms are the same so f = y y – 2 y X + X X. Mathematical Marketing 6

What Is the Derivative? Our objective function is the sum f = y y

What Is the Derivative? Our objective function is the sum f = y y – 2 y X + X X Now we need to determine the derivative, and set it to a column of k zeroes. The derivative of a sum is equal to the sum of the derivatives, so we can handle it in pieces. Mathematical Marketing 7

A Quickie Review of Some Derivative Rules The derivative of a constant: The derivative

A Quickie Review of Some Derivative Rules The derivative of a constant: The derivative of a linear combination: The derivative of a transpose: The derivative of a quadratic form: Mathematical Marketing 8

The Derivative of the Sum Is the Sum of the Derivatives f = y

The Derivative of the Sum Is the Sum of the Derivatives f = y y – 2 y X + X X (The derivative of a constant) (The derivative of a quadratic form) (The derivative of a linear combination and the derivative of a transpose) Mathematical Marketing 9

Beta Gets a Hat Add these all together and set equal to zero And

Beta Gets a Hat Add these all together and set equal to zero And with some algebra (This one has a name) (This one has a hat) Mathematical Marketing 10

Is Our Formula Any Good? Mathematical Marketing 11

Is Our Formula Any Good? Mathematical Marketing 11

What Do We Really Mean by “Good”? Unbiasedness Consistency Sufficiency as n . does

What Do We Really Mean by “Good”? Unbiasedness Consistency Sufficiency as n . does not depend on Efficiency is smaller than other estimators Mathematical Marketing 12

Two Key Assumptions The behavior of the estimator is driven by the error input

Two Key Assumptions The behavior of the estimator is driven by the error input According to the Gauss-Markov Assumption, V(e) = looks like Mathematical Marketing 13

The Likelihood Principle Consider a sample of 3: 10, 11 and 12. What is

The Likelihood Principle Consider a sample of 3: 10, 11 and 12. What is ? 5 10 11 12 Mathematical Marketing 10 11 12 14

Maximum Likelihood According to ML, we should pick values for the parameters that maximize

Maximum Likelihood According to ML, we should pick values for the parameters that maximize the probability of the sample. To do this we need to follow these steps: q Derive the probability of an observation q Assuming independent observations, calculate the likelihood of the sample using multiplication q Take the log of the sample likelihood q Derive the derivative of the log likelihood with respect to the parameters q Figure out what the parameters must be so that the derivative is equal to a vector of zeroes • With linear models we can do this analytically using algebra • With non-linear models we sometimes have to use brute force hill “climbing” routines Mathematical Marketing 15

The Probability of Observation yi Mathematical Marketing 16

The Probability of Observation yi Mathematical Marketing 16

Multiply Out the Probability of the Whole Sample Mathematical Marketing 17

Multiply Out the Probability of the Whole Sample Mathematical Marketing 17

Take the Log of the Sample Likelihood ln exp(a) = a ln 1 =

Take the Log of the Sample Likelihood ln exp(a) = a ln 1 = 0 ln ab = b ln a Mathematical Marketing 18

Figure Out the Derivative and Set Equal to Zero From here we are just

Figure Out the Derivative and Set Equal to Zero From here we are just a couple of easy algebraic steps away from the normal equations, and the least squares formula, If ML estimators exist for a model, that estimator is guaranteed to be consistent, asymptotically normally distributed and asymptotically efficient. Mathematical Marketing 19

Sums of Squares SSError = y y - y X(X X)-1 X y SSError

Sums of Squares SSError = y y - y X(X X)-1 X y SSError = SSTotal - SSPredictable Mathematical Marketing 20

Using the Covariance Matrix Instead of the Raw SSCP Matrix The covariance matrix of

Using the Covariance Matrix Instead of the Raw SSCP Matrix The covariance matrix of the y and x variables looks like Mathematical Marketing 21

Using Z Scores We can calculate a standard version of using Z scores Or

Using Z Scores We can calculate a standard version of using Z scores Or use the correlation matrix of all the variables Mathematical Marketing 22

The Concept of Partialing Imagine that we divided up the x variables into two

The Concept of Partialing Imagine that we divided up the x variables into two sets: so that the Beta's were also divided the same way: The model becomes Mathematical Marketing 23

The Normal Equations The normal equations would then be or Subtract Mathematical Marketing from

The Normal Equations The normal equations would then be or Subtract Mathematical Marketing from the first equation gives us 24

The Estimator for the First Set Solving for the Estimator for the first set

The Estimator for the First Set Solving for the Estimator for the first set yields The usual formula Factoring What is this? Mathematical Marketing 25

The P and M Matrices Define P = X(X X)-1 X and define M

The P and M Matrices Define P = X(X X)-1 X and define M = I – P, = I - X(X X)-1 X. Mathematical Marketing 26

An Intercept Only Model Mathematical Marketing 27

An Intercept Only Model Mathematical Marketing 27

The Intercept Only Model 2 The model becomes Mathematical Marketing 28

The Intercept Only Model 2 The model becomes Mathematical Marketing 28

The P Matrix in This Case Mathematical Marketing 29

The P Matrix in This Case Mathematical Marketing 29

The M Matrix Mathematical Marketing 30

The M Matrix Mathematical Marketing 30

Response Surface Models: Linear vs Quadratic Mathematical Marketing 31

Response Surface Models: Linear vs Quadratic Mathematical Marketing 31

Response Surface Models: The Sign of the Betas Mathematical Marketing 32

Response Surface Models: The Sign of the Betas Mathematical Marketing 32