Prediction variance in Linear Regression Assumptions on noise

  • Slides: 12
Download presentation
Prediction variance in Linear Regression • Assumptions on noise in linear regression allow us

Prediction variance in Linear Regression • Assumptions on noise in linear regression allow us to estimate the prediction variance due to the noise at any point. • Prediction variance is usually large when you are far from a data point. • We distinguish between interpolation, when we are in the convex hull of the data points, and extrapolation where we are outside. • Extrapolation is associated with larger errors, and in high dimensions it usually cannot be avoided.

Linear Regression •

Linear Regression •

Model based error for linear regression •

Model based error for linear regression •

Prediction variance • Linear regression model • Define • With some algebra • Standard

Prediction variance • Linear regression model • Define • With some algebra • Standard error then

 Interpolation, extrapolation and regression • Interpolation is often contrasted to regression or least-squares

Interpolation, extrapolation and regression • Interpolation is often contrasted to regression or least-squares fit • As important is the contrast between interpolation and extrapolation • Extrapolation occurs when we are outside the convex hull of the data points • For high dimensional spaces we must have extrapolation!

2 D example of convex hull • By generating 20 points at random in

2 D example of convex hull • By generating 20 points at random in the unit square we end up with substantial region near the origin where we will need to use extrapolation • Using the data in the notes, give a couple of alternative sets of alphas Approximately for the point (0. 4, 0. 4)

Example of prediction variance • For a linear polynomial RS y=b 1+b 2 x

Example of prediction variance • For a linear polynomial RS y=b 1+b 2 x 1+b 3 x 2 find the prediction variance in the region • (a) For data at three vertices (omitting (1, 1))

Interpolation vs. Extrapolation • At origin . At 3 vertices . At (1, 1)

Interpolation vs. Extrapolation • At origin . At 3 vertices . At (1, 1)

Standard error contours •

Standard error contours •

Data at four vertices • Now • And • Error at vertices • At

Data at four vertices • Now • And • Error at vertices • At the origin minimum is • How can we reduce error without adding points?

Graphical Comparison of Standard Errors Three points Four points

Graphical Comparison of Standard Errors Three points Four points

Homework • Redo the four point example, when the data points are not at

Homework • Redo the four point example, when the data points are not at the corners but inside the domain, at +-0. 8. What does the difference in the results tells you? • For a grid of 3 x 3 data points, compare the standard errors for a linear and quadratic fits.