Environmental Data Analysis with Mat Lab 2 nd

Environmental Data Analysis with Mat. Lab 2 nd Edition Lecture 19: Smoothing, Correlation and Spectra

SYLLABUS Lecture 01 Lecture 02 Lecture 03 Lecture 04 Lecture 05 Lecture 06 Lecture 07 Lecture 08 Lecture 09 Lecture 10 Lecture 11 Lecture 12 Lecture 13 Lecture 14 Lecture 15 Lecture 16 Lecture 17 Lecture 18 Lecture 19 Lecture 20 Lecture 21 Lecture 22 Lecture 23 Lecture 24 Lecture 25 Lecture 26 Using Mat. Lab Looking At Data Probability and Measurement Error Multivariate Distributions Linear Models The Principle of Least Squares Prior Information Solving Generalized Least Squares Problems Fourier Series Complex Fourier Series Lessons Learned from the Fourier Transform Power Spectra Filter Theory Applications of Filters Factor Analysis Orthogonal functions Covariance and Autocorrelation Cross-correlation Smoothing, Correlation and Spectra Coherence; Tapering and Spectral Analysis Interpolation Linear Approximations and Non Linear Least Squares Adaptable Approximations with Neural Networks Hypothesis testing Hypothesis Testing continued; F-Tests Confidence Limits of Spectra, Bootstraps

Goals of the lecture examine interrelationships between smoothing, correlation and power spectral density

review

Autocorrelation and Cross-correlation

Autocorrelation Measure of correlation in time series at different lags u(t) t

Autocorrelation Measure of correlation in time series at different lags t t lag, multiply and sum area no lag a(t) 0 lag, t

Autocorrelation Measure of correlation in time series at different lags t t lag, multiply and sum area small lag a(t) 0 lag, t

Autocorrelation Measure of correlation in time series at different lags t t lag, multiply and sum area large lag a(t) 0 lag, t

Autocorrelation Measure of correlation in time series at different lags t t lag, multiply and sum area a(t) 0 lag, t

Autocorrelation Measure of correlation in time series at different lags t t lag, multiply and sum area a(t) 0 lag, t a(t)=u(t)⋆u(t)

crooss-correlation Measure of correlation between two time series at different lags u(t) v(t) t t

crooss-correlation Measure of correlation between two time series at different lags u(t) v(t) t t c(t)=u(t)⋆v(t)

important relationships c(t) = u(t)⋆v(t) = u(-t)*v(t) c(ω) = u*(ω) v(ω) a(ω)= |u(ω)|2

rough time series u(t) t sharp autocorrelation a(t) 0 lag, t wide spectrum 0 frequency, ω |u(ω)|2

smooth time series u(t) t wide autocorrelation a(t) 0 lag, t narrow spectrum 0 frequency, ω |u(ω)|2

v(t) t rough timeseries a(t) 0 lag, t 0 v(t) lag, t t a(t) 0 lag, t

Part 1 Smoothing a Time Series

smoothing as filtering (example of 3 -point smoothing)

smoothing as filtering (example of 3 -point smoothing) non-causal

fix-up allow for a delay

fix-up allow for a delay dsmoothed and delayed = s * dobs causal filter, s

triangular smoothing filters 3 points si index, i si 21 points index, i

smoothing if Neuse River Hydrograph

question how does smoothing effect the autocorrelation of d

answer the autocorrelation of s acts as a smoothing filter on the autocorrelation of d

effect of smoothing on autocorrelation

effect of smoothing on autocorrelation of smoothed time series

effect of smoothing on autocorrelation of smoothed time series everything written as convolution

effect of smoothing on autocorrelation of smoothed time series everything written as convolution regrouped

effect of smoothing on autocorrelation of smoothing filter convolved with * autocorrelation of time series

answer the autocorrelation of s acts as a smoothing filter on the autocorrelation of d

Part 2 What Makes a Good Smoothing Filter?

dsmoothed(t) = s(t) * dobs(t) then by the convolution theorem

dsmoothed(t) = s(t) * dobs(t) then by the convolution theorem so what’s this look like?

example of a uniform or “boxcar” smoothing filter s(t) 1/T 0 T time, t

take Fourier Transform where sinc(x) = sin(πx) / (πx)

A) T=3 B) T=21

B) T=21 falls off with frequency (good)

B) T=21 bumpy side lobes (bad)

a box car filter does not suppress high frequencies evenly the challenge find a filter that suppresses high frequencies evenly

Normal Function Fourier Transform of a Normal Function is a Normal Function (which has no side lobes)

B) T=3 A) L=3 B) T=21

but a Normal Function is non-causal (unless you truncate it, in which case it is not exactly a Normal Function)

Box Car Normal Function simplicity sidelobes Triangle

Part 3 Designing a Filter that Suppresses Specific Frequencies

General form of the IIR Filter, f

z-transform of the IIR filter

General form of the IIR Filter z-transform

General form of the IIR Filter z-transform ratio of polynomials

z-transform of the IIR filter u(z) as a product of its factors roots of u(z) roots of v(z) as a v(z) z-transform ratio of polynomials product of its factors

so designing a filter is equivalent to specifying the roots of the two polynomails u(z) and v(z)

at this point we need to explore the relationship between the Fourier Transform and the z-transform

Answer the Fourier Transform is the z-transform evaluated at a specific set of z’s

Relationship between Fourier Transform and Z-transform since

Relationship between Fourier Transform and Z-transform Fourier Transform since

Relationship between Fourier Transform and Z-transform Fourier Transform discrete times and frequencies since

Relationship between Fourier Transform and Z-transform Fourier Transform discrete times and frequencies z-transform since

Relationship between Fourier Transform and Z-transform Fourier Transform discrete times and frequencies z-transform since specific choice of z’s

in words the Fourier Transform is the z-transform evaluated at a specific set of z’s

there are N specific z’s zk or with θ

they plot as equally-spaced points around a “unit circle” in the complex z-plane unit circle, |z|2=1 q real z zero frequency Nyquist frequency imag z

Back to the IIR Filter roots of u(z) roots of v(z)

Back to the IIR Filter (z-zju) is zero at z=zju produces a low amplitude region near z=zju called a “zero”

Back to the IIR Filter 1/(z-zkv) is infinite at z=zku produces a high amplitude region near z=zkv called a “pole”

so build a filter by placing the poles and zeros at strategic points in the complex z-plane

Rules zeros suppress frequencies poles amplify frequencies all poles must be outside the unit circle (so vinv converges) all poles, zeros must be in complex conjugate pairs (so filter is real)

A) B)

A) zero near zero frequency suppresses low frequencies “high pass filter” B) zero near the Nyquist frequency suppresses high frequencies “low pass filter”

A) B)

A) poles near ± a given frequency amplify that frequency “band pass filter” B) poles and zeros near ± a given frequency attenuate that frequency “notch filter”

something useful a tunable band pass filter |f(ω)|2 -fny -f 2 -f 1 0 f 1 f 2 +fny frequency, f

Chebychev band-pass filter: 4 zeros, 4 poles

Chebychev band-pass filter: 4 zeros, 4 poles 2 zeros pole

not quite as boxy as one might hope …

In Mat. Lab

Ground velocity at Palisades NY

Ground velocity at Palisades NY Low pass filter

Ground velocity at Palisades NY high pass filter