Correlation and spectral analysis Objective investigation of correlation

Correlation and spectral analysis • Objective: – investigation of correlation structure of time series – identification of major harmonic components in time series • Tools: – auto-covariance and auto-correlation function – cross-covariance and cross-correlation function – variance spectrum and spectral density function

Autocovariance and autocorrelation function • Autocovariance function of series Y(t): • Covariance at lag 0 = variance: • To ensure comparibility scaling of autocovariance by variance: • For a random series:

Estimation of autocovariance and autocorrelation function – Estimation of autocovariance: – Estimation of autocorrelation function: – Note that estimators are biased (division by n rather than by (n-k-1): gives smaller mean square error – 95% confidence limits for zero correlation:

Example autocorrelation function for monthly rainfall station PATAS (1979 -1989)

Example PATAS continued (1)

Example PATAS continued (2) Periodicity with T = 12 months due to fixed occurrence of monsoon rainfall

Autocorrelogram of daily rainfall at PATAS Lag = 1 month

Autocorrelation function of harmonic • it follows: – covariance of harmonic remains a harmonic of the same frequency f, so it preserves frequency information – information on phase shift vanishes – amplitude of covariance is equal to variance of Y(t)

Y(t) = harmonic series

Autocovariance and autocorrelation of harmonic

Cross-covariance and cross-correlation function • Cross-covariance: X X(t) Lag = k Y Time t Y(t+k) Time t

Cross-covariance and cross-correlation function • Definition cross-covariance function: • Definition of cross-correlation function: • Note: – for lag 0: XY(0) < 1 unless perfect correlation – maximum may occur at lag 0 – for two water level stations along river, maximum correlation will be equal to k = distance / celerity – hence upstream station X(t) will give maximum correlation with Y(t+k) at downstream site – or downstream site Y(t) will give maximum correlation with X(t-k) at upstream site

Estimation of cross-covariance and crosscorrelation function • Estimation of cross-covariance: • Estimation of cross-correlation: • Note that estimators are biased since n rather than (n-k-1) is used in divider, but estimator provides smaller mean square error

Variance spectrum and spectral density function • Plot of variance of harmonics versus their frequencies is called power or variance spectrum – If Sp(f) is the ordinate of continuous spectrum then the variance contributed by all frequencies in the frequency interval f, f+df is given by Sp(f). df. Hence: – Hydrological processes can be considered to be frequency limited, hence harmonics with f > fc do not significantly contribute to variance of the process. Then: – fc = Nyquist or cut-off frequency

Spectrum (2) • Scaling of variance spectrum by variance (to make them comparable) gives the spectral density function: • spectral density function is Fourier transform of autocorrelation function

Estimation of spectrum Replace YY(k) by r. YY(k) for k=0, 1, 2, …, M M = maximum lag for which acf is estimated M is to be carefully selected To reduce sampling variance in estimate a smoothing function is applied: the spectral density at fk is estimated as weighted average of density at fk-1 , fk+1 • Smoothing function is spectral window Number of • Spectral window has to be carefully designed degrees of • Appropriate window is Tukey window: freedom • • Bandwidth • In frequency domain it implies fk is weighted average according to: 1/4 fk-1 , 1/2 fk , 1/4 fk+1

Estimation of spectrum (2) • Spectral estimator s(f) for Sd(f) becomes: • To be estimated at: fc = 1/(2 t) • According to Jenkins and Watts number of frequency points should be 2 to 3 times (M+1) • (1 - )100% confidence limits:

Confidence limits for white noise (s(f) =2) Variance reduces with decreasing M; for n > 25 variance reduces only slowly

Estimation of spectrum (3) • To reduce sampling variance, M should be taken small, say M = 10 to 15 % of N (= series length) • However, small M leads to large bandwidth B • Large B gives smoothing over large frequency range • E. g. if one expects significant harmonics with periods 16 and 24 hours in hourly series: – – frequency difference is 1/16 - 1/24 = 1/48 hence: B < 1/48 so: M > 4 x 48/3 = 64 by choosing M = 10% of N, then N > 640 data points or about one month • Since it is not known in advance which harmonics are significant, estimation is to be repeated for different M • White noise: YY(k) = 0 for k > 0 it follows since 0 f 1/2, for white noise s(f) 2