Developing 4 DEnVar thoughts and progress Neill Bowler

Developing 4 D-En-Var: thoughts and progress Neill Bowler Andrew Lorenc, Adam Clayton, Dale Barker … (and more…) © Crown copyright Met Office

What are we building? 4 D-En-Var / 5 D-En-Var (Lorenc, 2003, Liu et al. , 2008) © Crown copyright Met Office

Hybrid data assimilation – 4 DVar • 4 D-Var cost-function: • Now the background term is split into two (Wang et al. , 2007): • Where © Crown copyright Met Office

Hybrid data assimilation – 4 DEn-Var • 4 D-Var hybrid Tangent-linear model Potentially 4 -dimensional • 4 D-En-Var hybrid 3 D hybrid • Benefit: Reduced cost (no tangent-linear) • Disadvantage: Tangent-linear implied from limited size ensemble © Crown copyright Met Office

Using the DEn. KF method • DEn. KF (Sakov & Oke, 2008): First update the mean • Then update the ensemble perturbations • En. Var: First analyse for each member • Then post-process to get the analysis states • Run an assimilation for each member • Find the ensemble mean • Revert the ensemble perturbation back towards the forecast © Crown copyright Met Office

Why build 4 D-En-Var / 5 D-En-Var? © Crown copyright Met Office

Key reasons for choosing 4 DEn-Var • DA and ensemble unified • Maintenance • Consistency of perturbations with DA (Berre et al. , 2006) • Simultaneous processing of obs • All observations processed at same time, so do not have to choose order of observations • Model-space localisation • Localisation in model space – avoids issues with satellite radiances and localisation between observation points • Localisation in transformed variables – improved balance • Hybrid covariances • Can run DA and ensemble using a hybrid background-error covariance • Not too expensive • Not as cheap as some options, but we can still run many members © Crown copyright Met Office

Using hybrid covariances • Berre et al (2006) indicated that ensemble setup should mimic analysis system • Kalman gain used in updating perturbations should be the same as used in updating analysis • Test using Lorenz ’ 95 40 -variable model (identical twin) • Hybrid covariance matrix used in updating the control forecast • Use hybrid covariances in updating perturbations? © Crown copyright Met Office

Using hybrid covariances • Obs error 1. 0, observations every grid-point • 5 ensemble members • Hybrid covariance used in control analysis © Crown copyright Met Office

Where is our building at? © Crown copyright Met Office

Initial coding • 4 D-En-Var has been coded in the Met Office VAR system • It includes the ability to do an ensemble of analyses • Adaptive inflation (used by MOGREPS) has been integrated with VAR code • A few flies: • There are still some bugs in some aspects of the code • Need to work on optimising performance © Crown copyright Met Office

Can we afford it? • Successful runs on 4 nodes of IBM Power 6 • Takes ~1 h for 23 analyses (1 deterministic + 22 ensemble) • Approximately 20 x cheaper than equivalent 4 D-Var • Requires most of the memory of those 4 nodes (50 GB per node, requires around 155 GB) • Increasing ensemble size and resolution increases these both • Can do analysis at lower resolution © Crown copyright Met Office

Example fields • Run 4 D-En-Var with 22 m for a single analysis cycle (6 Z on 1 st Dec 2011) • Run with all observations (conventional + satellite) • 80 iterations in solver • Compare with increments from the operational analysis (hybrid 4 D-Var, 50% ensemble, 80% climatological) • Gives an idea of some of the differences between the two systems © Crown copyright Met Office

Example increment: hybrid 4 D-Var theta L 15 (~850 h. Pa) © Crown copyright Met Office

Example increment: 4 D-En-Var theta L 15 (~850 h. Pa) © Crown copyright Met Office

Example increment: 4 D-En-Var Hybrid theta L 15 (~850 h. Pa) (70% ensemble, 30% static) © Crown copyright Met Office

Forecast ensemble spread MOGREPS theta L 15 (~850 h. Pa) © Crown copyright Met Office

Analysis ensemble spread 4 D-En -Var theta L 15 (~850 h. Pa) © Crown copyright Met Office

Ratio of ensemble spread 4 D-En. Var theta L 15 (~850 h. Pa) © Crown copyright Met Office

Ratio of ensemble spread 4 D-En. Var theta L 15 (~850 h. Pa) (70/30 hybrid) © Crown copyright Met Office

Conclusion • Developed 4 D-En-Var for use in data assimilation and ensemble • In testing and development phase • Optimise performance: improve convergence, parallelise minimisation • Compare with 4 D-Var, using the same input data from the ensemble (will require a larger ensemble) • Test 4 D-En-Var in ensemble mode, comparing with current ETKF • Develop an additive inflation scheme to represent model error • Test different methods of multiplicative inflation © Crown copyright Met Office

Thanks for listening © Crown copyright Met Office

References • Berre L, Stefanescu S, and Pereira M, 2006. The representation of the analysis effect in three error simulation techniques. Tellus, 58 A: 196– 209. • Greybush SJ, Kalnay E, Miyoshi T, Ide K, Hunt BR. 2011. Balance and ensemble Kalman filter localization techniques, Mon. Wea. Rev. 139: 511– 522 • Liu C, Xiao Q, Wang B. 2008. An ensemble-based four-dimensional variational data assimilation scheme. Part I: Technical formulation and preliminary test. Mon. Wea. Rev. 136: 3363– 3373 • Lorenc AC, 2003. The potential of the ensemble Kalman filter for NWP a comparison with 4 D-Var. Q. J. Royal. Met. Soc. , 129: 3183 -3203 • Sakov P, Oke PR. 2008. A deterministic formulation of the ensemble Kalman filter: an alternative to ensemble square root filters. Tellus 60 A: 361– 371 • Wang X, Snyder C, Hamill TM, 2007. On theoretical equivalence of differently proposed ensemble-3 DVar hybrid analysis schemes, Mon. Wea. Rev. , 135: 222 -227 © Crown copyright Met Office

Example increment: 4 D-En-Var u-wind © Crown copyright Met Office

Example increment: hybrid 4 D-Var u-wind © Crown copyright Met Office

Ratio of ensemble spread (smoothed, non-hybrid) © Crown copyright Met Office

Ratio of ensemble spread (smoothed, 70/30 hybrid) © Crown copyright Met Office

4 D-Var Hybrid MOGREPS-G 4 D-Var Linear model 21 Z © Crown copyright Met

4 D-En-Var MOGREPS-G 4 D-Ens-Var 21 Z © Crown copyright Met Office 0 Z

Ensemble 4 D-En-Var / 5 D-En -Var En. KF 4 D-Ens-Var MOGREPS-G 4 D-Ens-Var 21 Z © Crown copyright Met Office 0 Z 3 Z 6 Z 9 Z 12 Z 15 Z

Issue with observation localisation • Consider an idealised case (following Greybush et al, 2011)

Issue with observation localisation • Covariance localisation • Observation localisation © Crown copyright Met

• Covariance localisation: (a) (b) • Observation localisation: (a)+(b) © Crown copyright Met

Using hybrid covariances • Obs error 1. 0, observations every grid-point • 5 ensemble members • Hybrid covariance used in control analysis • Hybrid does not help in En. SRF © Crown copyright Met Office

What’s wrong with a squareroot hybrid? • The update equation for the analysis-error covariance is • If the Kalman gain is the optimal choice • Then this simplifies to • Square-root filters assume the simplified form of the analysis error covariance • If we use a hybrid, then we are not using the optimal Kalman gain, and a square-root filter may not be appropriate © Crown copyright Met Office