Variational cloud retrievals from radar lidar and radiometers

Variational cloud retrievals from radar, lidar and radiometers Robin Hogan Julien Delanoë Nicola Pounder University of Reading

Introduction • Best estimate of the atmospheric state from instrument synergy – Use a variational framework / optimal estimation theory • Some important measurements are integral constraints – E. g. microwave, infrared and visible radiances – Affected by all cloud types in profile, plus aerosol and precipitation – Hence need to retrieve different particle types simultaneously • Funded by ESA and NERC to develop unified retrieval algorithm – For application to Earth. CARE – Will be tested on ground-based, airborne and A-train data • Algorithm components – Target classification input – State variables – Minimization techniques: Gauss-Newton vs. Gradient-Descent – Status of forward models and their adjoints • Progress with individual target types – Ice clouds – Liquid clouds

1. New ray of data: define state vector Use classification to specify variables describing each species at each gate Ice: extinction coefficient , N 0’, lidar extinction-to-backscatter ratio Liquid: extinction coefficient and number concentration Rain: rain rate and mean drop diameter Aerosol: extinction coefficient, particle size and lidar ratio 2. Convert state vector to radar-lidar resolution Often the state vector will contain a low resolution description of the profile 3. Forward model 3 a. Radar model Including surface return and multiple scattering 3 b. Lidar model Including HSRL channels and multiple scattering 4. Compare to observations Check for convergence Converged 7. Calculate retrieval error Error covariances and averaging kernel 3 c. Radiance model Solar and IR channels Retrieval framework Ingredients developed before In progress Not yet developed 6. Iteration method Derive a new state vector Either Gauss-Newton or quasi-Newton scheme Not converged 5. Convert Jacobian/adjoint to state-vector resolution Initially will be at the radar-lidar resolution Proceed to next ray of data

Target classification • In Cloudnet we used radar and lidar to provide a detailed discrimination of target types (Illingworth et al. 2007): • A similar approach has been used by Julien Delanoe on Cloud. Sat and Calipso using the one-instrument products as a starting point: • More detailed classifications could distinguish “warm” and “cold” rain (implying different size distributions) and different aerosol types

Example from the AMF in Niamey 94 -GHz radar reflectivity Observations 532 -nm lidar backscatter Forward model at final iteration 94 -GHz radar reflectivity 532 -nm lidar backscatter

Results: radar+lidar only Retrievals in regions where radar or lidar detects the cloud Retrieved visible extinction coefficient Retrieved effective radius Large error where only one instrument detects the cloud Retrieval error in ln(extinction)

Results: radar, lidar, SEVERI radiances Delanoe & Hogan (JGR 2008) TOA radiances increase retrieved optical depth and decrease particle size near cloud top Retrieved visible extinction coefficient Retrieved effective radius Cloud-top error greatly reduced Retrieval error in ln(extinction)

Unified algorithm: state variables • Proposed list of retrieved variables held in the state vector x State variable Representation with height / constraint Ice clouds and snow Visible extinction coefficient One variable per pixel with smoothness constraint Number conc. parameter Cubic spline basis functions with vertical correlation Lidar extinction-to-backscatter ratio Cubic spline basis functions Riming factor Likely a single value per profile Liquid clouds Liquid water content Droplet number concentration One variable per pixel but with gradient constraint One value per liquid layer Rain rate Cubic spline basis functions with flatness constraint Normalized number conc. Nw One value per profile Melting-layer thickness scaling factor One value per profile A-priori Ice clouds follows None Delanoe & Hogan (2008); Temperature dependent Snow & riming in convective 20 sr clouds needs to be added Liquid clouds 1 currently being tackled None Basic rain to be Temperature dependent added shortly; Full representation None later Basic aerosols Dependent on whether to be added from melting ice or shortly; Full coallescence representation 1 via collaboration?

The cost function • The essence of the method is to find the state vector x that minimizes a cost function: The forward model H(x) predicts the observations from the state vector x Some elements of x are constrained by an a priori estimate + Smoothness constraints Each observation yi is weighted by the inverse of its error variance This term penalizes curvature in the extinction profile

Gauss-Newton method • See Rodgers’ book (p 85): write the cost function in matrix form: • Define its gradient (a vector): • …and its second derivative (a matrix): • Approximate J as quadratic and apply this: – Advantage: rapid convergence (instant convergence for linear problems) – Another advantage: get the error covariance of the solution “for free” – Disadvantage: need the Jacobian matrix of every forward model: can be expensive for larger problems

Gradient descent methods • Just use gradient information: – Advantage: we don’t need to calculate the Jacobian so forward model is cheaper! – Disadvantage: more iterations needed since we don’t know curvature of J(x) – Use a quasi-Newton method to get the search direction, such as BFGS used by ECMWF: builds up an approximate form of the second derivative to get improved convergence – Scales well for large x – Disadvantage: poorer estimate of the error at the end • Why don’t we need the Jacobian H? Typical convergence behaviour – The “adjoint” of a forward model takes as input the vector {. } and outputs the vector Jobs without needing to calculate the matrix H on the way – Adjoint can be coded to be only ~3 times slower than original forward model – Tricky coding for newcomers, although some automatic code generators exist

Forward model components • From state vector x to forward modelled observations H(x). . . Ice & snow Liquid cloud Rain Aerosol x Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor Ice/radar Ice/lidar Ice/radiometer Liquid/radar Liquid/lidar Liquid/radiometer Rain/radar Rain/lidar Rain/radiometer Aerosol/lidar Aerosol/radiometer Jacobian matrix H= y/ x Computationally expensive matrix-matrix multiplications: the most expensive part of the entire algorithm Sum the contributions from each constituent Radar scattering profile Lidar scattering profile Radiative transfer models Radar forward modelled obs Lidar forward modelled obs Radiometer scattering profile H(x) Radiometer forward modelled obs Gradient of radar lidar radiometer measurements s with respect to to radar to lidar inputs radiometer inputs Jacobian part of radiative transfer models

Equivalent adjoint method • From state vector x to forward modelled observations H(x). . . Ice & snow Liquid cloud Rain Aerosol x Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor Ice/radar Ice/lidar Ice/radiometer Liquid/radar Liquid/lidar Liquid/radiometer Rain/radar Rain/lidar Rain/radiometer Aerosol/lidar Aerosol/radiometer Gradient of cost function (vector) J/ x=HTy* Vector-matrix multiplications: around the same cost as the original forward operations Sum the contributions from each constituent Radar scattering profile Lidar scattering profile Radiative transfer models Radar forward modelled obs Lidar forward modelled obs Radiometer scattering profile H(x) Radiometer forward modelled obs Adjoint of radar model lidar model radiometer model (vector) Adjoint of radiative transfer models

Scattering models • First part of a forward model is the scattering and fall-speed model – Same methods typically used for all radiometer and lidar channels – Radar and Doppler model uses another set of methods Particle type Aerosol Radar (3. 2 mm) Aerosol not detected by radar Liquid droplets Mie theory Rain drops T-matrix: Brandes et al. (2002) shapes Ice cloud T-matrix (Hogan et particles al. 2010) Graupel and hail Mie theory Melting ice Wu & Wang (1991) Radar Doppler Aerosol not detected by radar Beard (1976) Thermal IR, Solar, UV Mie theory, Highwood refractive index Mie theory Westbrook & Heymsfield TBD Baran (2004) Mie theory • Graupel and melting ice still uncertain; but normal ice is decided. . .

Radiative transfer forward models • Computational cost can scale with number of points describing vertical profile N; we can cope with an N 2 dependence but not N 3 Radar/lidar model Applications Speed Jacobian Adjoint Single scattering: b’=b exp(-2 t) Radar & lidar, no multiple scattering N N 2 N Platt’s approximation b’=b exp(-2 ht) Lidar, ice only, crude multiple scattering N N 2 N Photon Variance-Covariance (PVC) method (Hogan 2006, 2008) Lidar, ice only, small-angle multiple scattering N or N 2 N Time-Dependent Two-Stream (TDTS) method (Hogan and Battaglia 2008) Lidar & radar, wide-angle multiple scattering N 2 N 3 N 2 Depolarization capability for TDTS Lidar & radar depol with multiple scattering N 2 • • N 2 Lidar uses PVC+TDTS (N 2), radar uses single-scattering+TDTS (N 2) Jacobian of TDTS is too expensive: N 3 We have recently coded adjoint of multiple scattering models Future work: depolarization forward model with multiple scattering Radiometer model Applications RTTOV (used at ECMWF & Met Office) Infrared and microwave radiances N Two-stream source function technique (e. g. Delanoe & Hogan 2008) Infrared radiances N N 2 LIDORT Solar radiances N N 2 • • Speed Jacobian Adjoint N Infrared will probably use RTTOV, solar radiances will use LIDORT Both currently being tested by Julien Delanoe N

Gradient constraint A. Slingo, S. Nichols and J. Schmetz, Q. J. R. Met. Soc. 1982 • We have a good constraint on the gradient of the state variables with height for: – LWC in stratocu (adiabatic profile, particularly near cloud base) – Rain rate (fast falling so little variation with height expected) • Not suitable for the usual “a priori” constraint • Solution: add an extra term to the cost function to penalize deviations from gradient c:

Example in liquid clouds • Using simulated observations: – Triangular cloud observed by a 1 - or 2 -field-of-view lidar – Retrieval uses Levenberg-Marquardt minimization with Hogan and Battaglia (2008) model for lidar multiple scattering Optical depth=30 Two FOVs: very good performance Footprint=100 m One 10 -m footprint: saturates at optical Footprint=600 m depth=5 One 100 -m footprint: multiple scattering helps!

Progress • Done: – C++: object orientation allows code to be completely flexible: observations can be added and removed without needing to keep track of indices to matrices, so same code can be applied to different observing systems – Code to generate particle scattering libraries in Net. CDF files – Adjoint of radar and lidar forward models with multiple scattering and HSRL/Raman support – Radar/lidar model interfaced and cost function can be calculated • In progress / future work: – Interface to BFGS algorithm (e. g. in GNU Scientific Library) – Implement ice, liquid, aerosol and rain constituents – Interface to radiance models – Test on a range of ground-based and spaceborne instruments – Test using ECSIM observational simulator – Apply to large datasets of ground-based observations…