The Joint Effort for Data assimilation Integration JEDI

The Joint Effort for Data assimilation Integration (JEDI) Observation Filters - Quality Control Joint Center for Satellite Data Assimilation (JCSDA) JEDI Academy - 10 -13 June 2019

UFO Observation “filters” • JEDI/UFO calls abstract “observation filters” before and after the actual operator • Filters are written once and used for many observation types • Several generic filters already exist - Entirely controlled from yaml configuration file(s) • More generic filters will be developed to cover most needs

Observations Processing Flow Cost. Function: : evaluate Cost. Jo: : initialize Obs. Filters: : Obs. Filters Observer: : Observer Obs. Operator: : variables Obs. Filters: : required. Geo. Va. Ls Cost. Function: : run. NL Model: : forecast Observer: : initialize Geo. Va. Ls: : Geo. Va. Ls loop over time steps Observer: : process State: : get. Values end loop over time steps Observer: : finalize Obs. Filters: : prior. Filter Obs. Operator: : simulate. Obs. Filters: : post. Filter Cost. Jo: : finalize Obs. Errors: : Obs. Errors ydep=ysimul-yobs Create and configure filters Metadata Gather list of required Geo. Va. Ls Create Geo. Va. Ls Fill Geo. Va. Ls Obs. Filters base class manages individual filters (Models and Simulate observations Obs. Operators don’t need to know). Call post filters H(x) Call prior filters Setup R, compute Jo

Observation Processing The observation operator is called from the Observer Observation filters are called immediately before/after the operator template <typename MODEL> void Observer<MODEL>: : do. Finalize() { Log: : trace() << "Observer: : do. Finalize start" << std: : endl; filters_->prior. Filter(*gvals_); hop_. simulate. Obs(*gvals_, yobs_, ybias_); filters_->post. Filter(yobs_); Log: : trace() << "Observer: : do. Finalize done" << std: : endl; } filters_ contains the list of filters and calls them

UFO Observation “filters” • Observation filters are generic and have access to - Observation values and metadata - Model values at observations locations (Geo. Va. Ls) - Simulated observation value (for post-filter) - Their own private data • Filters modify - QC flags - Observation error - Their own data (anything passed to the constructor)

UFO Generic Observation Filters • Pre. QC - Handles flags set by pre-processing in IODA • Background. Check - Currently based on σo only (σb will be added) • Obs. Bounds. Check - Rejects observations values outside some bounds • Obs. Domain. Check - Rejects observations with metadata values outside some bounds (metadata can be location or any other attribute) • Black. List - Unconditionally rejects data for some variables, channels…

UFO Generic Observation Filters • All filters except the Black. List take an optional “where” clause to restrict their scope of application - The where clause is very generic and can be expressed in terms of observation metadata, value or Geo. Va. Ls value • Filters are controlled by the yaml configuration file - Filters are applied in the order in which they appear in the yaml - Most scientific work happens in the yaml file • Another filter (QCmanager) is used internally for book-keeping • Geo. Va. Ls. Writer (in OOPS) saves Geo. Va. Ls to file is a “filter”

yaml examples Obs. Types: - Obs. Space: name: Radiosonde Obs. Data. In: obsfile: Data/obs/sondes_obs_2018041500_m. nc 4 Obs. Data. Out: obsfile: Data/hofx/sondes_hyb-4 dvar-gfs_2018041500_m. nc 4 simulate: variables: [eastward_wind, northward_wind, air_temperature] Obs. Operator: name: Radiosonde Obs. Filters: - Filter: Pre. QC threshold: 3 apply_at_iterations: 0 - Filter: Background Check variables: [eastward_wind, northward_wind, air_temperature] threshold: 6. 0

yaml examples Obs. Types: - Obs. Operator: name: CRTM n_Absorbers: 3 n_Clouds: 2 n_Aerosols: 0 Sensor_ID: amsua_n 19 Endian. Type: little_endian Coefficient. Path: Data/ Obs. Space: name: amsua_n 19 Obs. Data. In: obsfile: Data/amsua_n 19_obs_2018041500_m. nc 4 simulate: variables: [brightness_temperature] channels: 1 -15 Obs. Filters: - Filter: Background Check variables: [brightness_temperature] channels: 1 -5 threshold: 2. 0

yaml examples - Obs. Operator: name: Gnssro. Ref Obs. Options: use_compress: 1 Obs. Filters: - Filter: Domain Check where: - variable: altitude minvalue: 0 maxvalue: 30000 - variable: earth_radius_of_curvature minvalue: 6250000 maxvalue: 6450000 - variable: geoid_height_above_reference_ellipsoid minvalue: -200 maxvalue: 200 - Filter: Background Check variables: - refractivity threshold: 3. 0

yaml examples -Obs. Space: name: Sea. Surface. Salinity Obs. Data. Out: {obsfile: . /Data/sss. out. nc} Obs. Data. In: {obsfile: . /Data/sss. nc} simulate: variables: [sea_surface_salinity] Obs. Operator: name: Sea. Surface. Salinity Obs. Filters: - Filter: Domain Check where: - variable: sea_area_fraction@Geo. Va. Ls minvalue: 0. 5 - Filter: Domain Check where: - variable: sea_surface_temperature@Geo. Va. Ls minvalue: 15

Filter Example The Domain Check filter is not very complicated: void Obs. Domain. Check: : prior. Filter(const Geo. Va. Ls & gv) const { const oops: : Variables vars(config_. get. String. Vector("observed")); std: : vector<bool> inside = process. Where(obsdb_, gv, config_); for (size_t jv = 0; jv < vars. size(); ++jv} ( for (size_t jobs = 0; jobs < obsdb_. nlocs(); ++jobs) { if (!inside[jobs] && flags_[jv][jobs] == 0) { flags_[jv][jobs] = QCflags: : domain; } { { {

Observation Filters: Next Developments • Thinning (code exists) • Buddy check • Duplicate check • Profile check (inversions…) • Var. QC • Maximum speed of ships, buoys, sondes, aircrafts… • Force use of @Meta. Data • Save more information in diagnostics file • One default yaml file per observation type

Final Comments Think generic!