The Joint Effort for Data assimilation Integration JEDI

The Joint Effort for Data assimilation Integration (JEDI) Interface for Observation Data Access (IODA) Subsystem Joint Center for Satellite Data Assimilation (JCSDA) JEDI Academy – 24 -27 February 2020

JEDI Overview FV 3 (GFS+GOES) (NOAA/NASA) MPAS (NCAR) A Next-Generation Unified DA System (credit: M. Miesch) Radiosondes Radiance (AMSU-A, …) IODA NEPTUNE (NRL) Aircraft LFRic (UKMO) Aerosols (AOD) MOM 6 (JCSDA/NOAA) … IODA • OOPS • SABER (BUMP) • UFO • CRTM • IODA Sea Ice (fraction, thickness) • Enables high leverage • For example, add your model • Then you have access to: • Obs data • Forward operators • DA flows • Etc. … 2 JEDI Academy, February, 2020

Interface for Observation Data Access (IODA) UFO/JEDI requires access to observation data Separation of concerns: isolate science from data storage Three levels: - Long term storage (historic database, R 2 D 2) - Files on disk (one DA cycle) - In memory handling of observations Two environments: - Plotting, analyzing, verifying on workstation - DA and other HPC applications (MPI, threads, GPUs…) IODA 3 JEDI Academy, February, 2020

Observation Data Handling Large Small Fast Slow Tape Disk SSD NVRAM Some OSs RAM Cache Register Compiler MARS + FDB Ideally, we want uniform and easy access to observation data Can we hide the complexity of the memory hierarchy from scientists (separation of concerns)? Files could become irrelevant in the future IODA 4 JEDI Academy, February, 2020

Observation Data Flow with IODA Plots Diagnostics Monitoring GTS Network Decoding IODA DA Preprocessing One observation data handling interface across the whole NWP chain IODA 5 JEDI Academy, February, 2020

) (x a ) H (x b H gs fla or C rr Q ue. E al. V Uniform view and interface for observation data access bs O Locations Access to Observations Interface for Observation Data Access based on this abstract view Ch 1 Ch 2 Ch 3 Ch 4 Ch 5 … T U V Q … Latitude Longitude Height Date. Time Station ID … Group by - flight - ascent - station. . . Obs. type Experiment Id … Abstract representation, not necessarily actual storage structure Channel frequencies … Variables IODA 6 JEDI Academy, February, 2020

Variable Naming IODA • • IODA schematic consists of tables containing variables The names of the tables are referred to as Groups • A specific variable is denoted by a group name plus a variable name – – Obs. Value, Obs. Error, Pre. QC, Hof. X, Meta. Dat, etc. Two styles of specifying these pairs • • Group, Variable@Group Meta. Data, latitude@Meta. Data Obs. Value, air_temperature@Obs. Value Hof. X, brightness_temperature@Hof. X JEDI Academy, February, 2020

IODA Requirements Requirement IODA Status Flexible Running DA tasks with ~30 obs types (10 conv, 20 radiance) Access to data and meta data Providing support for ~20 filter operations (constructed from generic filter ops in UFO), plus support for obs operators for the ~30 obs types Efficient I/O Testing larger DA runs using millions of observations Efficient compression Testing the construction of obs data archive Portable GNU, Intel and Clang compilers; Open. MPI, MPICH 2, IMPI implementations Running on supercomputers (Hera, Discover, Cheyenne, S 4), AWS EC 2 instances, and on laptops (mac, linux) Security Installation/execution can be done behind firewalls Support analysis Have capability to write netcdf and ODB formats giving access to a variety of diagnostic tools Data import Currently handle Netcdf, ODB, BUFR, Marine binary profile types, from various sources including GSI, NCEP, Met Office, GODAS and GODAE Ease of use Simple interface (get, put) for direct access by obs operators, and access via obs vectors Reliability Testing larger DA runs using millions of observations Replicate existing functionality GSI conversion path includes extraction of H(x) and filtering results from GSI runs for comparison with JEDI results, all obs assimilated by GSI have been converted for use in JEDI IODA 8 JEDI Academy, February, 2020

Current state of IODA ioda-converters NCEP (BUFR) bufr 2 ioda GODAE (Binary) profile 2 ioda Met Office (ODB) odb 2 ioda … IODA File Interface (Ioda. IO) Archive, File (IODA) Memory (Obs. Space) … Diagnostics 9 JEDI Academy, February, 2020

Objects related to observations • IODA JEDI Academy, February, 2020

IODA Implements the Obs. Space Interface Abstract Layer (OOPS) Generic applications Forecast 4 DVar 3 DVar EDA En. KF … Uses Interface layer State Model Covariance … Obs Operator Obs Space Implements Specific implementations IODA FV 3 MOM 6 MPAS 11 … SABER UFO IODA JEDI Academy, February, 2020

IODA-OOPS C++ Class Relationship • • OOPS provides an abstract interface layer – IODA provides concrete classes that implement two of the OOPS interface classes – – IODA Templated classes • Allows multiple implementations of underlying concrete classes Obs. Vector • Holds quantities such as y and H(x) Obs. Space • Analogous to a mathematical space that contains vectors • Provides an interface to observation data stored in files JEDI Academy, February, 2020

IODA Class Structure OOPS Obs. Vector<MODEL> Obs. Spaces<MODEL> spaces_[] Obs. Space<MODEL> obsdb_ data_ … IODA Obs. Vector IODA OOPS Obs. Space obsdb_ values_ database_ Obs Vector Data Obs Database 13 … JEDI Academy, February, 2020

Multiple Observation Spaces and Vectors • The total observation vector (e. g. , y) is chopped up into pieces according to observation type – • • • Radiosonde Radiance AOD UFO holds Obs. Operator objects that implement the various observation simulation algorithms OOPS manages these pieces with Observations, Obs. Spaces and Obs. Operators collector classes – IODA Different observation types require different algorithms to simulate those observations Corresponding Obs. Operator and Obs. Vector objects get paired up and OOPS chains these pieces together for the cost function minimization step JEDI Academy, February, 2020

IODA Data Flow y OOPS UFO IODA H(x) Observations Obs. Vector IODA Obs Data Obs. Vector x Observer Obs. Operators Obs. Operator Obs. Space 15 JEDI Academy, February, 2020

Interface for Observation Data Access The IODA API is used to access data: std: : vector<double> lats; obspace. get_db("Meta. Data", "latitude", lats); C++ std: : vector<float> viewing_angle; obspace. get_db("Meta. Data", "sensor_view_angle", viewing_angle); nlocs = obsspace_get_nlocs(obsdb) allocate(obs. Value(nlocs)) call obsspace_get_db(obsdb, "Obs. Value", "bending_angle", obs. Value) Fortran allocate(obselev(nlocs)) call obsspace_get_db(obsdb, "Meta. Data", "station_elevation", obselev) The API is independent of the file format or internal structure All applications (including plotting and diagnostics) should be based on this API IODA 16 JEDI Academy, February, 2020

YAML: Obs. Space Initialization • Jo IODA JEDI Academy, February, 2020 17

YAML: Obs. Space initialization from file • Obs. Data. Out is optional • Flexible channel selection (single, range) List of Obs. Spaces by obs type IODA JEDI Academy, February, 2020

YAML: Observation grouping, sorting Grouping is for keeping a set of observations ”atomic” Groups stay intact during distribution of observations across MPI tasks Example: A single radiosonde profile Sorting applies to each group Useful for filters such as those based on gradients Example: Remove radiosonde obs that exhibits too great a change from its neighbors IODA 19 JEDI Academy, February, 2020

YAML: Observation localization Used for LETKF applications Create a subset of obs based on proximity to a given location Uses “repeated block” syntax in YAML & -> block definition * -> instantiation of block IODA 20 JEDI Academy, February, 2020

YAML: Obs. Space algorithmic initialization Random values within specified ranges Copy directly from configuration Useful for testing Quickly throw together a small set of observations No IODA file required IODA 21 JEDI Academy, February, 2020

IODA Converters Obs category Language Input file type Obs types ODB Python Met office ODB 2 Aircraft, Radiosonde, AMSU-A GNSSRO Fortran BUFR GNSSRO GSI Python Ncdiag Conventional, Radiance (all types handled by GSI) Marine Python Netcdf, BUFR, Binary Argo. Clim 2, SST, Profile, Ship, Trak, ADT, In-situ, SSS NCEP Python, Fortran BUFR, prep. BUFR Aircraft, Radiosonde, AMSU-A IODA 22 JEDI Academy, February, 2020

IODA Observations Types (so far) IODA Conventional Observation Types Radiance Observation Types Aircraft AMSU-A SBUV-2 Radiosonde ABI VIIRS AOD Radar AHI Cryosat Sfc AIRS ICEC Sfc. Ship ATMS SEVERI VAD wind CRIS SNDR Wind profiler GNSSRO Satwind Marine insitu temp HIRS-4 Scatwind Marine profiler IASI SMAP Marine SST MHS Marine SST 23 JEDI Academy, February, 2020

Summary • IODA is transitioning from a prototype system into a production worthy system – • Actively growing Plans for next year are focused on enhancing performance Goal of having a single observation handling interface across the NWP chain is progressing well – IODA Currently handling ~30 obs types, with 10’s of millions of observations being assimilated At this point, IODA is handling the different aspects of the NWP chain with no known roadblocks JEDI Academy, February, 2020 24