ESMF Regridding Update Robert Oehmke Peggy Li Ryan

ESMF Regridding Update Robert Oehmke, Peggy Li, Ryan O’Kuinghttons, Mat Rothstein, Joseph Jacob NOAA Cooperative Institute for Research in Environmental Sciences University of Colorado, Boulder robert. oehmke@noaa. gov 3 rd Workshop on Coupling Technologies for Earth System Models April 21, 2015

Context • The Earth System Modeling Framework (ESMF) is software for building and coupling weather, climate, and related models. • This talk is about the regridding part of ESMF • The work presented here is finished and we expect it to come out in ESMF 7. 0. 0 due late spring/early summer • Also available before that as a development snapshot

ESMF Regridding (or remapping or interpolation) is the process of moving data from one grid to another while preserving qualities of the original data. ESMF regridding is: • Flexible: – Computes weights between a wide range of grids: structured and unstructured, global and regional, 2 D and 3 D, spherical and Cartesian – Options for interpolation method, pole treatment, masked points, … • Well Tested and Portable: – >200 regridding cases tested every night – >40 different OS/Compiler/MPI combinations tested every night • Parallel and Fast: – Able to compute weights in minutes which before took hours – Able to compute weights between very large grids • Community developed: – Supported by NASA, NOAA, DOD and NSF funding – Well established (since 2005) community processes for prioritization, support and review. – Development priorities set by users through quarterly Change Review Board (CRB) meetings

Interfaces • Complete F 95 API: – use ESMF – Derived types and methods – Investigating moving to Fortran 2003 • Limited C API: – #include “ESMC. h” – Structs and methods • Limited Python API: – Import ESMPy – Classes with methods • Applications: ! Create Grids src. Grid=ESMF_Grid. Create(…) dst. Grid=ESMF_Grid. Create(…) ! Create Fields src. Field=ESMF_Field. Create(src. Grid, …) dst. Field=ESMF_Field. Create(dst. Grid, …) ! Calc regrid sparse matrix (route. Handle) ESMF_Field. Regrid. Store(src. Field, dst. Field, route. Handle, …) do i=1, … ! Compute new src. Field …. ! Apply regrid sparse matrix (route. Handle) ESMF_Field. Regrid(src. Field, dst. Field, route. Handle) enddo – File-based interpolation weight generation: mpirun –np <N> ESMF_Regrid. Weight. Gen –s …. – (NEW!) File-based weight generation AND application of weights: mpirun –np <N> ESMF_Regrid –s…

Supported Geometry Types • Grid: – Structured representation of a region – A logically rectangular tile • Mesh: – Unstructured representation of a region – In 2 D: polygons with any number of sides • including concave – In 3 D: tetrahedrons & hexahedrons • Loc. Stream (Location Stream): – NEW! – Set of disconnected points • E. g. locations of observations – Very flexible and efficient – Can’t be used with every regrid method

Supported Grid File Formats • SCRIP: – Format used by SCRIP regridding tool – 2 D Spherical – Logically rectangular Grids or unstructured Meshes • ESMF unstructured: – Custom ESMF format – 2 D or 3 D / Spherical or Cartesian – Unstructured Meshes • UGRID: – Proposed CF convention – 2 D or 3 D / Spherical or Cartesian – Unstructured Meshes • GRIDSPEC: – CF convention – ESMF currently supports: 2 D spherical single tile Grids

Regrid Methods • Bilinear: – Destination is a linear combination of source cell corners – Weights based distance from corners – Loc. Streams can be destination • Higher order patch recovery: – – – Multiple polynomial patches represent region around source cell Destination is linear combination of patch values Yields better derivatives/smoother results than bilinear Based on “patch recovery” used in finite element modeling [1][2] Loc. Streams can be destination • Nearest neighbor: – Destination is equal to closest source point (or vise versa) – Loc. Streams can be source or destination • First order conservative: – Destination is combination of intersecting source cell areas – Preserves integral of data across interpolation

Other Options • (NEW!) Path between points in bilinear on a sphere: – Straight line – Great circle • Options for extrapolating across pole region: – – Full circle average N-point average Teeth No pole Full circle avg. N-point avg Teeth No Pole • Others: – Masking – (New!) Normalization options for conservative: destination area or fraction – User area – Ignore unmapped, Ignore degenerate

Spherical Regrid Support • Regrid works with spherical (lon, lat, radius) coordinates • All regrid methods supported between any pair of: – 2 D Global or 2 D regional logically rectangular Grids – 2 D Unstructured Meshes composed of polygons with any number of sides – 2 D Multi-patch grids (e. g. cubed spheres) currently supported via Meshes • (NEW! More Accurate!) Bilinear supported between any pair of: – 3 D Meshes composed of hexahedrons – 3 D Global or regional logically rectangular Grids • Loc. Streams supported for above depending on regrid method 3 D Global Spherical Grid FIM Unstructured Grid Regional Grid

Cartesian Regrid Support • Regrid works with Cartesian (x, y, z) coordinates • All regrid methods between any pair of: – 2 D Meshes composed of polygons with any number of sides – 2 D logically rectangular Grids • Bilinear, conservative, or nearest neighbor between any pair of: – 3 D Meshes composed of hexahedrons – 3 D logically rectangular Grids • Loc. Streams supported for above depending on regrid method 2 D Unstructured Mesh From www. ngdc. noaa. gov 3 D Grid 3 D Unstructured Mesh

Regrid Weight Calculation Performance First-Order Conservative Interpolation Weight Calculation (2 km unstructured land only grid to 1/8 degree global grid) 30 Time in Seconds 25 20 15 10 5 0 64 128 256 512 1024 Number of Processors 2048 Platform: IBM IData. Plex cluster (Yellowstone at NCAR) Grid size: ~30 million cells and ~4 millions cells 4096

Other Tools Using ESMF Regrid • Ultrascale Visualization Climate Data Analysis Tool (UV-CDAT): – Package designed for analyzing large climate data sets – Uses ESMF regridding via ESMPy – Recently won Federal Laboratory Consortium technology transfer award • Cf-python: – Python package for manipulating cf data and files – Uses ESMF regridding via ESMPy • NCAR Command Language (NCL): – Language for scientific data analysis and visualization – Uses ESMF regridding via ESMF_Regrid. Weight. Gen application

Selected Users • NOAA Environmental Modeling System (NEMS): – Under development – Aimed to be the backbone of next generation forecast systems – ESMF regridding used to transfer data between models • Community Earth System Model (CESM): – Has used ESMF regridding for 5+ years – Currently uses ESMF_Regrid. Weight. Gen, but are working towards online regridding • Met Office: – Multiple users of ESMF regridding there – Have been adding functionality for them (e. g. concave, great circle, …) • NOAA Space Weather Prediction Center (SWPC): – Working towards coupled 3 D ionosphere model using ESMF – New 3 D spherical bilinear reduces error by 40 x over Cartesian

Scheduled for Upcoming Releases • Higher-order conservative regridding (7. 1. 0) • Breaking up grid files to increase maximum grid size possible for interpolation weight generation (7. 1. 0) • Extrapolation of points that lie outside the source grid (7. 1. 0) • Dynamic masking during sparse matrix multiply (7. 1. 0) • Additional weight file formats for ESMF_Regrid. Weight. Gen (7. 1. 0) – Other SCRIP format – Slim version • Representation of higher-order elements (7. 2. 0)

References • Patch interpolation: 1. Khoei S. A. , Gharehbaghi A. R. The superconvergent patch recovery technique and data transfer operators in 3 d plasticity problems. Finite Elements in Analysis and Design, 43(8), 2007. 2. Hung K. C, Gu H. , Zong Z. A modified superconvergent patch recovery method and its application to large deformation problems. Finite Elements in Analysis and Design, 40(5 -6), 2004. If you have questions or requests, come talk to me, or email: esmf_support@list. woc. noaa. gov