Coupling ROMS and WRF using MCT Coupling design

Coupling ROMS and WRF using MCT Coupling design and implementation

ROMS 2. 0 ¯ROMS 2. 0 – new version ªchanged from F 77 to F 90/F 95 ªexplicit interfaces subject to “strong typing” ªallocation is via dereferenced pointer structures ªparallel framework includes shared- and distributed- memory paradigms Modeling Environment for Atmospheric Discovery National Computational Science

WRF-Weather Research and Forecasting ¯ Designed for: ª 1 -10 km horizontal grid resolution ªadvanced data assimilation and model physics ªboth research and operations ªperformance and maintainability ¯ Developed at: ªNCAR ªNOAA – NCEP/FSL/GFDL ªEPA – Atmospheric Modeling Division ªUniversity Community Modeling Environment for Atmospheric Discovery National Computational Science

WRF – Software design Modeling Environment for Atmospheric Discovery National Computational Science

The Coupler ¯WRF and ROMS are coupled using the Model Coupling Toolkit (MCT) developed at Argonne National Labs ¯MCT handles the passing of variables between the ocean and atmosphere models, as well as regridding and time averaging Modeling Environment for Atmospheric Discovery National Computational Science

The Model Coupling Toolkit ¯ Cons ª ª ¯ Component model processing element (PE) pool sizes remain constant Components can exchange only real and integer data as groups of vectors Pros ª ª ª Any number of components (ocean, atmos, ice, wave spray, etc) Any decomposition Any number of processors-per component local re-gridding supported by sparse-matrix multiplication The MCT user can supply: ± ± Consistent numbering schemes for grid points Integer ID for each component An MPI communicator for each component Interpolation Matrix Elements Modeling Environment for Atmospheric Discovery National Computational Science

Parallel data passing Each component model in the MCT framework passes the coupler information about the decomposition of the data arrays in the form of a Global Segment Map This allows for parallel data transfer, and the ability to regrid from the atmosphere grid to the ocean on local processors. A sample decomposition, and resulting Global Segment Map is shown in the next slide. Modeling Environment for Atmospheric Discovery National Computational Science

atmosphere Numbering of gridpoints 17 13 9 5 1 18 14 10 6 2 19 20 15 16 11 12 7 8 3 4 2 Global. Seg. Map Examples Pe_loc start 0 1 8 1 9 8 2 17 4 1 0 ocean Processor Decomposition 17 213 9 05 1 18 14 10 6 2 19 20 3 15 16 11 12 7 81 3 4 Modeling Environment for Atmospheric Discovery length Pe_loc start 0 1 2 0 5 2 0 9 2 1 3 2 1 7 2 1 11 2 2 13 2 2 17 2 3 15 2 3 19 2 Total number of segments = 3 length Total number of segments = 10 National Computational Science

Coupling through I/O API ¯ WRF I/O API abstracted to allow changing packages (net. CDF, HDF 5, etc) ¯ passing 2 -D fields at the air-sea interface equivalent to file I/O ¯ standard interface allows ªeasy interchange of components ªeasy interchange of variables passed ¯ ROMS changes (all CPP “switches”) ªnew module “mod_io_couple. F” ªocean. F subroutinized – takes communicator as argument, passed to distribute. F ªcommunicator pass to distribute. F, MPI initialization and finalization done by coupler ªinput file now handles tiling Modeling Environment for Atmospheric Discovery National Computational Science

Variables Passed Atmospheric Model (WRF) SST t QLW QSENS QSW QLATENT Oceanic Model SST - Sea Surface Temp t - Wind Stress E-P Q - heat (short wave and long wave) E - Evaporation P - Precipitation (ROMS) Modeling Environment for Atmospheric Discovery National Computational Science

Alpha coupling --The first runs The initial coupling took a sample test problem from WRF that integrates the evolution of a supercell. ROMS and WRF were run on the same grid so no sparse-matrix multiplication was needed, and the fast dynamics of the 3 hour simulation meant that ROMS and WRF could be run at similar timestep sizes to eliminate the need for time averaging. ROMS was given an initial stratification typical of summertime off the Florida coast, where supercells have been observed, and was forced only by the WRF-generated surface winds Modeling Environment for Atmospheric Discovery National Computational Science

Initial Model Results Updraft cells (yellow/white) drove strong convergent surface winds (white arrows) resulting in strong (2 -3 m/s) surface currents. Surface height (color slice) variations of a half-meter over 20 km were observed, and waves propagate along thermocline (shown in blue). Modeling Environment for Atmospheric Discovery National Computational Science

Initial Model Results, cont’d Modeling Environment for Atmospheric Discovery National Computational Science

Future Work ¯Currently only SST and Winds are passed between models - other variables need to be passed and implemented into model forcing ¯Allow models to run on separate grids (regridding with MCT) ¯Time averaging of fields using MCT accumulator needs to be implemented ¯Coupling across the Tera. Grid (next slide) Modeling Environment for Atmospheric Discovery National Computational Science

Coupled Modeling Across the Tera. Grid NCSA IA-32 Cluster (Myrinet) ROM S PSC Alpha Cluster (Quadrics) Wind Stress WRF SST In a collaborative effort between the NOAA PMEL and FSL laboratories, NCAR and Argonne, a version of the coupled WRF/ROMS model is being developed in which the ROMS component runs on one Tera. Grid machine and the WRF component runs on a second Tera. Grid machine. Intra-component communication occurs over, for example, Myrinet or Quadrics while inter-component communication (exchange of boundary conditions) will occur over the Tera. Grid fiber-optic backbone. MPICH-G 2, a globus-enabled version of MPI will provide the communication library used to implement all communication. The coupled model is divided into components using the Lawrence Berkeley Laboratory Multi-Program Component Handshaking (MPH) software. Parallel regridding of boundary conditions exchanged between the two models will be implemented using the Argonne Model Coupling Toolkit (MCT). Modeling Environment for Atmospheric Discovery National Computational Science

Questions? Modeling Environment for Atmospheric Discovery National Computational Science

Abstract WRF/ROMS coupling design and implementation using the Model Coupling Toolkit Rob Jacob, Dale Haidvogel, Al Hermann, John Michalakes, Christopher Moore Dan Schaffer WRF and ROMS are coupled using the Model Coupling Toolkit (MCT) developed at Argonne National Laboratory. MCT is a Fortran 90 library built on top of MPI with data types and methods that simplify the construction of distributedmemory parallel couplers. The coupler design itself is parallel, avoiding bottlenecks by allowing for parallel exchange of fields between models on different grids, and time averaging over the coupling period. The coupling is wired into the mediation layer in WRF, and into the I/O layer in ROMS. The initial coupling passes wind-stress from WRF to ROMS and sea surface temperature from ROMS to WRF. Longwave and shortwave heat and evaporation/precipitation will soon be added as variables passed. A simulation is run with WRF creating a supercell, and ROMS integrating the oceanic response. WRF creates the typical “hook-return” convective updraft usually seen in storms that generate supercells, as well as high precipitation and updraft-splitting. The ROMS response shows upwelling/downwelling patterns centered on the supercell updraft location, and oceanic circulation that mimics that of measured oceanic response to offshore supercell storms. Future work includes developing an API that will allow coupling and I/O using MCT and HDF 5, and utilizing this API in both WRF and ROMS, as well as coupling across the Tera. Grid using MPICH-G 2. Modeling Environment for Atmospheric Discovery National Computational Science

MCT Distribution Architecture { Model Coupling Toolkit Modeling Environment for Atmospheric Discovery High-level MCT classes Low-level MCT classes Message-Passing Environment Utilities (MPEU) National Computational Science