NSF NCAR NASA GSFC DOE LANL NOAA NCEP

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF Release 1. 0 Demo GFDL FMS Suite NASA GSFC PSAS NCAR/LANL CCSM Nancy Collins, Jon Wolfe, NCAR Second ESMF Community Meeting, Princeton, NJ May 15, 2003 MITgcm NCEP Forecast NSIPP Seasonal Forecast

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Outline • ESMF Demo Structure • ESMF Demo Data and Control Flow • Examples of ESMF Usage in Demo Code • ESMF Demo Output May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF Demo Overview • One Component Calculates Compressible Flow around Obstacles, with energy equation (Flow. Solver. Mod). Initial Flow is left to right over heated Obstacles. • Second Component Calculates Injector properties (Injector. Mod), coupled to Flow. Solver. • Similar to river run-off into an Ocean model. May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Demo Component Structure All code in these files is user-written code, not part of the ESMF Framework. Demo Application Component Demo Gridded Component Injector Gridded Component May 15, 2003 2 -way Coupler Component Solver Gridded Component

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Component Control Flow The red arrows show execution ordering. Injector Inject Run Export State CPL Run Data Fields Import State May 15, 2003 Coupler Import State Data Fields CPL Run Export State Solver Run

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Component Data Flow The green arrows show Data movement, black arrow show Data references. Coupler Injector Solver Export State Inject Run Data Fields Import State May 15, 2003 CPL Run Import State Data Fields CPL Run Export State Solver Run

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Component Data Layouts Each Component has a different Layout. The coupler routes data between processing elements. Injector 6 x 2 Layout Export State Data Fields Import State May 15, 2003 Coupler 1 x 12 Layout Import State Data Fields Export State Solver 3 x 4 Layout

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Layouts in the Demo • The demo runs 4, 8, 12 and 16 -way. For the 12 way decomposition the Injector component decomposes the data in a 6 x 2 pattern, and the Solver uses 3 x 4. This is controlled by the type of Layout associated with each Component. May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH User code vs. ESMF code The code flow is: • Main is user-written • Creates Application Component • Creates Top level Gridded component(s) • Calls registration routine May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH User vs. ESMF code (cont. ) • ESMF Framework provides language independent layer between calls • Framework checks arguments, supplies defaults, verifies DE Layout, calls into user code • User component code runs, fills in States, and returns error code • Returns thru Framework back to calling code May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Application Component - Creation This example code creates the top level application component, which initializes the ESMF Framework as well. comp. App = ESMF_App. Comp. Create("Coupled Flow Application", rc=rc) Query application for default layout. call ESMF_App. Comp. Get(comp. App, layout=layout. App, rc=rc) Create the Gridded component, passing in the default layout. comp. Gridded = ESMF_Grid. Comp. Create("Coupled Flow Demo", & layout=layout. App, rc=rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Application Component - Grid Init In the Demo, the Grid is created with input extents and other Grid parameters: grid = ESMF_Grid. Create(i_max=i_max, j_max=j_max, & x_min=x_min, x_max=x_max, & y_min=y_min, y_max=y_max, & layout=layout. App, & horz_gridtype=ESMF_Grid. Type_XY, & horz_stagger=ESMF_Grid. Stagger_C, & horz_coord_system=ESMF_Coord. System_Cartesian, & halo_width=1, name="source grid", rc=rc) The Grid can then be attached to the Gridded Component with a Set call: call ESMF_Grid. Comp. Set(comp. Gridded, grid=grid, rc=rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Application Component - Clock Init call ESMF_Calendar. Init(gregorian. Calendar, ESMF_CAL_GREGORIAN, rc) call ESMF_Time. Interval. Init(time. Step, S=int(2, kind=ESMF_IKIND_I 8), rc=rc) call ESMF_Time. Init(start. Time, YR=int(2003, kind=ESMF_IKIND_I 8), & MM=s_month, DD=s_day, H=s_hour, M=s_min, & S=int(0, kind=ESMF_IKIND_I 8), & cal=gregorian. Calendar, rc=rc) call ESMF_Time. Init(stop. Time, YR=int(2003, kind=ESMF_IKIND_I 8), & MM=e_month, DD=e_day, H=e_hour, M=e_min, & S=int(0, kind=ESMF_IKIND_I 8), & cal=gregorian. Calendar, rc=rc) call ESMF_Clock. Init(clock, time. Step, start. Time, stop. Time, rc=rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Application Component - Init/Run/Finalize Part of a User-written Component’s integration subroutinization into three separate callable routines: call ESMF_Grid. Comp. Initialize(comp. Gridded, flowstate, clock, & rc=rc) call ESMF_Grid. Comp. Run(comp. Gridded, flowstate, clock, rc=rc) call ESMF_Grid. Comp. Finalize(comp. Gridded, flowstate, clock, rc=rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in Components - Registry This is an example from one of the Components registering its callback routines. call ESMF_Grid. Comp. Set. Entry. Point(comp, ESMF_SETINIT, Flow_Init 1, 1, rc) call ESMF_Grid. Comp. Set. Entry. Point(comp, ESMF_SETINIT, Flow_Init 2, 2, rc) call ESMF_Grid. Comp. Set. Entry. Point(comp, ESMF_SETRUN, Flow. Solve, 0, rc) call ESMF_Grid. Comp. Set. Entry. Point(comp, ESMF_SETFINAL, Flow_Final, 0, & rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Gridded Component - Layout Create The following code creates 2 sublayouts on the same set of PEs (processing elements) as the top level Component, but each of the sublayouts has a different connectivity: cname. IN = "Injector model" layout. IN = ESMF_DELayout. Create(delist, 2, (/ mid, by 2 /), (/ 0, 0 /), rc) INcomp = ESMF_Grid. Comp. Create(cname. IN, layout=layout. IN, rc=rc) cname. FS = "Flow Solver model" layout. FS = ESMF_DELayout. Create(delist, 2, (/ quart, by 4 /), (/ 0, 0 /), rc) FScomp = ESMF_Grid. Comp. Create(cname. FS, layout=layout. FS, rc=rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Gridded Component - State Create The following code creates Import and Export States for the Injector subcomponent. All information being passed to other subcomponents will be described by these States. INimp = ESMF_State. Create("Injection Input", ESMF_STATEIMPORT, & cname. IN) INexp = ESMF_State. Create("Injection Feedback", ESMF_STATEEXPORT, & cname. IN) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in Components Import/Export States For initialization, add all fields to the import state. Only the ones needed will be copied over to the export state for coupling. call ESMF_State. Add. Data(import_state, field_sie, rc) call ESMF_State. Add. Data(import_state, field_u, rc) call ESMF_State. Add. Data(import_state, field_v, rc) This code is adding names only to the export list, marked by default as "not needed". The coupler will mark the ones needed based on the requirements of the component(s) this is coupled to. call ESMF_State. Add. Data(export_state, "SIE", rc) call ESMF_State. Add. Data(export_state, "U", rc) call ESMF_State. Add. Data(export_state, "V", rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Coupler - Init The Coupler sets the “is needed” flag for each Field in the Export State which will be needed by some other Component: call ESMF_State. Get. Data(statelist, & "Coupler States Flow. Solver to Injector", flowstates, rc) call ESMF_State. Get. Data(flowstates, "Flow. Solver Feedback", fromflow, rc) call ESMF_State. Set. Needed(fromflow, "SIE", ESMF_STATEDATAISNEEDED, rc) call ESMF_State. Set. Needed(fromflow, "V", ESMF_STATEDATAISNEEDED, rc) call ESMF_State. Set. Needed(fromflow, "RHO", ESMF_STATEDATAISNEEDED, rc) call ESMF_State. Set. Needed(fromflow, "FLAG", ESMF_STATEDATAISNEEDED, rc) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in Components - Field Creation and Array Usage In this example, we create a Field from an Array. Spec, which designates the rank, type, and kind of the data. call ESMF_Array. Spec. Init(arrayspec, rank=2, type=ESMF_DATA_REAL, & kind=ESMF_KIND_R 4) field_sie = ESMF_Field. Create(grid, arrayspec, relloc=ESMF_CELL_CENTER, & name="SIE", rc=status) Once the Field has been created, we have to get a pointer to the Array inside it and then its data, called “sie. ” Inside the Component "sie" can be used like an array made by an F 90 allocation but will reference the data inside "field_sie. " call ESMF_Field. Get. Data(field_sie, array_temp, rc=status) call ESMF_Array. Get. Data(array_temp, sie, ESMF_DATA_REF, status) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in the Coupler - Route The following piece of code provides an example of calling Route between two Fields in the Coupler Component. The first two lines get the Fields from the States, each corresponding to a different subcomponent. One is an Export State and the other is an Import State. call ESMF_State. Get. Data(mysource, datanames(i), srcfield, rc=status) call ESMF_State. Get. Data(mydest, datanames(i), dstfield, rc=status) The Route routine uses information contained in the Fields and the Coupler Layout object to call the Communication routines to move the data. Because many Fields may share the same Grid association, the same routing information may be needed repeatedly. Route information is cached so the precomputed information can be retained. call ESMF_Field. Route(srcfield, dstfield, cpllayout, status) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH ESMF in Components - Halo The following piece of code provides an example of Haloing the data in a Field. Currently the Field. Halo routine assumes the entire Halo is updated completely; i. e. the user cannot specify halo width or side separately. Field. Halo uses the Route object to transfer data from the exclusive computational domain of one DE to the Halo region of another. call ESMF_Field. Halo(field_rhou, status) May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Running the Demo • Officially supported platforms: – May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Demo Input • Will be ESMF Config files • Is currently F 90 namelist input • Separate files for Top Level Application Component, Gridded Injector Component, and Flow Solver Component • See Users Manual, Section 10, for details on the specific variables May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Demo Output • Controlled by the configuration files • Every N timesteps 3 data files are produced: – SIE. nnn - Internal Energy – U. nnn - U velocity – V. nnn - V velocity • Array header information gives size of each dimension, data type. • Use your favorite Vis tool to make May 15, 2003 images

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Demo Output Visualized May 15, 2003

NSF NCAR / NASA GSFC / DOE LANL / NOAA NCEP GFDL / MIT / U MICH Other barrier shapes May 15, 2003