Some Coding Structure in WRF Software Architecture Features

Some Coding Structure in WRF

Software Architecture Ø Features F 90 w/ structures and dynamic memory allocation Modules Run-time configurable Hierarchical Software Design Multi-level parallel decomposition shared-, distributed-, hybrid

Multi-level parallel decomposition l Single version of code for efficient execution on: l l l Logic al domai n 1 Patch, divided into multiple tiles Distributed-memory Shared-memory Hybrid-memory Model domains are decomposed for parallelism on two-levels Patch: section of model domain allocated to a distributed memory node Tile: section of a patch allocated to a shared-memory processor within a node; this is also the scope of a model layer subroutine. Distributed memory parallelism is over patches; shared memory parallelism is over tiles within patches

Three Sets of Dimensions Domain size: ids, ide, jds, jde, kds, kde Memory size: ims, ime, jms, jme, kms, kme Tile size: its, ite, jts, jte, kts, kte

template for model layer subroutine SUBROUTINE model ( & arg 1, arg 2, arg 3, … ids, ide, jds, jde, ims, ime, jms, jme, its, ite, jts, jte, , argn, kds, kde, kms, kme, kts, kte & & & ) ! Domain dims ! Memory dims ! Tile dims IMPLICIT NONE ! Define Arguments (S and I 1) data REAL, DIMENSION (ims: ime, kms: kme, jms: jme) : : arg 1, . . . REAL, DIMENSION (ims: ime, jms: jme) : : arg 7, . . . ! Define Local Data (I 2) REAL, DIMENSION (its: ite, kts: kte, jts: jte) : : loc 1, . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its, ids), MIN(ite, ide) loc(i, k, j) = arg 1(i, k, j) + … END DO • Domain dimensions • Size of logical domain • Used for bdy tests, etc.

template for model layer subroutine SUBROUTINE model ( & arg 1, arg 2, arg 3, … ids, ide, jds, jde, ims, ime, jms, jme, its, ite, jts, jte, , argn, kds, kde, kms, kme, kts, kte & & & ) • Domain dimensions • Size of logical domain • Used for bdy tests, etc. ! Domain dims ! Memory dims ! Tile dims IMPLICIT NONE ! Define Arguments (S and I 1) data REAL, DIMENSION (ims: ime, kms: kme, jms: jme) : : arg 1, . . . REAL, DIMENSION (ims: ime, jms: jme) : : arg 7, . . . ! Define Local Data (I 2) REAL, DIMENSION (its: ite, kts: kte, jts: jte) : : loc 1, . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its, ids), MIN(ite, ide) loc(i, k, j) = arg 1(i, k, j) + … END DO logical patch

Distributed Memory Communications Example code fragment that requires communication between patches Note the tell-tale +1 and – 1 expressions in indices for rr and H 1 arrays on right-hand side of assignment. These are horizontal data dependencies because the indexed operands may lie in the patch of a neighboring processor. That neighbor’s updates to that element of the array won’t be seen on this processor. We have to communicate. (dyn_eh/module_diffusion. F ) SUBROUTINE horizontal_diffusion_s (tendency, rr, var, . . . DO j = jts, jte DO k = kts, ktf DO i = its, ite mrdx=msft(i, j)*rdx mrdy=msft(i, j)*rdy tendency(i, k, j)=tendency(i, k, j)(mrdx*0. 5*((rr(i+1, k, j)+rr(i, k, j))*H 1(i+1, k, j)(rr(i-1, k, j)+rr(i, k, j))*H 1(i , k, j))+ mrdy*0. 5*((rr(i, k, j+1)+rr(i, k, j))*H 2(i, k, j+1)(rr(i, k, j-1)+rr(i, k, j))*H 2(i, k, j ))msft(i, j)*(H 1 avg(i, k+1, j)-H 1 avg(i, k, j)+ H 2 avg(i, k+1, j)-H 2 avg(i, k, j) )/dzetaw(k) ) ENDDO. . . & & & &

template for model layer subroutine SUBROUTINE model ( & arg 1, arg 2, arg 3, … ids, ide, jds, jde, ims, ime, jms, jme, its, ite, jts, jte, , argn, kds, kde, kms, kme, kts, kte & & & ) • Domain dimensions • Size of logical domain • Used for bdy tests, etc. ! Domain dims ! Memory dims ! Tile dims IMPLICIT NONE ! Define Arguments (S and I 1) data REAL, DIMENSION (ims: ime, kms: kme, jms: jme) : : arg 1, . . . REAL, DIMENSION (ims: ime, jms: jme) : : arg 7, . . . ! Define Local Data (I 2) REAL, DIMENSION (its: ite, kts: kte, jts: jte) : : loc 1, . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its, ids), MIN(ite, ide) loc(i, k, j) = arg 1(i, k, j) + … END DO logical patch

• template for model layer subroutine SUBROUTINE model ( & arg 1, arg 2, arg 3, … ids, ide, jds, jde, ims, ime, jms, jme, its, ite, jts, jte, , argn, kds, kde, kms, kme, kts, kte & & & ) • ! Domain dims ! Memory dims ! Tile dims IMPLICIT NONE ! Define Arguments (S and I 1) data REAL, DIMENSION (ims: ime, kms: kme, jms: jme) : : arg 1, . . . REAL, DIMENSION (ims: ime, jms: jme) : : arg 7, . . . ! Define Local Data (I 2) REAL, DIMENSION (its: ite, kts: kte, jts: jte) : : loc 1, . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its, ids), MIN(ite, ide) loc(i, k, j) = arg 1(i, k, j) + … END DO Domain dimensions • Size of logical domain • Used for bdy tests, etc. Memory dimensions • Used to dimension dummy arguments • Do not use for local arrays jme 1 node logical patch halo ims ime jms

• template for model layer subroutine SUBROUTINE model ( & arg 1, arg 2, arg 3, … ids, ide, jds, jde, ims, ime, jms, jme, its, ite, jts, jte, , argn, kds, kde, kms, kme, kts, kte & & & ) • ! Domain dims ! Memory dims ! Tile dims IMPLICIT NONE ! Define Arguments (S and I 1) data REAL, DIMENSION (ims: ime, kms: kme, jms: jme) : : arg 1, . . . REAL, DIMENSION (ims: ime, jms: jme) : : arg 7, . . . ! Define Local Data (I 2) REAL, DIMENSION (its: ite, kts: kte, jts: jte) : : loc 1, . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its, ids), MIN(ite, ide) loc(i, k, j) = arg 1(i, k, j) + … END DO • Domain dimensions • Size of logical domain • Used for bdy tests, etc. Memory dimensions • Used to dimension dummy arguments • Do not use for local arrays Tile dimensions • Local loop ranges • Local array dimensions jme jte tile its jts ite halo ims ime jms

Data structure l WRF Data Taxonomy l State data l Intermediate data type 1 (L 1) l Intermediate data type 2 (L 2)

Data structure State data l l l Persist for the duration of a domain Represented as fields in domain data structure Arrays are represented as dynamically allocated pointer arrays in the domain data structure Declared in Registry using state keyword Always memory dimensioned; always thread shared Only state arrays can be subject to I/O and Interprocessor communication

• template for model layer subroutine SUBROUTINE model ( & arg 1, arg 2, arg 3, … ids, ide, jds, jde, ims, ime, jms, jme, its, ite, jts, jte, , argn, kds, kde, kms, kme, kts, kte & & & ) • ! Domain dims ! Memory dims ! Tile dims IMPLICIT NONE ! Define Arguments (S and I 1) data REAL, DIMENSION (ims: ime, kms: kme, jms: jme) : : arg 1, . . . REAL, DIMENSION (ims: ime, jms: jme) : : arg 7, . . . ! Define Local Data (I 2) REAL, DIMENSION (its: ite, kts: kte, jts: jte) : : loc 1, . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its, ids), MIN(ite, ide) loc(i, k, j) = arg 1(i, k, j) + … END DO Domain dimensions • Size of logical domain • Used for bdy tests, etc. Memory dimensions • Used to dimension dummy arguments • Do not use for local arrays jme 1 node logical patch halo ims ime jms

Data structure L 1 Data l Data that persists for the duration of 1 time step on a domain and then released l Declared in Registry using i 1 keyword l Typically automatic storage (program stack) in solve routine l Typical usage is for tendency or temporary arrays in solver l Always memory dimensioned and thread shared l Typically not communicated or I/O

Data structure L 2 Data l L 2 data are local arrays that exist only in model-layer subroutines and exist only for the duration of the call to the subroutine l L 2 data is not declared in Registry, never communicated and never input or output l L 2 data is tile dimensioned and thread local; overdimensioning within the routine for redundant computation is allowed l the responsibility of the model layer programmer l should always be limited to thread-local data

• template for model layer subroutine SUBROUTINE model ( & arg 1, arg 2, arg 3, … ids, ide, jds, jde, ims, ime, jms, jme, its, ite, jts, jte, , argn, kds, kde, kms, kme, kts, kte & & & ) • ! Domain dims ! Memory dims ! Tile dims IMPLICIT NONE ! Define Arguments (S and I 1) data REAL, DIMENSION (ims: ime, kms: kme, jms: jme) : : arg 1, . . . REAL, DIMENSION (ims: ime, jms: jme) : : arg 7, . . . ! Define Local Data (I 2) REAL, DIMENSION (its: ite, kts: kte, jts: jte) : : loc 1, . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its, ids), MIN(ite, ide) loc(i, k, j) = arg 1(i, k, j) + … END DO • Domain dimensions • Size of logical domain • Used for bdy tests, etc. Memory dimensions • Used to dimension dummy arguments • Do not use for local arrays Tile dimensions • Local loop ranges • Local array dimensions jme jte tile its jts ite halo ims ime jms

The Registry l "Active data-dictionary” for managing WRF data structures l Database describing attributes of model state, intermediate, and configuration data l l l Program for auto-generating sections of WRF from database: l l l l l Dimensionality, number of time levels, staggering Association with physics I/O classification (history, initial, restart, boundary) Communication points and patterns Configuration lists (e. g. namelists) 570 Registry entries 30 -thousand lines of automatically generated WRF code Allocation statements for state data, I 1 data Argument lists for driver layer/mediation layer interfaces Interprocessor communications: Halo and periodic boundary updates, transposes Code for defining and managing run-time configuration information Code forcing, feedback and interpolation of nest data Automates time consuming, repetitive, error-prone programming Insulates programmers and code from package dependencies Allow rapid development Documents the data

Registry data base l l Currently implemented as a text file: Registry/Registry Types of entry: l State – Describes state variables and arrays in the domain structure l Dimspec – Describes dimensions that are used to define arrays in the model l L 1 – Describes local variables and arrays in solve l Typedef – Describes derived types that are subtypes of the domain structure l Rconfig – Describes a configuration (e. g. namelist) variable or array l Package – Describes attributes of a package (e. g. physics) l Halo – Describes halo update interprocessor communications l Period – Describes communications for periodic boundary updates l Xpose – Describes communications for parallel matrix transposes

State/L 1 Entry (Registry) l Elements l l Entry: The keyword “state” Type: The type of the state variable or array (real, double, integer, logical, character, or derived) l l Sym: The symbolic name of the variable or array Dims: A string denoting the dimensionality of the array or a hyphen (-) Use: A string denoting association with a solver or 4 D scalar array, or a hyphen Num. TLev: An integer indicating the number of time levels (for arrays) or hypen (for variables) l l l Stagger: String indicating staggered dimensions of variable (X, Y, Z, or hyphen) IO: String indicating whether and how the variable is subject to I/O and Nesting DName: Metadata name for the variable Descrip: Metadata description of the variable Example # Type Sym Dims Use Tlev Stag IO Dname Descrip # definition of a 3 D, two-time level, staggered state array state i 1 real ru real ww 1 ikj dyn_em 2 1 X Z irh "RHO_U" "X WIND COMPONENT“

State Entry– different output times l Example In Registry state real ru ikj dyn_em 2 X irh 01 "RHO_U" In namelist. input auxhist 1_outname auxhist 1_interval frames_per_auxhist 1_begin_y auxhist 1_begin_mo auxhist 1_begin_d auxhist 1_begin_h auxhist 1_begin_m auxhist 1_begin_s io_form_auxhist 1 = 'pm_output_d<domain>_<date>' = 10000, 5 = 30, 24 =0 =0 =1 =0 =0 =0 = 2, This will give you a five minute output interval on domain 3 starting after 1 day simulation. "XX“

Dimspec entry l Elements l l l l Entry: The keyword “dimspec” Dim. Name: The name of the dimension (single character) Order: The order of the dimension in the WRF framework (1, 2, 3, or ‘-‘) How. Defined: specification of how the range of the dimension is defined Coord. Axis: which axis the dimension corresponds to, if any (X, Y, Z, or C) Dat. Name: metadata name of dimension Example #<Table> dimspec <Dim> i j k l <Order> <How defined> <Coord-axis> 1 standard_domain x 3 standard_domain y 2 standard_domain z 2 namelist=num_soil_layers z <Dat. Name> west_east south_north bottom_top soil_layers

Package Entry (Registry) l Elements l l l Entry: the keyword “package”, Package name: the name of the package: e. g. “kesslerscheme” Associated rconfig choice: the name of a rconfig variable and the value of that variable that choses this package Package state vars: unused at present; specify hyphen (-) Associated 4 D scalars: the names of 4 D scalar arrays and the fields within those arrays this package uses Example # specification of microphysics options package passiveqv mp_physics==0 package kesslerscheme mp_physics==1 package linscheme mp_physics==2 package ncepcloud 3 mp_physics==3 package ncepcloud 5 mp_physics==4 - moist: qv, qc, qr, qi, qs, qg moist: qv, qc, qr, qi, qs # namelist entry that controls microphysics option rconfig integer mp_physics namelist, namelist_04 max_domains 0

Comm entries: halo and period l Elements l l Entry: keywords “halo” or “period” Commname: name of comm operation Description: defines the halo or period operation l For halo: npts: f 1, f 2, . . . [; npts: f 1, f 2, . . . ]* l For period: width: f 1, f 2, . . . [; width: f 1, f 2, . . . ]* Example # first exchange in eh solver halo HALO_EH_A dyn_em 24: u_2, v_2, ru_1, ru_2, rv_1, rv_2, w_2, t_2; 4: pp, pip # a periodic boundary update period PERIOD_EH_A dyn_em 2: u_1, u_2, ru_1, ru_2, v_1, v_2, rv_1, rv_2, rw_1, rw_2

4 D Tracer Arrays l l State arrays, used to store arrays of 3 D fields such as moisture tracers, chemical species, ensemble members, etc. First 3 indices are over grid dimensions; last dimension is the tracer index Each tracer is declared in the Registry as a separate state array but with f and optionally also t modifiers to the dimension field of the entry The field is then added to the 4 D array whose name is given by the use field of the Registry entry

Package Entry (Registry) state real qv ikjft moist 2 - i 01 rhusdf=(bdy_interp: dt, rqv_bt) "QVAPOR" state real qc ikjft moist 2 - i 01 rhusdf=(bdy_interp: dt, rqc_bt) "QCLOUD" state real qr ikjft moist 2 - i 01 rhusdf=(bdy_interp: dt, rqr_bt) "QRAIN" state real qi ikjft moist 2 - i 01 rhusdf=(bdy_interp: dt, rqi_bt) "QICE" state real qs ikjft moist 2 - i 01 rhusdf=(bdy_interp: dt, rqs_bt) "QSNOW" 1" state real qg ikjft moist 2 - i 01 rhusdf=(bdy_interp: dt, rqg_bt) "QGRAUP" "Water vapor mixing ratio" "kg kg-1" "Cloud water mixing ratio" "kg kg-1" "Rain water mixing ratio" "Ice mixing ratio" "Snow mixing ratio" "Graupel mixing ratio" "kg kg-1"

4 D Tracer Arrays l The extent of the last dimension of a tracer array is from PARAM_FIRST_SCALAR to num_tracername l Both defined in Registry-generated frame/module_state_description. F l PARAM_FIRST_SCALAR is a defined constant (2) l Num_tracername is computed at run-time in set_scalar_indices_from_config (module_configure) l Calculation is based on which of the tracer arrays are associated with which specific packages in the Registry and on which of those packages is active at run time (namelist. input)

4 D Tracer Arrays l l l Each tracer index (e. g. P_QV) into the 4 D array is also defined in module_state_description and set in set_scalar_indices_from_config Code should always test that a tracer index greater than or equal to PARAM_FIRST_SCALAR before referencing the tracer (inactive tracers have an index of 1) Loops over tracer indices should always run from PARAM_FIRST_SCALAR to num_tracername -EXAMPLE

4 D Tracer Array Example • 4 D moisture field, moist_1(i, k, j, ? ) ? = P_QV (water vapor) P_QC (cloud water) P_QI (cloud ice) P_QR (rain) P_QS (snow) P_QG (graupel) IF (qi_flag) then (the memory of cloud ice is allocated). . .

Directory Structure Registry

WRF Mass-Coordinate Model Integration Procedure WRFV 3/dyn_em/solve_em. F Begin time step Runge-Kutta loop (steps 1, 2, and 3) (i) advection, p-grad, buoyancy using (ii) if step 1 (first_rh_part 1/part 2) physics, save for steps 2 and 3 (iii) assemble dynamics tendencies Acoustic step loop (i) advance U, V, then w, (ii) time-average U, V, End acoustic loop Advance scalars using time-averaged U, V, End Runge-Kutta loop Other physics (currently microphysics) End time step

phy_prep phy_init radiation_driver surface_driver INIT … WRF … pbl_driver solve_em part 1 cumulus_driver DYNAMICS . moist_physics_prep microphysics_driver

Physics Calculate decoupled variable tendencies • Cumulus parameterization • Boundary layer parameterization • Radiation parameterization Update decoupled variables directly • Microphysics

Physics three-level structure solve_em Physics_driver SELECT CASE (CHOICE) CASE ( NOPHY ) CASE ( SCHEME 1 ) CALL XXX CASE ( SCHEME 2 ) CALL YYY . CASE DEFAULT END SELECT Individual physics scheme ( XXX )

Rules for WRF physics · Naming rules module_yy_xxx. F (module) yy = ra is for radiation bl is for PBL sf is for surface and surface layer cu is for cumulus mp is for microphysics. xxx = individual scheme ex, module_cu_grell. F

Rules for WRF physics · Naming rules RXXYYTEN (tendencies) XX = variable (th, u, v, qc, … ) YY = ra bl cu ex, RTHBLTEN is for radiation is for PBL is for cumulus

Rules for WRF physics · Naming rules · One scheme one module · Coding rules (later)

WRF Physics Features • Unified global constatnts (module_model_constants. F) REAL , PARAMETER : : r_d REAL , PARAMETER : : r_v REAL , PARAMETER : : cp REAL , PARAMETER : : cv. . = 287. = 461. 6 = 7. *r_d/2. = cp-r_d

WRF Physics Features • Unified global constatnts (module_model_constants. F) • Unified common calculations (saturation mixing ratio) • Vertical index (kms is at the bottom)

Implement a new physics scheme · Prepare your code · Create a new module · Declare new variables and a new package in Registry · Modify namelist · Do initialization · Modify solve_em. F · Modify phy_prep

Implement a new physics scheme · Modify cumulus_driver. F (use cumulus parameterization as an example) · Modify calculate_phy_ten · Modify phy_cu_ten (module_physics_addtendc. F) · Modify Makefile · Compile and test

phy_prep phy_init radiation_driver surface_driver INIT … WRF … pbl_driver solve_em part 1 cumulus_driver DYNAMICS . moist_physics_prep microphysics_driver

Prepare your code 1. F 90 a) Replace continuation characters in the 6 th column with f 90 continuation `&‘ at end of previous line F 77 Subroutine kessler(QV, T, + its, ite, jts, jte, kts, kte, + ims, ime, jms, jme, kms, kme, + ids, ide, jds, jde, kds, kde) F 90 Subroutine kessler(QV, T, . . . & its, ite, jts, jte, kts, kte, & ims, ime, jms, jme, kms, kme, & ids, ide, jds, jde, kds, kde )

Prepare your code 1. F 90 a) Replace continuation characters in the 6 th column with f 90 continuation `&‘ at end of previous line b) Replace the 1 st column `C` for comment with `!` F 77 c This is a test F 90 ! This is a test

Prepare your code 1. F 90 2. No common block common/var 1/T, q, p, … WRF Subroutine sub(T, q, p, …. ) real, intent(out), & dimension(ims: ime, kms: kme, jms: jme): : T, q, p

Prepare your code 1. F 90 2. No common block 3. Use “ implicit none ” 4. Use “ intent ” Subroutine sub(T, q, p, …. ) implicit none real, intent(out), & dimension(ims: ime, kms: kme, jms: jme): : T real, intent( in), & dimension(ims: ime, kms: kme, jms: jme): : q real, intent(inout), & dimension(ims: ime, kms: kme, jms: jme): : p

Prepare your code 1. F 90 2. No common block 3. Use “ implicit none ” 4. Use “ intent ” 5. Variable dimensions Subroutine sub(global, …. ) implicit none real, intent(out), & dimension(ims: ime, kms: kme, jms: jme): : global real, dimension(its: ite, kts: kte, jts: jte): : local

Prepare your code 1. F 90 2. No common block 3. Use “ implicit none ” 4. Use “ intent ” 5. Variable dimensions 6. Do loops do j = jts, jte do k = kts, kte do i = its, ite. . . enddo

Implement a new physics scheme · Create a new module ex, module_cu_exp. F (plug in all your codes) · Go Registry and declare a new package (and new variables) (WRFV 1/Registry) package kfscheme cu_physics==1 - - package bmjscheme cu_physics==2 - - package expscheme cu_physics==3 - -

Implement a new physics scheme · Create a new module ex, module_cu_exp. F (plug in all your codes) · Go Registry and declare a new package (and new variables) (WRFV 1/Registry) Cloud microphysics package kesslerscheme mp_physics==1 - moist: qv, qc, qr package linscheme mp_physics==2 - moist: qv, qc, qr, qi, qs, qg package wsm 3 mp_physics==3 - moist: qv, qc, qr package wsm 5 mp_physics==4 - moist: qv, qc, qr, qi, qs

Implement a new physics scheme · Create a new module ex, module_cu_exp. F (plug in all your codes) · Go Registry and declare a new package (and new variables) (WRFV 1/Registry) · Modify namelist. input and assign cu_physics = 3

(dyn_em) (start_em. F) INIT * start_domain_em (dyn_em) WRF ……. * solve_em (phys) (module_physics_init. F) phy_init cu_init

phys/module_physics_init. F · Pass new variables down to cu_init (dyn_em) (start_em. F) INIT * start_domain_em (dyn_em) WRF ……. * solve_em (phys) (module_physics_init. F) phy_init cu_init

phys/module_physics_init. F · Pass new variables down to cu_init · Go subroutine cu_init Include the new module and create a new SELECT case

phys/module_physics_init. F Subroutine cu_init(…). USE module_cu_kf USE module_cu_bmj USE module_cu_exp. cps_select: SELECT CASE(config_flags%cu_physics) CASE (KFSCHEME) CALL kfinit(. . . ) CASE (BMJSCHEME) CALL bmjinit(. . . ) Match the package CASE (EXPSCHEME) name in Registry CALL expinit(. . . ) CASE DEFAULT END SELECT cps_select Put into module_cu_exp. F

phy_prep phy_init INIT … WRF … solve_em part 1 DYNAMICS . moist_physics_prep microphysics_driver

phy_prep/moist_physics_prep • Calculate required variables • Convert variables from C grid to A grid

phy_prep phy_init radiation_driver INIT … surface_driver pbl_driver WRF … solve_em part 1 cumulus_driver DYNAMICS . moist_physics_prep microphysics_driver Expcps

Three-level structure solve_em Physics_driver SELECT CASE (CHOICE) CASE ( NOPHY ) CASE ( SCHEME 1 ) CALL XXX CASE ( SCHEME 2 ) CALL YYY . CASE DEFAULT END SELECT Individual physics scheme ( XXX )

cumulus_driver. F · Go physics driver (cumulus_driver. F) Include the new module and create a new SELECT CASE in driver Check available variables in drivers (variables are explained inside drivers)

Module_cumulus_driver. F MODULE module_cumulus_driver CONTAINS Subroutine cumulus_driver (…. ). . . !-- RQICUTEN ! !-- RAINC (mm) !-- RAINCV !-- NCA ! !-- u_phy !-- v_phy !-- th_phy !-- t_phy !-- w !-- moist (kg/kg) !-- dz 8 w !-- p 8 w Qi tendency due to cumulus scheme precipitation (kg/kg/s) accumulated total cumulus scheme precipitation (mm) counter of the cloud relaxation time in KF cumulus scheme (integer) u-velocity interpolated to theta points (m/s) v-velocity interpolated to theta points (m/s) potential temperature (K) vertical velocity (m/s) moisture array (4 D - last index is species) dz between full levels (m) pressure at full levels (Pa)

Module_cumulus_driver. F MODULE module_cumulus_driver CONTAINS Subroutine cumulus_driver. USE module_cu_kf USE module_bmj_kf USE module_cu_exp cps_select: SELECT CASE(config_flags%cu_physics) CASE (KFSCHEME) CALL KFCPS(. . . ) CASE (BMJSCHEME) CALL BMJCPS(. . . ) Match the package CASE (EXPSCHEME) name in Registry CALL EXPCPS(. . . ) CASE DEFAULT END SELECT cps_select Put in module_cu_exp. F

phy_prep phy_init radiation_driver surface_driver INIT … WRF … pbl_driver solve_em part 1 cumulus_driver DYNAMICS . moist_physics_prep microphysics_driver

phy_prep part 1 cumulus_driver part 2 calculate_phy_tend expcps solve_em update_phy_ten DYNAMICS . message passing ? phy_cu_ten

phys/module_physics_addtendc. F Subroutine phy_cu_ten (… ). CASE(BMJSCHEME). CASE (EXPSCHEME) CALL add_a 2 a (rt_tendf, RTHCUTEN, … ) CALL add_a 2 c_u(ru_tendf, RUBLTEN, … ) CALL add_a 2 c_v(rv_tendf, RVBLTEN, … ). if ( QI_FLAG ) & CALL add_a 2 a(moist_tendf(ims, kms, jms, P_QV), RQVCUTEN, . . & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & its, ite, jts, jte, kts, kte ).