Integrated Parallel Accurate Reservoir Simulator Introduction Contributor John

Integrated Parallel Accurate Reservoir Simulator Introduction Contributor: John Wheeler, Mary Wheeler, Krzysztof Banas, Steve Bryant, Srinivas Chippada, Rick Dean, Joe Eaton, Carter Edwards, Xiuli Gai, Sebastien Lacroix, Wonsuck Lee, Qin Lu, Bahareh Momken, Myeong. Hwan Noh, Manish Parashar, Victor Parr, Malgorzata Peszynska, Beatrice Riviere, Shuyu Sun, Yuri Vassilevski, Ivan Yotov

Framework Characteristics l l Support 3 D transient flow of multiphase with multicomponent Multiple physical models • air/water, oil/water, three-phase, black-oil model … Multiple accurate and efficient numerical methods • IMPES, sequential, fully implicit Discretization techniques • • • Cell-centered finite difference (black-oil, oil/water, air/water models) Discontinuous Galerkin (two phase IMPES model) Galerkin FEM (poroelastic model) Multipoint flux MFE (single phase implicit model) Mortar finite element

Framework Characteristics l A suit of linear solvers and preconditioners • l Wells control • • l LSOR, Multigrid, Domain decomposition/Newton Krylov Arbitrary number of wells with one or more completion Pressure and rate specified well control General purpose 2 D function utilities • • • Piecewise constant Piecewise linear Quadratic splines with optional poles Cubic splines with optional poles User defined functions

Framework Characteristics l Free-form keyword input l Restart capability l Dynamic grid refinement of the primary grid on each fault bloc l Languages • • • FORTRAN 77 is used whenever possible for portability concern Classical C is used for memory management and user interface C++ is used for the multiblock interface libraries (MACE).

Advanced Features l Portability • PC under DOS/Windows or Linux and PC clusters, RS 6 K and SP 2, SGI, Cray T 3 E … l Modularity l Parallel l Multiblock l Multiphysics l Multinumerics l Multiscale

IPARS Framework IPARS FRAMEWORK CS Interface I/O Visualization Processors Architecture Solvers Geomechanics Physics Grids, Algorithms Well Management Recovery Processes

IPARS Directories (IPARSv 2/) Directory Functionality drive/ Driver file: ipars. df input/ Initial, transient data input memman/ Memory management print/ Standard and restart output util/ Utility routines wells/ Well management Linear solver/ Linear solvers Visualization visual/ Visualization files Parallel parall/ Parallel communications MACE/ Multiblock interface libraries IPARS Framework Routines Multiblock

IPARS Directories (IPARSv 2/) IPARS physical model Routines Directory Physical Model air/, airv 2/ Fully implicit air/water model black/ Fully implicit black-oil model dgimpes/ Oil/water DG IMPES model dgmisci/ DG miscible displacement hydro/ Fully implicit hydrology model hydroe/ IMPES hydrology model mmodel/ Multi model poroe/ Poroelastic model single/ Single phase explicit model singlei/ Single phase fully implicit model trchem/ Transport chemistry model

IPARS Physical Models Directory Physical Model Make files make/modular/ Make files for machine, framework, physical model, parallel, multiblock, linear solver, and visualization Size files size/modular/ Size files for machine, framework, physical model, parallel, multiblock, linear solver and visualization Documentation doc/ Data data/ Framework and physical model documentation files Example input files

IPARS Driver Flow Platform specific initialization Scalar input Initialize utilities Setup geometry Memory allocation Driver IPARS Initial data input Well/rock data Grid element input Initialization Model specific iivdat aivdat … Time stepping - Read transient data - Set time step - Make one time step - Standard, well output - Visualization output Model specific istep astep … End Model specific iisdat aisdat … Model specific iarray aarray … Model specific iiadat aiadat … Newton iteration - Jacobian, residual - Convergence test - Solve linear system - Update solution

Physical Models Model Input file keyword Model keyword Air/water AIR_WATER_MODEL $AIR (11) Black-oil BLACK_OIL_IMPLICIT $BLACKI (2) DG IMPES DG_IMPES $DGIMPES (14) DG miscible DG_MISCI $DGMISCI (15) Hydrology implicit HYDROLOGY_IMPLICIT $HYDROI (5) Hydrology IMPES HYDROLOGY_IMPES $HYDROE (6) Multi model MULTI_MODEL $MMODEL (7) Poroelastic POROELASTIC $POROE (18) Single phase explicit SINGLE_PHASE_EXPLICIT $SINGLEE (9) Single phase implicit SINGLE_PHASE_IMPLICIT $SINGLEI (12) Transport chemistry TRANSPORT_CHEMSTRY $TRCHEM (17)

Model Make and Size Files Model Make file make/modular/ Size file size/modular/ Air/water Black-oil DG IMPES DG miscible Hydrology implicit Hydrology IMPES Multi model Poroelastic Single phase explicit Single phase implicit Transport chemistry air. mak, airv 2. mak air. siz, airv 2. siz blacki. mak blacki. siz dgimpes. mak dgimpes. siz dgmisci. mak dgmisci. siz hydroi. mak hydroi. siz hydroe. mak hydroe. siz mmodel. mak mmodel. siz poroe. mak poroe. siz single. mak single. siz singlei. mak singlei. siz trchem. mak trchem. siz

Example Input Files Model Example File: IPARSv 2/data/ Air/water Black-oil DG IMPES DG miscible Hydrology implicit Hydrology IMPES Multi model Poroelastic Single phase explicit Single phase implicit Transport chemistry Visualization air 1. dat, air*. dat, vauclin. dat blackh 1. dat, black*. dat IPARSv 2/data/dgimpes/ hydro 1. dat, hydro*. dat impes. dat, PCG. dat mmodel. dat, mm*. dat poroe 1. dat, poroe*. dat single 1. dat singli 1. dat tr_andra. dat, tr*. dat hydrovis. dat, hydro. PV 3. dat

IPARS Documentation Files: IPARS/doc/ Description frame. txt framein. txt, reader. txt, readpar. txt Boundary. txt MBflow. txt mem. txt mmodel. txt README size. txt Stone. doc table. txt General description of IPARS framework input/output General boundary conditions Multiblock capability Memory management Multi-model formulation How to build an ipars exectuable IPARS compiling preprocessor Three-phase relative permeability Input block data of table or functions

IPARS Documentation Files: IPARS/doc/ Description vis. txt, vis 1. txt, vis. PV 3. txt well. txt, welman. txt ygmres. txt IPARS/doc/models/ mem. txt mmodel. txt Visualization Well data input and management GMRES linear solver and preconditioners Physical model documentation Memory management Multi-model formulation

IPARS Make Files IPARSv 2/make/modular/ Files Framework Physical models Linear solvers Visualization Parallel Platform Multiblock frame. mak, frame_t. mak e. g. blacki. mak, hydroi. mak, poroe. mak … lsor. mak, pcg. mak, gmres. mak, ygmres. mak vis. mak, vis. PV 3. mak parall. mak, parall_c. mak, parall_m. mak e. g. lnx. mak, sun. mak … mace. mak

Compiler Preprocessor (size) Preprocessing: the preprocessor searches a set of sources files for a specific set of symbols, replace those symbols with other symbols or numbers, and write the result to a new set of files. l Predefine array dimensions l Turn on/off certain function calls l Include or exclude certain physical models l Include or exclude visualization capability l Include or exclude parallel communications l Include or exclude multiblock interface cal.

Compiler Preprocessor (size) l Example 1: predefine array demensions • • • l In a size control file, frame 1. siz Replace_Symbols MXBLKS 3 In a source file, mcontro. dh INTEGER blkmodel($MXBLKS) In the target file after preprocessed, mcontrl. h INTEGER blkmodel(3) Example 2: exclude physical models • In source file ipars. df $BLKCKI CALL ISTEP(KERR) $HYDROI CALL HSTEP(KERR) $TRCHEM CALL TRSTEP(KERR)

Compiler Preprocessor (size) • l In the target file after preprocessed, ipars. f CALL ISTEP(KERR) (if blacki. siz is included) C CALL HSTEP(KERR) C CALL TRSTEP(KERR) Example 3: exclude parallel communications • • In source file istep. df $MANY CALL UPDATE(N_PWAT, 1) In the target file after preprocessed, istep. f C CALL UPDATE(N_PWAT, 1) (if lnx. siz is included) CALL UPDATE(N_PWAT, 1) (if beo. siz is included)

IPARS Size Files IPARSv 2/make/modular/ Files Framework Physical models Linear solvers Visualization Parallel/platform Multiblock e. g. frame 1. siz, frame 36. siz … e. g. blacki. siz, hydroi. siz, poroe. siz … lsor. siz, pcg. siz, ticama. siz, yticama. siz visual. siz, vis. PV 3. siz e. g. beo. siz mortar. siz

Integrated Parallel Accurate Reservoir Simulator Build and Run Contributor: John Wheeler, Mary Wheeler, Krzysztof Banas, Steve Bryant, Srinivas Chippada, Rick Dean, Joe Eaton, Carter Edwards, Xiuli Gai, Sebastien Lacroix, Wonsuck Lee, Qin Lu, Bahareh Momken, Myeong. Hwan Noh, Manish Parashar, Victor Parr, Malgorzata Peszynska, Beatrice Riviere, Shuyu Sun, Yuri Vassilevski, Ivan Yotov

Checkout Source Code l l l Require permission to use IPARS • Contact Mary F. Wheeler or Hector Klie Require a user account at ICES • contact sysnet@ices. utexas. edu Check to be a member of “wheeler 3” group <water>/ices/gxl/ groups gxl : wheeler oden wheeler 3 csmprinters l Check on your machine that cvs is working and it is in your path (cvs-Concurrent Versions System: http: //www. nongnu. org/cvs/) <water>/ices/gxl/ which cvs /usr/bin/cvs

Checkout Source Code l Check that your machines “sees” CVS repository, /ws/csm/SOURCE/ <water>/ices/gxl/ ls /ws/csm/SOURCE Comm/ deflate/ IPARSv 2/ kelp 1. 2/ ktuf/ NDAGH/ pss/ config-defs/ gct/ IPARSv 3/ kparcel/ MB/ oldkparcel/ Utility/ CVSROOT/ IPARS/ kelp 1. 1/ kscript/ mparcel/ Parssim/ • l If something doesn't work, then stop here and contact sysnet@ices. utexas. edu Make a directory for IPARS source code <water>/ices/gxl/ mkdir IPARS <water>/ices/gxl/ cd IPARS l Check out a complete source code of IPARSv 2 <water>/ices/gxl/IPARS/ cvs -d /ws/csm/SOURCE checkout IPARSv 2 <water>/ices/gxl/IPARS/ ls IPARSv 2/

Checkout Source Code <water>/ices/gxl/IPARS/ cd IPARSv 2 <water>/ices/gxl/IPARS/ ls air/ airv 2/ blackn/ CVS/ data/ dgimpes/ dgmisci/ doc/ drive/ exec/ fixtypes* fixtypes. sed* MACE/ hydro/ make/ hydroe/ MBLib/ input/ mblk/ IPARSv 2/ memman/ lapack/ mmodel/ parall/ solve/ work/ poroe/ STL. tar. gz print/ trchem/ single/ util/ singlei/ visual/ size/ wells/

Build IPARS Executable l Read the following write-ups and proceed • • l IPARSv 2/doc/README IPARSv 2/make/modular/README Create an executable preprocessor (size) for IPARS from sizeux. mak <water>/ices/gxl/IPARSv 2/ cd size <water>/ices/gxl/IPARSv 2/size/ make -f sizeux. mak • • Preprocessor size is created and placed in IPARSv 2/work/ <water>/ices/gxl/IPARSv 2/size/ ls. . /work/ CVS/ size* The preprocessor size replaces in the source code special character strings (keywords) by their replacements, as specified in *. siz files.

Build IPARS Executable l Create a “Master” size file from ipars. siz <water>/ices/gxl/IPARSv 2/ cd work <water>/ices/gxl/IPARSv 2/work/ cp. . /size/modular/ipars. siz. / l Edit ipars. siz • • • Choose a framework size file (frame 1. siz or frame 36. siz) Choose a physical model size file Choose a linear solver size file Choose a platform size file Include or exclude visualization Include or exclude mutliblock

Build IPARS Executable l Create a “Master” make file from unix. mak <water>/ices/gxl/IPARSv 2/ cd work <water>/ices/gxl/IPARSv 2/work/ cp. . /make/modular/unix. mak Makefile l Edit Makefile • • • Correct the path to the preprocessor size Correct the path to preprocessor “Master” sizefile (ipars. siz) Correct the path to the “include makefiles” (default is IPARSv 2/make/modular ) Choose a physical model Choose a linear solver Choose a platform

Build IPARS Executable • • • l Include or exclude visualizatoin Include or exclude parallell communications Include or exclude multiblock MACE library Make an executable <water>/ices/gxl/IPARSv 2/work/ ls CVS/ ipars. siz Makefile size* <water>/ices/gxl/IPARSv 2/work/ make l Check if ipars is created <water>/ices/gxl/IPARSv 2/work/ ls ipars* (Note: If ipars does not exist, stop and ask for help)

Create Data Input Files l Make a directory to run ipars <water>/ices/gxl/IPARSv 2/work/ mkdir Run <water>/ices/gxl/IPARSv 2/work/ cd Run <water>/ices/gxl/IPARSv 2/work/Run/ ln -s. . /ipars. / Note: ipars can be run in the same place as it was built l Input file should be compatible with the executable as far as model/solver/vis etc. are concerned. l Dimensions should be compatible with the MAX bounds for arguments as defined in sizefiles.

Create Data Input Files l When creating an input file it is better to modify an existing file than start from scratch. • • l Do not use TAB symbols or unusual things may happen ! Use $ symbol to comment things out in the input file. Copy an existing data input file to the directory where ipars will be run • Assume hydrology implicit model is build <water>/ices/gxl/IPARSv 2/work/Run cp. . /data/hydro 1. dat mytest. dat l Modify the input file • Modify mytest. dat and change TIMEEND to 10. 0 (days)

Run IPARS l Interactive mode <water>/ices/gxl/IPARSv 2/work/Run/. /ipars ENTER INPUT FILE NAME: hydro 1. dat ENTER OUTPUT FILE NAME: hydro 1. out ENTER INPUT RESTART FILE NAME (RETURN IF NONE): ENTER OUTPUT RESTART FILE PRIMARY NAME (RETURN IF NONE): # of GMRES itns = 8 res. err=. 679143548 # of GMRES itns = 9 res. err=. 00119301397 # of GMRES itns = 9 res. err= 3. 63559161 E-06 STEP 1 TIME 1. 20 OIL 1. 0000000 WAT 1. 0000000 NEWT 3 LINR 9 # of GMRES itns = 9 res. err=. 0319111682 # of GMRES itns = 9 res. err=. 00196011434 # of GMRES itns = 9 res. err= 3. 29029604 E-06 …… STEP 6 TIME 10. 00 OIL 1. 0000000 WAT 1. 0000000 NEWT 2 LINR 10 STOP 0 statement executed 4. 110 u 0. 030 s 0: 10. 92 37. 9% 0+0 k 0+0 io 376 pf+0 w

Run IPARS l Use batch file IPARS. IN <water>/ices/gxl/IPARSv 2/work/Run cat IPARS. IN hydro 1. dat hydro 1. out <water>/ices/gxl/IPARSv 2/work/Run/. /ipars # of GMRES itns = 8 res. err=. 679143548 # of GMRES itns = 9 res. err=. 00119301397 # of GMRES itns = 9 res. err= 3. 63559161 E-06 STEP 1 TIME 1. 20 OIL 1. 0000000 WAT 1. 0000000 NEWT 3 LINR 9 # of GMRES itns = 9 res. err=. 0319111682 # of GMRES itns = 9 res. err=. 00196011434 # of GMRES itns = 9 res. err= 3. 29029604 E-06 …… STEP 6 TIME 10. 00 OIL 1. 0000000 WAT 1. 0000000 NEWT 2 LINR 10 STOP 0 statement executed 4. 110 u 0. 030 s 0: 10. 92 37. 9% 0+0 k 0+0 io 376 pf+0 w Note: two empty lines at the end of IPARS. IN

Standard Ouput l Check for errors in output, in particular, for any string ERR* in the standard output file l If everything went well, you should have some results in the output files according to what was requested in the input file. Example: Look at the results in the output file hydro 1. out and in the well file HYDRO 1. WEL <water>/ices/gxl/IPARSv 2/work/Run/ ls hydro 1. dat hydro 1. out HYDRO 1. WEL ipars Standard output Well output

IPARS Restart Run l Specify restart output time in a data file. • • • TIMRES = Time for restart output DTIMRES = Time between scheduled restart outputs Example: Modifty hydro 1. dat TIMRES = 2. 0 DTIMRES = 2. 0 l Specify restart input/output files when running ipars • • Give restart output file name in an initial run Give restart input/output file names in a restart run

IPARS Restart Run l Initial runs • Interactive mode <water>/ices/gxl/IPARSv 2/work/Run/. /ipars ENTER INPUT FILE NAME: hydro 1. dat ENTER OUTPUT FILE NAME: hydro 1. out ENTER INPUT RESTART FILE NAME (RETURN IF NONE): ENTER OUTPUT RESTART FILE PRIMARY NAME (RETURN IF NONE): myrestart # of GMRES itns = 8 res. err=. 679143548 # of GMRES itns = 9 res. err=. 00119301397 # of GMRES itns = 9 res. err= 3. 63559161 E-06 STEP 1 TIME 1. 20 OIL 1. 0000000 WAT 1. 0000000 NEWT 3 LINR 9 # of GMRES itns = 9 res. err=. 0319111682 # of GMRES itns = 9 res. err=. 00196011434 # of GMRES itns = 9 res. err= 3. 29029604 E-06 …… STEP 6 TIME 10. 00 OIL 1. 0000000 WAT 1. 0000000 NEWT 2 LINR 10 STOP 0 statement executed 4. 110 u 0. 030 s 0: 10. 92 37. 9% 0+0 k 0+0 io 376 pf+0 w

IPARS Restart Run • Batch file, initial run <water>/ices/gxl/IPARSv 2/work/Run/ hydro 1. dat hydro 1. out cat IPARS. IN myrestart <water>/ices/gxl/IPARSv 2/work/Run/. /ipars # of GMRES itns = 8 res. err=. 679143548 # of GMRES itns = 9 res. err=. 00119301397 # of GMRES itns = 9 res. err= 3. 63559161 E-06 STEP 1 TIME 1. 20 OIL 1. 0000000 WAT 1. 0000000 NEWT 3 LINR 9 # of GMRES itns = 9 res. err=. 0319111682 # of GMRES itns = 9 res. err=. 00196011434 # of GMRES itns = 9 res. err= 3. 29029604 E-06 …… STEP 6 TIME 10. 00 OIL 1. 0000000 WAT 1. 0000000 NEWT 2 LINR 10 STOP 0 statement executed 4. 110 u 0. 030 s 0: 10. 92 37. 9% 0+0 k 0+0 io 376 pf+0 w

IPARS Restart Run l Restart output files <water>/ices/gxl/IPARSv 2/work/Run/ ls myrestart* myrestart. 10 myrestart. 2 myrestart. 4 myrestart. 6 myrestart. 8 l Restart Runs <water>/ices/gxl/IPARSv 2/work/Run/. /ipars ENTER INPUT FILE NAME: hydro 1. dat ENTER OUTPUT FILE NAME: hydro 1. out ENTER INPUT RESTART FILE NAME (RETURN IF NONE): myrestart. 8 ENTER OUTPUT RESTART FILE PRIMARY NAME (RETURN IF NONE): myrestart <water> /ices/gxl/ipars/IPARSv 2/work/Run/. ipars # of GMRES itns = 9 res. err=. 0288425703 # of GMRES itns = 9 res. err=. 000279604748 # of GMRES itns = 9 res. err= 3. 0205166 E-07 STEP 9 TIME 9. 02 OIL 1. 0000000 WAT 1. 0000000 NEWT 3 LINR 9 # of GMRES itns = 9 res. err=. 00233964203 # of GMRES itns = 9 res. err= 1. 22894244 E-06 STEP 10 TIME 10. 00 OIL 1. 0000000 WAT 1. 0000000 NEWT 2 LINR 9 STOP 0 statement executed

IPARS Restart Run l Restart output files <water>/ices/gxl/IPARSv 2/work/Run/ ls myrestart* myrestart. 10 myrestart. 2 myrestart. 4 myrestart. 6 myrestart. 8 l Restart Runs <water>/ices/gxl/IPARSv 2/work/Run/ hydro 1. dat hydro 1. out cat IPARS. IN myrestart <water>/ices/gxl/IPARSv 2/work/Run/. /ipars # of GMRES itns = 8 res. err=. 679143548 # of GMRES itns = 9 res. err=. 00119301397 # of GMRES itns = 9 res. err= 3. 63559161 E-06 STEP 1 TIME 1. 20 OIL 1. 0000000 WAT 1. 0000000 NEWT 3 LINR 9 # of GMRES itns = 9 res. err=. 0319111682 # of GMRES itns = 9 res. err=. 00196011434 # of GMRES itns = 9 res. err= 3. 29029604 E-06 …… STEP 6 TIME 10. 00 OIL 1. 0000000 WAT 1. 0000000 NEWT 2 LINR 10 STOP 0 statement executed 4. 110 u 0. 030 s 0: 10. 92 37. 9% 0+0 k 0+0 io 376 pf+0 w

IPARS Visualization l Source file: IPARSv 2/visual/ l Documentation: IPARSv 2/doc/vis. txt, vis 1. txt l Example data file: IPARSv 2/doc/hydrovis. dat l To build vis. include visual. mak in Makefile and comment off vis. siz in ipars. siz

IPARS Visualization l Table visualizatoin (input file keyword) VIS_TABOUTTYPE = 0 VIS_TABOUTTYPE = 1 VIS_TABOUTTYPE = 2 $ default : no vis output $ suggested: output for all table $ ask for specific tables and stop • Vis. output files are postprocessed using tecplot macros IPARSv 2/visual/scripts/mtectab, tectab • Macros for tecplot visualization will be created

IPARS Visualization l Grid element array variables (e. g. Sw, Sg) VISFLAG = 1: (ijk): one block box, single proc, scalars only VISFLAG = 2: p. c. unstructured FE: for piecewise constant data VISFLAG = 3: default unstructured FE format: requires map from cell centers to nodes (interpolation) VISFLAG = 4: slices (2 D, vertical only) VISFLAG = 5: based on VISFLAG=3 produces output in binary format with all time steps lumped in one file, created especially for the needs of ADR/Data. Cutter project; code in visual 5. dc • Vis. output files are postprocessed using tecplot macros IPARSv 2/visual/scripts/mteccos

IPARS Visualization

IPARS Visualization l Vis. control in data input file • Table function visualization VIS_TABOUTTYPE = 1 • $ Set vis. output type Specify the number of variables (scalars and/or vectors) to vis. VIS_SCL = 2 VIS_NVEC = 1 • Specify VISFLAG = 3 • Specify the name of variables to visualize VIS_SCL_NAMES(1) = "POIL" VIS_SCL_NAMES(2) = "COIL" VIS_VEC_NAMES(1) = "VEL_OIL“

IPARS Visualization • Specify the name of visualization output files. VIS_FAME = “Tec. VIS” • • Vis. output time step control VISOUT = 100. 0 $ Time for visualization output DVISOUT = 20. 0 $ Time step size for next visualization output Request to visualize permeability PERMOUT • $ In the initial data input section Request to visualize initial variables VISINIT $ In the initial data input section Note: VISINIT and PERMOUT can not be turned on together • Visualization control parameters can be changed in transient data blocks

IPARS Visualization l Example: IPARSv 2/data/hydrovis. dat Table function visualization • Copy IPARSv 2/data/hydrovis. dat to the ipars run directory <water>/ices/gxl/IPARSv 2/work/Run/ cp. . /data/hydrovis. dat. / • Modify hydrovis. dat to request table visualization Comment out $VIS_TABOUTTYPE = 1 • Run ipars executable using hydrovis. dat as input <water>/ices/gxl/IPARSv 2/work/Run/. /ipars

IPARS Visualization • Post-process the data (as described in IPARSv 2/doc/vis. txt); here is one way to do it: - Copy all files from IPARSv 2/visual/scripts to where you have all output <water>/ices/gxl/IPARSv 2/work/Run/ cp. . /visual/scripts/*. / - Ensure that you have the correct path to preplot. <water>/ices/gxl/IPARSv 2/work/Run/ which preplot /lusr/tecplot/bin/preplot - Run mtectab to create tecplot macros <water>/ices/gxl/IPARSv 2/work/Run/ mtectab - Load the resulting macros <water>/ices/gxl/IPARSv 2/work/Run/ ls *. mc KOSW___1. mcr KWSW___1. mcr PCOW___1. mcr

IPARS Visualization Example: IPARSv 2/data/hydrovis. dat l Scalar and vector variables • • • Copy IPARSv 2/data/hydrovis. dat to the ipars run directory <water>/ices/gxl/IPARSv 2/work/Run/ cp. . /data/hydrovis. dat. / Modify hydrovis. dat to request scalar and vector visualizations Comment out vis. parameters in the transient data block Run ipars executable using hydrovis. dat as input <water>/ices/gxl/IPARSv 2/work/Run/. /ipars Run mteccos to postprocess and create macros <water>/ices/gxl/IPARSv 2/work/Run/ mteccos

IPARS Visualization Example: IPARSv 2/data/hydrovis. dat l Tecplot visualization • Check macros <water>/ices/gxl/IPARSv 2/work/Run/ ls *. mcr Tec. TRAN 3. mcr Tec. TRAN 5. mcr Tec. TRANanim. mcr Tec. TRANfirst. mcr Tec. TRANload. mcr • Load the macros to the Tecplot macro menu 1. Load the "load" macro: Tec. TRANload. mcr 2. Load Tec. TRANfirst. mcr 3. Load Tec. TRANanim. mcr This can be done while your simulation continues to run (assuming that all the files have been nicely closed)

IPARS Visualization l IPARS physical model vis variable names • Hydrology implicit model “POIL” “PWAT” “COIL” “SWAT” • $ Oil phase pressure $ Water phase pressure $ Oil concentration $ Water saturation Hydrology IMPES model “POIL” “PWAT” “COIL” “CWAT” “SWAT” $ Oil phase pressure $ Water phase pressure $ Oil concentration $ Water saturation

Integrated Parallel Accurate Reservoir Simulator Parallel, Multiblock and Multiphysics Contributor: John Wheeler, Mary Wheeler, Krzysztof Banas, Steve Bryant, Srinivas Chippada, Rick Dean, Joe Eaton, Carter Edwards, Xiuli Gai, Sebastien Lacroix, Wonsuck Lee, Qin Lu, Bahareh Momken, Myeong. Hwan Noh, Manish Parashar, Victor Parr, Malgorzata Peszynska, Beatrice Riviere, Shuyu Sun, Yuri Vassilevski, Ivan Yotov

IPARS Parallel Implementation l Source file: IPARSv 2/parall/ l Documentation: IPARSv 2/doc/mem. txt l Basic principles parallel execution of IPARSv 2 • • • Parallel decomposition of grid Encapsulated calls to - UPDATE (exchange values in ghost cells) - GLOBAL_REDUCE operations (compute scalar values across all processors) Main files - many. df or manyf. f and manyc. c to handle UPDATE calls - putil. df to handle GLOBAL_REDUCE and BCAST operations

IPARS Parallel Runs

IPARS Parallel Documentation l To build an ipars executable for parallel machines • • Include parall. mak or parall_c. mak in Makefile Comment out a platform size file for parallel machines in ipars. siz Comment out frame 36. siz in ipars. siz file Include parall_m. mak if the multi-model is going to be buil. Note: frame 36. siz may need to be modified for your own applications

IPARS Parallel Runs on Bevo Example: build ipars on bevo cluster and run data/hydro 1. dat l Documentation on bevo: http: //www. ices. utexas. edu/bevo/ l To get an account on bevo, you must contact Prof. Mary F. Wheeler and then our Sysnet to get setup.

Obtain IPARS Source Code l Logon to bevo l Set environment variable CVSROOT in your. cshrc file to do remote CVS checkout setenv CVSROOT : ext: water. ices. utexas. edu: /ws/csm/SOURCE l Make a directory for ipars source code gxl@bevo: ~>mkdir ipars l CVS checkout IPARSv 2 gxl@bevo: ~>cd ipars gxl@bevo: ~/ipars>cvs checkout IPARSv 2

Choose Compilers l Logon to devel gxl@bevo: ~/ipars> ssh devel l Set environment variables for your preferred compilers gxl@bevo: ~> menu Welcome to the Bevo Development Node! ------------------This startup script will set the proper environment variables for your usage of the various compilers available on this system. Please select from the following options: Which compiler will you be using? 1) GNU 3. 2 for ethernet ( C/C++/F 77 ) 2) GNU 3. 2 for myrinet ( C/C++/F 77 ) 3) Intel Compilers for myrinet ( C/C++/F 77/F 90 ) 4) Intel Compilers for ethernet ( C/C++/F 77/F 90 ) 5) PGI compilers for ethernet ( C/C++/F 77/F 90 ) 6) PGI compilers for myrinet ( C/C++/F 77/F 90 ) 7) GNU 3. 2 for myrinet (-fno-second-underscore) Enter a number > c 7

Choose Compilers Resetting PATH variable Setting environment variables for GNU 3. 2 for Myrinet (-fno_second_underscore) Would you like to set this as your default compiler? (y or n) > c y Note: You should always select an option for myrinet, unless you have a special need to use ethernet! If you need to use ethernet, contact the sysadmin l Once you made a selection, you will have access to a number of compiler commands: (mpicc, mpi. CC, mpif 77, mpif 90). Use these commands to compile your program.

Build IPARS Executable on Bevo l Build preprocessor size for ipars compling l Prepare Makefile and ipars. siz files just like single processor case • • l Comment out beo. siz instead of lnx. siz in ipars. siz Comment out frame 36. siz Include bevo-myrit-gccf. mak instead of lnx. mak in Makefile Include parall_c. mak in Makefile Build IPARv 2 executable gxl@bevo: ~/ipars/IPARSv 2/work> make ipars

IPARS Parallel Runs l l Exit devel Prepare an input file and copy it to the directory where you want to run ipars gxl@bevo: ~/ipars/IPARSv 2/work> mkdir Run gxl@bevo: ~/ipars/IPARSv 2/work>cd Run gxl@bevo: ~/ipars/IPARSv 2/work>cp. . /data/hydro 1. dat. / NOTE: Input file can be any that runs in serial mode. Special (additional keywords) may include METHOD. l Bevo uses SGE queing system to submit, manage, and monitor the jobs being performed. You must write a submit script in order to submit a job to the queue.

IPARS Parallel Runs l Create a job submit script file (gxljob) Number of processors #!/bin/bash #$ -pe mpich 16 Job name #$ -cwd #$ -N gxljob /opt/bevo_mpich/myrit/pgcc/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines programname l Create batch file IPARS. IN gxl@bevo: ~/ipars/IPARSv 2/work/Run> cat IPARS. IN Hydro 1. dat Hydro 1. out l Submit bevojob to the queue gxl@bevo: ~/ipars/IPARSv 2/work/Run> qsub gxljob

IPARS Parallel Runs l Check the status of your job gxl@bevo: ~/ipars/IPARSv 2/work/Run> qstat job-ID prior name user state submit/start at queue master ja-task-ID ------------------------------------------------------------------------------------------------------- 774 0 gxljob gxl qw 08/07/2005 23: 39: 47 Job name gxl@bevo: [11: 39 pm]: ~/ipars/IPARSv 2/work/Run > qstat job-ID prior name user state submit/start at queue ----------------------------------------------------------------------------------------- 774 0 gxljob gxl gxl t t 08/07/2005 23: 39: 55 compute-0 - MASTER compute-0 - SLAVE compute-1 - SLAVE 0 gxljob gxl t 08/07/2005 23: 39: 55 compute-1 - SLAVE

Parallel Run Results l Cancel your job gxl@bevo: ~/ipars/IPARSv 2/work/Run> qdel job. ID l View hydro 1. out for standard output l View jobname. ojob. ID for screen output l View jobname. ejob. ID for error message

IPARS Multiblock – Source Code l Multiblock code in IPARSv 2 exists mainly in the "mortar" flavor. l Calls to mortar-specific routines are associated with keyword $MORTAR l CALL MORTAR_STEP(NERR) The currently used framework "mortar" code is based on MACE (Multiblock Adaptive Computational Engine) MACE library: IPARSv 2/MACE l IPARS-specific multiblock framework code is in MACE/utilities/mblk_mace/

IPARS Multiblock – Source Code l Model dependent code for mortar multiblock is in model directories. Example: see object list in hydroi. mak MBOBJ = hm_util$(O) hfault$(O) hm_pc$(O) hm_vars$(O) • hm_util, hfault: initialization and setup of mortar boundary conditions • • hm_vars: different choices of primary unknowns on the interface hm_pc: implementation of mortar interface preconditioner

IPARS Multiblock – Documentation l Therory and practice of "mortars“: papers by Mary F. Wheeler, Ivan Yotov, Qin Lu, Manish Parashar and Malgorzata Peszynska. l Multiblock mortar implementation: doc/MB. txt and in doc/MBflow. txt.

IPARS Multiblock – Build l Logon to bevo and then to devel l Type menu and choose compilers • l Example: choose 7 (GNU 3. 2 for myrinet with -fno-second-underscore) Prepare MACE libraries • • • gxl@compute-devel-0 -0: ~/IPARSv 2/MACE> make SYSTEM=bevo all • SYSTEM must be compatible with the statement in the machine makefile for IPARS so the master Makefile can find appropriate MACE library. For example, in beo-myrit-mb. mak: This command will build IPARSv 2/MACE/lib/bevo/libmace. a The value of variable SYSTEM must be the name of a configuration that is defined in MACE/arch_makes/make. defn. bevo SYSTEM = bevo MACESYSFLAG = -DLINUX -DWant_c_files -DDEBUG_PRINT -DACE_NO_MPI

IPARS Multiblock – Build l Build IPARS executable • Comment out mace. mak in master Makefile: this will automatically "pick up" all the framework AND all model dependent multiblock code • Comment out mortar. siz in master sizefile (ipars. siz): this will make active all the mortar code "hidden" under the keyword $MORTAR. • For parallel machines comment parall_c. mak and a machine make file e. g. bevo-myrit-mb. mak • Build ipars executable gxl@compute-devel-0 -0: ~/IPARSv 2/work> make ipars Note: MACE is a C++ templated library and therefore there exist limitations in its current portability. It has been tested to work for gnu and intel compilers. However, it works only for gnu compilers on bevo (IPARSv 2/make/modular/bevo-myrit-mb. mak)

IPARS Multiblock – Data Input l Define Mortars in an input file • • Examples: IPARSv 2/data/h*mb*, IPARSv 2/data/hmb 2 b. dat Use MB_OUTLEVEL to control the level of output <water>/ices/gxl/IPARSv 2/data/ cat hmb 2 b. dat TITLE(2)="MULTIBLOCK HYDROLOGY MODEL TEST 1" DESCRIPTION()= "BLOCK LENGTH (FT) WIDTH (FT) HEIGHT (FT) " 1 400 24 2 x 8 x 8 0, 0, 0" " 2 400 24 2 x 10 0, 400, 0" "16800 GRID ELEMENTS SIZE CORNER" DATE : 12/21/97" HYDROLOGY_IMPLICIT TIMEEND = 3. $ I/O OPTIONS OUTLEVEL = 2 SPLINEOUT GEOMOUT PROCOUT DEBUGS BUGKEY(1) BUGKEY(2) BUGKEY(6) BUGKEY(8)

IPARS Multiblock – Data Input $ INTERFACE GMRES INFO MB_GMRES_RESTART = 30 MB_GMRES_MAXITER = 30 $ Restart for interface GMRES $ Maximum number of interface GMRES iterations MB_GMRES_TOL = 0. 01 $ Tolerance for interface GMRES MB_NEWT_MAXITER = 10 MB_NEWT_TOL = 0. 01 $ Maximum number of interface Newton iterations $ Tolerance for interface $ FAULT BLOCK AND MESH DATA MB_NUMBLKS = 2 MB_NUMMORTARS = 1 $ Total number of blocks $ Total number of mortars (interfaces) $%%%%%%% Block Difiition %%%%%%% $%%%%%%% Block (1) %%%%%%% $ In the section below the SECOND index (or the only index) $ corresponds to the block number BLOCKNAME(1) = "BLOCK 1" $ Name of this block DOWN(1 TO 3, 1) = 1 0 0 NX(1) = 2 NY(1) = 8 NZ(1) = 8 DX(1 TO 2, 1) = 12. DY(1 TO 8, 1) = 50. DZ(1 TO 8, 1) = 50. XYZ 111(, 1) = 3*0.

IPARS Multiblock – Data Input $%%%%%%%%%%%%%% Mortar definition %%%%%%% Block (1) %%%%%%% MB_MYMORTAR_NUMBER(1) = 1 $ Number of mortars adjacent to this block $ In the section below mortar information is given from the block's $ perspective. The FIRST index corresponds to the block number $ and the SECOND index corresponds to the mortar number. $ NOTE that the mortar numbering is for this block only and always starts from one! MB_MYMORTAR_ID(1, 1) = 1 MB_MYMORTAR_FACE(1, 1) = 4 MB_MYMORTAR_LB 1(1, 1) = 0. MB_MYMORTAR_UB 1(1, 1) = 24. MB_MYMORTAR_LB 2(1, 1) = 0. MB_MYMORTAR_UB 2(1, 1) = 400. MB_MYMORTAR_DIM 1(1, 1) = 2 MB_MYMORTAR_DIM 2(1, 1) = 8 MB_MYMORTAR_FLG(1, 1) = 1 Block number Mortar number $ Global ID for this mortar $ Block face adjacent to this mortar $ Lower bound for first mortar dimension $ Upper bound for first mortar dimension $ Lower bound for second mortar dimension $ Upper bound for second mortar dimension $ Mortar size in first dimension $ Mortar size in second dimension $ Grid flag: 1 -> Uniform; $ 0 -> Read grid from $ file 'junk' - this option needs to be properly $ implemented by reading grid file name from the $ data file

IPARS Multiblock – Data Input $%%%%%%% Block Difinition %%%%%%% $%%%%%%% Block (2) %%%%%%% $ In the section below the SECOND index (or the only index) $ corresponds to the block number BLOCKNAME(2) = "BLOCK 2" DOWN(1 TO 3, 2) = 1 0 0 NX(2) = 2 NY(2) = 10 NZ(2) = 10 DX(1 TO 2, 2) = 12. DY(1 TO 10, 2) = 40. DZ(1 TO 10, 2) = 40. XYZ 111(, 2) = 0 , 400 , 0 $%%%%%%% Mortar definition %%%%%%% $%%%%%%% Block (2) %%%%%%% MB_MYMORTAR_NUMBER(2) = 1 MB_MYMORTAR_ID(2, 1) = 1 MB_MYMORTAR_FACE(2, 1) = 3 MB_MYMORTAR_LB 1(2, 1) = 0. MB_MYMORTAR_UB 1(2, 1) = 24. MB_MYMORTAR_LB 2(2, 1) = 0. MB_MYMORTAR_UB 2(2, 1) = 400. MB_MYMORTAR_DIM 1(2, 1) = 2 MB_MYMORTAR_DIM 2(2, 1) = 8 MB_MYMORTAR_FLG(2, 1) = 1

IPARS Multiblock – Data Input $%%%%%%%%% MORTAR INFO %%%%%%%% $ In the section below the SECOND (or the only) index corresponds $ to the global mortar ID number MB_MORTAR_TYPE(1) = 1 MB_MORTAR_ALIGN(1) = 1 MB_MORTAR_NEIGH(1, 1) = 1 MB_MORTAR_NEIGH(2, 1) = 2 $ Mortar type $ Mortar alignment flag (should be set to 1 if no mapping is used) $ $ ID number of first mortar neighbor $ ID number of second mortar neighbor

IPARS Multiphysics l Multimodels in IPARSv 2 • • Multimodel coupling in adjacent domains Multimodel coupling in overlapping domain Single-phase model Black-oil model Geomechanics model Single phase model Adjacent domains l l Overlapping domains Source code: IPARSv 2/mmodel/ Documentation: IPARSv 2/doc/mmodel. txt

IPARS Multiphysics Multimodel coupling Coupling Algorithm Model Keyword Directory Adjacent domains Coupling across interface using mortar multiblock $MMODEL Multimodel Overlapping domains Coupling in time using operator splitting $TRCHEM Transport & chemistry $POROE Poroelasticity

IPARS Multiphysics $MMODEL Directory Physical Models mmodel hydroe, hydroi, blacki, single MULTI_MODEL BLOCKMODEL(1) = "HYDROLOGY_IMPLICIT" BLOCKMODEL(2) = "SINGLE_PHASE_IMPLICIT" trchem reactive transport, singlei, hydroi TRANSPORT_CHEMISTRY FLOW_MODEL = "SINGLE_PHASE_IMPLICIT" poroe blacki, poroelasticity Multimodel $TRCHEM Tran. and chem. $POROE Poroelasticity Input File Keywords POROELASTICITY FLOW_MODEL = “BLACK_OIL_IMPLICIT”

IPARS Multiphysics Build Physical Models $MMODEL Multimodel $TRCHEM Tran. and chem. $POROE Poroelasticity Required Size Included Makefiles Files hydroe, hydroi, blacki, single Mortar. siz, mace. siz mmodel. mak, mace. mak hydroe. mak, hydroi. mak, blacki. mak, single. mak parall_m. mak (for parallel) reactive transport, singlei, hydroi trchem. siz, either singlei. siz or hydroi. siz trchem. mak, either singlei. mak or hydroi. mak blacki, poroelasticity poroe. siz, blacki. siz poroe. mak, blacki. mak

Multiphysics Linear Solvers Physical Models Linear Solvers singlee PCG, GMRES singlei LSOR, GMRES hydroe PCG hydroi LSOR, GMRES blacki LSOR, GMRES

Integrated Parallel Accurate Reservoir Simulator Develop Your Own Model Contributor: John Wheeler, Mary Wheeler, Krzysztof Banas, Steve Bryant, Srinivas Chippada, Rick Dean, Joe Eaton, Carter Edwards, Xiuli Gai, Sebastien Lacroix, Wonsuck Lee, Qin Lu, Bahareh Momken, Myeong. Hwan Noh, Manish Parashar, Victor Parr, Malgorzata Peszynska, Beatrice Riviere, Shuyu Sun, Yuri Vassilevski, Ivan Yotov

IPARS Model Dependent Calls Platform specific initialization Scalar input Initialize utilities Setup geometry Memory allocation Driver IPARS Initial data input Well/rock data Grid element input Initialization Time stepping - Read transient data - Set time step - Make one time step - Standard, restart output - Visualization output End Model specific *isdat. df Model specific *array. df Model specific *iadat. df Model specific *ivdat. df Model specific *step. df Model specific *stdout. df *rest. df Model specific *visual. df Newton iteration *prop. df *tran. df *bwell. df

Model Keywords l Set an input file keyword for your model • l Example: THERMAL Assign a model number to your model • Example: MODEL = 19 l Use the keyword and model number to register the model in IPARSv 2/input/idata. df l Set a model keyword for preprocessing • • Example: $THERMAL All your model dependent calls from the framework routines should be hidden under this keyword $THERMAL CALL TMSTEP( )

Model Keywords l Make a directory for your model • Example: IPARSv 2/thermal/ l Name your files to distinguish from other models • Example: thivdat. df, thbwell. df, thstep. df l Create a model makefile and size file • Example: thermal. mak, thermal. siz

Model Keywords l Hook up your model dependent calls to the framwork Example: in IPARSv 2/ipars. df $BLACKP IF (MODEL. EQ. 1) CALL PSTEP(NERR) $BLACKI. . . $TRCHEM $POROE IF (MODEL. EQ. 2) CALL ISTEP(NERR) $THERMAL IF (MODEL. EQ. 17) CALL TRSTEP(NERR) IF (MODEL. EQ. 18) CALL PESTEP(NERR) IF(MODEL. EQ. 19) CALL THSTEP(NERR)

Hook up with IPARS Framework Platform specific initialization Scalar input Initialize utilities Setup geometry Memory allocation Driver IPARS Initial data input Well/rock data Grid element input Initialization Time stepping - Read transient data - Set time step - Make one time step - Standard, restart output - Visualization output End Model specific thisdat. df Model specific tharray. df Model specific thiadat. df Model specific thivdat. df Model specific thstep. df Model specific thstdout. df threst. df Model specific thvisual. df Newton iteration thprop. df thtran. df thbwell. df

Input and Output l l General keyword input • CALL GETVAL(‘KEYWORD[unit] ', VNAME, ‘TYPE', DIM 1, DIM 2, DIM 3, DIM 4, NDUM, NERR) • Example: CALL GETVAL('OILVIS[cp] ', OILVIS, 'R 8', 0, 0, NDUM, NERR) Grid element array input • • l CALL GETGRDA (‘KEYWORD ', KARY, NDUM, NERR) Example: CALL GETGRDA (‘SOIL ', 1, NDUM, NERR) Grid element array output • • CALL GEAOUT(ARRAY_NUMBER, IDIM 4, KARY) Example: CALL GEAOUT(N_POIL, 1, 1)

Memory Management l Grid decomposition JDIM KDIM

Allocate Memory In Fortran routine: CALL ALCGEA(‘NVAR’, DTYPE, LDIM, N_ARY, KERR) In C routine: void alcgea (char* nvar, int* dtype, int* ldim, int* n_ary, int* err) Id = id + 1 aryadd[nb][id] = address Array of pointers *n_ary = id address Allocated memory (idim*jdim*kdim*ldim)

Usage of Memory In Fortran routine: e. g. print. Var(…) CALLWORK(print. Var, N_ARY) In C routine: void callwork (void (*sub. Name) (), n_ary) (*sub. Name) (idim, jdim, kdim, …, aryadd[nb][n_ary])