ITM General Meeting 2011 IPP Garching The ITM

ITM General Meeting 2011 IPP Garching The ITM General Grid Description Presented by H. -J. Klingshirn on behalf of IMP 3 TF Leader : G. Falchetto, Deputies: R. Coelho, D. Coster EFDA CSU Contact Person: D. Kalupin ITM General Meeting 2011 Conference name, data and Presenter

Outline 1. The ITM General Grid Description • Standard use case • General approach • Advanced & experimental features 2. Software • Grid Service Library • UALConnector & Vis. It 3. Outlook & Issues ITM General Meeting 2011 2

The ITM General Grid Description: Recapitulation Basic idea: separation of physics and numerics • The CPOs define what quantities are stored (physics) • How they are stored (numerics) is up to the code Central components 1. Standardized method & conventions how to write down spatial discretizations 2. Standardized data structures for use in CPOs 3. Dedicated software for handling grids and data (the Grid Service Library) ITM General Meeting 2011 3

The user perspective What does this mean for the user? • CPO Designer – CPOs have to be defined using the GGD data types • Code developer – When writing to the CPO, the grid has to be defined explicitly – A code reading the CPO has to interpret the given grid • Which can also mean to indicate “can’t handle this” – Code coupling still requires careful thought: • define standard discretization(s) for specific coupling scenarios, or • write general coupling code that can handle different discretizations • Code user – General purpose tools can be used to work with CPOs using the General Grid Description ITM General Meeting 2011 4

CPO Layout (example: edge CPO) Grid data structure Standard grid data structure: complexgrid (defined in utilities. xsd) • typically placed in same hierarchy level as associated fields • multiple grid definitions per CPO possible ITM General Meeting 2011 5

CPO Layout (example: edge CPO) Data fields • CPO data fields: standard data type complexgrid_scalar • use arrays of scalars to allow storage on multiple subgrids ITM General Meeting 2011 6

CPO I/O in codes: Writing grids grid data structure node positions standardized coordinate types Grid id / name ITM General Meeting 2011 7

Simple 2 d structured grid 0. 0 1. 0 2. 0 Z 0. 0 1. 0 2. 0 3. 0 4. 0 5. 0 R 2 d structured grid in the R, Z plane ITM General Meeting 2011 8

Subgrid: nodes Behind your back, the high-level interface creates subgrids: lists of grid objects 0. 0 1. 0 2. 0 Z 0. 0 1. 0 2. 0 3. 0 4. 0 5. 0 R • subgrids are identified by their index • this example: module itm_grid_structured: constant GRID_STRUCT_NODES ITM General Meeting 2011 9

Subgrid: edges 0. 0 1. 0 2. 0 Z 0. 0 1. 0 2. 0 3. 0 4. 0 5. 0 R Module itm_grid_structured: constant GRID_STRUCT_EDGES ITM General Meeting 2011 10

Subgrid: faces (2 d cells) 0. 0 1. 0 2. 0 Z 0. 0 1. 0 2. 0 3. 0 4. 0 5. 0 R Module itm_grid_structured: constant GRID_STRUCT_FACES ITM General Meeting 2011 11

Writing data Fortran interface grid data structure CPO data field data array Subgrid index subgrid index Subgrids are central to reading and writing data ITM General Meeting 2011 12

Reading grids: Python interface • High-level handling of CPOs Create wrapper object for CPO Get grid object • Subgrids Subgrid objects act as sequences of itm. grid. base. Object ITM General Meeting 2011 13

Reading grids: Python interface (ctd. ) • Plots via matplotlib Subgrid lookup ITM General Meeting 2011 14

Reading data: Python interface Find used data fields in CPOs & return them as itm. grid. data. Scalar. Data Retrieve data Find used data arrays Automatic lookup of data for specific objects ITM General Meeting 2011 15

Describing discretizations: General approach Z edge 1. 0 2. 0 node 0. 0 face (2 d cell) 0. 0 1. 0 2. 0 3. 0 4. 0 5. 0 R The general grid description identifies and describes grid objects: nodes, edges, … ITM General Meeting 2011 16

Describing discretizations: Space decomposition 5 4 2. 0 Z If possible, decompose the discretization into spaces: independent discretizations of individual coordinate directions 3 3 2 1. 0 Space 2 4 Here: two one-dimensional spaces separately discretizing the R and Z direction 2 1 edge, node: space objects 0. 0 1 0. 0 Node index Edge index ITM General Meeting 2011 1. 0 1 2. 0 2 1 3 2 3. 0 4 3 Space 1 5. 0 5 4 R 6 5 17

Storing subobject information 0. 0 1 2. 0 2 1 3. 0 3 2 4. 0 4 3 5. 0 5 4 R 6 5 0 d space objects: nodes Node positions n-dimensional space objects (n>0): assembled from (n-1)d space objects Boundaries of objects Connectivity EFDA-ITM-WS/CC These fields store space object indices 18

Describing discretizations: Space combination/multiplication Z 4 1 d grid objects (edges): 1 d + 0 d space objects 2. 0 5 0 d grid objects (nodes): 0 d + 0 d space objects 3 3 2 1. 0 Space 2 4 2 d grid objects (faces): 1 d + 1 d space objects 2 1 The full grid is obtained by forming all possible combinations of space objects 0. 0 1 0. 0 Node index Edge index ITM General Meeting 2011 1. 0 1 2. 0 2 1 3 2 3. 0 4 3 Space 1 5. 0 5 4 R 6 5 19

Object Descriptor: ((c 1, c 2, . . . , cn) (i 1, i 2, . . . , in)) Object index: ij = subobject index in space j Object class: cj = subobject dim. in space j ((0, 1) (5, 4)) ((1, 0) (2, 4)) ((0, 0) (3, 3)) 5 4 Space 2 4 3 3 2 ((1, 1) (4, 2)) 2 1 1 2 1 ITM General Meeting 2011 1 3 2 4 3 5 4 6 5 20

Global object order The space objects have an explicit (local) order in their respective space (simply the order in which they are defined) For the implicitly defined grid objects, a global order is imposed by adopting a simple counting convention (think linear address computation for multidimensional Fortran arrays): Grid objects of a common object class are counted by varying the leftmost index of the index tuple first. This imposes an ordering of grid objects of a common class. Every grid object can therefore be uniquely identified by: Its object descriptor: ((c 1, c 2, . . . , cn) (i 1, i 2, . . . , in)) EFDA-ITM-WS/CC or Its object class and global index ig: ((c 1, c 2, . . . , cn) ig) 21

Storing data on grids: subgrids data is stored on a subgrid = list of grid objects the data is then stored simply as a vector, one entry per object in the subgrid Subgrid index Actual data: one entry per subgrid object, in the order defined in the subgrid (vector, matrix: for complex data representations) EFDA-ITM-WS/CC 22

Subgrid definition Object class tuple A subgrid is a selection of a subset of grid objects (of common dimension) There can be an arbitrary number of subgrids (which are part of the grid description). A specific subgrid is identified by its index. Object index set (for implicit list) Object index list (for explicit list) A subgrid is a list of object lists. Each object list can be either explicit: an explicit list of object descriptors implicit: an implicit list of object descriptors, selecting a range or an entire class of objects EFDA-ITM-WS/CC 23

Subgrids: Implicit object list notation Object List Descriptor: ((c 1, c 2, . . . , cn) (s 1, s 2, . . . , sn)) Object class: cj = subobject dim. in space j (same as in object descriptor) Object index set: specifies multiple indices sj = [i 1, i 2, …, i. N] → explicit list of indices sj = (i 1, i 2) → range of indices from i 1 to i 2 sj = UNDEFINED → all possible indices Object order: implicit object lists inherit the object order intrinsic to the underlying grid definition Why is this important? Space splitting and implicitly defined object order allow efficient handling of datasets on very large (5 d, 6 d, . . . ) structured grids. EFDA-ITM-WS/CC 24

Subgrids Example: SOLPS-B 2, single null Outer midplane nodes Core boundary faces All 2 d cells ITM General Meeting 2011 25

Advanced features: alternate geometries Alternate geometries: Explicitly include node positions in alternate coordinate systems ITM General Meeting 2011 26

More features • Additional object properties – measures: length, area, volume … – metric information: jacobian, metric tensor – Identification of x-point nodes • Periodicity – either directly through object connectivity – or indirectly through node aliasing …we are quickly moving into area of experimental features – much of this still has to be tested in real applications …. ITM General Meeting 2011 27

Experimental features: non-standard representations “Standard” geometry & data representation: • Grid geometry directly given by node positions and object boundaries • Data fields: one constant value per grid object (node value, area/volume average) Non-standard representations: • Flag in space data structure indicates alternate interpretation of space definition • The rest is up to you ITM General Meeting 2011 28

Example: Fourier representation, space data structure =(COORDTYPE_. . . ) (unused) = GEO_TYPE_FOURIER = GEO_TYPE_ID_FOURIER Holds mode numbers & node positions for DFT • The space data structure holds information required for an unambiguous inversion of the DFT • Geometry data with additional degrees of freedom also possible for implicitly defined grid objects ITM General Meeting 2011 29

Example: Fourier representation, data field structure matrix(i, : , 1) = real matrix(i, : , 2) = imaginary components of DFT for object with index i in subgrid To spare us pain, the DFT definition of FFTW is used. …but of course you can just as well propose your own! ITM General Meeting 2011 30

Experimental features: vector data type • A complexgrid_vector is a vector of complexgrid_scalars • The components can possibly be aligned to something: • another vector quantity • a set of base vectors, possibly defined as part of the grid • This is work in progress, application driven ITM General Meeting 2011 31

Grid Service Library • Does what we can’t do in the CPO definition: adds functionality for specific data structures • Current approach: – Provide dedicated implementations for separate languages, exploiting their strengths (like UAL) – Some basic functionality present in all implementations, consistency ensured by unit test framework – Advanced functionality will diverge depending on typical use cases for the different languages • Current languages: – Fortran 90: procedural, typical for codes – Python: object oriented, typical for post-processing tools/glue scripts ITM General Meeting 2011 Can serve as starting point for other implementations 32

Getting the service library • GForge: maintained in ITM_SHARED project, grid branch • Getting a local copy: svn co http: //gforge. efda-itm. eu/svn/itmshared/branches/grid cd grid Sets up F 90&Python environment source setup. csh testgrid setup Writes some example grids to database testgrid all Runs integration unit tests for all languages …Test all implementations: OK • Public copy currently provided at klingshi/bin/itm-grid (should move to a better place in the future) • Your environment has to be set up for 4. 09 a • Instructions at https: //www. efda-itm. eu/ITM/html/imp 3_gridservicelibrary. html ITM General Meeting 2011 33

Grid Service Library: Directory layout • grid/ • f 90/ • src/ • service/ • examples/ • test/ • python/ • itm/ • grid/ • visit/ • test/ Fortran service library modules Example programs Unit tests service library classes ualconnector/Visit integration Unit tests Documentation: • General: IMP 3 section of documentation website • Documentation partially generated from source (Doxygen/Sphinx) make doc; make doc_release https: //www. efda-itm. eu/ITM/doxygen/imp 3/grid_service_library/ ITM General Meeting 2011 34

Grid Service Library: Fortran 90 Structured in modules. Some more interesting ones: • itm_grid_access: accessing basic grid properties • itm_grid_object: handling grid objects • itm_grid_subgrid: handling subgrids • itm_grid_structured: high-level interface for structured grids • itm_grid_simplex: high-level interface for simplex grids (triangles…) Subroutines & functions acting on on standard data types: …which is a new property required from the UAL implementations, especially for strongly typed languages ITM General Meeting 2011 35

Grid Service Library: Python Basic design: • classes wrapping data structures, which implement methods acting on them (could include functionality in the UAL objects, but this would lead to lots of complications) • Python Inspection capabilities allow dynamic analysis of CPO structure and contents (without prior knowledge) Main classes (class name: wrapped data structure) • itm. grid. cpo_tools. Cpo: general CPO wrapper • itm. grid. base. Grid: complexgrid • itm. grid. base. Sub. Grid: complexgrid_subgrid • Itm. grid. data. Scalar. Data: complexgrid_scalar …objects typically created through the itm. grid. cpo_tools. Cpo CPO wrapper object ITM General Meeting 2011 36

ualconnector: Vis. It integration • The general grid description enables general operations with grids and data • First example: general visualization tools for complex discretizations and data sets • Example: LLNL Vis. It (https: //wci. llnl. gov/codes/visit/) – Coupling mechanism: • use Python Grid Service library to analyse CPO contents • transform grids & data fields into Vis. It-specific data format (completely general step) • pass data to Vis. It via the libsim interface (ualconnector Python program) UALConnector: Python GSL + Python libsim Vis. It 2. 3 ITM General Meeting 2011 37

ualconnector: Basic usage ~klingshi/bin/itm-grid/ualconnector -s 17151, 898, 100. 0 -c edge can specify multiple CPOs -s 17151, 899, 100. 0 -c edge -u coster -t aug -v 4. 09 a Options: -s run, shot, time -c cpo-name -u username -t Tokamak name -v data version have to be specified in pairs • this will automatically launch & connect a Vis. It 2. 3 instance • your environment has to be set up for data version 4. 09 a • currently only makes sense for the edge CPO ITM General Meeting 2011 38

Vis. It GUI ITM General Meeting 2011 39

UALConnector: details • CPO analysis done using python inspect – Recursive search in data structures for standard types (complexgrid*), with checks for data – No a-priori information about CPO structure needed • Grid transformation: GGD Vis. It – 1 d grid curve – 2 d grid 2 d unstructured grid (currently only triangles, quads) – 3 d grid 3 d unstructured grid (general polyhedra) • Vis. It libsim library – Python binding available since June 2011 – designed for instrumentation and in-situ visualization of ((massively) parallel) simulations no concept of time ITM General Meeting 2011 40

Outlook • Upcoming data version release (4. 09 b) – Minor improvements to data structures (multiple grids, alternate geometries) – Implementation in at least one additional CPO (wall) • Consolidation & further development of GSL – C++ version – Improved data handling and high-level functions – Applications! • Further explore complex data representation ITM General Meeting 2011 41

Current issues • Compatibility of GSL concept with automatically generated UAL interface data structures – the service library relies on consistent types in the strongly typed languages • Performance of general grid operations (especially in Python): often very inefficient – optimization (doable? ) – dedicated implementations for special cases (doable!) ITM General Meeting 2011 42