Parallel Algorithm Oriented Mesh Datastructure JeanFranois Remacle Joe

Parallel Algorithm Oriented Mesh Datastructure Jean-François Remacle, Joe E. Flaherty and Mark S. Shephard Rensselaer Polytechnic Institute remacle@scorec. rpi. edu Outline Basics of Mesh Representation n Parallel Extensions n Software issues n Examples n n http: //www. scorec. rpi. edu/AOMD

AOMD-PAOMD and PAOMD deals with meshes n A core for basic mesh representation (topological adjacencies) n Advanced design patterns : Iterator, Observer, Visitor. . . Some extensions Parallel services : message passing, load balancing, partitioning Meshing Toolbox : quality measures, mesh modifications, cavity mesher, … Calculus : coordinate systems, integration <> AOMD supports n Geometry based analysis design, classification n Parasolid, Pro. E, STL (not open source yet) Hybrid meshes Hexes, Tets, Quads … Non conforming meshes (hanging nodes, AMR) n Curved meshes (Bézier) n AOMD is an Open Source Project

Motivations Advanced analysis techniques Are automated with automatic mesh generation from geometric models n Employ variable order elements based on other than Lagrange basis n Use various weak forms that required alternative mesh relationships n Adaptive the mesh as the simulation proceeds n To meet these needs the mesh data structure must n n n Understand the relationship of mesh entities with the geometric model ensure mesh validity and associate physical parameters with the mesh Understand the interactions of various mesh entities - different relationships used during automatic mesh generation and analysis Support assignment of independent geometry to the entities in the mesh must control geometric approximation with higher order methods Be able to associate dof to various mesh entities - provides flexibility to support variable p-order and different collections of dofs Effectively maintain relationships during modification - needed in mesh generation, mesh adaptation and mesh modification for evolving geometry

Automated Adaptive Analysis AOMD-Mesh. Sim Trellis AOMD FEM PUM

Topological Mesh Data Structure Classic node point coordinates / element connectivities do not meet this need Mesh representations based on topological entities and their adjacencies fill the need: Can be proven to be complete and unique can effectively support all relationships and associations needed by any mesh generation or analysis procedure n Provide a shape independent abstraction for associating geometry n Effectively supports the linkage to the geometric model since systems maintain a model topology n Various approaches to support mesh topology have been taken A new approach considered here curved mesh for p-version analysis

Modern Geometric Modeling Systems n Employ non-manifold boundary representation Provide access to model and its geometry through a geometric modeling kernel driven by topological entities n Simulation processes (mesh generation, p-version analysis. . . ) can directly interact with the modeler n

Basics of mesh representation A geometrical domain G Is the highest level representation of the domain d n is composed of geometrical entities Gi n A Mesh M is a discrete representation of G d d n is composed of mesh entities Mi , i=1, … N (M) together with their adjacencies Mid {M q } n Mesh entities Mid n 4 different topological kinds n Vertices (d=0), edges (d=1), faces (d=2) and regions (d=3) The unique association of a mesh entity, Midi, to a geometric model entity, Gjdj, where di<dj is denoted by Midi G j dj

Adjacencies sets Examples of complete adjacencies sets Circular One-Level Upward Adjacenties Downward Adjacenties Face to Edge Incomplete Complete

Cost of a mesh representation

Memory cost of some mesh representation

Higher order adjacencies First order adjacencies Direct access, unit cost n Full representation : only way to have all adjacencies as first order n Higher order adjacencies sets 3 2 Regions know faces : Mi {M } 2 1 n Faces know edges : Mi {M } 2 1 n Edges know vertices : Mi {M } 3 2 1 0 n Third order adjacencies sets : Mi {M } n

Functionally complete representation Minimum information n Equally dimension classified entities must be present n All vertices, all regions, all edges classified on model edges and all faces classified on model faces This is a sufficient minimum, not necessary but this choice allow to complete the representation without geometrical checks

Basics of the Algorithm Oriented Mesh Database Mesh entity description A mesh entity described by a set of lower dimension entities : Mid {M q } , d > q n All vertices are always required n Vertices are atomic mesh entities, must be differentiated (e. g. , using i. D (Mi 0) n Mesh entities comparison Two entities are equal if their set of vertices are equal n allows to compare mesh entities (<, >, =) n Not absolutely general but key to practical implementation n

Downward adjacencies ordering : templates Entity described using their boundaries n unique description i. e. non ambiguous shape Weaker hypothesis Need for ordering, templates n Used for computing uses n T ev and T fe n Same vertices but different entities Invert templates

Mesh Entity i. D Need of search in AOMD Add, search and remove operations are crucial n Comparing entities is always possible but. . . n std: : set<m. Entity*, less. Than. Entity> log behavior is not acceptable n Hash tables n n n n Elements in a hash table not sorted complexity : worst is linear, average is constant std: : hash_set<m. Entity*, hash. Entity, equal. Entity> Hash function needed, deterministic and stateless, a mesh entity i. D(M 1) = i. D(M 2) M 1 = M 2 true i. D(M 1) = i. D(M 2) M 1 = M 2 false i. D is a function of vertices for being independent of the representation i. D(M 1) = i. D(M 2) and M 1 M 2 should not happen too often, efficiency of the hash table because equal. Entity is to be used in this case

Choice if the i. D Efficiency Ne is the number of elements n Nk is the number of keys n

Higher order finite elements Counting of degrees of freedom (Szabo basis) Stokes problem, tet mesh n Number of dofs, use previous statistics

Basics of the Parallel AOMD Basics of parallel AOMD n Partition boundaries treated like model boundaries Equal order mesh entities must exist on partition boundaries (partition faces, edges and vertices) Mesh vertices must have a unique global label n On processor : serial AOMD n Implementation aspects Simplicity, no master, no owner n Round of communication standardized, no MPI calls visible, messages automatically packed n

Parallel AOMD - Mesh Adaptation Target is transient applications with thousands of mesh adaptation steps Want fast and simple adaptation n Need efficient interprocessor communications n Mesh Refinement Apply templates n Include support of non-conforming meshes n Refined entities with remote copies must be split on all partitions n Round of communication needed to ensure unique vertex ID’s n Mesh Coarsening (will be same for local mesh modifications) Collect all mesh entities involved onto one partition n Carry out operation using serial operators on processor n

Dynamic Load Balancing and Mesh Migration Need dynamic load balancing after mesh adaptation Procedures build on balancing procedures in Zoltan (from Sandia) n PAOMD used to provide Zoltan needed entities and connections n Load balancing procedure indicates which mesh entities are to be migrated to which processor n PAOMD only migrates minimum set, unless user specifically asks to migrate other entities n classification after load balancing and before migration configuration after migration

Mesh Migration Steps in process Collect the mesh entities to be migrated to another partition n Determine needed higher order mesh entities to be migrated (use AOMD to determine minimal set needed) n Collect entities and any user attached data n Perform communications to send entities and update links (following methods of the Rensselaer Partition Model (RPM)) n Message passing n At PAOMD operator level it appears messages are sent one at a time n This would lead to unacceptable communication costs Message packing used - AUTOPACK (from Argonne) Automatically controls message packing process Includes information and tools to optimize message size for network architecture used Communications costs n In the examples that follow communications were on the order of 1% of the total costs

Implementation issues n Orientation of entities computed on the fly Language n C++ and generic programming STL, significant new feature of the language Programming with concepts n C++ and OO programming n generic and OO are complementary Trade off efficiency vs. flexibility We believe not Templates, functors… generic programming is efficient Classical example, quick sort stl: : sort is 4 times faster (with VC 6) than C qsort Parallel Autopack, automatic message packing n Zoltan, dynamic load balancing and partitioning n STL, associative containers, algorithms n

Mesh refinement class AOMD_Ref. Callback { public : virtual int operator () (const mesh. Entity *) const = 0; virtual void callback (std: : list<mesh. Entity *> &before, std: : list<mesh. Entity *> &after ) const = 0; }; Conformal or not (hanging nodes or mixed meshes) n Typically n class my. AOMD_Ref. Callback : public AOMD_Ref. Callback; AOMD: : Ref. Unref(the. Mesh, my. AOMD_Ref. Callback);

Communications class AOMD_Round. Of. Comm { public : virtual char * send. Buffer (const mesh. Entity *, int dest_proc, size_t &sizebuf) const = 0; virtual char * recv. Buffer (const mesh. Entity *, int src_proc, size_t sizebuf) const = 0; }; Messages are packed (autopack) n Typically n class my. AOMD_Round. Of. Comm : public AOMD_Round. Of. Comm; AOMD: : round. Of. Comm(the. Mesh, my. AOMD_Round. Of. Comm);

Load balancing class AOMD_LBCallback { public : virtual char * send. Buffer (const mesh. Entity *, int dest_proc, size_t &sizebuf) const = 0; virtual char * recv. Buffer (const mesh. Entity *, int src_proc, size_t sizebuf) const = 0; }; Messages are packed (autopack) n Typically n class my. AOMD_LBCallback : public AOMD_LBCallback; AOMD: : LB(the. Mesh, my. AOMD_LBCallback);

Demonstration of Load Balancing Refined to moving level function - non-conforming triangular mesh

http: //www. scorec. rpi. edu/DG Desing specifications n Solve any conservation law, in any dimension, in parallel and adaptively, using any system of curvilinear coordinates, any discretization, any spatial basis and any time stepping scheme class Conservation. Law : fluxes, numerical fluxes fn and g, right hand side r, equation of state, initial and boundary conditions : the physics class Integrator : specialization's for geometrical elements class Metric : Euclidian, axisymmetric class Function. Space : Orthogonal or not, class Solver : forward Euler, Runge-Kutta, matrix free GMRES. Examples n Navier-Stokes, Euler, Burgers, Maxwell-Boltzmann. . .

Demonstration of Load Balancing

2 -D Examples n 2 D Rayleigh Taylor Red (heavy), blue (light), On left, 4 levels of refinement, On right, 2 levels of refinement, Faster growth with more refinement as expected

2 -D Animation of Instability Linear DG elements, 30, 000 to 800, 000 dof Atwood Number, A = 1/3 10 fourier modes in “random” distribution time for the bubble to reach top of the window (y = 0. 5) : 5 sec This calculation: alpha = 0. 06 Experiments: alpha = 0. 058 - 0. 065 Theory (Glimm, et al) alpha 0. 045 = 0. 06

Refined 3 -D Meshes for Rayleigh Taylor Instability non-conforming hexahedron mesh light fluid 24 steps of refinement 72 steps of refinement 104 steps of refinement heavy fluid

Rayleigh Taylor Instability 64 processors of the PSC alpha cluster 1 106 to 2. 0 107 dof’s 128 processors of Blue Horizon 108 dof’s

The cannon

Conclusions PAOMD advantages Quite small piece of software, documented n Focused, mesh management only n Asks for minimum user knowledge about parallel issues n Efficient implementation n Future work n n n Terascale computers, more than 1000 processors (in progress, ASCI & Sci. DAC projects) Anisotropic mesh refinement (in progress, with X Li) TSTT Mesh component Hardware heterogeneity, machine and network models have to be added in partitioners (in progress) Modification of the design, storing less