The Charm Par FUM Framework PARallel Framework for

The Charm++ Par. FUM Framework PARallel Framework for Unstructured Meshing presented by: Isaac Dooley Parallel Programming Lab University Illinois Urbana-Champaign Charm Workshop 2005 http: //charm. cs. uiuc. edu

Par. FUM Overview Charm Workshop 2005

Why use the Par. FUM Framework? • Publicly Available (included with Charm++) • Makes parallelizing a serial code faster and easier • Handles mesh partitioning • Handles communication • Handles load balancing (via Charm++) • Bindings for Fortran, C++, C • Provides Advanced features • Parallel Adaptivity and Incremental Mesh Modification • Visualization Support with Para. View or Net. FEM • Collision Detection Library • Iterative Matrix Solver Library (IFEM) Charm Workshop 2005 3

Features Overview • Originally Designed for Finite Element/Volume Problems • Parallel Partitioning • Field registration and updating for shared nodes, ghosts -- “User can own Data” • Arbitrary element types, and arbitrary user data associated to elements or nodes • Boundary condition representation for faces, edges and nodes • Efficient ID translation mechanism for communication • Mesh Modification Support Charm Workshop 2005 3 -D Fractography in Par. FUM 4 Rocket Burn Simulation, CSAR

Par. FUM Framework Users • CSAR • Rocflu: Fluids solver • Roc. Rem: Remeshing / Parallel Solution Transfer • CPSD • Space. Time meshing • Frac 3 D • Fracture Mechanics • Dendritic Growth • Metal Solidification process Charm Workshop 2005 5

Outline for the rest of this talk • • Par. FUM concepts & philosophy Program Structure Synchronization of Ghost & Shared entities Topological Relationships (Adjacencies) Parallel Mesh Adaptivity Obtaining & Compiling Par. FUM Libraries within Par. FUM • Collision Detection • Visualization Tools • One Sample Program Charm Workshop 2005 6

Par. FUM Concepts Charm Workshop 2005

Par. FUM Basics • Nodes • Points in the problem domain’s space. • Elements • Discretized areas or volumes representing portions of a problem domain • Arbitrary Dimension and Type • Each element is defined by a set of nodes • Data Attributes, can be associated with nodes or Elements • Data can be owned by either Framework or Application Charm Workshop 2005 8

Serial Mesh Element Surrounding Nodes E 1 N 3 N 4 E 2 N 1 N 2 N 4 E 3 N 2 N 4 N 5 Charm Workshop 2005 9

Partitioning • Partition the Par. FUM Mesh into multiple chunks • Distribute elements, replicate shared nodes and/or add ghosts • Keep track of communication • Partition so that communication is minimized Charm Workshop 2005 10

Partitioned Mesh Element Surrounding Nodes E 1 N 3 N 4 E 2 N 1 N 2 N 3 Element Surrounding Nodes E 1 N 2 N 3 Shared Nodes A B N 2 N 1 N 4 N 3 Charm Workshop 2005 11

Par. FUM Parallel Model: Shared Nodes • “Shared Node” model • Element computations based on values of surrounding nodes • Node values are sum of surrounding elements • Example: Mechanical Simulation • Element stresses are computed from element’s nodal displacements Charm Workshop 2005 12

Par. FUM Parallel Model: Ghosts • “Ghost” model • Element computations based only on values of surrounding nodes and elements • Example: Fluid Dynamics • Element pressures and velocities come from neighboring element pressures and velocities, and node locations Charm Workshop 2005 13

Virtualization • Charm++ Runtime System • Applications built using migratable objects • Each Par. FUM mesh chunk mapped to a migratable object. • Virtualization = multiple migratable objects per processor • Load Balancing • High(90 -100%) Processor Utilization and Scaling. User View System Implementation Charm Workshop 2005 14

Benefit of Virtualization to Structural Dynamics Applications Par. FUM Application Charm Workshop 2005 On Eight Physical Processors 15

Par. FUM Program Structure Charm Workshop 2005

Structure of an Par. FUM Application • Consists of at least two user-written subroutines • init • driver • init is called on chunk 0 • driver is called on every chunk Charm Workshop 2005 17

init() subroutine init read the serial mesh and configuration data inform the framework about the mesh end subroutine Charm Workshop 2005 18

driver() subroutine driver get local mesh chunk time loop Par. FUM computations communication more Par. FUM computations end time loop end subroutine Charm Workshop 2005 19

Structure of an Par. FUM Application init() driver Update Charm Workshop 2005 20

Data Management In Par. FUM • Framework Owns Data • User gives framework mesh in init using FEM_Mesh_Data() • User Accesses mesh by calling FEM_Mesh_Data() in driver to copy data into the user’s arrays • User Owns Data • User registers mesh with framework in init using FEM_Register_entity() and FEM_Register_array() • User uses their own data arrays in driver • User must also supply a resize function, to be called when framework partitions mesh and resizes mesh in adaptive refinement calls. Charm Workshop 2005 21

Setting/Getting Mesh Data void Mesh_data( int mesh, int entity, int attr, void *data, int first, int length, Get/Set, and multi-mesh support NODE, ELEM, SPARSE (+GHOST) DATA, CONN, SYM, GLOBALNO, … User data: width x length array Apply to fir st…first+length-1 User data formatting int datatype, int width ); Charm Workshop 2005 22

Mesh Access: Example 1 Mesh_data( FEM_Mesh_default_read(), () FEM_NODE, FEM_DATA+23, coord, 0, n. Nodes, FEM_DOUBLE, 3 Read from default mesh Access nodal data Some user defined attribute #23 User data: 3 x n. Nodes array Get data for nodes 0 to n. Nodes 3 doubles for each node ); Charm Workshop 2005 23

Mesh Access: Example 2 Mesh_data( FEM_Mesh_default_write(), () FEM_ELEM, FEM_CONN, conn, 0, n. Elem, FEM_INDEX_0, 3 ); Write to default mesh Modify element data Element Connectivity Array: 3 x n. Elem For elements 0 to n. Elem 3 integers for each element (0 -based C-style indexing) Charm Workshop 2005 24

Mesh Access: Example 3 Mesh_data( FEM_Mesh_default_read(), () FEM_ELEM+FEM_GHOST, FEM_CONN, conn, 0, n. Elem, FEM_INDEX_0, 3 ); Read from default mesh Access ghost element data Charm Workshop 2005 25

Par. FUM Ghost Elements Charm Workshop 2005

Ghost Elements: Overview • Most Par. FUM programs communicate via shared nodes • Some computations require read-only copies of remote elements—“ghosts” • Stencil-type finite volume computation • Many kinds of mesh modification Charm Workshop 2005 27

Ghosts: 2 D Example Serial Mesh 1 2 3 4 Right Chunk Left Chunk 1 2 Ghost of 3 Ghost of 2 Charm Workshop 2005 3 4 28

Defining Ghost Layers: • Add ghost elements layer-by-layer from init • A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes • For 2 D, a tuple might be a 2 -node edge • For 3 D, a tuple might be a 4 -node face • You specify a ghost layer with Add_ghost_layer(tuple. Size, ghost. Nodes) • ghost. Nodes indicates whether to add ghost nodes as well as ghost elements. Charm Workshop 2005 29

Ghosts: Node adjacency /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1, 0); /* 1 node per tuple */ const static int tri 2 node[]={0, 1, 2}; FEM_Add_ghost_elem(0, 3, tri 2 node); 1 2 0 Charm Workshop 2005 30

Ghosts: Edge adjacency /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2, 0); /* 2 nodes per tuple */ const static int tri 2 edge[]={0, 1, 2, 2, 0}; FEM_Add_ghost_elem(0, 3, tri 2 edge); 1 2 0 Charm Workshop 2005 31

Par. FUM Ghosts & Shared Node Synchronization Charm Workshop 2005

Node Fields • Framework handles combining data for shared nodes and keeps them in sync • Framework does not understand meaning of node fields, only their location and types • Framework needs to be informed of locations and types of fields • Create_field once, Update_field every timestep Charm Workshop 2005 33

Create a Field integer function FEM_Create_simple_field( datatype, len) integer, intent(in) : : datatype, len Charm Workshop 2005 34

Update Field: Shared Nodes Can be used to sum forces contributed to a node in FE Method subroutine FEM_Update_Field(fid, nodes) integer, intent(in) : : fid varies, intent(inout) : : nodes Charm Workshop 2005 35

Update Field: Ghosts Can be used to update ghost copies of local nodes(with current displacements) in FE Method subroutine Update_ghost_field(fid, el. Type, elts) integer, intent(in) : : fid, el. Type varies, intent(inout) : : elts Charm Workshop 2005 36

Topological Relationships: Adjacencies Charm Workshop 2005

Adjacencies • User registers element to node connectivity • Framework can derive other relationships: • Element to Element • Node to Node • Node to Element • User specifies a “tuple” to specify when two arbitrary elements are adjacent. The framework uses this to construct the other adjacencies if needed. Charm Workshop 2005 38

Adjacencies • User must explicitly have framework generate the adjacency tables const int triangle. Faces[6] = {0, 1, 1, 2, 2, 0}; FEM_Add_elem 2 face_tuples(mesh, 0, 2, 3, triangle. Faces); FEM_Mesh_create_elem_adjacency(mesh); FEM_Mesh_create_node_elem_adjacency(mesh); FEM_Mesh_create_node_adjacency(mesh); Charm Workshop 2005 39

Adjacencies • The adjacency tables are created as attributes which can be accessed via Mesh_Data() or by simple accessor functions. Get a list of elements adjacent to element e: e 2 e_get. All(int e, int *neighbors); Get a list of elements adjacent to node n: n 2 e_get. All(int n, int **adjelements, int *sz); Charm Workshop 2005 40

Adjacencies • Caveats or Limitations: • Currently some of the adjacency functions only work correctly for meshes containing one type of element. • Two nodes are adjacent if and only if the two nodes are listed in the connectivity for some element. Thus in a rectangular grid, two nodes on opposite corners of a rectangle will be listed as adjacent nodes. Charm Workshop 2005 41

How Do I get a copy of Par. FUM and Run a Par. FUM Application? Charm Workshop 2005

Par. FUM or FEM? • FEM is the old name for Par. FUM • FEM and Par. FUM are currently the same library in Charm++ • The FEM framework as described in the manual does not contain some new features: • Adjacency Data Structures, Adaptivity, Mesh Modification • The codebase currently uses the old naming conventions of FEM, functions start with “FEM_” • The current version of FEM/Par. FUM can no longer run apart from Charm++ Charm Workshop 2005 43

Where to Get It ? Par. FUM is included in Charm++ CVS distribution CSH: setenv CVSROOT ": pserver: checkout@charm. cs. uiuc. edu: /cvsroot" Or BASH: export CVSROOT=": pserver: checkout@charm. cs. uiuc. edu: /cvsroot" You should now be able to do a > cvs login (no password needed, just type [Enter] at prompt) and then > cvs co -P charm Par. FUM-FEM is in charm/src/libs/ck-libs/fem Charm Workshop 2005 44

How to Build It ? > cd charm and do >. /build LIBS net-linux -O This will make a net-linux directory, with bin, include, lib etc subdirectories. Charm Workshop 2005 45

How to Compile & Link ? • Use “charmc”: available under bin • a multi-lingual compiler driver, understands f 90 • Knows where modules and libraries are • Portable across machines and compilers • Linking • use “-language femf” : for F 90 • Use “–language fem” : for C/C++ • See example Makefiles • charm/examples/fem/… Charm Workshop 2005 46

How to Run ? • Charmrun • • A portable parallel job execution script Specify number of processors: +p. N Specify number of chunks: +vp. N Special “nodelist” file for net-* versions. /charmrun. /pgm +vp 100 +p 16 Charm Workshop 2005 47

Example Nodelist File: $(HOME)/. nodelist group main ++shell ssh host tur 0001. cs. uiuc. edu host tur 0002. cs. uiuc. edu host tur 0003. cs. uiuc. edu host tur 0004. cs. uiuc. edu . /charmrun pgm +p 4 +vp 70 Charm Workshop 2005 48

Par. FUM Parallel Mesh Adaptivity Charm Workshop 2005

Mesh Adaptivity in Par. FUM • • • 2 D refinement 2 D coarsening 3 D refinement 3 D coarsening -- coming soon Support for user defined mesh operations. Charm Workshop 2005 50

Mesh Modification Primitives • Par. FUM provides incremental asynchronous parallel mesh modificiation primitives: • • • Add_node() Remove_node() Add_element() Remove_element() Higher Level Operations are built using these primitives • • edge_bisect() edge_flip() edge_collapse() … Charm Workshop 2005 51

Par. FUM Structure SDG Application API (serial or parallel) Par. FUM Mesh Adaptivity Edge Flip, Edge Bisect, Edge Contract, … Mesh Adjacency e 2 e, e 2 n, n 2 e, n 2 n Generate, Modify … Mesh Modification Lock(), Unlock() Add/Remove Node() Add/Remove Element() Nodes: Elements: Local, Shared, Ghost Local, Ghost Charm Workshop 2005

Mesh Modification Examples Edge Flip: Remove elements e 1 Remove element e 2 Add element (n 1, n 2, n 4) Add element (n 2, n 3, n 4) Charm Workshop 2005 53

Mesh Modification Examples Edge Bisect: Remove elements e 1 Remove element e 2 Add node Add element (n 1, n 2, n 5) Add element (n 3, n 5, n 2) Add element (n 4, n 5, n 3) Add element (n 4, n 1, n 5) Charm Workshop 2005 54

Mesh Modification in Parallel Mesh on Processor 1 before edge flip Mesh on Processor 2 after edge flip Charm Workshop 2005 55

Mesh Modification in Par. FUM Primitive Operations must do the following: • Perform the operation on local and all applicable remote processors • Convert local nodes to shared nodes when they become part of the new boundary • Update ghost layers(nodes and elements) for all applicable processors. The ghost layers can grow or shrink Charm Workshop 2005 56

Parallel Mesh Refinement/Coarsening Parallel Refinement and Coarsening algorithms in action: A wave propagates through a block across 8 processors Charm Workshop 2005 57

Parallel 3 D Refinement • • Built from same primitives Longest edge based refinement Longest face based refinement Currently implementing 3 D coarsening Charm Workshop 2005 58

Refinement and Coarsening in a 2 D Par. FUM Application Shock propagation and reflection down the length of the bar Adaptive mesh modification to capture the shock propagation Charm Workshop 2005 59

Solution Comparison Initial Mesh Adaptive Fine Mesh Charm Workshop 2005 60

Par. FUM Collision Detection Charm Workshop 2005

Charm++ Collision Detect collisions/intersections/contacts between objects scattered across processors Built on Charm++ Arrays Overlay regular 3 D sparse grid of voxels (boxes) Send objects to all voxels they touch Collect collisions from each voxel Collision response is left to caller Charm Workshop 2005 62

Par. FUM Mesh Visualization Charm Workshop 2005

Mesh Visualization in Par. FUM • Currently we support: • Net. FEM Online Visualization • Net. FEM Offline Visualization • Para. View Offline Visualization Charm Workshop 2005 64

Net. FEM Client: pretty pictures Wave dispersion off a crack (simplified frac 3 d) Charm Workshop 2005 65

Par. FUM Unified Visualization Interface n=Net. FEM_Begin(…); Start output for one timestep, specifying Output method(online/offline) Net. FEM_Nodes(…); Net. FEM_Vector(…); Specify number of nodes and coord. Give an array of nodal vector data Give a second array of nodal data Net. FEM_Elements(…); Net. FEM_Scalar(…); Specify number of elements and conn. Give an array of scalar element data Give a second array of element data Give a third array of element data Net. FEM_End(n); Charm Workshop 2005 66

Par. FUM Unified Visualization Interface Net. FEM n=Net. FEM_Begin(2, t, Net. FEM_POINTAT); Net. FEM_Nodes(n, nnodes, (double *)g. coord, "Position (m)"); Net. FEM_Vector(n, (double *)g. d, "Displacement (m)"); Net. FEM_Vector(n, (double *)g. v, "Velocity (m/s)"); Net. FEM_Elements(n, nelems, 3, (int *)g. conn, "Triangles"); Net. FEM_Scalar(n, g. S 11, 1, "X Stress (pure)"); Net. FEM_Scalar(n, g. S 22, 1, "Y Stress (pure)"); Net. FEM_Scalar(n, g. S 12, 1, "Shear Stress (pure)"); Net. FEM_End(n); Charm Workshop 2005 67

Net. FEM Online Visualization • To allow the Net. FEM client to connect, you add Net. FEM registration calls to your server • Register nodes and element types • Register data items: scalars or spatial vectors associated with each node or element • You provide the display name and units for each data item • Link your program with “-module netfem” • Run with “++server”, and connect! Charm Workshop 2005 68

Net. FEM: Setup • n=Net. FEM_Begin(FEM_My_partition(), timestep, dim, Net. FEM_POINTAT) • Call this each time through your timeloop; or skip • timestep identifies this data update • dim is the spatial dimension—must be 2 or 3 • Returns a Net. FEM handle n used by everything else • Net. FEM_POINTAT for online visualization • Net. FEM_End(n) • Finishes update n Charm Workshop 2005 69

Net. FEM: Nodes • Net. FEM_Nodes(n, nnodes, coord, ”Position (m)”) • Registers node locations with Net. FEM—future vectors and scalars will be associated with nodes • n is the handle returned by Net. FEM_Begin • nnodes is the number of nodes • coord is a dim by nnodes array of doubles • The string describes the coordinate system and meaning of nodes • Currently, there can only be one call to nodes Charm Workshop 2005 70

Net. FEM: Node Displacement Charm Workshop 2005 71

Net. FEM: Elements • Net. FEM_Elements(n, nelem, nodeper, conn, ”Triangles”) • Registers elements with Net. FEM—future vectors and scalars will be associated with these elements • n is the handle returned by Net. FEM_Begin • nelem is the number of elements • nodeper is the number of nodes per element • conn is a nodeper by nelem array of node indices • The string describes the kind of element • Repeat to register several kinds of element • Perhaps: Triangles, squares, pentagons, … Charm Workshop 2005 72

Net. FEM: Element Stress Charm Workshop 2005 73

Net. FEM: Vectors • Net. FEM_Vector(n, val, ”Displacement (m)”) • Registers a spatial vector with each node or element • Whichever kind was registered last • n is the handle returned by Net. FEM_Begin • val is a dim by nitems array of doubles • There’s also a more general Net. FEM_Vector_field in the manual • The string describes the meaning and units of the vectors • Repeat to register multiple sets of vectors • Perhaps: Displacement, velocity, acceleration, rotation, … Charm Workshop 2005 74

Net. FEM: Element Stress Charm Workshop 2005 75

Net. FEM: Element Velocity Charm Workshop 2005 76

Net. FEM: Zoom in Charm Workshop 2005 77

Net. FEM: Outline Elements Charm Workshop 2005 78

Offline Visualization with Net. FEM • Call Net. FEM_Begin() with Net. FEM_WRITE, • Par. FUM application will then write out binary output dump files to a new directory called “Net. FEM” containing directories for each timestep • Visualize with command netfem Net. FEM/10 • In offline mode, the ``update'' button fetches the next extant timestep directory. Charm Workshop 2005 79

Offline Visualization with Para. View • The directory Net. FEM can be converted to a Para. View/VTK compatible XML format. • Converter can be built by calling “make” in charm/tmp/libs/ck-libs/netfem/Paraview. Converter/ • Run the resulting program Net. FEM_To_Paraview to convert the Net. FEM directory to a new directory called Para. View. Data. • Para. View can open the files inside this directory either as individual chunks, or as the entire mesh. • Scan through the timesteps using the standard timestep slider. Charm Workshop 2005 80

Offline Visualization with Para. View • Limitations: • Multiple element types with varying numbers of data attributes may not work • Current converter produces ASCII XML files, not binary, so they use excessive disk space. • Benefits: • Advanced visualization functionality • More stable than offline Net. FEM which crashes if you try to access non-existent timesteps, or huge meshes Charm Workshop 2005 81

Migration and Load Balancing Charm Workshop 2005

Advanced API: Migration • Chunks may not be computationally equal: Results in load imbalance • Multiple chunks per processor • Chunks cannot have writable global data • Automatic load balancing • Migrate chunks to balance load • How to migrate allocated data for chunks ? • Embed it in a user-defined type Charm Workshop 2005 83

Chunk Data Example MODULE my_block_mod TYPE my_block INTEGER : : n 1, n 2 x, n 2 y REAL*8, POINTER, DIMENSION(: , : ) : : arr END TYPE END MODULE Charm Workshop 2005 84

Pack/Unpack (PUP) Routine SUBROUTINE pup_my_block(p, m) USE my_block_mod USE pupmod INTEGER : : p TYPE(my_block) : : m call fpup_int(p, m%n 1) call fpup_int(p, m%n 2 x) call fpup_int(p, m%n 2 y) IF (fpup_is. Unpacking(p)) THEN ALLOCATE(m%arr(m%n 2 x, m%n 2 y)) END IF call fpup_doubles(p, m%arr, m%n 2 x*m%n 2 y) IF (fpup_is. Deleting(p)) THEN DEALLOCATE(m%arr) END IF END SUBROUTINE Charm Workshop 2005 85

Registering Chunk Data !- Fortran driver subroutine use my_block_mod interface subroutine pup_my_block(p, m) use my_block_mod INTEGER : : p TYPE(my_block) : : m end subroutine end interface TYPE(my_block) : : m CALL FEM_Register(m, pup_my_block) Charm Workshop 2005 86

Migration • Every chunk driver calls FEM_Migrate() • Framework calls PUP • for getting the size of packed data • For packing data • Chunk migrates to new processor • Framework calls PUP for unpacking • Driver returns from FEM_Migrate() Charm Workshop 2005 87

A sample Par. FUM Program Charm Workshop 2005

Simple 2 D • Can be found in charm/examples/fem/simple 2 D • Generates simple but interesting visualizations • A constrained triangle Finite Element Code • Here we will show the init and driver routines, almost exactly copied from pgm. C, omitting some comments and trivial code fragments. Charm Workshop 2005 89

extern "C" void init(void) { Ck. Printf("init startedn"); double start. Time=Cmi. Wall. Timer(); int n. Pts=0; //Number of nodes vector 2 d *pts=0; //Node coordinates int n. Ele=0; conn. Rec *ele=NULL; // Omitted for readability: // Read in mesh from users files to ele and pts // Tell framework we will be writing to the mesh int fem_mesh=FEM_Mesh_default_write(); Charm Workshop 2005 90

FEM_Mesh_data( fem_mesh, // Add nodes to the current mesh FEM_NODE, // We are registering nodes FEM_DATA+0, // Register the point locations which (double *)pts, // The array of point locations 0, // First node is 0 n. Pts, // The number of points FEM_DOUBLE, // Coordinates are doubles 2); // Points have dimension 2 (x, y) FEM_Mesh_data( fem_mesh, // Add elements to the current mesh FEM_ELEM+0, // Element type 0 FEM_CONN, // Register the connectivity table for this (int *)ele, // The array of point locations 0, // 0 based indexing n. Ele, // The number of elements FEM_INDEX_0, // We use zero based node numbering 3); // Elements have three nodes Charm Workshop 2005 91

delete[] ele; delete[] pts; Ck. Printf("Finished with initn"); } Charm Workshop 2005 92

extern "C" void driver(void){ int nnodes, nelems, ignored; int i, my. Id=FEM_My_partition(); my. Globals g; FEM_Register(&g, (FEM_Pup. Fn)pup_my. Globals); int mesh=FEM_Mesh_default_read(); // Get node data nnodes=FEM_Mesh_get_length(mesh, FEM_NODE); g. coord=new vector 2 d[nnodes]; FEM_Mesh_data(mesh, FEM_NODE, FEM_DATA+0, (double*)g. coord, 0, nnodes, FEM_DOUBLE, 2); Charm Workshop 2005 93

// Read element data from framework nelems=FEM_Mesh_get_length(mesh, FEM_ELEM+0); g. nnodes=nnodes; g. nelems=nelems; g. conn=new conn. Rec[nelems]; g. S 11=new double[nelems]; g. S 22=new double[nelems]; g. S 12=new double[nelems]; FEM_Mesh_data(mesh, FEM_ELEM+0, FEM_CONN, (int *)g. conn, 0, nelems, FEM_INDEX_0, 3); Charm Workshop 2005 94

//Initialize associated data g. R_net=new vector 2 d[nnodes]; //Net force g. d=new vector 2 d[nnodes]; //Node displacement g. v=new vector 2 d[nnodes]; //Node velocity g. a=new vector 2 d[nnodes]; //Node acceleration for (i=0; i<nnodes; i++) g. R_net[i]=g. d[i]=g. v[i]=g. a[i]=vector 2 d(0. 0); int fid=FEM_Create_simple_field(FEM_DOUBLE, 2); Charm Workshop 2005 95

int t. Steps=5000; for (int t=0; t<t. Steps; t++) { //Structural mechanics //Compute forces on nodes exerted by elements CST_NL(g. coord, g. conn, g. R_net, g. d, mat. Const, nnodes, nelems, g. S 11, g. S 22, g. S 12); //Communicate net force on shared nodes. FEM_Update_field(fid, g. R_net); //Advance node positions advance. Nodes(dt, nnodes, g. coord, g. R_net, g. a, g. v, g. d, 0); /* perform migration-based load balancing */ if (t%1024==0) FEM_Migrate(); Charm Workshop 2005 96

if (t%1024==0) { //Publish data for Visualization Net. FEM n=Net. FEM_Begin(FEM_My_partition(), t, 2, Net. FEM_POINTAT); Net. FEM_Nodes(n, nnodes, (double*)g. coord, "Position (m)"); Net. FEM_Vector(n, (double*)g. d, "Displacement (m)"); Net. FEM_Vector(n, (double*)g. v, "Velocity (m/s)"); Net. FEM_Elements(n, nelems, 3, (int*)g. conn, "Triangles"); Net. FEM_Scalar(n, g. S 11, 1, "X Stress (pure)"); Net. FEM_Scalar(n, g. S 22, 1, "Y Stress (pure)"); Net. FEM_Scalar(n, g. S 12, 1, "Shear Stress (pure)"); Net. FEM_End(n); } } // end timeloop }// end driver Charm Workshop 2005 97

Conclusions Charm Workshop 2005

Conclusions: Par. FUM • Par. FUM is a functional, freely available Unstructured Mesh Framework. • Par. FUM has a wide range of useful features: • Load Balancing, Fault Tolerance, … • Parallel Partitioning • Parallel Incremental Adaptivity, Remeshing, and Refinement • Visualization Tools • Collision/Contact Detection Library • Matrix Solver Charm Workshop 2005 99

The End Thanks Charm Workshop 2005