Ghost Elements Ghost Elements Overview n Most FEM

Ghost Elements

Ghost Elements: Overview n Most FEM programs communicates via shared nodes, using FEM_Update_field n Some computations require read-only copies of remote elements—“ghosts” Stencil-type finite volume computations n The push form of matrix-vector product n Many kinds of mesh modification n n Ghosts are a recent addition to the FEM framework

Ghosts: 2 D Example Serial Mesh 1 2 3 4 Right Chunk Left Chunk 1 2 Ghost of 3 Ghost of 2 3 4

Building Ghosts: n Add ghost elements layer-by-layer from init n A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes n For 2 D, a tuple might be a 2 -node edge n For 3 D, a tuple might be a 4 -node face n You specify a ghost layer with FEM_Add_ghost_layer(tuple. Size, ghost. Nodes) n ghost. Nodes indicates whether to add ghost nodes as well as ghost elements.

Building Ghosts: n FEM_Add_ghost_elem(e, t, elem 2 tuple) ne is the element type n t is the number of tuples per element n elem 2 tuple maps an element to its tuples: A tuple. Size by t array of integers n Contains element-local node numbers n n Repeat this call for each ghost element type

Ghosts: Node adjacency /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1, 0); /* 1 node per tuple */ const static int tri 2 node[]={0, 1, 2}; FEM_Add_ghost_elem(0, 3, tri 2 node); 1 0 2

Ghosts: Edge adjacency /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2, 0); /* 2 nodes per tuple */ const static int tri 2 edge[]={0, 1, 2, FEM_Add_ghost_elem(0, 3, tri 2 edge); 1 0 2, 0}; 2

Extracting and Using Ghosts n Ghosts are always given larger numbers than non-ghosts—that is, ghosts are at the end n FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) n 0 Return the index of the first ghost node or element g n FEM_Update_ghost_field(fid, e, data) n e Obtain other processor’s data (formatted like fid) for each ghost element of type e

Ghost Elements: Sub-Mesh

Ghost Elements

Ghosts and Symmetries

Ghosts and Symmetries n In addition to cross-processor ghosts, can build ghosts to model problem symmetries Translational and rotational periodicities n Mirror symmetry n n FEM_Add_linear_periodicity(n. Faces, n. Per, faces. A, faces. B, n. Nodes, node. Locs) n Identify these two lists of faces under linear periodicity, and build ghosts to match

Symmetry Ghosts: 2 D Example Serial Mesh 1 2 3 4 Horizontal Periodicity Left Chunk Sym. Ghost 4 1 2 Right Chunk Ghost of 3 Ghost of 2 3 4 Sym. Ghost 1

Symmetry Ghost Elements

Symmetry-Ghost Elements

Ghosts and Symmetries: Update

Net. FEM Client

Net. FEM Client: Pretty pictures of wave dispersion around a crack

Net. FEM: Zoom in

Net. FEM: Outline Elements

Net. FEM: Point Nodes

Net. FEM Server

Net. FEM Server Side: Overview n To allow the Net. FEM client to connect, you add Net. FEM registration calls to your server Register nodes and element types n Register data items: scalars or spatial vectors associated with each node or element n You provide the display name and units for each data item n n Link your program with “-module netfem” n Run with “++server”, and connect!

Net. FEM Server Side: Setup n n=Net. FEM_Begin(FEM_My_partition(), timestep, dim, Net. FEM_POINTAT) Call this each time through your timeloop; or skip n timestep identifies this data update n dim is the spatial dimension—must be 2 or 3 n Returns a Net. FEM handle n used by everything else n n Net. FEM_End(n) n Finishes update n

Net. FEM Server Side: Nodes n Net. FEM_Nodes(n, nnodes, coord, ”Position (m)”) Registers node locations with Net. FEM—future vectors and scalars will be associated with nodes n n is the handle returned by Net. FEM_Begin n nnodes is the number of nodes n coord is a dim by nnodes array of doubles n The string describes the coordinate system and meaning of nodes n n Currently, there can only be one call to nodes

Net. FEM: Node Displacement

Net. FEM Server Side: Elements n Net. FEM_Elements(n, nelem, nodeper, conn, ”Triangles”) n n n n Registers elements with Net. FEM—future vectors and scalars will be associated with these elements n is the handle returned by Net. FEM_Begin nelem is the number of elements nodeper is the number of nodes per element conn is a nodeper by nelem array of node indices The string describes the kind of element Repeat to register several kinds of element n Perhaps: Triangles, squares, pentagons, …

Net. FEM: Element Stress

Net. FEM Server Side: Vectors n Net. FEM_Vector(n, val, ”Displacement (m)”) n Registers a spatial vector with each node or element n n is the handle returned by Net. FEM_Begin val is a dim by nitems array of doubles n n n Whichever kind was registered last There’s also a more general Net. FEM_Vector_field in the manual The string describes the meaning and units of the vectors Repeat to register multiple sets of vectors n Perhaps: Displacement, velocity, acceleration, rotation, …

Net. FEM: Element Velocity

Net. FEM Server Side: Scalars n Net. FEM_Scalar(n, val, s, ”Displacement (m)”) n Registers s scalars with each node or element n n is the handle returned by Net. FEM_Begin val is an s by nitems array of doubles n n Whichever kind was registered last There’s also a more general Net. FEM_Scalar_field in the manual n s is the number of doubles for each node or element n The string describes the meaning and units of the scalars Repeat to register multiple sets of scalars n Perhaps: Stress, plasticity, node type, damage, …

Net. FEM Server Side: 2 D Example integer : : t, n, numnp, numel real*8, dimension(2, numnp) : : coor, d, v, a integer, dimension(3, numel) : : conn n=Net. FEM_Begin(FEM_My_partition(), t, 2, Net. FEM_POINTAT) CALL Net. FEM_Nodes(n, numnp, coor, 'Position (m)') CALL Net. FEM_Vector(n, d, 'Displacement (m)') CALL Net. FEM_Vector(n, v, 'Velocity (m/s)') CALL Net. FEM_Vector(n, a, 'Acceleration (m/s^2)') CALL Net. FEM_Elements(n, numel, 3, conn, 'Triangles') CALL Net. FEM_Scalar(n, stress, 1, 'Stress (pure)') CALL Net. FEM_End(n)

Net. FEM: Conclusion n Easy, general way to get output from an FEM computation n Client configures itself based on server n Client can be run anywhere (from home!) n Server performance impact minimal (1 s!) n Future work: Support multiple chunks per processor n Non-network, file-based version n Movie mode n

Multiple Modules

Multiple Modules n Use of 2 or more CHARM++ frameworks in the same program n n n FEM—multiple unstructured mesh chunks MBLOCK—multiple structured mesh blocks AMPI—Adaptive MPI-on-Charm++ All based on the Threaded CHARM++ framework (TCHARM) For example, we may want to use AMPI in our FEM program for exchanging information between FEM chunks

Details Can compose FEM programs with other modules by just calling that module’s attach routine from init() n For example: n void init(void) { //Start AMPI, to allow drivers to use MPI calls: MPI_Attach(“my. AMPIFEM”); //. . Use FEM_Set() calls as usual. . }

Example #include ‘fem. h’ #include ‘mpi. h’ void driver(void) { //. . use FEM_Get calls as usual. . //Broadcast “data” from chunk 0: MPI_Bcast(&data, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD); //. . . timeloop: Use FEM_Update_field calls as usual. . . if (data. To. Send) MPI_Send(&data, 4, MPI_INT, dest, tag, MPI_COMM_WORLD); else MPI_Recv(&data, 4, MPI_INT, src, tag, MPI_COMM_WORLD, &status); }

Multiple Modules: Conclusion n Easy to use other modules from FEM framework n n Just call MPI_Attach from init, and link with “-module ampi” We could also have specified how to combine frameworks by writing a special startup routine named TCHARM_User_setup() n n n Not FEM-centric: overrides the normal call to init() Allows you to call main computation routine something other than driver() See TCHARM manual for details