Charm FEM Framework Tutorial Orion Sky Lawlor olawloruiuc

  • Slides: 121
Download presentation
Charm++ FEM Framework Tutorial Orion Sky Lawlor olawlor@uiuc. edu 2003/10/20 1

Charm++ FEM Framework Tutorial Orion Sky Lawlor olawlor@uiuc. edu 2003/10/20 1

Roadmap Why use FEM? n FEM Concepts n FEM Basic Calls n FEM Advanced

Roadmap Why use FEM? n FEM Concepts n FEM Basic Calls n FEM Advanced Calls n Extra Features n 2

n Why use FEM? 3

n Why use FEM? 3

Why use the FEM Framework? n Makes parallelizing a serial code faster and easier

Why use the FEM Framework? n Makes parallelizing a serial code faster and easier Handles mesh partitioning n Handles communication n Handles load balancing (via Charm) n n Allows extra features IFEM Matrix Library n Net. FEM Visualizer n Collision Detection Library n 4

Why not use the FEM Framework? n Does not help you write serial code

Why not use the FEM Framework? n Does not help you write serial code n n Another thing to learn n n But it does help with parallel But it has a manual and examples Another thing to break n But it runs on native MPI as well as AMPI 5

Scalability of FEM Framework 6

Scalability of FEM Framework 6

“Overhead” of Multipartitioning 7

“Overhead” of Multipartitioning 7

Load balancer in action Automatic Load Balancing in Crack Propagation 1. Elements Added 3.

Load balancer in action Automatic Load Balancing in Crack Propagation 1. Elements Added 3. Chunks Migrated 2. Load Balancer Invoked 8

FEM Framework Users: Frac 3 D fracture mechanics n Basic structures code, but with

FEM Framework Users: Frac 3 D fracture mechanics n Basic structures code, but with “cohesive elements” for fracture n Authors: Scott Brietenfeld, Philippe Geubelle n 9

FEM Framework Users: CSAR Rocflu fluids solver, a part of GENx n Finite-volume fluid

FEM Framework Users: CSAR Rocflu fluids solver, a part of GENx n Finite-volume fluid dynamics code n Uses FEM ghost elements n Author: Andreas Haselbacher n 10 Robert Fielder, Center for Simulation of Advanced Rockets

FEM Framework Users: DG Dendritic Growth n Simulate metal solidification process n Solves mechanical,

FEM Framework Users: DG Dendritic Growth n Simulate metal solidification process n Solves mechanical, thermal, fluid, and interface equations n Implicit, uses Bi. CG n Adaptive 3 D mesh n Authors: Jung-ho Jeong, John Danzig n 11

Net. FEM Client: pretty pictures n Wave dispersion off a crack (simplified frac 3

Net. FEM Client: pretty pictures n Wave dispersion off a crack (simplified frac 3 d) 12

n FEM Concepts 13

n FEM Concepts 13

FEM Basics FEM programs manipulate elements and nodes n Element is a portion of

FEM Basics FEM programs manipulate elements and nodes n Element is a portion of problem domain, surrounded by nodes n Node is one point in the domain n 14

Serial FEM Mesh Elemen t Surrounding Nodes E 1 N 3 N 4 E

Serial FEM Mesh Elemen t Surrounding Nodes E 1 N 3 N 4 E 2 N 1 N 2 N 4 E 3 N 2 N 4 N 5 15

Partitioned Mesh Element Surrounding Nodes E 1 N 3 N 4 E 2 N

Partitioned Mesh Element Surrounding Nodes E 1 N 3 N 4 E 2 N 1 N 2 N 3 Element Surrounding Nodes E 1 N 2 N 3 Shared Nodes A B N 2 N 1 N 4 N 3 16

FEM Parallel Model: Shared Nodes n “Shared Node” model Element computations based on values

FEM Parallel Model: Shared Nodes n “Shared Node” model Element computations based on values of surrounding nodes n Node values are sum of surrounding elements n n Example: Mechanical Simulation Element stresses are computed from locations of element’s nodes n Sum up forces at nodes n 17

FEM Mesh: Node Communication Summing forces from other processors only takes one call: FEM_Update_field

FEM Mesh: Node Communication Summing forces from other processors only takes one call: FEM_Update_field Adds values from shared nodes 18

FEM Parallel Model: Ghosts n “Ghost” model n n Element computations based only on

FEM Parallel Model: Ghosts n “Ghost” model n n Element computations based only on values of surrounding nodes and elements Example: Fluid Dynamics n Element pressures and velocities come from neighboring element pressures and velocities, and node locations 19

FEM Mesh: Ghosts Serial Mesh 1 2 3 4 Partitioning Right Chunk Left Chunk

FEM Mesh: Ghosts Serial Mesh 1 2 3 4 Partitioning Right Chunk Left Chunk 1 2 Ghost of 3 Communication Ghost of 2 3 4 20

FEM Program Structure read mesh, connectivity, boundary conditions time loop element loop- Element deformation

FEM Program Structure read mesh, connectivity, boundary conditions time loop element loop- Element deformation applies forces to surrounding nodes node loop- Forces and boundary conditions change node positions end time loop write out mesh data for postprocessing 21

Parallelization Partition the FEM Mesh into multiple chunks n Distribute elements, replicate shared nodes

Parallelization Partition the FEM Mesh into multiple chunks n Distribute elements, replicate shared nodes and/or add ghosts n n n Keep track of communication Partition so that communication is minimized 22

Parallel FEM Program read/get chunk mesh data, connectivity, shared nodes chunk time loop element

Parallel FEM Program read/get chunk mesh data, connectivity, shared nodes chunk time loop element loop- Element deformation applies forces to surrounding nodes <update forces on shared nodes> node loop- Forces and boundary conditions change node positions end time loop write chunk mesh data for postprocessing 23

FEM Framework Program n Consists of at least two userwritten subroutines init n driver

FEM Framework Program n Consists of at least two userwritten subroutines init n driver n init is called on processor 0 n driver is called on every chunk n 24

init subroutine init read the serial mesh and configuration data inform the framework about

init subroutine init read the serial mesh and configuration data inform the framework about the mesh end subroutine 25

driver subroutine driver get local mesh chunk time loop FEM computations communication more FEM

driver subroutine driver get local mesh chunk time loop FEM computations communication more FEM computations end time loop end subroutine 26

Structure of an FEM Application init() driver Update 27

Structure of an FEM Application init() driver Update 27

A Serial Program Processing Input Output 28

A Serial Program Processing Input Output 28

A Parallel Framework Program Parallel Processing Output Input Parallel Infrastructure 29

A Parallel Framework Program Parallel Processing Output Input Parallel Infrastructure 29

Real Names of Pieces Driver Finalize or Mesh_updated Init Charm++ FEM Framework 30

Real Names of Pieces Driver Finalize or Mesh_updated Init Charm++ FEM Framework 30

d n a w e N ! d e v o r p Im

d n a w e N ! d e v o r p Im FEM Mesh Access Calls 31

Old FEM Mesh Access Routines FEM_Set_elem_data FEM_Set_elem_conn FEM_Get_elem_data FEM_Get_elem_conn FEM_Get_elem_ghost FEM_Set_node_data FEM_Get_node_data FEM_Get_node_ghost FEM_Set_sparse_elem

Old FEM Mesh Access Routines FEM_Set_elem_data FEM_Set_elem_conn FEM_Get_elem_data FEM_Get_elem_conn FEM_Get_elem_ghost FEM_Set_node_data FEM_Get_node_data FEM_Get_node_ghost FEM_Set_sparse_elem FEM_Get_sparse_length FEM_Get_sparse FEM_Get_sym Also, F 90 versions of each Missing: FEM_Get_sparse_ghost 32

New FEM Mesh Access Routine (!) void FEM_Mesh_data( int mesh, Get/Set, and multi-mesh support

New FEM Mesh Access Routine (!) void FEM_Mesh_data( int mesh, Get/Set, and multi-mesh support int entity, NODE, ELEM, SPARSE (+GHOST) int attr, DATA, CONN, SYM, GLOBALNO, … void *data, User data: width x length array int first, int length, Apply to first…first+length-1 int datatype, int width User data formatting ); 33

New Mesh Access: Example FEM_Mesh_data( mesh, e. g. , FEM_Mesh_default_read() FEM_NODE, We’re changing node

New Mesh Access: Example FEM_Mesh_data( mesh, e. g. , FEM_Mesh_default_read() FEM_NODE, We’re changing node values FEM_DATA+23, User data (tag 23) coord, An array of 3 x n. Nodes doubles 0, n. Nodes, Change all the nodes 3 FEM_DOUBLE, 3 3 doubles/node X Y ); n. Nodes X X X Y Y Y Z Z Z 34

New Mesh Access: Advantages n n n Novice users have only one routine to

New Mesh Access: Advantages n n n Novice users have only one routine to learn User’s read and write routines can be similar or even identical (just like PUP) Smart users can easily abstract over parameters n n E. g. , “higher-order function” that sets both ghosts and real elements for any entity type Library developers can easily, naturally add: n n New entities: FEM_SPARSE, FEM_GHOST New data: FEM_COORD, 35 FEM_NODE_PRIMARY

New Mesh Access: Disadvantages n n 8 parameters is too many to keep straight

New Mesh Access: Disadvantages n n 8 parameters is too many to keep straight Invalid combinations detected at runtime, instead of compile time as with the old way n n E. g. , FEM_NODE doesn’t have FEM_CONN, so there’s no FEM_Set_node_conn Solution: Keep the old routines n n n But implement them in terms of the new routine Mix and match with new routine Needed for backward compatability, too 36

FEM Communication Calls 37

FEM Communication Calls 37

Node Fields n n Framework handles combining data for shared nodes and keeps them

Node Fields n n Framework handles combining data for shared nodes and keeps them in sync Framework does not understand meaning of node fields, only their location and types Framework needs to be informed of locations and types of fields Create_field once, Update_field every timestep 38

Create a Field integer function FEM_Create_simple_field( datatype, len) integer, intent(in) : : datatype, len

Create a Field integer function FEM_Create_simple_field( datatype, len) integer, intent(in) : : datatype, len 39

Update Field: Shared Nodes subroutine FEM_Update_Field(fid, nodes) integer, intent(in) : : fid varies, intent(inout)

Update Field: Shared Nodes subroutine FEM_Update_Field(fid, nodes) integer, intent(in) : : fid varies, intent(inout) : : nodes 40

FEM Ghost Layers 41

FEM Ghost Layers 41

Ghost Elements: Overview Most FEM programs communicates via shared nodes, using FEM_Update_field n Some

Ghost Elements: Overview Most FEM programs communicates via shared nodes, using FEM_Update_field n Some computations require readonly copies of remote elements— “ghosts” n Stencil-type finite volume computations n The push form of matrix-vector product n Many kinds of mesh modification n n Ghosts are a recent addition to the FEM framework 42

Ghosts: 2 D Example Serial Mesh 1 2 3 4 Right Chunk Left Chunk

Ghosts: 2 D Example Serial Mesh 1 2 3 4 Right Chunk Left Chunk 1 2 Ghost of 3 Ghost of 2 3 4 43

Building Ghosts: n n n Add ghost elements layer-by-layer from init A chunk will

Building Ghosts: n n n Add ghost elements layer-by-layer from init A chunk will include ghosts of all the elements it is connected to by “tuples”— sets of nodes For 2 D, a tuple might be a 2 -node edge For 3 D, a tuple might be a 4 -node face You specify a ghost layer with FEM_Add_ghost_layer(tuple. Size, ghost. Node s) ghost. Nodes indicates whether to add ghost nodes as well as ghost elements. 44

Building Ghosts: n n FEM_Add_ghost_elem(e, t, elem 2 tuple) e is the element type

Building Ghosts: n n FEM_Add_ghost_elem(e, t, elem 2 tuple) e is the element type t is the number of tuples per element elem 2 tuple maps an element to its tuples: n n n A tuple. Size by t array of integers Contains element-local node numbers Repeat this call for each ghost element type 45

Ghosts: Node adjacency /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1, 0); /* 1

Ghosts: Node adjacency /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1, 0); /* 1 node per tuple */ const static int tri 2 node[]={0, 1, 2}; FEM_Add_ghost_elem(0, 3, tri 2 node); 1 2 0 46

Ghosts: Edge adjacency /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2, 0); /* 2

Ghosts: Edge adjacency /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2, 0); /* 2 nodes per tuple */ const static int tri 2 edge[]={0, 1, 2, 2, 0}; FEM_Add_ghost_elem(0, 3, tri 2 edge); 1 2 0 47

Extracting and Using Ghosts n n Ghosts are always given larger numbers than non-ghosts—that

Extracting and Using Ghosts n n Ghosts are always given larger numbers than non-ghosts—that is, ghosts are at the end FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) n 0 n Return the index of the first ghost node or element g e FEM_Update_ghost_field(fid, e, data) n Obtain other processor’s data (formatted like fid) for each ghost element of type e 48

Update Field: Ghosts subroutine FEM_Update_ghost_field(fid, el. Type, elts) integer, intent(in) : : fid, el.

Update Field: Ghosts subroutine FEM_Update_ghost_field(fid, el. Type, elts) integer, intent(in) : : fid, el. Type varies, intent(inout) : : elts 49

Ghost Example: Mesh 50

Ghost Example: Mesh 50

Ghost Example: Ghost Elements 51

Ghost Example: Ghost Elements 51

FEM Installation 52

FEM Installation 52

Where to Get It ? FEM Framework is included in Charm++ distribution, available under

Where to Get It ? FEM Framework is included in Charm++ distribution, available under CVS CSH: setenv CVSROOT ": pserver: checkout@charm. cs. uiuc. edu: /cvsroot" Or BASH: export CVSROOT=": pserver: checkout@charm. cs. uiuc. edu: /cvsroot" You should now be able to do a > cvs login (no password needed, just type [Enter] at prompt) and then > cvs co -P charm to get the entire Charm++ source: FEM is in charm/src/libs/ck-libs 53

How to Build It ? > cd charm and do >. /build FEM net-linux

How to Build It ? > cd charm and do >. /build FEM net-linux -O This will make a net-linux directory, with bin, include, lib etc subdirectories. Platforms: net-sol, mpi-origin, mpi-linux etc. 54

How to Compile & Link ? n Use “charmc”: available under bin n n

How to Compile & Link ? n Use “charmc”: available under bin n n Linking n n n a multi-lingual compiler driver, understands f 90 Knows where modules and libraries are Portable across machines and compilers use “-language femf” : for F 90 Use “–language fem” : for C/C++ See example Makefiles n charm/pgms/charm++/fem/… 55

How to Run ? n Charmrun A portable parallel job execution script n Specify

How to Run ? n Charmrun A portable parallel job execution script n Specify number of processors: +p. N n Specify number of chunks: +vp. N n Special “nodelist” file for net-* versions n 56

Example Nodelist File: $(HOME)/. nodelist group main host tur 0001. cs. uiuc. edu host

Example Nodelist File: $(HOME)/. nodelist group main host tur 0001. cs. uiuc. edu host tur 0002. cs. uiuc. edu host tur 0003. cs. uiuc. edu etc… . /charmrun pgm +p 4 57

Advanced FEM Calls 58

Advanced FEM Calls 58

Advanced: FEM Migration 59

Advanced: FEM Migration 59

Advanced API: Migration n n Chunks may not be computationally equal: Results in load

Advanced API: Migration n n Chunks may not be computationally equal: Results in load imbalance Multiple chunks per processor Chunks cannot have writable global data Automatic load balancing n n Migrate chunks to balance load How to migrate allocated data for chunks ? n Embed it in a user-defined type 60

Chunk Data Example MODULE my_block_mod TYPE my_block INTEGER : : n 1, n 2

Chunk Data Example MODULE my_block_mod TYPE my_block INTEGER : : n 1, n 2 x, n 2 y REAL*8, POINTER, DIMENSION(: , : ) : : arr END TYPE END MODULE 61

Pack/Unpack (PUP) Routine SUBROUTINE pup_my_block(p, m) USE my_block_mod USE pupmod INTEGER : : p

Pack/Unpack (PUP) Routine SUBROUTINE pup_my_block(p, m) USE my_block_mod USE pupmod INTEGER : : p TYPE(my_block) : : m call fpup_int(p, m%n 1) call fpup_int(p, m%n 2 x) call fpup_int(p, m%n 2 y) IF (fpup_is. Unpacking(p)) THEN ALLOCATE(m%arr(m%n 2 x, m%n 2 y)) END IF call fpup_doubles(p, m%arr, m%n 2 x*m%n 2 y) IF (fpup_is. Deleting(p)) THEN DEALLOCATE(m%arr) END IF END SUBROUTINE 62

Registering Chunk Data !- Fortran driver subroutine use my_block_mod interface subroutine pup_my_block(p, m) use

Registering Chunk Data !- Fortran driver subroutine use my_block_mod interface subroutine pup_my_block(p, m) use my_block_mod INTEGER : : p TYPE(my_block) : : m end subroutine end interface TYPE(my_block) : : m CALL FEM_Register(m, pup_my_block) 63

Migration Every chunk driver calls FEM_Migrate n Framework calls PUP n for getting the

Migration Every chunk driver calls FEM_Migrate n Framework calls PUP n for getting the size of packed data n For packing data n Chunk migrates to new processor n Framework calls PUP for unpacking n Driver returns from FEM_Migrate n 64

Advanced: Complicated Fields 65

Advanced: Complicated Fields 65

Node Fields 66

Node Fields 66

FEM_Create_Field function integer : : FEM_Create_Field( base_type, vec_len, offset, dist) integer, intent(in) : :

FEM_Create_Field function integer : : FEM_Create_Field( base_type, vec_len, offset, dist) integer, intent(in) : : base_type, vec_len, offset, dist Base_type • FEM_BYTE- INTEGER*1, or CHARACTER*1 • FEM_INT- INTEGER*4 • FEM_REAL- REAL*4 • FEM_DOUBLE- DOUBLE PRECISION, or REAL*8 67

Create_field Example ! 3 D Force for each node ! stored as 3*n real*8

Create_field Example ! 3 D Force for each node ! stored as 3*n real*8 array REAL*8 ALLOCATABLE, DIMENTION(: ) : : node. Force INTEGER : : fid . . . allocate node. Force as 3*n_nodes. . . fid = FEM_Create_Field(FEM_DOUBLE, 3, foffsetof(node. Force(1), node. Force(1)), foffsetof(node. Force(1), node. Force(4)) 68

Create_field Example ! 3 D force is contained as f. XYZ variable ! in

Create_field Example ! 3 D force is contained as f. XYZ variable ! in a user-defined type node_type TYPE(node_type), ALLOCATABLE, DIMENTION(: ) : : nodes INTEGER : : fid . . . allocate nodes array as n_nodes. . . fid = FEM_Create_Field(FEM_DOUBLE, 3, foffsetof(nodes(1), nodes(1)%f. XYZ), foffsetof(nodes(1), nodes(2)) ) 69

Advanced: Symmetry Ghosts 70

Advanced: Symmetry Ghosts 70

Ghosts and Symmetries n In addition to cross-processor ghosts, can build ghosts to model

Ghosts and Symmetries n In addition to cross-processor ghosts, can build ghosts to model problem symmetries Translational and rotational periodicities n Mirror symmetry n n FEM_Add_linear_periodicity(n. Fac es, n. Per, faces. A, faces. B, n. Nodes, node. Locs) n Identify these two lists of faces under linear periodicity, and build ghosts to match 71

Symmetry Ghosts: 2 D Example Serial Mesh 1 2 3 4 Horizontal Periodicity Left

Symmetry Ghosts: 2 D Example Serial Mesh 1 2 3 4 Horizontal Periodicity Left Chunk Sym. Ghost 4 1 2 Right Chunk Ghost of 3 Ghost of 2 3 4 Sym. Ghost 1 72

Ghost Elements 73

Ghost Elements 73

Symmetry-Ghost Elements 74

Symmetry-Ghost Elements 74

Ghosts and Symmetries: Update 75

Ghosts and Symmetries: Update 75

Advanced: FEM on MPI 76

Advanced: FEM on MPI 76

FEM on MPI FEM routines perform their communication using MPI calls n This means

FEM on MPI FEM routines perform their communication using MPI calls n This means you can call FEM routines from an MPI program; or call MPI routines from an FEM program n 77

Chunk Data Example PROGRAM main include ‘femf. h’ include ‘mpif. h’ CALL MPI_Init() CALL

Chunk Data Example PROGRAM main include ‘femf. h’ include ‘mpif. h’ CALL MPI_Init() CALL FEM_Init(MPI_COMM_WORLD) . . . MPI or FEM routines. . . END PROGRAM 78

Extra FEM Features 79

Extra FEM Features 79

Collision Detection 80

Collision Detection 80

Charm++ Collision Detection n Detect collisions (intersections) between objects scattered across processors n Built

Charm++ Collision Detection n Detect collisions (intersections) between objects scattered across processors n Built on Charm++ Arrays n Overlay regular 3 D sparse grid of voxels (boxes) n Send objects to all voxels they touch n Collect collisions from each voxel 81 Collision response is left to caller n

Collision Detection Algorithm: n Sparse 3 D voxel grid (implemented as Charm array) 82

Collision Detection Algorithm: n Sparse 3 D voxel grid (implemented as Charm array) 82

Serial Scaling 83

Serial Scaling 83

Parallel Scaled Problem 84

Parallel Scaled Problem 84

Mesh Adaptation 85

Mesh Adaptation 85

Parallel Mesh Refinement n n n To refine, split the longest edge: n But

Parallel Mesh Refinement n n n To refine, split the longest edge: n But if split neighbor has a longer edge, split his edge first n Refinement propagates across mesh, but preserves mesh quality Initial 2 D parallel implementation built on Charm++ Interfaces with FEM Framework 86

d n a r B w! Ne IFEM: Iterative FEM Linear Solver Interface 87

d n a r B w! Ne IFEM: Iterative FEM Linear Solver Interface 87

IFEM: Intro to Matrix-based FEM n Normal computations run elementby-element n n Matrix-based computations

IFEM: Intro to Matrix-based FEM n Normal computations run elementby-element n n Matrix-based computations are holistic n n n Only consider one element at a time One big matrix represents the whole system Solve the matrix == solve the problem Row of matrix solves for “Degree of Freedom” n n n E. g. , a node’s X coordinate Typically a row has only a few nonzero entries Around 106 degrees of freedom--106 88 rows

106 Rows is a Big Matrix n Normal matrix operations are O(n 3) n

106 Rows is a Big Matrix n Normal matrix operations are O(n 3) n n Almost everybody uses iterative solvers n n Can’t use Gaussian elimination, Jacobi, QR, … Based on a fast “guess-and-check” method Each iteration consists of a matrix-vector product Typically converges in 10 to 100 iterations Popular iterative solvers: n n n Conjugate gradient (CG) Bidirectional Conjugate Gradient (Bi. CG) Generalized modified residual (GMRES) 89

ILSI: Iterative Linear Solver Interfac n Provides solvers with operations they need: n Parallel

ILSI: Iterative Linear Solver Interfac n Provides solvers with operations they need: n Parallel matrix-vector product (written by user) Parallel dot-product n class ILSI_Comm { virtual void matrix. Vector. Product(const double *src, double *dest) =0; virtual double dot. Product(const double *a, const double *b) =0; }; n Iterative solvers use these operations to solve the matrix n n Writing iterative solvers: an active research area Krylov Subspaces, Residuals, Quadratic 90

Matrix Example: n Hooke’s spring law: F=-kx n F 1=-k(D 1 -D 2) n

Matrix Example: n Hooke’s spring law: F=-kx n F 1=-k(D 1 -D 2) n Linear relation between node force and node displacement n Element corresponds to little 2 x 2 matrix n D 1 F 1 -k k = F 2 k -k D 2 D 1 D 2 91

Matrix Example: 2 Elements A D 1 B D 2 D 3 F 1

Matrix Example: 2 Elements A D 1 B D 2 D 3 F 1 -k k 0 = F 2 k -k-k k D 1 F 3 D 3 0 k -k D 2 92

Matrix Example: Derivation A D 1 B D 2 F 1 -k k =

Matrix Example: Derivation A D 1 B D 2 F 1 -k k = F 2 k –k-k k A F 3 k -k B D 3 D 1 D 2 D 3 93

Matrix Example: FEM A B Shared Node D 1 D 2 F 1 -k

Matrix Example: FEM A B Shared Node D 1 D 2 F 1 -k k = F 2 A k -k D 2 D 3 D 1 D 2 Shared Node F 2 B -k k = F 3 k -k D 2 D 3 Later, sum partial forces across shared nodes to get: F 2=F 2 A+F 2 B 94

IFEM: Bottom Line n n If you can do matrix-vector product, you can run

IFEM: Bottom Line n n If you can do matrix-vector product, you can run iterative solvers like conjugate gradient Matrix-vector product amounts to the usual FEM calculation (displacements to forces), plus the usual FEM communication (update shared nodes) n n You don’t even need to store the matrix Users can easily solve iterative problems with the FEM framework and IFEM solvers! 95

Net. FEM 96

Net. FEM 96

Net. FEM Client: pretty pictures n Wave dispersion off a crack (simplified frac 3

Net. FEM Client: pretty pictures n Wave dispersion off a crack (simplified frac 3 d) 97

Net. FEM: Easy Visualization n Can interact with a running FEM application n n

Net. FEM: Easy Visualization n Can interact with a running FEM application n n Uses Charm++ CCS (Client-server) library Easy to “publish” attributes Net. FEM n=Net. FEM_Begin(2, t, Net. FEM_POINTAT); Net. FEM_Nodes(n, nnodes, (double *)g. coord, "Position (m)"); Net. FEM_Vector(n, (double *)g. d, "Displacement (m)"); Net. FEM_Vector(n, (double *)g. v, "Velocity (m/s)"); Author: Orion Lawlor Net. FEM_Elements(n, nelems, 3, (int *)g. conn, "Triangles"); Net. FEM_Scalar(n, g. S 11, 1, "X Stress (pure)"); Net. FEM_Scalar(n, g. S 22, 1, "Y Stress (pure)"); Net. FEM_Scalar(n, g. S 12, 1, "Shear Stress (pure)"); Net. FEM_End(n); 98

Net. FEM: Easy visualization 99

Net. FEM: Easy visualization 99

Net. FEM: Zoom in 100

Net. FEM: Zoom in 100

Net. FEM: Outline Elements 101

Net. FEM: Outline Elements 101

Net. FEM: Point Nodes 102

Net. FEM: Point Nodes 102

Net. FEM Server Side: Overview n To allow the Net. FEM client to connect,

Net. FEM Server Side: Overview n To allow the Net. FEM client to connect, you add Net. FEM registration calls to your server Register nodes and element types n Register data items: scalars or spatial vectors associated with each node or element n You provide the display name and units for each data item n Link your program with “-module netfem” n Run with “++server”, and 103 connect! n

Net. FEM Server Side: Setup n n=Net. FEM_Begin(FEM_My_partit ion(), timestep, dim, Net. FEM_POINTAT) Call

Net. FEM Server Side: Setup n n=Net. FEM_Begin(FEM_My_partit ion(), timestep, dim, Net. FEM_POINTAT) Call this each time through your timeloop; or skip n timestep identifies this data update n dim is the spatial dimension—must be 2 or 3 n Returns a Net. FEM handle n used by everything else n n Net. FEM_End(n) n Finishes update n 104

Net. FEM Server Side: Nodes n Net. FEM_Nodes(n, nnodes, coord, ”Posit ion (m)”) n

Net. FEM Server Side: Nodes n Net. FEM_Nodes(n, nnodes, coord, ”Posit ion (m)”) n n n Registers node locations with Net. FEM— future vectors and scalars will be associated with nodes n is the handle returned by Net. FEM_Begin nnodes is the number of nodes coord is a dim by nnodes array of doubles The string describes the coordinate system and meaning of nodes Currently, there can only be one call to nodes 105

Net. FEM: Node Displacement 106

Net. FEM: Node Displacement 106

Net. FEM Server Side: Elements n Net. FEM_Elements(n, nelem, nodeper, conn, ”Triangles”) n n

Net. FEM Server Side: Elements n Net. FEM_Elements(n, nelem, nodeper, conn, ”Triangles”) n n n n Registers elements with Net. FEM—future vectors and scalars will be associated with these elements n is the handle returned by Net. FEM_Begin nelem is the number of elements nodeper is the number of nodes per element conn is a nodeper by nelem array of node indices The string describes the kind of element Repeat to register several kinds of element n Perhaps: Triangles, squares, pentagons, 107…

Net. FEM: Element Stress 108

Net. FEM: Element Stress 108

Net. FEM Server Side: Vectors n Net. FEM_Vector(n, val, ”Displacement (m)”) n Registers a

Net. FEM Server Side: Vectors n Net. FEM_Vector(n, val, ”Displacement (m)”) n Registers a spatial vector with each node or element • Whichever kind was registered last n n n is the handle returned by Net. FEM_Begin val is a dim by nitems array of doubles • There’s also a more general Net. FEM_Vector_field in the manual n n The string describes the meaning and units of the vectors Repeat to register multiple sets of vectors n Perhaps: Displacement, velocity, 109

Net. FEM: Element Velocity 110

Net. FEM: Element Velocity 110

Net. FEM Server Side: Scalars n Net. FEM_Scalar(n, val, s, ”Displacement (m)”) n Registers

Net. FEM Server Side: Scalars n Net. FEM_Scalar(n, val, s, ”Displacement (m)”) n Registers s scalars with each node or element • Whichever kind was registered last n n n is the handle returned by Net. FEM_Begin val is an s by nitems array of doubles • There’s also a more general Net. FEM_Scalar_field in the manual n n n s is the number of doubles for each node or element The string describes the meaning and units of the scalars Repeat to register multiple sets of scalars 111

Net. FEM Server Side: 2 D Example integer : : t, n, numnp, numel

Net. FEM Server Side: 2 D Example integer : : t, n, numnp, numel real*8, dimension(2, numnp) : : coor, d, v, a integer, dimension(3, numel) : : conn n=Net. FEM_Begin(FEM_My_partition(), t, 2, Net. FEM_POINTAT) CALL Net. FEM_Nodes(n, numnp, coor, 'Position (m)') CALL Net. FEM_Vector(n, d, 'Displacement (m)') CALL Net. FEM_Vector(n, v, 'Velocity (m/s)') CALL Net. FEM_Vector(n, a, 'Acceleration (m/s^2)') CALL Net. FEM_Elements(n, numel, 3, conn, 'Triangles') CALL Net. FEM_Scalar(n, stress, 1, 'Stress (pure)') CALL Net. FEM_End(n) 112

Net. FEM: Conclusion n n Easy, general way to get output from an FEM

Net. FEM: Conclusion n n Easy, general way to get output from an FEM computation Client configures itself based on server Client can be run anywhere (from home!) Server performance impact minimal (1 s!) Future work: n n n Support multiple chunks per processor Non-network, file-based version Movie mode 113

Conclusions: Charm++ FEM Easy to parallelize existing codes n Flexible: shared nodes or ghosts

Conclusions: Charm++ FEM Easy to parallelize existing codes n Flexible: shared nodes or ghosts n High performance n Extra features n Visualization with Net. FEM n Collision n Matrix methods with IFEM n 114

Old FEM Mesh Routines 115

Old FEM Mesh Routines 115

Framework Calls n FEM_Set_Mesh n n n FEM_Create_Field n n Registers a node data

Framework Calls n FEM_Set_Mesh n n n FEM_Create_Field n n Registers a node data field with the framework, supports user data types FEM_Update_Field n n n Called from initialization to set the serial mesh Framework partitions mesh into chunks Updates node data field across all processors Handles all parallel communication Other parallel calls (Reductions, etc. ) 116

FEM_*_Elem subroutine FEM_Set_Elem(el. Type, n. El, double. Per. El, node. Per. El) integer, intent(in)

FEM_*_Elem subroutine FEM_Set_Elem(el. Type, n. El, double. Per. El, node. Per. El) integer, intent(in) : : el. Type, n. El, double. Per. El, node. Per. El subroutine FEM_Get_Elem(el. Type, n. El, double. Per. El, node. Per. El) integer, intent(in) : : el. Type integer, intent(out) : : n. El, double. Per. El, node. Per. El 118

FEM_*_Node subroutine FEM_Set_Node(n. Node, double. Per. Node) integer, intent(in) : : n. Node, double.

FEM_*_Node subroutine FEM_Set_Node(n. Node, double. Per. Node) integer, intent(in) : : n. Node, double. Per. Node subroutine FEM_Get_Node(n. Node, double. Per. Node) integer, intent(out) : : n. Node, double. Per. Node 119

Element Connectivity subroutine FEM_Set_Elem_Conn_r(el. Type, conn) integer, intent(in) : : el. Type integer, intent(in),

Element Connectivity subroutine FEM_Set_Elem_Conn_r(el. Type, conn) integer, intent(in) : : el. Type integer, intent(in), dimention(node. Per. El, n. El) : : conn subroutine FEM_Get_Elem_Conn_r(el. Type, conn) integer, intent(in) : : el. Type integer, intent(out), dimention(node. Per. El, n. El) : : conn 120

Elements subroutine FEM_Set_Node_Data_r(data) REAL*8, intent(in), dimension(double. Per. Node, n. Node) : : data subroutine

Elements subroutine FEM_Set_Node_Data_r(data) REAL*8, intent(in), dimension(double. Per. Node, n. Node) : : data subroutine FEM_Get_Node_Data_r(data) REAL*8, intent(out), dimension(double. Per. Node, n. Node) : : data 121

Utility function integer : : FEM_Num_Partitions() function integer : : FEM_My_Partition() function double precision

Utility function integer : : FEM_Num_Partitions() function integer : : FEM_My_Partition() function double precision : : FEM_Timer() subroutine FEM_Print_Partition() subroutine FEM_Print(str) character*, intent(in) : : str 123