CADD ComponentAveraged Domain Decomposition Dan Gordon Rachel Gordon

CADD: Component-Averaged Domain Decomposition Dan Gordon Rachel Gordon Computer Science University of Haifa Aerospace Engg. Technion January 13, 2008 Component-Averaged Domain Decomposition 1

Outline of Talk u CADD – Algebraic explanation u CADD – DD explanation u The Kaczmarz algorithm (KACZ) u KACZ CARP (a CADD method) u Applications of CARP u CARP-CG: CG acceleration of CARP u Sample results January 13, 2008 Component-Averaged Domain Decomposition 2

CADD – Algebraic Explanation – 1 u The equations are divided into blocks (not necessarily disjoint) u Initial estimate: vector x=(x 1, …, xn) u Suppose x 1 is a variable (component of x) that appears in 3 blocks u x 1 is “cloned” as y 1 , z 1 , t 1 in the different blocks. u Perform one (or more) iterative operation(s) on each block (independently, in parallel) January 13, Component-Averaged Domain January 13, 2008 Component-Averaged Domain Decomposition 3

CADD – Algebraic Explanation – 2 u The internal iterations in each block produce 3 new values for the clones of x 1 : y 1’ , z 1’ , t 1’ u The next iterative value of x 1 is x 1’ = (y 1’ + z 1’ + t 1’)/3 u The next iterate is x’ = (x 1’ , . . . , xn’) u Repeat iterations as needed for convergence January 13, 2008 Component-Averaged Domain Decomposition 4

CADD – non-overlapping domains x 0 y 1 clone of x 1 external grid point of A domain A January 13, 2008 domain B Component-Averaged Domain Decomposition 5

CADD – overlapping domains domain B x 0 y 1 clone of x 1 external grid point of A A ∩B January 13, 2008 domain A Component-Averaged Domain Decomposition 6

CADD – Parallel Implementation u Every processor is in charge of one domain u Mode: BLOCK-PARALLEL – All processors operate in parallel (each on its domain) – Processors exchange clone values – All values are updated – New values = average of clones January 13, 2008 Component-Averaged Domain Decomposition 7

CADD Vs. Standard DD Methods u Difference is in handling external grid points: u CADD: A clone of the external grid point is modified by the domain's equations and contributes to new value of grid point. u Standard DD: Value of external grid point is fixed (contributes to RHS) January 13, 2008 Component-Averaged Domain Decomposition 8

KACZ: The Kaczmarz algorithm u Iterative method, due to Kaczmarz (1937). Rediscovered for CT as ART u Basic idea: Consider the hyperplane defined by each equation u Start from an arbitrary initial point u Successively project current point onto the next hyperplane, in cyclic order January 13, 2008 Component-Averaged Domain Decomposition 9

KACZ: Geometric Description initial point eq. 1 eq. 3 January 13, 2008 eq. 2 Component-Averaged Domain Decomposition 10

KACZ with Relaxation Parameter u KACZ can be used with a relaxation parameter λ u λ=1: project exactly on the hyperplane u λ<1: project in front of hyperplane u λ>1: project beyond the hyperplane u Cyclic relaxation: Eq. i is assigned a relaxation parameter λi January 13, 2008 Component-Averaged Domain Decomposition 11

Convergence Properties of KACZ u KACZ with relaxation (0< λ <2) converges for consistent systems: – Herman, Lent & Lutz, 1978 – Trummer, 1981 u For inconsistent systems, KACZ converges cyclically: – Tanabe, 1971 – Eggermont, Herman & Lent, 1981 (for cyclic relaxation parameters). January 13, 2008 Component-Averaged Domain Decomposition 12

Another view of KACZ u Given the system u Consider system Ax = b the "normal equations" AATy = b, x=ATy u Well-known: KACZ is simply SOR applied to the normal equations u The relaxation parameter of KACZ is the usual relax. par. of SOR January 13, 2008 Component-Averaged Domain Decomposition 13

CARP: Component-Averaged Row Projections u u u CARP is a CADD method with KACZ iterations in each domain: The system Ax=b is divided into blocks B 1, …, Bn (need not be disjoint) Each processor is assigned one (or more) block All the blocks are processed in parallel Results from blocks are “merged” to form the next iterate Merging is done by CADD averaging January 13, 2008 Component-Averaged Domain Decomposition 14

Overview of CARP domain A domain B KACZ iterations averaging KACZ iterations cloning KACZ in superspace (with cyclic relaxation) January 13, 2008 Component-Averaged Domain Decomposition 15

Convergence of CARP u Averaging Lemma: the component- averaging and cloning operations of CARP are equivalent to KACZ rowprojections in a certain superspace (with λ = 1) u CARP is equivalent to KACZ in the superspace, with cyclic relaxation parameters – known to converge January 13, 2008 Component-Averaged Domain Decomposition 16

"Historical" Note u The term "component averaging" was first used by Censor, Gordon & Gordon (2001) for CAV and BICAV, used for image reconstruction in CT u These are Cimmino-type algorithms, with weights related to the sparsity of the system matrix u CADD, CARP use this concept in a different sense January 13, 2008 Component-Averaged Domain Decomposition 17

CARP Application: Solution of stiff linear systems from PDEs u Elliptic PDEs w/large convection term result in stiff linear systems (large off -diagonal elements) u CARP is very robust on these systems, as compared to leading solver/preconditioner combinations u Downside: Not always efficient January 13, 2008 Component-Averaged Domain Decomposition 18

CARP Application: Electron Tomography (joint work with J. -J. Fernández) u 3 D reconstructions: Each processor is assigned a block of consecutive slices. Data is in overlapping blobs. u The blocks are processed in parallel. u The values of shared variables are transmitted between the processors which share them, averaged, and redestributed. January 13, 2008 Component-Averaged Domain Decomposition 19

CARP-CG: CG acceleration of CARP u CARP is KACZ in some superspace (with cyclic relaxation parameter) u Björck & Elfving (BIT 79): developed CGMN, which is a (sequential) CGacceleration of KACZ (double sweep, fixed relax. parameter) u We extended this result to allow cyclic relaxation parameters u Result: CARP-CG January 13, 2008 Component-Averaged Domain Decomposition 20

CARP-CG: Properties u Same robustness as CARP u Very significant improvement in performance on stiff linear systems derived from elliptic PDEs u Very competitive runtime compared to leading solver/preconditioner combinations on systems derived from convection-dominated PDEs u Improved performance in ET January 13, 2008 Component-Averaged Domain Decomposition 21

CARP-CG on PDEs u CG acceleration of projection methods was done before, but with block projections, requiring multicoloring. CARP-CG avoids this. u Tests were run on 9 convectiondominated PDEs, comparing CARPCG with restarted GMRES, CGS and Bi-CGSTAB, with and without various preconditioners. Also tested: CGNR. u Domain: unit cube – 80 x 80. January 13, 2008 Component-Averaged Domain Decomposition 22

512, 000 equations January 13, 2008 Component-Averaged Domain Decomposition 23

512, 000 equations January 13, 2008 Component-Averaged Domain Decomposition 24

512, 000 equations January 13, 2008 Component-Averaged Domain Decomposition 25

512, 000 equations January 13, 2008 Component-Averaged Domain Decomposition 26

512, 000 equations January 13, 2008 Component-Averaged Domain Decomposition 27

CARP-CG: ET results January 13, 2008 Component-Averaged Domain Decomposition 28

Future research on CADD u Different internal solvers: 1. Guaranteed convergence? 2. Symmetrizability? u Combine CADD with multigrid u CFD applications u Additional biomedical applications u CADD with preconditioners u Compare CADD with other DD methods, such as additive Schwarz January 13, 2008 Component-Averaged Domain Decomposition 29

CARP Publications u CARP: SIAM J. Sci. Comput. 27 (2005) 1092 -1117. http: //cs. haifa. ac. il/~gordon/carp. pdf u Electron tomography: J. Par. & Dist. Comput. (2007), in press. http: //cs. haifa. ac. il/~gordon/eltom. pdf u CARP-CG: January 13, 2008 Submitted for publication. Component-Averaged Domain Decomposition 30