Matrix Addition and Multiplication ComputationCommunication Ratio ITCS 45145
- Slides: 17
Matrix Addition and Multiplication Computation/Communication Ratio ITCS 4/5145 Parallel Computing UNC-Charlotte, B. Wilkinson, Jan 21, 2014. slides 11 A. ppt 1
Matrices — A Review An n x m matrix 2
Matrix Addition Sequential code to compute A + B could simply be for (i = 0; i < n; i++) for (j = 0; j < n; j++) { c[i][j] = a[i][j] * b[i][j]; } Requires n 2 multiplications and n 2 additions Sequential time complexity of O(n 2). Very easy to parallelize as each result independent although may not get speedup in message-passing environment. 3
Matrix Addition, C = A + B Add corresponding elements of each matrix to form elements of result matrix. Given elements of A as ai, j and elements of B as bi, j, each element of C computed as: Add A B C Could have one process(or) compute one or more C elements. 4
Workpool/master slave implementation Slaves Return one element of C C A B Send one element of A and B to slave Master Each slave process doing just one addition not a good partition of work. Could try each slave doing more work - next. Compute node Source/sink 5
Each process adding a pair of complete rows Adds one row of A with one row of B to create one row of C (rather than each process adding single elements) Add A B C 6
Workpool implementation Slaves (one for each row) Return one row of C C A B Send one row of A and B to slave Master Although slaves do more, need more data sent! Compute node Source/sink 7
Matrix Multiplication Sequential code to compute A x B could simply be for (i = 0; i < n; i++) // for each row of A for (j = 0; j < n; j++) { // for each column of B c[i][j] = 0; for (k = 0; k < n; k++) c[i][j] = c[i][j] + a[i][k] * b[k][j]; } Requires n 3 multiplications and n 3 additions Sequential time complexity of O(n 3). Very easy to parallelize as each result independent 8
Matrix Multiplication, C = A * B Multiplication of two matrices, A and B, produces matrix C whose elements, ci, j (0 <= i < n, 0 <= j < m), computed as follows: where A is an n x l matrix and B is an l x m matrix. 9
Workpool implementation Slaves (one for each element of result) Return one element of C C A Send one row of A and one column of B to slave B Master Following example 3 x 3 arrays and 9 slaves Compute node Source/sink 10
Usually size of matrices (n) much larger than number of processors (p). So divide matrix into s 2 submatrices. Each submatrix has n/s x n/s elements. One processor produces each submatrix result (p = s 2). Block Matrix Multiplication Can be applied to all parallelization methods. for (p = 0; p < s; p++) for (q = 0; q < s; q++) { Cp, q = 0; /* clear elements of submatrix*/ for (r = 0; r < m; r++) /* submatrix multiplication */ Cp, q = Cp, q + Ap, r * Br, q; /*add to accum. submatrix*/ } Means multiply submatrix Ap, r and Br, q using matrix multiplication and add to submatrix Cp, q using matrix addition. 11
Workpool implementation Slaves (one for each element of result) Return s x s submatrix C C A Send s rows of A and B s column of B to slave Master Compute node Source/sink 12
Computation/Communication Ratio where tcomp is computation time and tcomm is communication time. Normally communication very costly Typically approximately linear cost with increasing message data Startup time Time 13
Computation/Communication Ratio Could write as time complexities. Suppose: Computation = O(n 2) Communication = O(n) (i. e. linear) where n is number of data items By increasing n, eventually n can be found when computation can dominate communication – that is good. 14
Matrix Addition With pairs of rows added by each process: With communication linear with data Computation = O(n) Communication = O(n) Not good in this context as communication constant large 15
Matrix Multiplication ? Discussion 16
Questions 17
- What are the rules for adding decimals
- Inverse property of addition and multiplication
- Multiplication and addition rule genetics
- Addition subtraction and multiplication of polynomials
- What is the addition principle
- General multiplication rule probability
- Matrix transpose times matrix
- Itcs 6114
- Itcs 6114
- Ecu itcs
- Liceo scientifico primo levi bollate
- Itcs primo levi
- Maths addition subtraction multiplication division
- If multiplication is repeated addition what is division
- Four fundamental operations
- Multiplication prioritaire
- Augmented matrix
- Cannon algorithm