7.1 Gaussian Elimination 363 can be used to solve several linear systems with the same matrix A and different right-hand side vectors b without repeating the elimination process. 7.1.1.2 Pivoting Forward elimination and LU decomposition require the division by a (k) kk and so these methods can only be applied when a (k) kk = 0. That is, even if det A = 0 and the system Ax = y is solvable, there does not need to exist a decomposition A = LU when a (k) kk is a zero element. However, for a solvable linear system, there exists a matrix resulting from permutations of rows of A, for which an LU decomposition is possible, i.e., BA= LU with a permutation matrix B describing the permutation of rows of A. The permutation of rows of A, if necessary, is included in the elimination process. In each elimination step, a pivot element is determined to substitute a (k) kk . A pivot element is needed when a (k) kk = 0 and when a (k) kk is very small which would induce an elimination factor, which is very large leading to imprecise computations. Pivoting strategies are used to find an appropriate pivot element. Typical strategies are column pivoting, row pivoting, and total pivoting. Column pivoting considers the elements a (k) kk ···a (k) nk of column k and determines the element a (k) rk , k ≤ r ≤ n, with the maximum absolute value. If r = k,therowsr and k of matrix A (k) and the values b (k) k and b (k) r of the vector b (k) are exchanged. Row pivoting determines a pivot element a (k) kr , k ≤ r ≤ n, within the elements a (k) kk ···a (k) kn of row k of matrix A (k) with the maximum absolute value. If r = k, the columns k and r of A (k) are exchanged. This corresponds to an exchange of the enumeration of the unknowns x k and x r of vector x. Total pivoting determines the element with the maximum absolute value in the matrix ˜ A (k) = (a (k) ij ), k ≤ i, j ≤ n, and exchanges columns and rows of A (k) depending on i = k and j = k. In practice, row or column pivoting is used instead of total pivoting, since they have smaller computation time, and total pivoting may also destroy special matrix structures like banded structures. The implementation of pivoting avoids the actual exchange of rows or columns in memory and uses index vectors pointing to the current rows of the matrix. The indexed access to matrix elements is more expensive but in total the indexed access is usually less expensive than moving entire rows in each elimination step. When supported by the programming language, a dynamic data storage in the form of separate vectors for rows of the matrix, which can be accessed through a vector pointing to the rows, may lead to more efficient implementations. The advantage is that matrix elements can still be accessed with a two-dimensional index expression but the exchange of rows corresponds to a simple exchange of pointers. 7.1.2 Parallel Row-Cyclic Implementation A parallel implementation of the Gaussian elimination is based on a data distribution of matrix A and of the sequence of matrices A (k) , k = 2, ,n, which can be a row-oriented, a column-oriented, or a checkerboard distribution, see Sect. 3.4. In this section, we consider a row-oriented distribution. 364 7 Algorithms for Systems of Linear Equations From the structure of the matrices A (k) it can be seen that a blockwise row- oriented data distribution is not suitable because of load imbalances: For a blockwise row-oriented distribution processor P q ,1≤ q ≤ p, owns the rows (q − 1) ·n/p + 1, ,q ·n/ p so that after the computation of A (k) with k = q ·n/ p +1thereisno computation left for this processor and it becomes idle. For a row-cyclic distribution, there is a better load balance, since processor P q ,1≤ q ≤ p, owns the rows q, q+p, q +2p, , i.e., it owns all rows i with 1 ≤ i ≤ n, and q = ((i −1) mod p)+1. The processors begin to get idle only after the first n − p stages, which is reasonable for p  n. Thus, we consider a parallel implementation of the Gaussian elimination with a row-cyclic distribution of matrix A and a column-oriented pivoting. One step of the forward elimination computing A (k+1) and b (k+1) for given A (k) and b (k) performs the following computation and communication phases: 1. Determination of the local pivot element: Each processor considers its local elements of column k in the rows k, ,n and determines the element (and its position) with the largest absolute value. 2. Determination of the global pivot element: The global pivot element is the local pivot element which has the largest absolute value. A single-accumulation operation with the maximum operation as reduction determines this global pivot element. The root processor of this global communication operation sends the result to all other processors. 3. Exchange of the pivot row: If k =r for a pivot element a (k) rk ,therowk owned by processor P q and the pivot row r owned by processor P q  have to be exchanged. When q = q  , the exchange can be done locally by processor P q . When q = q  , then communication with single transfer operations is required. The elements b k and b r are exchanged accordingly. 4. Distribution of the pivot row: Since the pivot row (now row k) is required by all processors for the local elimination operations, processor P q sends the elements a (k) kk , ,a (k) kn of row k and the element b (k) k to all other processors. 5. Computation of the elimination factors: Each processor locally computes the elimination factors l ik for which it owns the row i according to Formula (7.2). 6. Computation of the matrix elements: Each processor locally computes the elements of A (k+1) and b (k+1) using its elements of A (k) and b (k) according to Formulas (7.3) and (7.4). The computation of the solution vector x in the backward substitution is inherently sequential, since the values x k , k = n, ,1, depend on each other and are com- puted one after another. In step k, processor P q owning row k computes the value x k according to Formula (7.5) and sends the value to all other processors by a single- broadcast operation. A program fragment implementing the computation phases 1–6 and the back- ward substitution is given in Fig. 7.2. The matrix A and the vector b arestoredina two- and a one-dimensional array a and b, respectively. Some of the local functions are already introduced in the program in Fig. 7.1. The SPMD program uses the variable me to store the individual processor number. This processor number, the 7.1 Gaussian Elimination 365 Fig. 7.2 Program fragment with C notation and MPI operations for the Gaussian elimination with row-cyclic distribution 366 7 Algorithms for Systems of Linear Equations current value of k, and the pivot row are used to distinguish between different com- putations for the single processors. The global variables n and p are the system size and the number of processors executing the parallel program. The parallel algorithm is implemented in the program in Fig. 7.2 in the following way: 1. Determination of the local pivot element: The function max col loc(a,k) determines the row index r of the element a[r][k], which has the largest local absolute value in column k for the rows ≥ k. When a processor has no element of column k for rows ≥ k, the function returns −1. 2. Determination of the global pivot element: The global pivoting is performed by an MPI Allreduce() operation, implementing a single-accumulation with a subsequent single-broadcast. The MPI reduction operation MPI MAXLOC for data type MPI DOUBLE INT consisting of one double value and one integer value is used. The MPI operations have been introduced in Sect. 5.2. The MPI Allreduce() operation returns y with the pivot element in y.val and the processor owning the corresponding row in y.node. Thus, after this step all processors know the global pivot element and the owner for possible communication. 3. Exchange of the pivot row: Two cases are considered: • If the owner of the pivot row is the processor also owning row k (i.e., k%p == y.node), the rows k and r are exchanged locally by this processor for r = k. Row k is now the pivot row. The function copy row(a,b,k,buf) copies the pivot row into the buffer buf, which is used for further communication. • If different processors own the row k and the pivot row r,rowk is sent to the processor y.node owning the pivot row with MPI Send and MPI Recv operations. Before the send operation, the function copy row(a,b,k,buf) copies row k of array a and element k of array b into a common buffer buf so that only one communication operation needs to be applied. After the com- munication, the processor y.node finalizes its exchange with the pivot row. The function copy exchange row(a,b,r,buf,k) exchanges the row r (still the pivot row) and the buffer buf. The appropriate row index r is known from the former local determination of the pivot row. Now the former row k is the row r and the buffer buf contains the pivot row. Thus, in both cases the pivot row is stored in buffer buf. 4. Distribution of the pivot row: Processor y.node sends the buffer buf to all other processors by an MPI Bcast() operation. For the case of the pivot row being owned by a different processor than the owner of row k, the content of buf is copied into row k by this processor using copy back row(). 5. and 6. Computation of the elimination factors and the matrix elements: The computation of the elimination factors and the new arrays a and b is done in parallel. Processor P q starts this computation with the first row i > k with i mod p = q. For a row-cyclic implementation of the Gaussian elimination, an alternative way of storing array a and vector b can be used. The alternative data structure consists 7.1 Gaussian Elimination 367 Fig. 7.3 Data structure for the Gaussian elimination with n = 8andp = 4 showing the rows stored by processor P 1 . Each row stores n +1 elements consisting of one row of the matrix a and the corresponding element of b b 1 b a a a a 11 a 18 558 51 of a one-dimensional array of pointers and n one-dimensional arrays of length n +1 each containing one row of a and the corresponding element of b. The entries in the pointer-array point to the row-arrays. This storage scheme not only facilitates the exchange of rows but is also convenient for a distributed storage. For a distributed memory, each processor P q stores the entire array of pointers but only the rows i with i mod p = q; all other pointers are NULL-pointers. Figure 7.3 illustrates this storage scheme for n = 8. The advantage of storing an element of b together with a is that the copy operation into a common buffer can be avoided. Also the computation of the new values for a and b is now only one loop with n+1 iterations. This implementation variant is not shown in Fig. 7.2. 7.1.3 Parallel Implementation with CheckerboardDistribution A parallel implementation using a block–cyclic checkerboard distribution for matrix A can be described with the parameterized data distribution introduced in Sect. 3.4. The parameterized data distribution is given by a distribution vector ((p 1 , b 1 ), (p 2 , b 2 )) (7.7) with a p 1 × p 2 virtual processor mesh with p 1 rows, p 2 columns, and p 1 · p 2 = p processors. The numbers b 1 and b 2 are the sizes of a block of data with b 1 rows and b 2 columns. The function G : P → N 2 maps each processor to a unique position in the processor mesh. This leads to the definition of p 1 row groups R q ={Q ∈ P | G(Q) = (q, ·)} with |R q |=p 2 for 1 ≤ q ≤ p 1 and p 2 column groups C q ={Q ∈ P | G(Q) = (·, q)} with |C q |=p 1 for 1 ≤ q ≤ p 2 . The row groups as well as the column groups are a partition of the entire set of processors, i.e., 368 7 Algorithms for Systems of Linear Equations p 1  q=1 R q = p 2  q=1 C q = P and R q ∩R q  =∅=C q ∩C q  for q = q  .Rowi of the matrix A is distributed across the local memories of the processors of only one row group, denoted Ro(i)inthe following. This is the row group R k with k =  i−1 b 1  mod p 1 + 1. Analogously, column j is distributed within one column group, denoted as Co( j), which is the column group C k with k =  j−1 b 2  mod p 2 +1. Example For a matrix of size 12×12 (i.e., n = 12), p = 4 processors {P 1 , P 2 , P 3 , P 4 } and distribution vector ((p 1 , b 1 ), (p 2 , b 2 )) = ((2, 2), (2, 3)), the virtual processor mesh has size 2 ×2 and the data blocks have size 2 ×3. There are two row groups and two column groups: R 1 ={Q ∈ P | G(Q) = (1, j), j = 1, 2}, R 2 ={Q ∈ P | G(Q) = (2, j), j = 1, 2}, C 1 ={Q ∈ P | G(Q) = ( j, 1), j = 1, 2}, C 2 ={Q ∈ P | G(Q) = ( j, 2), j = 1, 2} . The distribution of matrix A is shown in Fig. 7.4. It can be seen that row 5 is dis- tributed in row group R 1 and that column 7 is distributed in column group C 1 . Using a checkerboard distribution with distribution vector (7.7), the computation of A (k) has the following implementation, which has a different communication pat- tern than the previous implementation. Figure 7.5 illustrates the communication and computation phases of the Gaussian elimination with checkerboard distribution. Fig. 7.4 Illustration of a checkerboard distribution for a12×12 matrix. The tuples denote the position of the processors in the processor mesh owning the data block 1 2 3 4 5 6 7 8 9 10 11 12 12345 6789101112 (1,1) (1,2) (1,1) (1,2) (2,1) (2,2) (2,1) (2,2) (1,1) (2,1) (1,2) (2,2) (1,1) (2,1) (1,2) (2,2) (1,1) (2,1) (1,2) (2,2) (1,1) (2,1) (1,2) (2,2) 7.1 Gaussian Elimination 369 Fig. 7.5 Computation phases of the Gaussian elimination with checkerboard distribution k k k k k k (5a) broadcast of the k k r (5) computation of the matrix elements (4) broadcast of the k k r pivot row pivot row k k (5) computation of the elimination factors elimination factors (3) exchange of the k k (1) determination of the (2) determination of the global pivot elements local pivot elements This figure will be printed in b/w 370 7 Algorithms for Systems of Linear Equations 1. Determination of the local pivot element: Since column k is distributed across the processors of column group Co(k), only these processors determine the element with the largest absolute value for row ≥ k within their local elements of column k. 2. Determination of the global pivot element: The processors in group Co(k) perform a single-accumulation operation within this group, for which each pro- cessor in the group provides its local pivot element from phase 1. The reduction operation is the maximum operation also determining the index of the pivot row (and not the number of the owning processor as before). The root processor of the single-accumulation operation is the processor owning the element a (k) kk . After the single-accumulation, the root processor knows the pivot element a (k) rk and its row index. This information is sent to all other processors. 3. Exchange of the pivot row: The pivot row r containing the pivot element a (k) rk is distributed across row group Ro(r). Row k is distributed across the row group Ro(k), which may be different from Ro(r). If Ro(r) = Ro(k), the processors of Ro(k) exchange the elements of the rows k and r locally within the columns they own. If Ro(r) = Ro(k), each processor in Ro(k) sends its part of row k to the corresponding processor in Ro(r); this is the unique processor which belongs to the same column group. 4. Distribution of the pivot row: The pivot row is needed for the recalculation of matrix A, but each processor needs only those elements with column indices for which it owns elements. Therefore, each processor in Ro(r) performs a group- oriented single-broadcast operation within its column group sending its part of the pivot row to the other processors. 5. Computation of the elimination factors: The processors of column group Co(k) locally compute the elimination factors l ik for their elements i of column k according to Formula (7.2). 5a. Distribution of the elimination factors: The elimination factors l ik are needed by all processors in the row group Ro(i). Since the elements of row i are dis- tributed across the row group Ro(i), each processor of column group Co(k) performs a group-oriented single-broadcast operation in its row group Ro(i)to broadcast its elimination factors l ik within this row group. 6. Computation of the matrix elements: Each processor locally computes the elements of A (k+1) and b (k+1) using its elements of A (k) and b (k) according to Formulas (7.3) and (7.4). The backward substitution for computing the n elements of the result vector x is done in n consecutive steps where each step consists of the following computations: 1. Each processor of the row group Ro(k) computes that part of the sum  n j=k+1 a (n) kj x j which contains its local elements of row k. 2. The entire sum  n j=k+1 a (n) kj x j is determined by the processors of row group Ro(k) by a group-oriented single-accumulation operation with the processor P q as root which stores the element a (n) kk . Addition is used as reduction operation. 7.1 Gaussian Elimination 371 3. Processor P q computes the value of x k according to Formula (7.5). 4. Processor P q sends the value of x k to all other processors by a single-broadcast operation. A pseudocode for an SPMD program in C notation with MPI operations imple- menting the Gaussian elimination with checkerboard distribution of matrix A is given in Fig. 7.6. The computations correspond to those given in the pseudocode for the row-cyclic distribution in Fig. 7.2, but the pseudocode additionally uses several functions organizing the computations on the groups of processors. The functions Co(k) and Ro(k) denote the column or row groups owning column k or row k, respectively. The function member(me,G) determines whether processor me belongs to group G. The function grp leader() determines the first processor in a group. The functions Cop(q) and Rop(q) determine the column or row group, respectively, to which a processor q belongs. The function rank(q,G) returns the local processor number (rank) of a processor in a group G. 1. Determination of the local pivot element: The determination of the local pivot element is performed only by the processors in column group Co(k). 2. Determination of the global pivot element: The global pivot element is again computed by an MPI MAXLOC reduction operation, but in contrast to Fig. 7.2 the index of the row of the pivot element is calculated and not the processor number owning the pivot element. The reason is that all processors which own a part of the pivot row need to know that some of their data belongs to the current pivot row; this information is used in further communication. 3. Exchange of the pivot row: For the exchange and distribution of the pivot row r, the cases Ro(k) == Ro(r) and Ro(k)! = Ro(r) are distinguished. • When the pivot row and the row k are stored by the same row group, each processor of this group exchanges its data elements of row k and row r locally using the function exchange row loc() and copies the ele- ments of the pivot row (now row k) into the buffer buf using the function copy row loc(). Only the elements in column k or higher are considered. • When the pivot row and the row k are stored by different row groups, communication is required for the exchange of the pivot row. The function compute partner(Ro(r),me) computes the communication partner for the calling processor me, which is the processor q ∈ Ro(r) belonging to the same column group as me. The function compute size(n,k,Ro(k)) computes the number of elements of the pivot row, which is stored for the call- ing processor in columns greater than k; this number depends on the size of the row group Ro(k), the block size, and the position k. The same function is used later to determine the number of elimination factors to be communicated. 4. Distribution of the pivot row: For the distribution of the pivot row r, a processor takes part in a single-broadcast operation in its column group. The roots of the broadcast operation performed in parallel are the processors q ∈ Ro(r).The participants of a broadcast are the processors q  ∈ Cop(q), either as root when q  ∈ Ro(r) or as recipient otherwise. 372 7 Algorithms for Systems of Linear Equations Fig. 7.6 Program of the Gaussian elimination with checkerboard distribution . distribution of matrix A and a column-oriented pivoting. One step of the forward elimination computing A (k+1) and b (k+1) for given A (k) and b (k) performs the following computation and communication. matrix A and different right-hand side vectors b without repeating the elimination process. 7.1.1.2 Pivoting Forward elimination and LU decomposition require the division by a (k) kk and so these methods. to Formula (7.2). 6. Computation of the matrix elements: Each processor locally computes the elements of A (k+1) and b (k+1) using its elements of A (k) and b (k) according to Formulas (7.3) and