MPI_Cart_sub

MPI_Cart_sub

MPI_Cart_sub creates new communicators for subgrids of up to (N-1) dimensions from an N-dimensional cartesian grid.

Often, after we have created a cartesian grid, we wish to further group elements of this grid into subgrids of lower dimensions. For instance, the subgrids of a 2D cartesian grid are 1D grids of
the individual rows or columns. Similarly, for a 3D cartesian grid, the subgrids can either be 2D or 1D.

Fortran Syntax

Subroutine MPI_Cart_sub(old_comm, belongs, new_comm, ierr)

C Syntax

int MPI_Cart_sub(MPI_Comm old_comm, int *belongs, MPI_Comm *new_comm)

Example in Fortran

For a 2D cartesian grid, create subgrids of rows and columns. Create cartesian topology for processes.


!Create 2D cartesian topology for processes
      call MPI_Cart_create(MPI_COMM_WORLD, ndim, dims, 
     &       period, reorder, comm2D, ierr)
      call MPI_Comm_rank(comm2D, id2D, ierr)
      call MPI_Cart_coords(comm2D, id2D, ndim, coords2D, ierr)
!Create 1D row subgrids
      belongs(0) = .false.
      belongs(1) = .true.  ! this dimension belongs to subgrid
      call MPI_Cart_sub(comm2D, belongs, commrow, ierr)
!Create 1D column subgrids
      belongs(0) = .true.  ! this dimension belongs to subgrid
      belongs(1) = .false.
      call MPI_Cart_sub(comm2D, belongs, commcol, ierr)

Shown in Figure a below is a 3-by-2 cartesian topology where the index pair “i,j” represents row “i” and column “j”. The number
in parentheses represents the rank number associated with the 2D cartesian grid. Figure b shows the row subgrids while Figure c shows the column subgrids.

Figure a.
2D Cartesian Grid
0,0 (0) 0,1 (1)
1,0 (2) 1,1 (3)
2,0 (4) 2,1 (5)
Figure b.
Row Subgrids
0,0 (0) 0,1 (1)
1,0 (2) 1,1 (3)
2,0 (4) 2,1 (5)
Figure c.
Column Subgrids
0,0 (0) 0,1 (1)
1,0 (2) 1,1 (3)
2,0 (4) 2,1 (5)

Here is a fortran example demonstrating the column subgrid.

As another example, lets look at a 3D cartesian grid of 3x2x4. Calling MPI_Cart_sub with belongs
array defined as


      belongs(0) = .true.
      belongs(1) = .true.
      belongs(2) = .false.

yields four 3-by-2 subgrids. The output of a fortran code for this particular arrangement is shown below:

   MPI_Cart_sub example: 3x2x4 cartesian grid ==> 4 (3x2) subgrids

     Iam      3D    3D cartesian coords.        2D    2D subgrid 
    Rank    Rank                              Rank      coords.
       0       0|       0       0       0|       0|       0       0
      18      18|       2       0       2|       4|       2       0
       8       8|       1       0       0|       2|       1       0
      14      14|       1       1       2|       3|       1       1
      16      16|       2       0       0|       4|       2       0
      12      12|       1       1       0|       3|       1       1
       1       1|       0       0       1|       0|       0       0
       7       7|       0       1       3|       1|       0       1
       6       6|       0       1       2|       1|       0       1
      10      10|       1       0       2|       2|       1       0
      23      23|       2       1       3|       5|       2       1
       4       4|       0       1       0|       1|       0       1
       5       5|       0       1       1|       1|       0       1
       2       2|       0       0       2|       0|       0       0
       3       3|       0       0       3|       0|       0       0
      21      21|       2       1       1|       5|       2       1
       9       9|       1       0       1|       2|       1       0
      15      15|       1       1       3|       3|       1       1
      19      19|       2       0       3|       4|       2       0
      20      20|       2       1       0|       5|       2       1
      17      17|       2       0       1|       4|       2       0
      13      13|       1       1       1|       3|       1       1
      11      11|       1       0       3|       2|       1       0
      22      22|       2       1       2|       5|       2       1

The above table is illustrated below in four figures, each representing one of the four 2D subgrids of size (3×2). The top set of numbers denote the 3D cartesian grid coordinates with the process rank in that grid enclosed in parentheses. The lower, colored, sets represent the corresponding numbers in the 2D subgrids.

Figure d.
Subgrid for K = 0
0,0,0 (0)
0,0 (0)		 0,1,0 (4)
0,1 (1)
1,0,0 (8)
1,0 (2)		 1,1,0 (12)
1,1 (3)
2,0,0 (16)
2,0 (4)		 2,1,0 (20)
2,1 (5)
Figure e.
Subgrid for K = 1
0,0,1 (1)
0,0 (0)		 0,1,1 (5)
0,1 (1)
1,0,1 (9)
1,0 (2)		 1,1,1 (13)
1,1 (3)
2,0,1 (17)
2,0 (4)		 2,1,1 (21)
2,1 (5)
Figure f.
Subgrid for K = 2
0,0,2 (2)
0,0 (0)		 0,1,2 (6)
0,1 (1)
1,0,2 (10)
1,0 (2)		 1,1,2 (14)
1,1 (3)
2,0,2 (18)
2,0 (4)		 2,1,2 (22)
2,1 (5)
Figure g.
Subgrid for K = 3
0,0,3 (3)
0,0 (0)		 0,1,3 (7)
0,1 (1)
1,0,3 (11)
1,0 (2)		 1,1,3 (15)
1,1 (3)
2,0,3 (19)
2,0 (4)		 2,1,3 (23)
2,1 (5)

We have just demonstrated the use of MPI_Cart_sub to divide a cartesian grid into subgrids of lower dimensions. On occasions, the information regarding a subgrid may not be available, as in the case where the subgrid communicator was created in one routine and is used in another. In such a situation, MPI_Cartdim_get may be called to find out the dimensions of the subgrid. Armed with this information, additional information may be obtained by calling MPI_Cart_get. We will discuss these routines next.

Note that:

  • full length of each dimension of the original grid are used in the subgrids.
  • there is a comparable MPI_Comm_split routine to perform similar function.
  • MPI_Cartdim_get and MPI_Cart_get can be used to acquire background information of grid (such as dimensions, periodicity).