Abstract
Parallel programs often express operations on a
subset (process group) of all the participating processes or ranks.
Subcommunicators in MPI are an example of such process
groups. Often, these process groups are used only for simple
collective communication (broadcast, reduction, allreduce) over
the members of the process group.
Current algorithms to create process groups tend to be
centralized schemes which store or manipulate data structures
of size proportional to the number of members in the process
group. In extreme scale parallel architectures, these algorithms
will consume a prohibitive amount of memory to manage the
process group and also suffer from scalability limits.
In this paper, we contend that MPI subcommunicators pack
more features than are necessary for a sizeable fraction of the
use cases. We make a case for system-ranked process groups,
intended primarily for simple collective operations. As opposed
to MPI communicators, member ranks in system-ranked process
groups are assigned by the runtime system.
This paper presents distributed algorithms for the creation of
spanning trees for rudimentary collective communication over
system-ranked process groups that are obtained by splitting an
original parent process group. Our schemes use only a small
constant amount of memory per node and also perform better
than a reference centralized scheme even at modest process
counts. We also demonstrate performance results up to 131, 072
cores of BlueGene/P.
Our algorithms can apply to the creation of MPI subcom-
municators as well as to equivalent entities in other parallel
programming paradigms.