OpenMDAO: unexpected behaviour of ParallelGroup - openmdao

Question

In the below example where two ParallelGroups are set up to require different number of procs to compute their serial sub-components, it seems that a group that should not fill up the global comm in fact does, and executes redundant/repeated cases. In previous versions the unassigned procs would not execute their sub-components. It runs without errors though.
from mpi4py import MPI
import openmdao.api as om
class Exec(om.ExplicitComponent):
def __init__(self, val=-10):
super().__init__()
self.val = val
def setup(self):
self.add_output('y', 0.)
def compute(self, inputs, outputs):
print('HELLO from Exec1, %s, global rank %i, val=%f' % (self.name, MPI.COMM_WORLD.rank, self.val))
outputs['y'] = self.val
class Exec2(om.ExplicitComponent):
def __init__(self, val):
super().__init__()
self.val = val
def setup(self):
self.add_input('x', 0.)
self.add_output('y', 0.)
def compute(self, inputs, outputs):
print('HELLO from Exec2, %s, global rank %i, val=%f' % (self.name, MPI.COMM_WORLD.rank, self.val))
outputs['y'] = self.val * inputs['x']
class Summer(om.ExplicitComponent):
def __init__(self, ncase):
super().__init__()
self.ncase = ncase
def setup(self):
for i in range(self.ncase):
self.add_input('y%i' % i, 0.)
self.add_output('sum', 0.)
def compute(self, inputs, outputs):
for i in range(self.ncase):
outputs['sum'] += inputs['y%i' % i]
p = om.Problem()
ncase = 3
par1 = p.model.add_subsystem('par1', om.ParallelGroup())
p.model.add_subsystem('summer1', Summer(ncase))
for i in range(ncase):
par1.add_subsystem('ex_%i' % i, Exec(val=float(i)), min_procs=1)
p.model.connect('par1.ex_%i.y' % i, 'summer1.y%i' % i)
ncase2 = 4
par2 = p.model.add_subsystem('par2', om.ParallelGroup())
p.model.add_subsystem('summer2', Summer(ncase2))
for i in range(ncase2):
par2.add_subsystem('ex_%i' % i, Exec2(float(i)), min_procs=1)
p.model.connect('summer1.sum', 'par2.ex_%i.x' % i)
p.model.connect('par2.ex_%i.y' % i, 'summer2.y%i' % i)
p.setup()
p.run_model()

Answer 1

According to the docs, the processor allocation is done mainly based on proc_weight using a round robin style of allocation.
The algorithm used for the allocation starts, assuming that the number of processes is greater than or equal to the number of subsystems, by assigning the min_procs for each subsystem. It then adds any remaining processes to subsystems based on their weights, being careful not to exceed their specified max_procs, if any.
So the behavior is as expected. OpenMDAO seeks to allocate all the given processors. It is then up to the component author to use them wisely.
If you really want to, you could modify the component so that anything other than Rank 0 did nothing, but I would not recommend that.
As an asside, you should not reference COMM_WORLD like that. Each component has a local comm you should use instead. Here is a modified version of your example:
from mpi4py import MPI
import openmdao.api as om
class Exec(om.ExplicitComponent):
def __init__(self, val=-10):
super().__init__()
self.val = val
def setup(self):
self.add_output('y', 0.)
def compute(self, inputs, outputs):
print('HELLO from Exec1, %s, global rank %i, comm size %i, val=%f' % (self.pathname, self.comm.rank, self.comm.size, self.val))
outputs['y'] = self.val
class Exec2(om.ExplicitComponent):
def __init__(self, val):
super().__init__()
self.val = val
def setup(self):
self.add_input('x', 0.)
self.add_output('y', 0.)
def compute(self, inputs, outputs):
print('HELLO from Exec2, %s, global rank %i, comm size %i, val=%f' % (self.pathname, self.comm.rank, self.comm.size, self.val))
outputs['y'] = self.val * inputs['x']
class Summer(om.ExplicitComponent):
def __init__(self, ncase):
super().__init__()
self.ncase = ncase
def setup(self):
for i in range(self.ncase):
self.add_input('y%i' % i, 0.)
self.add_output('sum', 0.)
def compute(self, inputs, outputs):
for i in range(self.ncase):
outputs['sum'] += inputs['y%i' % i]
p = om.Problem()
ncase = 2
par1 = p.model.add_subsystem('par1', om.ParallelGroup())
p.model.add_subsystem('summer1', Summer(ncase))
for i in range(ncase):
par1.add_subsystem('ex_%i' % i, Exec(val=float(i)), min_procs=1)
p.model.connect('par1.ex_%i.y' % i, 'summer1.y%i' % i)
ncase2 = 4
par2 = p.model.add_subsystem('par2', om.ParallelGroup())
p.model.add_subsystem('summer2', Summer(ncase2))
for i in range(ncase2):
par2.add_subsystem('ex_%i' % i, Exec2(float(i)), min_procs=1, max_procs=1)
p.model.connect('summer1.sum', 'par2.ex_%i.x' % i)
p.model.connect('par2.ex_%i.y' % i, 'summer2.y%i' % i)
p.setup()
p.run_model()
Running that on 4 processors gives:
HELLO from Exec1, par1.ex_0, global rank 0, comm size 2, val=0.000000
HELLO from Exec1, par1.ex_0, global rank 1, comm size 2, val=0.000000
HELLO from Exec1, par1.ex_1, global rank 0, comm size 2, val=1.000000
HELLO from Exec1, par1.ex_1, global rank 1, comm size 2, val=1.000000
HELLO from Exec2, par2.ex_3, global rank 0, comm size 1, val=3.000000
HELLO from Exec2, par2.ex_0, global rank 0, comm size 1, val=0.000000
HELLO from Exec2, par2.ex_1, global rank 0, comm size 1, val=1.000000
HELLO from Exec2, par2.ex_2, global rank 0, comm size 1, val=2.000000
So you see that in par1 each component is given a comm of size 2. This is the duplication that you call wasteful. I argue that it is not wasteful though, due to details of how you have set up the components. Both Exec and Exec2 are serial components (i.e. they do not have self.options['distributed'] = True). OpenMDAO always duplicates any serial component across all ranks in the local group comm that owns that component. The value of this duplication is lower MPI communication overhead. Since the value is computed locally, you can do a local transfer to any other serial components on that proc (rather than having to broadcast from the root).
If you prefer not to have the local value used, you could choose to set src_indices=[0] in the connect statement yourself. Then you would force OpenMDAO to broadcast from the root of that comm. The duplicate calculation would still occur though. It should not waste any time, since the duplicate proc would have been sitting idle while the root proc did the calculations. You can argue that it wastes some electricity because of the extra calculations. In most cases, this cost would be trivially small, but if you are concerned about it, you can change the components to be distributed, and set the variable sizes to 0 on all ranks except the root. Then you can set things up to no duplicate calcs.
Our experience is that most of the time, communication overhead is what you want to avoid. This is why we designed it to duplicate, but you do have the freedom to work around it if you like.