Why are promoted names invalid when running in parallel? - openmdao

Question

I am trying to run a simple mathematical problem in parallel in OpenMDAO 2.5.0. The problem is an adapted version of the example in the OpenMDAO docs found here: http://openmdao.org/twodocs/versions/latest/features/core_features/grouping_components/parallel_group.html. It has some extra components and connections and uses promotions instead of connections.
from openmdao.api import Problem, IndepVarComp, ParallelGroup, ExecComp, Group, NonlinearBlockGS
prob = Problem()
model = prob.model
model.add_subsystem('p1', IndepVarComp('x1', 1.0), promotes=['x1'])
model.add_subsystem('p2', IndepVarComp('x2', 1.0), promotes=['x2'])
cycle = model.add_subsystem('cycle', Group(), promotes=['*'])
parallel = cycle.add_subsystem('parallel', ParallelGroup(), promotes=['*'])
parallel.add_subsystem('c1', ExecComp(['y1=(-2.0*x1+z)/3']), promotes=['x1', 'y1', 'z'])
parallel.add_subsystem('c2', ExecComp(['y2=(5.0*x2-z)/6']), promotes=['x2', 'y2', 'z'])
cycle.add_subsystem('c3', ExecComp(['z=(3.0*y1+7.0*y2)/10']), promotes=['y1', 'y2', 'z'])
model.add_subsystem('c4', ExecComp(['z2 = y1+y2']), promotes=['z2', 'y1', 'y2'])
cycle.nonlinear_solver = NonlinearBlockGS()
prob.setup(mode='fwd')
prob.set_solver_print(level=2)
prob.run_model()
print(prob['z2'])
print(prob['z'])
print(prob['y1'])
print(prob['y2'])
When I run this code in series, it works as expected with no errors.
However, when I run this code in parallel with:
mpirun -n 2 python Test.py
I get this error for the first process:
RuntimeError: The promoted name y1 is invalid because it refers to multiple inputs: [cycle.c3.y1 ,c4.y1]. Access the value from the connected output variable cycle.parallel.c1.y1 instead.
and this error for the second process:
RuntimeError: The promoted name y2 is invalid because it refers to multiple inputs: [cycle.c3.y2 ,c4.y2]. Access the value from the connected output variable cycle.parallel.c2.y2 instead.
So my question is: why does this example give an error in the promotes names when running in parallel, while it is running without problems in series? Is it only allowed to use connections when running in parallel or are promoted variables okay as well?

Answer 1

As of OpenMDAO V2.5, you have to be a little careful about how you access problem variables in parallel.
If you look at the full stack track of your model, you can see that the error is being thrown right at the end when you call
print(prob['y1'])
print(prob['y2'])
What is happening here is that you have set the model up so that y1 only exists on proc 0 and y2 only exists on proc 1. Then you try to get values that don't exist on that proc and you get (an admittedly not very clear) error.
You can fix this with the following minor modification to your script:
from openmdao.api import Problem, IndepVarComp, ParallelGroup, ExecComp, Group, NonlinearBlockJac
prob = Problem()
model = prob.model
model.add_subsystem('p1', IndepVarComp('x1', 1.0), promotes=['x1'])
model.add_subsystem('p2', IndepVarComp('x2', 1.0), promotes=['x2'])
cycle = model.add_subsystem('cycle', Group(), promotes=['*'])
parallel = cycle.add_subsystem('parallel', ParallelGroup(), promotes=['*'])
parallel.add_subsystem('c1', ExecComp(['y1=(-2.0*x1+z)/3']), promotes=['x1', 'y1', 'z'])
parallel.add_subsystem('c2', ExecComp(['y2=(5.0*x2-z)/6']), promotes=['x2', 'y2', 'z'])
cycle.add_subsystem('c3', ExecComp(['z=(3.0*y1+7.0*y2)/10']), promotes=['y1', 'y2', 'z'])
model.add_subsystem('c4', ExecComp(['z2 = y1+y2']), promotes=['z2', 'y1', 'y2'])
cycle.nonlinear_solver = NonlinearBlockJac()
prob.setup(mode='fwd')
prob.set_solver_print(level=2)
prob.run_model()
print(prob['z2'])
print(prob['z'])
if prob.model.comm.rank == 0:
print(prob['y1'])
if prob.model.comm.rank == 0:
print(prob['y2'])
There are a few minor issues with this. 1) it means your script is now different for serial and parallel. 2) Its annoying. So we're working on a fix that will let things work more cleanly by automatically doing an MPI broadcast when you try to get a value thats no on your proc. That will be released in V2.6.
One other small note. I changed you NL solver over to NonLinearBlockJac. That is for Block Jacobi, which is designed to work in parallel. You could also use the Newton solver in parallel The Gauss-Seidel solver won't actually allow you to get parallel speedups.