Multiple Evaluations of the External Code Component With the same input? OpenMDAO - openmdao

Question

I am setting up an optimization routine with three components using SLSQP, external code, explicit comp and FD differentiation.
Comp1 : IndepVarComp with output ='x'
Comp2 : External Code input ='x' / output = 'f','g' - derivatives
='fd'
Comp3 : Explicit Comp input ='f','g' /
output = 'obj' - analytical
derivatives declared
More specifically the Comp2 is similar to the paraboloid example and Comp3 is just a linear function. There are no constraints.
If I use the central FD on the Comp2, it evaluates the External Code with the same input value (x) 4 times before it changes it, with the forward FD this is two times. I understand vaguely what the optimizer is trying to do but
repeating the same calculation is just a computational burden.
My remedy is to save the output file with the corresponding input name and in the second iteration check if this file exists, if so skip the calculation and read in from file. This works fine.
My questions are :
--- I was wondering should there be an embedded way to avoid re-evaluating the same function?
--- Even though I claim I understand a bit what the optimizer does, it is not clear why it tries to find a gradient without stepping.
Important Note :
This does not happen if Comp2 has two inputs, one output.
It happens again if Comp2 has two inputs, two outputs
Below an example output of the first few iterations.
ExtInX = [10.], , ExtOutF = 79.0, ExtOutG = 373.0
OBJECTIVE 589362.0
----------------------------------------------------------------------
ExtInX = [10.], , ExtOutF = 79.0, ExtOutG = 373.0
OBJECTIVE 589362.0
----------------------------------------------------------------------
ExtInX = [10.], , ExtOutF = 79.0, ExtOutG = 373.0
ExtInX = [10.], , ExtOutF = 79.0, ExtOutG = 373.0
ExtInX = [10.], , ExtOutF = 79.0, ExtOutG = 373.0
ExtInX = [10.], , ExtOutF = 79.0, ExtOutG = 373.0
ExtInX = [10.000001], , ExtOutF = 79.00001700000098, ExtOutG = 373.0001500000209
ExtInX = [9.999999], , ExtOutF = 78.99998300000101, ExtOutG = 372.9998500000211
ExtInX = [8.07102609], , ExtOutF = 49.928382692675996, ExtOutG = 154.61605669448198
OBJECTIVE 250797.01369955452
----------------------------------------------------------------------
**ExtInX = [8.07102609], , ExtOutF = 49.928382692675996, ExtOutG = 154.61605669448198
ExtInX = [8.07102609], , ExtOutF = 49.928382692675996, ExtOutG = 154.61605669448198
ExtInX = [8.07102609], , ExtOutF = 49.928382692675996, ExtOutG = 154.61605669448198
ExtInX = [8.07102609], , ExtOutF = 49.928382692675996, ExtOutG = 154.61605669448198**
ExtInX = [8.07102709], , ExtOutF = 49.92839583472901, ExtOutG = 154.61613684041134
ExtInX = [8.07102509], , ExtOutF = 49.92836955062501, ExtOutG = 154.61597654858318
ExtInX = [4.88062761], , ExtOutF = 18.178645674383997, ExtOutG = 21.29321703417343
OBJECTIVE 38811.35361617729
AFTER THE FIRST ANSWER
Thank you for your answer.
From that I understand that in 2.2 the caching has to be done by the user and
one can do that in different ways. This is fine.
And the improvement you mention in 2.3 seems to fix one thing.
The single extra call in the beginning. I am more concerned about the
several calls with the same input during iterations.
I am adding the four code patches below. One should be able to run the optimizer
as long as the others are in the same folder.
In this case the external code is pure python. But simultaneously I am
developing an optimizer that uses an 'actual' external code. It is again
wrapped in python though. So some system calls are sent to the executable
and python handles the outputs from the external code, python does some post
processing and outputs a text file which is read by the OpenMDAO external code
component. So not sure if that one counts as pure python.
Regarding the constraints : In the below code there is none. In the other code
that I had explained there are some lifetime constraints for car components
(e.g. xx, lower=20).
CODE :
Optimizer.py : Driver
and other 3 .py files
https://gist.github.com/anonymous/2c9d5d182cbb24a2334c97b57a954802

Answer 1

In OpenMDAO v2.2 there is no build in way to handle result caching. So you have to roll your own solution there. Instead of saving the full output file, you could do the caching in memory and just store the last known input values. If they haven't changed, you can just leave the output values where they are.
In OpenMDAO V2.2 the drivers are coded to make one extra function call before even starting the optimization. This is done because of the way OpenMDAO handles linear constraints. It will pre-compute the derivatives associated with the linear constraints and store them.
In OpenMDAO V2.3 we've modified the drivers so they only make this extra call if you have linear constraints in your problem. So that, at least, explains one of your extra functions calls.
After examining your test problem there does appear to be a bug in the way OpenMDAO is finite-differencing the components. The problem shows up even if you use forward mode, and it appears related to the way you're declaring the partials.
There isn't anything wrong with the way your declared them, but with an alternate method of declaration I was able to get it to behave properly:
from openmdao.api import ExternalCode
import numpy as np
class ParaboloidExternalCode(ExternalCode):
def setup(self):
self.add_input('x', val=0.0)
self.add_input('y', val=0.0)
self.add_output('f_xy', val=0.0)
self.add_output('g_xy', val=0.0)
self.input_file = 'paraboloid_input.dat'
self.output_file = 'paraboloid_output.dat'
# providing these is optional; the component will verify that any input
# files exist before execution and that the output files exist after.
self.options['external_input_files'] = [self.input_file,]
self.options['external_output_files'] = [self.output_file,]
self.options['command'] = [
'python', 'extcode_paraboloid.py', self.input_file, self.output_file
]
# self.declare_partials(of='*', wrt='*', method='fd', form='central', step=1e-6)
self.declare_partials(of='f_xy', wrt=['x','y'], method='fd', form='central', step=1e-6)
self.declare_partials(of='g_xy', wrt=['x','y'], method='fd', form='central', step=1e-6)
def compute(self, inputs, outputs):
x = inputs['x']
y = inputs['y']
# generate the input file for the paraboloid external code
with open(self.input_file, 'w') as input_file:
input_file.write('%f\n%f\n' % (x,y))
# the parent compute function actually runs the external code
super(ParaboloidExternalCode, self).compute(inputs, outputs)
file_contents = np.loadtxt(self.output_file)
outputs['f_xy'] = file_contents [0]
outputs['g_xy'] = file_contents [1]
print('ExtInX = {}, ExtInY = {},, ExtOutF_XY = {}, ExtOutG_XY = {} '.format(x,y,file_contents [0],file_contents[1]) )
I got rid of the of='*', wrt='*' and replaced it with two separate calls which explicitly name the variables. I was also able to replicate this problem without the external code at all all, using an in-memory paraboloid. I've entered the bug into the the development backlog and we'll get it sorted out. In the meantime, this workaround solves the problem.

Answer 2

Please note that there was a bug in OpenMDAO V2.2 that did cause extra evaluations. It was fixed via OpenMDAO PR # 553