I'm trying to demux an array with 3 axis as seen in the following sample code
import openmdao.api as om
p = om.Problem()
ivc = p.model.add_subsystem('idv', om.IndepVarComp(), promotes=['*'])
ivc.add_output(name='f0', shape=(2,3), units='m', val=[[111,112,113], [121,122,123]])
ivc.add_output(name='f1', shape=(2,3), units='m', val=[[211,212,213], [221,222,223]])
ivc.add_output(name='f2', shape=(2,3), units='m', val=[[311,312,313], [321,322,323]])
ivc.add_output(name='f3', shape=(2,3), units='m', val=[[411,412,413], [421,422,423]])
mux_comp = p.model.add_subsystem(name='mux', subsys=om.MuxComp(vec_size=4))
mux_comp.add_var('r', shape=(2,3), axis=0, units='m')
demux_comp = p.model.add_subsystem(name='demux', subsys=om.DemuxComp(vec_size=4))
demux_comp.add_var('g', shape=(4,2,3), axis=0, units='m')
p.model.connect('f0', 'mux.r_0')
p.model.connect('f1', 'mux.r_1')
p.model.connect('f2', 'mux.r_2')
p.model.connect('f3', 'mux.r_3')
p.model.connect('mux.r', 'demux.g')
p.setup()
p.run_model()
print(p['mux.r'])
print(p['mux.r'].shape)
print(p['demux.g_0'])
print(p['demux.g_1'])
print(p['demux.g_2'])
print(p['demux.g_3'])
print(p['demux.g_0'].shape)
When this runs, I get the following error
RuntimeError: DemuxComp (demux): d(g_0)/d(g): declare_partials has been called with rows and cols, which should be arrays of equal length, but rows is length 6 while cols is length 2.
As the error only pertained to the partials, I took a look into the demux_comp.py code in the OpenMDAO library and modified the declare_partials line from
self.declare_partials(of=out_name, wrt=var, rows=rs, cols=cs, val=1.0)
to
self.declare_partials(of=out_name, wrt=var, val=1.0)
This allowed the code to run successfully and output the proper demuxed variables. Will this have any adverse effects on the rest of my code and optimizations?
It looks like you've uncovered a bug. We'll get on fixing this.
While your fix will allow it to run, the partials will be incorrect. In the mean time you'd be better off replacing the declare partials call with instructions to use finite difference or complex step.
self.declare_partials(of=out_name, wrt=var, method='cs')
For complex-step, or method='fd' for finite difference.
Related
this will be a long question. I’m trying to define my own custom objective function
I want the XGBClassifier, so I run
from xgboost import XGBClassifier
the documentation of xgboost says:
A custom objective function can be provided for the objective parameter. In this case, it should have the signature
objective(y_true, y_pred) -> grad, hess :
y_true: array_like of shape [n_samples], The target values
y_pred: array_like of shape [n_samples], The predicted values
grad: array_like of shape [n_samples], The value of the gradient for each sample point.
hess: array_like of shape [n_samples], The value of the second derivative for each sample point
Now, I’ve coded this custom:
def guess_averse_loss(y_true, y_pred):
y_true = y_true.astype(int)
y_pred = y_pred.astype(int)
... stuffs ...
return grad, hess
everything is compatible with the previous documentation.
If I run:
classifier=XGBClassifier(eval_metric=custom_weighted_accuracy,objective=guess_averse_loss,**params_common_model)
classifier.train(X_train, y_train)
(where custom_weighted_accuracy is a custom metric defined by me following the documentation of scikitlearn)
I get the error:
-> first_term = np.multiply(cost_matrix[y_true, y_pred], np.exp(y_pred - y_true))
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (4043,) (4043,5)
So, y_pred enters the function as a matrix (n_samples x n_classes) where the element ij is the probability that the sample i belongs to the class j.
Then, I modify the line as
first_term = np.multiply(cost_matrix[y_true, np.argmax(y_pred, axis=1)],np.exp(np.argmax(y_pred, axis=1) - y_true))
so it passes from a matrix to an array,
This leads to the error:
unknown custom metric
so it seems that the problem now is the metric.
I try to remove the custom obj function using the default one and another error comes:
XGBoostError: Check failed: in_gpair->Size() % ngroup == 0U (3 vs. 0) : must have exactly ngroup * nrow gpairs
WHAT CAN I DO???
You read what I've tried, I'm excepting some suggestion to solve this problems
I am currently working on doing graph classification on the IMDB-Binary dataset using deep learning and specifically the pytorch geometric environment.
I have split my data into test/train samples that are list of tuples containing a graph and its label. One thing I've had to do is to treat the different graph as a "Batch", a large disconnected graph, using torch_geometric.data.Batch. To start, I am using a data loader with the following collate function
def collate(samples) :
graphs,labels = map(list,zip(*samples))
datalist = make_datalist(graphs)
datalist = Batch.from_data_list(datalist)
return datalist, torch.tensor(labels)
and my classifier is the following :
class Classifier(nn.Module):
def __init__(self, in_dim, hidden_dim, n_classes):
super(Classifier, self).__init__()
self.conv1 = GraphConv(in_dim, hidden_dim)
self.conv2 = GraphConv(hidden_dim, hidden_dim)
self.classify = nn.Linear(hidden_dim, n_classes)
def forward(self, g):
# Use node degree as the initial node feature. For undirected graphs, the in-degree
# is the same as the out_degree.
h = g.in_degrees
# Perform graph convolution and activation function.
h = F.relu(self.conv1(g, h))
h = F.relu(self.conv2(g, h))
g.ndata['h'] = h
# Calculate graph representation by averaging all the node representations.
hg = dgl.mean_nodes(g, 'h')
return self.classify(hg)
Which simply averages the nodes representations of each graph, and feeds it to a MLP
The problem I come up with is that during the prediction of our batch, I have the error
AttributeError: 'Batch' object has no attribute 'local_var'
and I can't find where it may come from, would anyone know ?
Thank you for taking the time to read !
I am also experimenting with Pytorch geometric and its' data set capabilities.
Maybe following information will help someone in the future:
I'm facing AttributeErrors when forgetting to set #property annotated getters/setters for my data set class attributes. See https://docs.python.org/3.7/library/functions.html#property
I think to answer your question we need more information about your make_datalist function.
However, here are the links to the batch class:
https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html
https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/data/batch.html#Batch
And indeed, there is nothing like a local_var variable.
I noticed that prob.compute_totals() returns a wrong answer when prob.model.approx_totals() is not specified before. Having the partial derivative manually defined or computed by finite differences doesn't change anything, the answer remains wrong when not calling prob.model.approx_totals() before. Also, the call to compute_totals is actually faster when approx_totals is called before, compared to when it's not. This seems counter-intuitive with manually defined partials, since approx_totals is supposed to add an unnecessary finite-difference computation.
Here is a MWE with the Sellar example taken from the OpenMDAO documentation. I also noticed the same behaviour in OpenAeroStruct, even though the differences are smaller than in this example.
import openmdao.api as om
from openmdao.test_suite.components.sellar_feature import SellarMDA
prob = om.Problem()
prob.model = SellarMDA()
prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'
prob.driver.options['tol'] = 1e-8
prob.model.add_design_var('x', lower=0, upper=10)
prob.model.add_design_var('z', lower=0, upper=10)
prob.model.add_objective('obj')
prob.model.add_constraint('con1', upper=0)
prob.model.add_constraint('con2', upper=0)
prob.setup()
prob.set_solver_print(level=0)
prob.model.approx_totals() # Commenting this line gives the wrong result
prob.run_driver()
totals = prob.compute_totals(of=['obj'],wrt=['x','z'])
print("""
Obj = {}
x = {}
z = {}
y1 = {}
y2 = {}
Totals = {}""".format(prob['obj'][0],prob['x'][0],prob['z'][0],prob['y1'][0],prob['y2'][0],totals))
The good result, with approx_totals :
Optimization terminated successfully. (Exit mode 0)
Current function value: 3.183393951729169
Iterations: 6
Function evaluations: 6
Gradient evaluations: 6
Optimization Complete
-----------------------------------
Obj = 3.183393951729169
x = 0.0
z = 1.977638883487764
y1 = 3.1600000000897133
y2 = 3.755277766976125
Totals = OrderedDict([(('obj', 'x'), array([[0.94051147]])), (('obj', 'z'), array([[3.50849282, 1.72901602]]))])
The wrong result, whithout approx_totals :
Optimization terminated successfully. (Exit mode 0)
Current function value: 3.1833939532752136
Iterations: 11
Function evaluations: 12
Gradient evaluations: 11
Optimization Complete
-----------------------------------
Obj = 3.1833939532752136
x = 4.401421628747386e-15
z = 1.9776388839289216
y1 = 3.1600000016563765
y2 = 3.755277767857951
Totals = OrderedDict([(('obj', 'x'), array([[0.99341446]])), (('obj', 'z'), array([[3.90585351, 1.97002055]]))])
In this example, the problem is that you have a cycle in SellarMDA, but the model does not contain a linear solver that can compute the total derivatives across the cycle. One way you can check on this is to run "openmdao check myfilename.py" at the command-line. I ran it on your model, and got the following warnings:
INFO: checking comp_has_no_outputs
INFO: checking dup_inputs
INFO: checking missing_recorders
WARNING: The Problem has no recorder of any kind attached
INFO: checking out_of_order
INFO: checking solvers
WARNING: Group 'cycle' contains cycles [['d1', 'd2']], but does not have an iterative linear solver.
INFO: checking system
There are a couple of remedies for this. You could manually add a different linear solver such as DirectSolver or PETScKrylov to the "cycle" group. You could also import SellarMDALinearSolver instead of SellarMDA. SellarMDALinearSolver uses a Newton solver for converging the cycle, and a DirectSolver for computing the derivatives. SellarMDA uses NonlinearBlockGS to converge the cycle, but unfortunately does not contain an appropriate linear solver to compute the derivatives. These components are used in a variety of testing roles, but I think in retrospect, we should probably add a LinearBlockGS to SellarMDA in the future, so that total derivatives can be computed without modification. For now though, you'll have to use SellarMDALinearSolver or add the solver yourself.
BTW, I suspect the optimization was slower because the derivatives were so bad. It took twice as many iterations, though it still somehow managed to get pretty close to the answer.
You mentioned similar symptoms in your OpenAeroStruct model. I would suspect that either 1) a subcomponent has an error in its analytical derivatives, or 2) the linear solvers are not set up correctly (maybe you have a cycle somewhere without a good linear solver in that group or parent group.) I think that Problem.check_partials and Problem.check_totals will give you more insight on where the problem could be. There is more info on these here.
Below there is a sample code where the BSplineComp is combined either with an ExplicitComp or ExternalCodeComp.
Both of these two do the same calculation and both of the components' gradients are calculated using finite difference.
If I run the version Bspline+ExplicitComp the result is achieved within 2,3 iterations.
If I run the version Bspline+ExternalCodeComp I have to wait a lot. In this case it is trying to find the gradient of the output with respect to each input. So for example there are 9 control points that are interpolated to 70 points in the bspline component. Then the external component has to be evaluated as many as the interpolated points (70 times)
So in a case where the bspline is combined with an expensive external code the finite difference requires as much as the number of points it is interpolated to which becomes the bottleneck of the computation.
Based on this input I have two questions
1- If external code component is based on the explicit component what is the major difference that causes the behaviour difference? (considering both have an input of shape=70)
2- In the previously mentioned scneario where the bspline is combined with an expensive external code would there be a more efficient way of combining them apart from the way it is shown here.
MAIN CODE: 'external' variable is the flag for toggling external/explicit code comp. set that true/false for running the two cases explained above.
from openmdao.components.bsplines_comp import BsplinesComp
from openmdao.api import IndepVarComp, Problem, ExplicitComponent,ExecComp,ExternalCodeComp
from openmdao.api import ScipyOptimizeDriver, SqliteRecorder, CaseReader
import matplotlib.pyplot as plt
import numpy as np
external=True # change this to true for the case with external code comp. or false for the case with explicit comp.
rr=np.arange(0,70,1)
"Explicit component for the area under the line calculation"
class AreaComp(ExplicitComponent):
def initialize(self):
self.options.declare('lenrr', int)
self.options.declare('rr', types=np.ndarray)
def setup(self):
self.add_input('h', shape=lenrr)
self.add_output('area')
self.declare_partials(of='area', wrt='h', method='fd')
def compute(self, inputs, outputs):
rr = self.options['rr']
outputs['area'] = np.trapz(rr,inputs['h'])
class ExternalAreaComp(ExternalCodeComp):
def setup(self):
self.add_input('h', shape=70)
self.add_output('area')
self.input_file = 'paraboloid_input.dat'
self.output_file = 'paraboloid_output.dat'
# providing these is optional; the component will verify that any input
# files exist before execution and that the output files exist after.
self.options['external_input_files'] = [self.input_file]
self.options['external_output_files'] = [self.output_file]
self.options['command'] = [
'python', 'extcode_paraboloid.py', self.input_file, self.output_file
]
# this external code does not provide derivatives, use finite difference
self.declare_partials(of='*', wrt='*', method='fd')
def compute(self, inputs, outputs):
h = inputs['h']
# generate the input file for the paraboloid external code
np.savetxt(self.input_file,h)
# the parent compute function actually runs the external code
super(ExternalAreaComp, self).compute(inputs, outputs)
# parse the output file from the external code and set the value of f_xy
f_xy=np.load('a.npy')
outputs['area'] = f_xy
prob = Problem()
model = prob.model
n_cp = 9
lenrr = len(rr)
"Initialize the design variables"
x = np.random.rand(n_cp)
model.add_subsystem('px', IndepVarComp('x', val=x))
model.add_subsystem('interp', BsplinesComp(num_control_points=n_cp,
num_points=lenrr,
in_name='h_cp',
out_name='h'))
if external:
comp=ExternalAreaComp()
model.add_subsystem('AreaComp', comp)
else:
comp = AreaComp(lenrr=lenrr, rr=rr)
model.add_subsystem('AreaComp', comp)
case_recorder_filename2 = 'cases4.sql'
recorder2 = SqliteRecorder(case_recorder_filename2)
comp.add_recorder(recorder2)
comp.recording_options['record_outputs']=True
comp.recording_options['record_inputs']=True
model.connect('px.x', 'interp.h_cp')
model.connect('interp.h', 'AreaComp.h')
model.add_constraint('interp.h', lower=0.9, upper=1, indices=[0])
prob.driver = ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'
prob.driver.options['disp'] = True
#prob.driver.options['optimizer'] = 'COBYLA'
#prob.driver.options['disp'] = True
prob.driver.options['tol'] = 1e-9
model.add_design_var('px.x', lower=1,upper=10)
model.add_objective('AreaComp.area',scaler=1)
prob.setup(check=True)
#prob.run_model()
prob.run_driver()
cr = CaseReader(case_recorder_filename2)
case_keys = cr.system_cases.list_cases()
cou=-1
for case_key in case_keys:
cou=cou+1
case = cr.system_cases.get_case(case_key)
plt.plot(rr,case.inputs['h'],'-*')
The external code extcode_paraboloid.py is below
import numpy as np
if __name__ == '__main__':
import sys
input_filename = sys.argv[1]
output_filename = sys.argv[2]
h=np.loadtxt(input_filename)
rr=np.arange(0,70,1)
rk= np.trapz(rr,h)
np.save('a',np.array(rk))
In both cases your code takes 3 iterations to run. The wall time for the external code is much much longer simply because of the cost of file-io plus the requirement to make a system call to spool up a new process each time your function is called.
Yep, system calls are that expensive and file i/o isn't cheap either. If you have a more costly analysis its less of a big deal, but you can see why it should be avoided if at all possible.
In this case you can reduce your FD cost though. Since you have only 9 bspline variables, you have correctly deduced that you could run far fewer FD steps. You want to use the approximate semi-total derivative feature in OpenMDAO v2.4 to set up FD across the group instead of across each individual component.
Its as simple as this:
.
.
.
if external:
comp=ExternalAreaComp()
model.add_subsystem('AreaComp', comp)
else:
comp = AreaComp(lenrr=lenrr, rr=rr)
model.add_subsystem('AreaComp', comp)
model.approx_totals()
.
.
.
maybe someone is able to help me here. I am trying to compute the cross entropy loss of a given output of my network
print output
Variable containing:
1.00000e-02 *
-2.2739 2.9964 -7.8353 7.4667 4.6921 0.1391 0.6118 5.2227 6.2540
-7.3584
[torch.FloatTensor of size 1x10]
and the desired label, which is of the form
print lab
Variable containing:
x
[torch.FloatTensor of size 1]
where x is an integer between 0 and 9.
According to the pytorch documentation (http://pytorch.org/docs/master/nn.html)
criterion = nn.CrossEntropyLoss()
loss = criterion(output, lab)
this should work, but unfortunately I get a weird error
TypeError: FloatClassNLLCriterion_updateOutput received an invalid combination of arguments - got (int, torch.FloatTensor, !torch.FloatTensor!, torch.FloatTensor, bool, NoneType, torch.FloatTensor, int), but expected (int state, torch.FloatTensor input, torch.LongTensor target, torch.FloatTensor output, bool sizeAverage, [torch.FloatTensor weights or None], torch.FloatTensor total_weight, int ignore_index)
Can anyone help me? I am really confused and tried almost everything I could imagined to be helpful.
Best
Please check this code
import torch
import torch.nn as nn
from torch.autograd import Variable
output = Variable(torch.rand(1,10))
target = Variable(torch.LongTensor([1]))
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)
This will print out the loss nicely:
Variable containing:
2.4498
[torch.FloatTensor of size 1]