I am kind of new on Chainer and I have been struggling with a weird situation recently.
I have a Chain to compute a CNN which I feed with a labeledDataSet.
But no results appears when I use the extensions. When I display the observation value it is empty. But the loss is indeed calculated and the parameters updated (at least they change) so I don't know where is the connection problem.
def convert(batch, device):
return chainer.dataset.convert.concat_examples(batch, device, padding=0)
def print_obs(t):
print("trainer.observation", trainer.observation)
print("updater.loss", updater.loss_func)
print("conv1", model.predictor.conv1.W[0][0])
print("conv20", model.predictor.conv20.W[0][0])
model.predictor.train = True
model.predictor.finetune = False ####or True ??
cuda.get_device(0).use()
model.to_gpu()
optimizer = optimizers.MomentumSGD(lr=learning_rate, momentum=momentum)
optimizer.use_cleargrads()
optimizer.setup(model)
optimizer.add_hook(chainer.optimizer.WeightDecay(weight_decay))
train, test = imageNet_data.train_val_test()
train_iter = iterators.SerialIterator(train, batch_size)
test_iter = iterators.SerialIterator(test, batch_size, repeat=False,shuffle=False)
with chainer.using_config('debug', True):
# Set up a trainer
updater = training.StandardUpdater(train_iter, optimizer, loss_func=model, converter=convert)
trainer = training.Trainer(updater, (10, 'epoch'), out="./backup/result")
trainer.extend(print_obs, trigger=(3, 'iteration'))
trainer.extend(extensions.LogReport())
trainer.extend(extensions.PrintReport(
['epoch', 'main/loss', 'validation/main/loss',
'main/accuracy', 'validation/main/accuracy', 'elapsed_time']))
trainer.run()
Maybe this is something is miss completely and which is quite obvious.. Thank you for all remarks that would help me a lot.
Chainer4.1, Ubuntu16
If you are using your own Link with the Trainer, you need to report metrics using chainer.report by your own.
See https://docs.chainer.org/en/stable/guides/report.html for instructions.
You can see some examples in Chainer repository:
https://github.com/chainer/chainer/blob/v4.1.0/chainer/links/model/classifier.py#L116
https://github.com/chainer/chainer/blob/v4.1.0/examples/imagenet/alex.py#L40
Related
I have started learning Python not to long ago and have decided to try making a linear search algorithim. The problem that seems to exist is that found is never = true therefore never triggering the print. The program is attached below. Any help would be greatly appreciated!
numbers = [55,37,12,13,89,47,3,24,21]
number_to_find = input("Enter a number to find:")
found = False
for index, single_num in enumerate(numbers):
if numbers[index] == number_to_find:
found = True
break
if found == True:
print(f"Found {number_to_find} at index {index}")
else:
print(f"Unable to find {number_to_find} in array")
I'm following this tutorial that codes a sentiment analysis classifier using BERT with the huggingface library and I'm having a very odd behavior. When trying the BERT model with a sample text I get a string instead of the hidden state. This is the code I'm using:
import transformers
from transformers import BertModel, BertTokenizer
print(transformers.__version__)
PRE_TRAINED_MODEL_NAME = 'bert-base-cased'
PATH_OF_CACHE = "/home/mwon/data-mwon/paperChega/src_classificador/data/hugingface"
tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME,cache_dir = PATH_OF_CACHE)
sample_txt = 'When was I last outside? I am stuck at home for 2 weeks.'
encoding_sample = tokenizer.encode_plus(
sample_txt,
max_length=32,
add_special_tokens=True, # Add '[CLS]' and '[SEP]'
return_token_type_ids=False,
padding=True,
truncation = True,
return_attention_mask=True,
return_tensors='pt', # Return PyTorch tensors
)
bert_model = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME,cache_dir = PATH_OF_CACHE)
last_hidden_state, pooled_output = bert_model(
encoding_sample['input_ids'],
encoding_sample['attention_mask']
)
print([last_hidden_state,pooled_output])
that outputs:
4.0.0
['last_hidden_state', 'pooler_output']
While the answer from Aakash provides a solution to the problem, it does not explain the issue. Since one of the 3.X releases of the transformers library, the models do not return tuples anymore but specific output objects:
o = bert_model(
encoding_sample['input_ids'],
encoding_sample['attention_mask']
)
print(type(o))
print(o.keys())
Output:
transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions
odict_keys(['last_hidden_state', 'pooler_output'])
You can return to the previous behavior by adding return_dict=False to get a tuple:
o = bert_model(
encoding_sample['input_ids'],
encoding_sample['attention_mask'],
return_dict=False
)
print(type(o))
Output:
<class 'tuple'>
I do not recommend that, because it is now unambiguous to select a specific part of the output without turning to the documentation as shown in the example below:
o = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'], return_dict=False, output_attentions=True, output_hidden_states=True)
print('I am a tuple with {} elements. You do not know what each element presents without checking the documentation'.format(len(o)))
o = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'], output_attentions=True, output_hidden_states=True)
print('I am a cool object and you can acces my elements with o.last_hidden_state, o["last_hidden_state"] or even o[0]. My keys are; {} '.format(o.keys()))
Output:
I am a tuple with 4 elements. You do not know what each element presents without checking the documentation
I am a cool object and you can acces my elements with o.last_hidden_state, o["last_hidden_state"] or even o[0]. My keys are; odict_keys(['last_hidden_state', 'pooler_output', 'hidden_states', 'attentions'])
I faced the same issue while learning how to implement Bert. I noticed that using
last_hidden_state, pooled_output = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'])
is the issue. Use:
outputs = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'])
and extract the last_hidden state using
output[0]
You can refer to the documentation here which tells you what is returned by the BertModel
I would like to test GQ-CNN which is network in Dex-Net on tensorRT.
I successfully converted tflite file to uff file but when I tried to inference with that network, there is an error I couldn't figure out.
[TensorRT] ERROR: Parameter check failed at: ../builder/Network.cpp::addLRN::149, condition: lrnWindow & 0x1
python3: uff/orders.cpp:330: void UffParser::addTranspose(ParserLayer&, std::vector<int>): Assertion `outputs.size() == 1' failed.
The error is appeared when building model.
I tried to find clue from google but there are no codes and no references.
There's only different thing compare with example code that works well.
(I wrote captions which codes I added. If I remove that codes and replace model file to single input network, it works well.)
I registered input twice like below code because GQ-CNN has multiple input.
So I guess that registering multiple input using uffparser could be the main reason of that error.
class ModelData(object):
MODEL_FILE = "./gqcnn.uff"
INPUT_NAME_1 = "Placeholder"
INPUT_SHAPE_1 = (1, 32, 32)
INPUT_NAME_2 = "Placeholder_1"
INPUT_SHAPE_2 = (2,)
OUTPUT_NAME = "softmax/Softmax"
def build_engine(model_file):
# For more information on TRT basics, refer to the introductory samples.
with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
builder.max_workspace_size = common.GiB(1)
builder.fp16_mode = True
#builder.int8_mode = True
# Parse the Uff Network
parser.register_input(ModelData.INPUT_NAME_1, ModelData.INPUT_SHAPE_1)
parser.register_input(ModelData.INPUT_NAME_2, ModelData.INPUT_SHAPE_2) # added code
parser.register_output(ModelData.OUTPUT_NAME)
parser.parse(model_file, network)
# Build and return an engine.
return builder.build_cuda_engine(network)
# do inference
with build_engine(ModelData.MODEL_FILE) as engine:
# Build an engine, allocate buffers and create a stream.
# For more information on buffer allocation, refer to the introductory samples.
inputs, outputs, bindings, stream = common.allocate_buffers(engine)
with engine.create_execution_context() as context:
for idx in range(len(val_images)) :
start = time.time()
val_image = val_images[idx]
val_pose = val_poses[idx] # added code
np.copyto(inputs[0].host, val_image)
np.copyto(inputs[1].host, val_pose) # added code
[prediction] = common.do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)
Is there anyone who succeeded to inference with multiple input model?
I've tried to handle non-converging points in the compute method of my ExplicitComponent, by raising an AnalysisError, as suggested in What is the best way to tell openMDAO driver or solver that it is impossible to evaluate the model at some point? (originally I wanted to make a comment in this thread, but I wasn't allowed due to my low Stack Overflow reputation score). However, this doesn't seem to solve my problem. What I expected was that error would be caught, the design point would be skipped and that the optimizer would continue to evaluate other points in order to find a solution. It is correct that the error is caught, but for some reason, the error is then reraised in ScipyOptimizeDriver.run. What is the purpose of this?
This is an example script for reproducing the behaviour:
import numpy as np
from openmdao.api import Problem, Group, IndepVarComp, ExplicitComponent, ScipyOptimizeDriver, ScipyKrylov, AnalysisError
class Test1Comp(ExplicitComponent):
def setup(self):
self.add_input('design_x', 1.0)
self.add_input('design_y', 1.0)
self.add_input('design_z', 0.5)
self.add_output('y', val=0.1)
self.add_output('z', val=0.1)
self.add_output('obj', val=0.0)
self.declare_partials(of='*', wrt='*', method='fd', form='central', step=1.0e-4)
def compute(self, inputs, outputs):
design_z = inputs['design_z']
design_x = inputs['design_x']
design_y = inputs['design_y']
# Let's assume we have a model that has problems converging around design_x = 5.0
if 0.49999 < design_x < 0.500001:
raise AnalysisError()
z = 4/(design_z + 1)
y = - design_z - 2*z
obj = (y/5.833333 - design_x)**2 + z/2.666667*100*(design_y - design_x**2)**2
outputs["z"] = z
outputs["y"] = y
outputs['obj'] = obj
if __name__ == "__main__":
prob = Problem()
model = prob.model = Group()
model.add_subsystem('d1', IndepVarComp('design_x', 1.0))
model.add_subsystem('d2', IndepVarComp('design_y', 1.0))
model.add_subsystem('d3', IndepVarComp('design_z', 0.5))
model.add_subsystem('comp', Test1Comp())
model.connect('d1.design_x', 'comp.design_x')
model.connect('d2.design_y', 'comp.design_y')
model.connect('d3.design_z', 'comp.design_z')
prob.driver = ScipyOptimizeDriver()
prob.driver.options["optimizer"] = 'SLSQP'
prob.driver.options['tol'] = 1e-8
model.add_design_var("d1.design_x", lower=0.5, upper=1.5)
model.add_design_var("d2.design_y", lower=0.5, upper=1.5)
model.add_design_var("d3.design_z", lower=0.0, upper=1.0)
model.add_objective('comp.obj')
model.linear_solver = ScipyKrylov()
model.linear_solver.options['maxiter'] = int(1200)
model.linear_solver.options['restart'] = int(20)
# prob.model.approx_totals()
prob.setup()
prob.run_driver()
print(prob['comp.y'])
print(prob['comp.z'])
Futrthermore, when looking at ExplicitComponent._solve_nonlinear, which is the method calling ExplicitComponent.compute, it appears to me that the natural way of communicating to OpenMDAO that a point is not converging would be to have ExplicitComponent.compute return True. See the source code for the method:
def _solve_nonlinear(self):
"""
Compute outputs. The model is assumed to be in a scaled state.
Returns
-------
boolean
Failure flag; True if failed to converge, False is successful.
float
absolute error.
float
relative error.
"""
super(ExplicitComponent, self)._solve_nonlinear()
with Recording(self.pathname + '._solve_nonlinear', self.iter_count, self):
with self._unscaled_context(
outputs=[self._outputs], residuals=[self._residuals]):
self._residuals.set_const(0.0)
failed = self.compute(self._inputs, self._outputs)
return bool(failed), 0., 0.
In summary, could someone clarify what is the recommended way of handling non-converging computations in ExplicitComponent.compute?
I have looked at your code, and you specified everything the correct way for telling an optimizer that the component could not evaluate the design point. The problem is that scipy.minimize (which is the optimizer underneath theScipyOptimizeDriver) does not know what do do when it hits a failed point (other than raising an exception), and has no way to report a failure back (at least to my knowledge).
However, pyOptSparseDriver can do something when a point fails: it an try progressing again in the gradient direction, but with a smaller stepsize. I took your code, and exchanged the ScipyOptimizeDriver with a pyOptSparseDriver, used the "SLSQP" optimizer in that package, and it worked around that problematic point just fine, reaching what I assume is a good optimum:
[0.69651727]
[-5.]
[2.]
My driver option code looks like this:
prob.driver = pyOptSparseDriver()
prob.driver.options["optimizer"] = 'SLSQP'
prob.driver.opt_settings['ACC'] = 1e-8
If you don't already have pyoptsparse, which is a separate package not maintained by us, you can get it from https://github.com/mdolab/pyoptsparse -- and you can build and install it with:
python setup.py build
python setup.py install
For your other question, I looked around through our code and found that the failure flag return on _solve_nonlinear is never used for anything. So, the only way to communicate a non-converging status to a driver or a higher-level solver is to raise the AnalysisError. Solvers canraise an AE when they don't converge, if "err_on_maxiter" on its options is set to True (the default is False).
As a final note, I think that while our error handling mechanisms are being used, we haven't thought of everything and are always open to suggestions for improvements.
This input series works perfectly
Bond.Valuation (bond.id = bondlab10, principal = 1000, price = 100,
settlement.date = "02-4-2013", trade.date = "1-31-2013")
This throws an error
Bond.Valuation (bond.id = bondlab10, principal = 1000, price = 100,
settlement.date = "01-5-2013", trade.date = "12-31-2012")
#Error in Key.Rate.Table[x, 3] = (spotrates("ns", BetaVector, m = Key.Rate.Table[x,
: replacement has length zero.
I am trying to use debug tools to step through the code but I am not having much luck. I get the following when I debug
debug at <tmp>#29: Bond.Term.Structure <<- Key.Rate.Analysis(bond.id = bondlab10,
Rate.Delta = Rate.Delta, BetaVector = BetaVector, principal = principal,
price = price, cashflow = Bond.CashFlow)
I know the function and the line but I have no idea what is wrong from the messages. Advise on debugging is appreciated. Basically, I have a long settlement here but this will be required for mortgage backed securities.
-Update, I found the problem but not from the debugger. Is there a way to watch R step through each calculation and see values?
If you are using Windows, Revolution R has good support for step-by-step debugging. It provides a Visual-Studio-style interface to R, with breakpoints and mouseovers on values of variables.
The academic version is free.
The lastest R-studio has built-in debugger. The community edition is free.