Related
I am downloading global data from tiles by NASA MODIS satellite using MODIStsp 2.0.9 in R. This would give me a single stitched TIFF file for the entire glob.
I am getting difference in the resolution of the image when I run the same code in Windows 10 vs linux.
PS: I am not able to understand why the resolution of the stitched global image should vary when the input parameters to the function call don't vary at all. The difference in large between the two OS.
MODIStsp(gui = FALSE,
out_folder = dropbox,
out_folder_mod = dropbox,
selprod = 'Surf_Temp_Daily_1Km (M*D11A1)',
bandsel = "LST_Day_1km", # daily surface temp
sensor = "Terra",
# your username for NASA http server
user = "user" ,
# your password for NASA http server
password = "pass",
start_date = '2002.01.01',
end_date = '2002.01.01',
#end_date = '2020.12.31',
verbose = TRUE,
spatmeth = "bbox",
bbox = c(-180.00,-90.00,180.00,90.00),
out_format = 'GTiff',
compress = 'None',
out_projsel = 'User Defined',
output_proj = 4326,
delete_hdf = FALSE,
parallel = TRUE,
reprocess = FALSE
)
Currently I'm tryin to convert given onnx file to tensorrt file, and do inference on the generated tensorrt file.
To do so, I used tensorrt python binding API, but
"Error Code 1: Cuda Driver (invalid resource handle)" happens and there is no kind description about this.
Can anyone help me to overcome this situation?
Thx in advance, and below is my code snippet.
def trt_export(self):
fp_16_mode = True
## Obviously, I provided appropriate file names
trt_file_name = "PATH_TO_TRT_FILE"
onnx_name = "PATH_TO_ONNX_FILE"
TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE)
EXPLICIT_BATCH = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
builder = trt.Builder(TRT_LOGGER)
network = builder.create_network(EXPLICIT_BATCH)
parser = trt.OnnxParser(network, TRT_LOGGER)
config = builder.create_builder_config()
config.max_workspace_size = (1<<30)
config.set_flag(trt.BuilderFlag.FP16)
config.default_device_type = trt.DeviceType.GPU
profile = builder.create_optimization_profile()
profile.set_shape('input', (1, 3, IMG_SIZE, IMG_SIZE), (12, 3, IMG_SIZE, IMG_SIZE), (32, 3, IMG_SIZE, IMG_SIZE)) # random nubmers for min. opt. max batch
config.add_optimization_profile(profile)
with open(onnx_name, 'rb') as model:
if not parser.parse(model.read()):
for error in range(parser.num_errors):
print(parser.get_error(error))
engine = builder.build_engine(network, config)
buf = engine.serialize()
with open(trt_file_name, 'wb') as f:
f.write(buf)
def validate_trt_result(self, input_path):
TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE)
trt_file_name = "PATH_TO_TRT_FILE"
trt_runtime = trt.Runtime(TRT_LOGGER)
with open(trt_file_name, 'rb') as f:
engine_data = f.read()
engine = trt_runtime.deserialize_cuda_engine(engine_data)
cuda.init()
device = cuda.Device(0)
ctx = device.make_context()
inputs, outputs, bindings = [], [], []
context = engine.create_execution_context()
stream = cuda.Stream()
index = 0
for binding in engine:
size = trt.volume(engine.get_binding_shape(binding)) * -1 # assuming one batch
dtype = trt.nptype(engine.get_binding_dtype(binding))
host_mem = cuda.pagelocked_empty(size, dtype)
device_mem = cuda.mem_alloc(host_mem.nbytes)
bindings.append(int(device_mem))
if engine.binding_is_input(binding):
inputs.append(HostDeviceMem(host_mem, device_mem))
context.set_binding_shape(index, [1, 3, IMG_SIZE, IMG_SIZE])
else:
outputs.append(HostDeviceMem(host_mem, device_mem))
index += 1
print(context.all_binding_shapes_specified)
input_img = cv2.imread(input_path)
input_r = cv2.resize(input_img, dsize = (256, 256))
input_p = np.transpose(input_r, (2, 0, 1))
input_e = np.expand_dims(input_p, axis = 0)
input_f = input_e.astype(np.float32)
input_f /= 255
numpy_array_input = [input_f]
hosts = [input.host for input in inputs]
trt_types = [trt.int32]
for numpy_array, host, trt_types in zip(numpy_array_input, hosts, trt_types):
numpy_array = np.asarray(numpy_array).astype(trt.nptype(trt_types)).ravel()
print(numpy_array.shape)
np.copyto(host, numpy_array)
[cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs]
#### ERROR HAPPENS HERE ####
context.execute_async_v2(bindings=bindings, stream_handle=stream.handle)
#### ERROR HAPPENS HERE ####
[cuda.memcpy_dtoh_async(out.host, out.device, stream) for out in outputs]
stream.synchronize()
print("TRT model inference result : ")
output = outputs[0].host
for one in output :
print(one)
ctx.pop()
Looks like ctx.push() line is missing before a line with memcpy_htod_async.
Such a error can happen if TensorFlow / PyTorch is also using CUDA in parallel with TensorRT.
See the related question/answer: https://stackoverflow.com/a/73996477/5655977
First of all, I am using TFX version 0.21.2 and Tensorflow version 2.1.
I have constructed a pipeline largely following the Chigaco taxi example. When the Trainer component is executed I can see the following in the logs:
INFO - Training complete. Model written to /root/airflow/tfx/pipelines/fish/Trainer/model/9/serving_model_dir
When checking the above directory it is empty. What am I missing?
This is my DAG definition file (import statements omitted):
_pipeline_name = 'fish'
_airflow_config = AirflowPipelineConfig(airflow_dag_config = {
'schedule_interval': None,
'start_date': datetime.datetime(2019, 1, 1),
})
_project_root = os.path.join(os.environ['HOME'], 'airflow')
_data_root = os.path.join(_project_root, 'data', 'fish_data')
_module_file = os.path.join(_project_root, 'dags', 'fishUtils.py')
_serving_model_dir = os.path.join(_project_root, 'serving_model', _pipeline_name)
_tfx_root = os.path.join(_project_root, 'tfx')
_pipeline_root = os.path.join(_tfx_root, 'pipelines', _pipeline_name)
_metadata_path = os.path.join(_tfx_root, 'metadata', _pipeline_name,
'metadata.db')
def _create_pipeline(pipeline_name: Text, pipeline_root: Text, data_root: Text,
module_file: Text, serving_model_dir: Text,
metadata_path: Text,
direct_num_workers: int) -> pipeline.Pipeline:
examples = external_input(data_root)
example_gen = CsvExampleGen(input=examples)
statistics_gen = StatisticsGen(examples=example_gen.outputs['examples'])
infer_schema = SchemaGen(
statistics=statistics_gen.outputs['statistics'],
infer_feature_shape=False)
validate_stats = ExampleValidator(
statistics=statistics_gen.outputs['statistics'],
schema=infer_schema.outputs['schema'])
trainer = Trainer(
examples=example_gen.outputs['examples'], schema=infer_schema.outputs['schema'],
module_file=_module_file, train_args= trainer_pb2.TrainArgs(num_steps=10000),
eval_args= trainer_pb2.EvalArgs(num_steps=5000))
model_validator = ModelValidator(
examples=example_gen.outputs['examples'],
model=trainer.outputs['model'])
pusher = Pusher(
model=trainer.outputs['model'],
model_blessing=model_validator.outputs['blessing'],
push_destination=pusher_pb2.PushDestination(
filesystem=pusher_pb2.PushDestination.Filesystem(
base_directory=_serving_model_dir)))
return pipeline.Pipeline(
pipeline_name=_pipeline_name,
pipeline_root=_pipeline_root,
components=[
example_gen,
statistics_gen,
infer_schema,
validate_stats,
trainer,
model_validator,
pusher],
enable_cache=True,
metadata_connection_config=metadata.sqlite_metadata_connection_config(
metadata_path),
beam_pipeline_args=['--direct_num_workers=%d' % direct_num_workers]
)
runner = AirflowDagRunner(config = _airflow_config)
DAG = runner.run(
_create_pipeline(
pipeline_name=_pipeline_name,
pipeline_root=_pipeline_root,
data_root=_data_root,
module_file=_module_file,
serving_model_dir=_serving_model_dir,
metadata_path=_metadata_path,
# 0 means auto-detect based on on the number of CPUs available during
# execution time.
direct_num_workers=0))
And this is my module file:
_DENSE_FLOAT_FEATURE_KEYS = ['length']
real_valued_columns = [tf.feature_column.numeric_column('length')]
def _eval_input_receiver_fn():
serialized_tf_example = tf.compat.v1.placeholder(
dtype=tf.string, shape=[None], name='input_example_tensor')
features = tf.io.parse_example(
serialized=serialized_tf_example,
features={
'length': tf.io.FixedLenFeature([], tf.float32),
'label': tf.io.FixedLenFeature([], tf.int64),
})
receiver_tensors = {'examples': serialized_tf_example}
return tfma.export.EvalInputReceiver(
features={'length' : features['length']},
receiver_tensors=receiver_tensors,
labels= features['label'],
)
def parser(serialized_example):
features = tf.io.parse_single_example(
serialized_example,
features={
'length': tf.io.FixedLenFeature([], tf.float32),
'label': tf.io.FixedLenFeature([], tf.int64),
})
return ({'length' : features['length']}, features['label'])
def _input_fn(filenames):
# TFRecordDataset doesn't directly accept paths with wildcards
filenames = tf.data.Dataset.list_files(filenames)
dataset = tf.data.TFRecordDataset(filenames, 'GZIP')
dataset = dataset.map(parser)
dataset = dataset.shuffle(2000)
dataset = dataset.batch(40)
dataset = dataset.repeat(10)
return dataset
def trainer_fn(trainer_fn_args, schema):
estimator = tf.estimator.LinearClassifier(feature_columns=real_valued_columns)
train_input_fn = lambda: _input_fn(trainer_fn_args.train_files)
train_spec = tf.estimator.TrainSpec(
train_input_fn,
max_steps=trainer_fn_args.train_steps)
eval_input_fn = lambda: _input_fn(trainer_fn_args.eval_files)
eval_spec = tf.estimator.EvalSpec(
eval_input_fn,
steps=trainer_fn_args.eval_steps,
name='fish-eval')
receiver_fn = lambda: _eval_input_receiver_fn()
return {
'estimator': estimator,
'train_spec': train_spec,
'eval_spec': eval_spec,
'eval_input_receiver_fn': receiver_fn
}
Thank you in advance for your help!
Posting the solution for anyone that is facing the same problem that I faced.
The reason that the model was not written in the filesystem was that the estimator needs a config argument to know where to write the model.
The following modification to the trainer_fn function should solve the problem.
run_config = tf.estimator.RunConfig(save_checkpoints_steps=999, keep_checkpoint_max=1)
run_config = run_config.replace(model_dir=trainer_fn_args.serving_model_dir)
estimator=tf.estimator.LinearClassifier(feature_columns=real_valued_columns,config=run_config)
I am trying to use to sen2r() function (Package sen2r_1.3.2) with default parameters but getting the following error:
Error in paste(c(...), collapse = sep) : argument is missing, with no default.
I know the error wants me to fill in some parameters, but the source manual clearly says that the default should work, and the parameters can be set subsequently upon launching the GUI.
Using the s2_gui() launches the shiny app, but keeps hanging when I try to "Save and Close"
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS
Also, can someone with a 'higher reputation' please create a sen2r tag, for easier subsequent communications?
Here is the traceback...
sen2r()
Error in paste(c(...), collapse = sep) :
argument is missing, with no default
> traceback()
7: paste(c(...), collapse = sep)
6: strsplit(paste(c(...), collapse = sep), "\n")
5: unlist(strsplit(paste(c(...), collapse = sep), "\n"))
4: strwrap(unlist(strsplit(paste(c(...), collapse = sep), "\n")),
width = width, indent = indent, exdent = exdent, prefix = prefix,
initial = initial)
3: print_message(type = "waiting", "It seems you are running this package for the first time. ",
"Do you want to verify/install the required dependencies using a GUI (otherwise, an\n automatic check will be performed)? (y/n) ",
)
2: .sen2r(param_list = param_list, pm_arg_passed = pm_arg_passed,
gui = gui, preprocess = preprocess, s2_levels = s2_levels,
sel_sensor = sel_sensor, online = online, order_lta = order_lta,
apihub = apihub, downloader = downloader, overwrite_safe = overwrite_safe,
rm_safe = rm_safe, step_atmcorr = step_atmcorr, sen2cor_use_dem = sen2cor_use_dem,
sen2cor_gipp = sen2cor_gipp, max_cloud_safe = max_cloud_safe,
timewindow = timewindow, timeperiod = timeperiod, extent = extent,
extent_name = extent_name, s2tiles_selected = s2tiles_selected,
s2orbits_selected = s2orbits_selected, list_prods = list_prods,
list_rgb = list_rgb, list_indices = list_indices, index_source = index_source,
rgb_ranges = rgb_ranges, mask_type = mask_type, max_mask = max_mask,
mask_smooth = mask_smooth, mask_buffer = mask_buffer, clip_on_extent = clip_on_extent,
extent_as_mask = extent_as_mask, reference_path = reference_path,
res = res, res_s2 = res_s2, unit = unit, proj = proj, resampling = resampling,
resampling_scl = resampling_scl, outformat = outformat, rgb_outformat = rgb_outformat,
index_datatype = index_datatype, compression = compression,
rgb_compression = rgb_compression, overwrite = overwrite,
path_l1c = path_l1c, path_l2a = path_l2a, path_tiles = path_tiles,
path_merged = path_merged, path_out = path_out, path_rgb = path_rgb,
path_indices = path_indices, path_subdirs = path_subdirs,
thumbnails = thumbnails, parallel = parallel, processing_order = processing_order,
use_python = use_python, tmpdir = tmpdir, rmtmp = rmtmp,
log = log, globenv = sen2r_env, .only_list_names = FALSE)
1: sen2r()
I ran s2_gui() as is...no parameters specified. But i am running the dependency check now, I suspect that should clear things up even for the GUI.
This error was due to a code bug, which was fixed (see the GitHub issue 292).
Until the sen2r CRAN version will be updated, the bug can be:
solved installling the sen2r GitHub version (remotes::install_github("ranghetti/sen2r")), or
bypassed launching check_gdal() before running sen2r().
This is a bug in the original code.
In the traceback that you provided, it included:
3: print_message(type = "waiting", "It seems you are running this package for the first time. ",
"Do you want to verify/install the required dependencies using a GUI (otherwise, an\n automatic check will be performed)? (y/n) ",
)
Notably, I'll truncate most of the strings:
3: print_message(type = "waiting", "It seems ... time. ",
"Do you ... performed)? (y/n) ",
) # ^-- extra comma, invalid R syntax
Notice how it ends with a comma and then a right-paren? Yup, that's a syntax error in R. (This has been submitted as issue 292 on the original repo.)
I am testing a program executed partially on a MPC603 and partially on a MPC555.
I have to verify that some data is correctly "moved" from one processor to the other via a DPRAM.
I am guessing that at some point "someone" makes a conversion but I don't know how to find what kind of conversion is done.
Here are some examples:
Pt_Dpram->acq1 at 0x8D00008 = 0x3EB2
acq1 = (0xA010538) = 1182451712 = 0x467AC800
Pt_Dpram->acq2 at 0x8D0000A = 0x5528
acq2 = (0xA010540) = 1185566720 = 0x46AA5000
Pt_Dpram->acq3 at 0x8D0000C = 0x416E
acq3 = (0xA010548) = 1107552036 = 0x4203E724
Pt_Dpram->acq4 at 0x8D0000E = 0x413C
acq4 = (0xA010550) = 1107526232 = 0x42038258
I got my answers from a collegue : the values in acqX are in Motorola binary format : http://en.wikipedia.org/wiki/SREC_(file_format)
Here is a small software that does the conversion : http://www.hexworkshop.com/onlinehelp/500/html/idhelp_baseconv.htm