I have a function that will download an image collection as a TFrecord or a geotiff.
Heres the function -
def download_image_collection_to_drive(collection, aois, bands, limit, export_format):
if collection.size().lt(ee.Number(limit)):
bands = [band for band in bands if band not in ['SCL', 'QA60']]
for aoi in aois:
cluster = aoi.get('cluster').getInfo()
geom = aoi.bounds().getInfo()['geometry']['coordinates']
aoi_collection = collection.filterMetadata('cluster', 'equals', cluster)
for ts in range(1, 11):
print(ts)
ts_collection = aoi_collection.filterMetadata('interval', 'equals', ts)
if ts_collection.size().eq(ee.Number(1)):
image = ts_collection.first()
p_id = image.get("PRODUCT_ID").getInfo()
description = f'{cluster}_{ts}_{p_id}'
task_config = {
'fileFormat': export_format,
'image': image.select(bands),
'region': geom,
'description': description,
'scale': 10,
'folder': 'output'
}
if export_format == 'TFRecord':
task_config['formatOptions'] = {'patchDimensions': [256, 256], 'kernelSize': [3, 3]}
task = ee.batch.Export.image.toDrive(**task_config)
task.start()
else:
logger.warning(f'no image for interval {ts}')
else:
logger.warning(f'collection over {limit} aborting drive download')
It seems whenever it gets to the second aoi it fails, Im confused by this as if ts_collection.size().eq(ee.Number(1)) confirms there is an image there so it should manage to get product id from it.
line 24, in download_image_collection_to_drive
p_id = image.get("PRODUCT_ID").getInfo()
File "/lib/python3.7/site-packages/ee/computedobject.py", line 95, in getInfo
return data.computeValue(self)
File "/lib/python3.7/site-packages/ee/data.py", line 717, in computeValue
prettyPrint=False))['result']
File "/lib/python3.7/site-packages/ee/data.py", line 340, in _execute_cloud_call
raise _translate_cloud_exception(e)
ee.ee_exception.EEException: Element.get: Parameter 'object' is required.
am I falling foul of immutable server side objects somewhere?
This is a server-side value, problem, yes, but immutability doesn't have to do with it — your if statement isn't working as you intend.
ts_collection.size().eq(ee.Number(1)) is a server-side value — you've described a comparison that hasn't happened yet. That means that doing any local operation like a Python if statement cannot take the comparison outcome into account, and will just treat it as a true value.
Using getInfo would be a quick fix:
if ts_collection.size().eq(ee.Number(1)).getInfo():
but it would be more efficient to avoid using getInfo more than needed by fetching the entire collection's info just once, which includes the image info.
...
ts_collection_info = ts_collection.getInfo()
if ts_collection['features']: # Are there any images in the collection?
image = ts_collection.first()
image_info = ts_collection['features'][0] # client-side image info already downloaded
p_id = image_info['properties']['PRODUCT_ID'] # get ID from client-side info
...
This way, you only make two requests per ts: one to check for the match, and one to start the export.
Note that I haven't actually run this Python code, and there might be some small mistakes; if it gives you any trouble, print(ts_collection_info) and examine the structure you actually received to figure out how to interpret it.
Related
I would like to test GQ-CNN which is network in Dex-Net on tensorRT.
I successfully converted tflite file to uff file but when I tried to inference with that network, there is an error I couldn't figure out.
[TensorRT] ERROR: Parameter check failed at: ../builder/Network.cpp::addLRN::149, condition: lrnWindow & 0x1
python3: uff/orders.cpp:330: void UffParser::addTranspose(ParserLayer&, std::vector<int>): Assertion `outputs.size() == 1' failed.
The error is appeared when building model.
I tried to find clue from google but there are no codes and no references.
There's only different thing compare with example code that works well.
(I wrote captions which codes I added. If I remove that codes and replace model file to single input network, it works well.)
I registered input twice like below code because GQ-CNN has multiple input.
So I guess that registering multiple input using uffparser could be the main reason of that error.
class ModelData(object):
MODEL_FILE = "./gqcnn.uff"
INPUT_NAME_1 = "Placeholder"
INPUT_SHAPE_1 = (1, 32, 32)
INPUT_NAME_2 = "Placeholder_1"
INPUT_SHAPE_2 = (2,)
OUTPUT_NAME = "softmax/Softmax"
def build_engine(model_file):
# For more information on TRT basics, refer to the introductory samples.
with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
builder.max_workspace_size = common.GiB(1)
builder.fp16_mode = True
#builder.int8_mode = True
# Parse the Uff Network
parser.register_input(ModelData.INPUT_NAME_1, ModelData.INPUT_SHAPE_1)
parser.register_input(ModelData.INPUT_NAME_2, ModelData.INPUT_SHAPE_2) # added code
parser.register_output(ModelData.OUTPUT_NAME)
parser.parse(model_file, network)
# Build and return an engine.
return builder.build_cuda_engine(network)
# do inference
with build_engine(ModelData.MODEL_FILE) as engine:
# Build an engine, allocate buffers and create a stream.
# For more information on buffer allocation, refer to the introductory samples.
inputs, outputs, bindings, stream = common.allocate_buffers(engine)
with engine.create_execution_context() as context:
for idx in range(len(val_images)) :
start = time.time()
val_image = val_images[idx]
val_pose = val_poses[idx] # added code
np.copyto(inputs[0].host, val_image)
np.copyto(inputs[1].host, val_pose) # added code
[prediction] = common.do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)
Is there anyone who succeeded to inference with multiple input model?
I am able to set up my graph for streaming just fine. Here's the initialization:
self.data_source = AjaxDataSource(data_url='my_route',
polling_interval=1000, mode='append', max_size=300)
Now I want to 'pause' the polling of the AjaxDataSource. I couldn't find a way to do this in the documentation. I'm NOT running a bokeh server, so bokeh server solutions I am unable to use.
I came up with one possible solution: just return empty data set to the function that is appending the data via AjaxDataSource. So in the example above, the my_route function would look something like this:
def my_route:
if not self.is_paused:
data = normal_data_to_graph
else:
data = []
return data
Once you set the polling_interval = None in Python, it will not request it. In CustomJS, you can start the paused request. Here, the source is an AJaxDataSource instance.
source.polling_interval = 1000; // the interval you want
source.intialized = false;
source.setup();
I would like to speed up figure generation in Bokeh by multiprocessing:
jobs = []
for label in list(peakLabels):
args = {'data': rt_proj_data[label],
'label': label,
'tools': tools,
'colors': itertools.cycle(palette),
'files': files,
'highlight': highlight}
jobs.append(args)
pool = Pool(processes=cpu_count())
m = Manager()
q = m.Queue()
plots = pool.map_async(plot_peaks_parallel, jobs)
pool.close()
pool.join()
def plot_peaks_parallel(args):
data = args['data']
label = args['label']
colors = args['colors']
tools = args['tools']
files = args['files']
highlight = args['highlight']
p = figure(title=f'Peak: {label}',
x_axis_label='Retention Time',
y_axis_label='Intensity',
tools=tools)
...
return p
Though I ran into this error:
MaybeEncodingError: Error sending result: '[Figure(id='1078', ...)]'. Reason: 'PicklingError("Can't pickle at 0x7fc7df0c0ea0>: attribute lookup ColumnDataSource. on bokeh.models.sources failed")'
Can I do something to the object p, so that it becomes pickleable?
Individual Bokeh objects are not serializable in isolation, including with pickle. The smallest meaningful unit of serialization in Bokeh is the Document, which is a specific collection of Bokeh objects guaranteed to be complete with respect to following references. However, I would be surprised if pickle works with Document either (AFAIK you are the first person to ask about it since the project started, it's never been a priority, or even looked into that I know of). Instead, I would suggest if you want to do something like this, to use Bokeh's own JSON serialization functions, such as json_item:
# python code
p_serialized = json.dumps(json_item(p))
This will properly serialize p in the context of the Document it is a part of. Then you can pass this to your page templates to display with the Bokeh JS embed API:
# javascript code
p = JSON.parse(p_serialized);
Bokeh.embed.embed_item(p, "mydiv")
I have an Elastic Search cluster with a lot of nice data, that I have created some nice Kibana dashboards for.
For the next level I decided to take a look at scripted fields to make some of the dashboards even nicer.
I want to translate some of the numeric fields into more easily understandable text values. As an example of what I want to do and what I have tried I will use the http response status code field, that most will understand quite easily but also illustrates the problem.
We log the numeric status code (200, 201, 302, 400, 404, 500 etc.) I can create a data table visualization that tells me the count for each of these status codes. But I would like to display the text reason in my dashboard.
I can create a painless script with a lot of IF statements like this:
if (doc['statuscode'].value == 200) {return "OK";}
if (doc['statuscode'].value == 201) {return "Created";}
if (doc['statuscode'].value == 400) {return "Bad Request";}
return doc['statuscode'].value;
But that isn't very nice I think.
But since I will most likely have about 150 different values and that list won't change very often, so I can live with maintaining a static map. But I haven't found any examples of implementing a map or dictionary in painless scripting.
I was thinking of implementing something like this:
Map reasonMap;
reasonMap[200] = 'OK';
reasonMap[201] = 'Created';
def reason = reasonMap[doc['statuscode'].value];
if (reason != null)
{
return reason;
}
return doc['statuscode'].value;
I haven't been able to make this code work though. The question is also if this will perform well enough for a map with up to 150 values.
Thanks
EDIT
After some trial and error... and a lot of googling, this is what I came up with that works (notice that the key needs to start with a character and not a number):
def reasonMap =
[
's200': 'OK',
's201': 'Created'
];
def key = 's' + doc['statuscode'].value
def reason = reasonMap[key];
if (reason != null)
{
return reason;
}
return doc['statuscode'].value;
Should it be
def reason = reasonMap[doc['statuscode']value];
It will perform well with a Map of 150 values.
Using the built-in "Import Data..." functionality we can import a properly formatted text file (like CSV and/or tab-delimited) as an image. It is rather straight forward to write a script to do so. However, my scripting approach is not efficient - which requires me to loop through each raw (use the "StreamReadTextLine" function) so it takes a while to get a 512x512 image imported.
Is there a better way or an "undocumented" script function that I can tap in?
DigitalMicrograph offers an import functionality via the File/Import Data... menu entry, which will give you this dialog:
The functionality evoked by this dialog can also be accessed by script commands, with the command
BasicImage ImageImportTextData( String img_name, ScriptObject stream, Number data_type_enum, ScriptObject img_size, Boolean lines_are_rows, Boolean size_by_counting )
As with the dialog, one has to pre-specify a few things.
The data type of the image.
This is a number. You can find out which number belongs to which image data type by, f.e., creating an image outputting its data type:
image img := Realimage( "", 4, 100 )
Result("\n" + img.ImageGetDataType() )
The file stream object
This object describes where the data is stored. The F1 help-documention explains how one creates a file-stream from an existing file, but essentially you need to specify a path to the file, then open the file for reading (which gives you a handle), and then using the fileHandle to create the stream object.
string path = "C:\\test.txt"
number fRef = OpenFileForReading( path )
object fStream = NewStreamFromFileReference( fRef, 1 )
The image size object
This is a specific script object you need to allocate. It wraps image size information. In case of auto-detecting the size from the text, you don't need to specify the actual size, but you still need the object.
object imgSizeObj = Alloc("ImageData_ImageDataSize")
imgSizeObj.SetNumDimensions(2) // Not needed for counting!
imgSizeObj.SetDimensionSize(0,10) // Not used for counting
imgSizeObj.SetDimensionSize(1,10) // Not used for counting
Boolean checks
Like with the checkboxes in the UI, you spefic two conditions:
Lines are Rows
Get Size By Counting
Note, that the "counting" flag is only used if "Lines are Rows" is also true. Same as with the dialog.
The following script improrts a text file with couting:
image ImportTextByCounting( string path, number DataType )
{
number fRef = OpenFileForReading( path )
object fStream = NewStreamFromFileReference( fRef, 1 )
number bLinesAreRows = 1
number bSizeByCount = 1
bSizeByCount *= bLinesAreRows // Only valid together!
object imgSizeObj = Alloc("ImageData_ImageDataSize")
image img := ImageImportTextData( "Imag Name ", fStream, DataType, imgSizeObj, bLinesAreRows, bSizeByCount )
return img
}
string path = "C:\\test.txt"
number kREAL4_DATA = 2
image img := ImportTextByCounting( path, kREAL4_DATA )
img.ShowImage()