I'm using CoreML for image segmentation. I have a VNCoreMLRequest to run a model that returns a MLMultiArray. To accelerate processing the model output, I use MPSNNReduceFeatureChannelsArgumentMax to reduce the multiarray output to a 2D array which I then convert to a grayscale image.
This works great on iOS, but when running on Mac as a catalyst build, the output 2D array is all zeros.
I'm running Version 12.2 beta 2 (12B5025f) on an iMac Pro. I'm not seeing any runtime errors. MPSNNReduceFeatureChannelsArgumentMax appears to not work on Mac Catalyst.
I'm able to reduce the channels directly on the cpu by looping through all the array dimensions but it's very slow. This proves the model output works, just the metal reduce features fails.
Anyone else using CoreML and Catalyst?
Here's the bit of code that doesn't work:
let buffer = self.queue.makeCommandBuffer()
let filter = MPSNNReduceFeatureChannelsArgumentMax(device: self.device)
filter.encode(commandBuffer: buffer!, sourceImage: probs, destinationImage: classes)
// add a callback to handle the buffer's completion and commit the buffer
buffer?.addCompletedHandler({ (_buffer) in
let argmax = try! MLMultiArray(shape: [1, softmax.shape[1], softmax.shape[2]], dataType: .float32)
classes.readBytes(argmax.dataPointer,
dataLayout: .featureChannelsxHeightxWidth,
imageIndex: 0)
// unmap the discrete segmentation to RGB pixels
guard var mask = codesToMask(argmax) else {
return
}
// display image in view
DispatchQueue.main.async {
self.imageView.image = mask
}
})
Related
I have this function for fitting a plane to a pointcloud using PCL's sac model fitting. I want the best result I can get, so I want to run with seg.setOptimizeCoefficients(true).
The problem is that a lot of the time, the pointcloud passed in, will not have enough points to optimise coefficients, so I get a continuous stream of:
[pcl::SampleConsensusModelPlane::optimizeModelCoefficients] Not enough inliers found to optimize model coefficients (0)! Returning the same coefficients.
I would like to have coefficient optimisation to run when it can, and when it can't to just carry on without polluting the CLI output with many red warning messages.
according to this issue this message just means that there are fewer than 3 inlier points for the SAC model fitting. I do extract the inlier points, so I could manually check if there are 3 or more. But I can't see how to do this first, and THEN find the optimized model coefficients. Is there a way?
inline void fit_plane_to_points(
const pcl::PointCloud<pcl::PointXYZI>::ConstPtr& det_points,
const pcl::ModelCoefficients::Ptr& coefficients,
const Eigen::Vector3f& vec,
const pcl::PointCloud<pcl::PointXYZI>::Ptr& inlier_pts) {
// if no det points to work with, don't try and segment
if (det_points->size() < 3) {
return;
}
// fit surface point samples to a plane
pcl::PointIndices::Ptr inlier_indices(new pcl::PointIndices);
pcl::SACSegmentation<pcl::PointXYZI> seg;
seg.setModelType(pcl::SACMODEL_PERPENDICULAR_PLANE);
// max allowed difference between the plane normal and the given axis
seg.setEpsAngle(sac_angle_threshold_);
seg.setAxis(vec);
seg.setMethodType(pcl::SAC_RANSAC);
seg.setDistanceThreshold(sac_distance_threshold_);
seg.setMaxIterations(1000);
seg.setInputCloud(det_points);
seg.setOptimizeCoefficients(sac_optimise_coefficients_);
seg.segment(*inlier_indices, *coefficients);
if (inlier_indices->indices.empty()) {
// if no inlier points don't try and extract
return;
}
// extract the planar points
pcl::ExtractIndices<pcl::PointXYZI> extract;
extract.setInputCloud(det_points);
extract.setIndices(inlier_indices);
extract.setNegative(false);
extract.filter(*inlier_pts);
return;
}
I would say the best way to do this is to disable setOptimizeCoefficients and then do that manually after seg.segment. You basically have to recreate these lines: https://github.com/PointCloudLibrary/pcl/blob/10235c9c1ad47989bdcfebe47f4a369871357e2a/segmentation/include/pcl/segmentation/impl/sac_segmentation.hpp#L115-L123 .
You can access the model via getModel() (https://pointclouds.org/documentation/classpcl_1_1_s_a_c_segmentation.html#ac7b9564ceba35754837b4848cf448d78).
Ultimately got it working, with advice from IBitMyBytes by setting seg.setOptimizeCoefficients(false); and then manually optimising after doing my own check:
// if we can, optimise model coefficients
if (sac_optimise_coefficients_ && inlier_indices->indices.size() > 4) {
pcl::SampleConsensusModel<pcl::PointXYZI>::Ptr model = seg.getModel();
Eigen::VectorXf coeff_refined;
Eigen::Vector4f coeff_raw(coefficients->values.data());
model->optimizeModelCoefficients(inlier_indices->indices,
coeff_raw, coeff_refined);
coefficients->values.resize(coeff_refined.size());
memcpy(&coefficients->values[0], &coeff_refined[0],
coeff_refined.size() * sizeof (float));
// Refine inliers
model->selectWithinDistance(coeff_refined, sac_distance_threshold_,
inlier_indices->indices);
}
I have a function that will download an image collection as a TFrecord or a geotiff.
Heres the function -
def download_image_collection_to_drive(collection, aois, bands, limit, export_format):
if collection.size().lt(ee.Number(limit)):
bands = [band for band in bands if band not in ['SCL', 'QA60']]
for aoi in aois:
cluster = aoi.get('cluster').getInfo()
geom = aoi.bounds().getInfo()['geometry']['coordinates']
aoi_collection = collection.filterMetadata('cluster', 'equals', cluster)
for ts in range(1, 11):
print(ts)
ts_collection = aoi_collection.filterMetadata('interval', 'equals', ts)
if ts_collection.size().eq(ee.Number(1)):
image = ts_collection.first()
p_id = image.get("PRODUCT_ID").getInfo()
description = f'{cluster}_{ts}_{p_id}'
task_config = {
'fileFormat': export_format,
'image': image.select(bands),
'region': geom,
'description': description,
'scale': 10,
'folder': 'output'
}
if export_format == 'TFRecord':
task_config['formatOptions'] = {'patchDimensions': [256, 256], 'kernelSize': [3, 3]}
task = ee.batch.Export.image.toDrive(**task_config)
task.start()
else:
logger.warning(f'no image for interval {ts}')
else:
logger.warning(f'collection over {limit} aborting drive download')
It seems whenever it gets to the second aoi it fails, Im confused by this as if ts_collection.size().eq(ee.Number(1)) confirms there is an image there so it should manage to get product id from it.
line 24, in download_image_collection_to_drive
p_id = image.get("PRODUCT_ID").getInfo()
File "/lib/python3.7/site-packages/ee/computedobject.py", line 95, in getInfo
return data.computeValue(self)
File "/lib/python3.7/site-packages/ee/data.py", line 717, in computeValue
prettyPrint=False))['result']
File "/lib/python3.7/site-packages/ee/data.py", line 340, in _execute_cloud_call
raise _translate_cloud_exception(e)
ee.ee_exception.EEException: Element.get: Parameter 'object' is required.
am I falling foul of immutable server side objects somewhere?
This is a server-side value, problem, yes, but immutability doesn't have to do with it — your if statement isn't working as you intend.
ts_collection.size().eq(ee.Number(1)) is a server-side value — you've described a comparison that hasn't happened yet. That means that doing any local operation like a Python if statement cannot take the comparison outcome into account, and will just treat it as a true value.
Using getInfo would be a quick fix:
if ts_collection.size().eq(ee.Number(1)).getInfo():
but it would be more efficient to avoid using getInfo more than needed by fetching the entire collection's info just once, which includes the image info.
...
ts_collection_info = ts_collection.getInfo()
if ts_collection['features']: # Are there any images in the collection?
image = ts_collection.first()
image_info = ts_collection['features'][0] # client-side image info already downloaded
p_id = image_info['properties']['PRODUCT_ID'] # get ID from client-side info
...
This way, you only make two requests per ts: one to check for the match, and one to start the export.
Note that I haven't actually run this Python code, and there might be some small mistakes; if it gives you any trouble, print(ts_collection_info) and examine the structure you actually received to figure out how to interpret it.
I'm trying to use SequenceReader<T> in .Net Core Preview 8 to parse Guacamole Protocol network traffic.
The traffic might look as follows:
5.error,14.some text here,1.0;
This is a single error instruction. There are 3 fields:
OpCode = error
Reason = some text here
Status = 0 (see Status Codes)
The fields are comma delimited (semi-colon terminated), but they also have the length prefixed on each field. I presume that's so that you could parse something like:
5.error,24.some, text, with, commas,1.0;
To produce Reason = some, text, with, commas.
Simple comma delimited parsing is simple enough to do (with or without SequenceReader). However, to utilise the length I've tried the following:
public static bool TryGetNextElement(this ref SerializationContext context, out ReadOnlySequence<byte> element)
{
element = default;
var start = context.Reader.Position;
if (!context.Reader.TryReadTo(out ReadOnlySequence<byte> lengthSlice, Utf8Bytes.Period, advancePastDelimiter: true))
return false;
if (!lengthSlice.TryGetInt(out var length))
return false;
context.Reader.Advance(length);
element = context.Reader.Sequence.Slice(start, context.Reader.Position);
return true;
}
Based on my understanding of the initial proposal, this should work, though also could be simplified I think because some of the methods in the proposal make life a bit easier than that which is available in .Net Core Preview 8.
However, the problem with this code is that the SequenceReader does not seem to Advance as I would expect. It's Position and Consumed properties remain unchanged when advancing, so the element I slice at the end is always an empty sequence.
What do I need to do in order to parse this protocol correctly?
I'm guessing that .Reader here is a property; this is important because SequenceReader<T> is a mutable struct, but every time you access .SomeProperty you are working with an isolated copy of the reader. It is fine to hide it behind a property, but you'd need to make sure you work with a local and then push back when complete, i.e.
var reader = context.Reader;
var start = reader.Position;
if (!reader.TryReadTo(out ReadOnlySequence<byte> lengthSlice,
Utf8Bytes.Period, advancePastDelimiter: true))
return false;
if (!lengthSlice.TryGetInt(out var length))
return false;
reader.Advance(length);
element = reader.Sequence.Slice(start, reader.Position);
context.Reader = reader; // update position
return true;
Note that a nice feature of this is that in the failure cases (return false), you won't have changed the state yet, because you've only been mutating your local standalone clone.
You could also consider a ref-return property for .Reader.
Experienced programmer playing around with Gamemaker2 Studio.
Trying to draw some random squares on the screen using a 2D array to store the "map"
Step 1 : declare a 2D array MyMap[25,25] this works
Step 2 : Set 100 random locations in Map[]=1 this works
I get a crash when I try to look up the values I have stored in the array.
Its crashing with:
**Execution Error - Variable Index [3,14] out of range [26,14] **
So it looks like it is trying to read 26 element, when you can see from my code the for next loop only goes to 20 and the array bound is 25.
Oddly enough it does the first two loops just fine?
Looking like a bug, I've spent so much time trying to work it out, anyone got an idea what is going on?
var tx=0;
var ty=0;
var t=0;
MyMap[25,25]=99; **// Works**
for( t=1; t<100; t+=1 ) **// Works**
{
MyMap[random(20),random(15)]=1
}
for( tx=1; tx<20; tx+=1 )
{
for( ty=1; ty<15; ty+=1 )
{
show_debug_message(string(tx) + ":" + string(ty))
t = MyMap[tx,ty]; /// **<---- Crashes Here**
if t=1 then {draw_rectangle(tx*32,ty*32,tx*32+32,ty*32+32,false) }
}
}
The line MyMap[random(20),random(15)]=1 does not initialize values in the entire array, creating a sparse array(where some elements do not exist).
The line MyMap[25,25]=99;
Should read:
for( tx=1; tx<20; tx+=1 )
{
for( ty=1; ty<15; ty+=1 )
{
MyMap[tx,ty]=99;
}
}
This will pre-initialize the all of the array values to 99. Filling out the array.
Then you can randomly assign the ones. (You will probably get less than 100 ones the due to duplicates in the random function and the random returning zeros.)
You should have the above code in the Create Event, or in another single fire or controlled fire event, and move the loops for the draw into the Draw Event.
All draw calls should be in the Draw Event. If the entire block were in Draw, it would randomize the blocks each step.
I 'm using OpenCV 2.3.1 and Qt and i am facing a problem with cv::calcOpticalFlowPyrLK. I am using oodFeaturesToTrack and calcOpticalFlowPyrLK to track a face that I have previously detected.
std::vector<cv::Point2f> Feat;
GrayFrame=FrameBuffer->GetFrame();
cv::goodFeaturesToTrack(GrayFrame,
Feat,
maxcorners,
qualitylevel,
mindistance);
while(1){
std::vector<cv::Point2f> NewFeat;
std::vector<uchar> status;
std::vector<float> err;
GrayFramePrev=GrayFrame.clone();
GrayFrame=FrameBuffer->GetFrame();
cv::calcOpticalFlowPyrLK(GrayFramePrev,
GrayFrame,
Feat,
NewFeat,
status,
err);
Feat=NewFeat;
}//while(1)
GrayFrame takes an image from a buffer where i store the images captured from a webcam both GrayFrame and GrayFramePrev contain the right image (GrayFrame -> FrameBuffer[i], GrayFramePrev -> FrameBuffer[i-1]) when they are at cv::calcOpticalFlowPyrLK and the Feat parameter contains points from goodFeaturesToTrack. But when calcOpticalFlowPyrLK is executed it returns via NewFeat the exact same points.
Please tell me what i am doing wrong with the `calcOpticalFlowPyrLK
Solution (For me)
I used the following parameters and it worked (i probably mixed up Feat and NewFeat also so that was the problem).
cv::calcOpticalFlowPyrLK(GreyFramePrev,
GreyFrame,
Feat,
NewFeat,
status,
err,
*WinSize,
maxLevel,
*TermCrit,
derivLamda,
LKflags,
minEigThreshold);
Parameter values
WinSize= new cv::Size(31,31);
maxLevel=3;
TermCrit= new cv::TermCriteria(CV_TERMCRIT_ITER|CV_TERMCRIT_EPS,20,0.03);
derivLamda=0;
LKflags=0;
minEigThreshold=0.001;