We're displaying time series data (utilisation of a compute resource, sampled hourly over months) on a stacked area chart using D3.js:
d3.json("/growth/instance_count_1month.json", function( data ) {
data.forEach(function(d) {
d.datapoints = d.datapoints.map(
function(da) {
// NOTE i'm not sure why this needs to be multiplied by 1000
return {date: new Date(da[1] * 1000),
count: da[0]};
});
});
x.domain(d3.extent(data[0].datapoints, function(d) { return d.date; }));
y.domain([0,
Math.ceil(d3.max(data.map(function (d) {return d3.max(d.datapoints, function (d) { return d.count; });})) / 100) * 100
]);
The result is rather spiky for my tastes:
Is there an easy way to simplify the data, either using D3 or another readily available library? I want to reduce the spikiness, but also reduce the volume of data to be graphed, as it will get out of hand.
I have a preference for doing this at the UI level, rather than touching the logging routines (even though redundant JSON data will have to be transferred.)
You have a number of options, you need to decided what is the best way forward for the type of data you have and the needs of it been used. Without knowing more about your data the best I can suggest is re-sampling. Simply report the data at longer intervals ('rolling up' the data). Alternatively you could use a rolling average or look at various line smoothing algorithms.
Related
firstly apologies: I am a beginner in Earth Engine, but googling my question hasn't yielded any results. I have a reasonable amount of experience with other languages/platforms.
I have a table of data with headers 'name' and 'value' with N entries, I also have a multiband images in which the bands are named the same as the 'name' column in my table.
I want to apply a functions to each band in the image, based on its corresponding value in the table.
I'm struggling to find a way to do this without using loops and getInfo(), both of which I understand are not efficient and generally frowned upon.
I think perhaps I'm missing something fundamental here regarding the interaction between local variables and things occuring serverside - help would be greatly appreciated!
You could perhaps iterate over the band names in the image:
var updatedImage = ee.Image(
image.bandNames().iterate(
function (bandName, acc) {
bandName = ee.String(bandName) // Must cast from ee.Element to actual type
var feature = features
.filter(ee.Filter.eq('name', bandName))
.first()
var value = feature.getNumber('value')
var bandWithFunctionApplied = image.select(bandName)
.add(value) // Apply some function
return ee.Image(acc)
.addBands(bandWithFunctionApplied)
},
ee.Image([])
)
)
https://code.earthengine.google.com/082411908a7525a4c8a87916b5ea88fc
I have this function for fitting a plane to a pointcloud using PCL's sac model fitting. I want the best result I can get, so I want to run with seg.setOptimizeCoefficients(true).
The problem is that a lot of the time, the pointcloud passed in, will not have enough points to optimise coefficients, so I get a continuous stream of:
[pcl::SampleConsensusModelPlane::optimizeModelCoefficients] Not enough inliers found to optimize model coefficients (0)! Returning the same coefficients.
I would like to have coefficient optimisation to run when it can, and when it can't to just carry on without polluting the CLI output with many red warning messages.
according to this issue this message just means that there are fewer than 3 inlier points for the SAC model fitting. I do extract the inlier points, so I could manually check if there are 3 or more. But I can't see how to do this first, and THEN find the optimized model coefficients. Is there a way?
inline void fit_plane_to_points(
const pcl::PointCloud<pcl::PointXYZI>::ConstPtr& det_points,
const pcl::ModelCoefficients::Ptr& coefficients,
const Eigen::Vector3f& vec,
const pcl::PointCloud<pcl::PointXYZI>::Ptr& inlier_pts) {
// if no det points to work with, don't try and segment
if (det_points->size() < 3) {
return;
}
// fit surface point samples to a plane
pcl::PointIndices::Ptr inlier_indices(new pcl::PointIndices);
pcl::SACSegmentation<pcl::PointXYZI> seg;
seg.setModelType(pcl::SACMODEL_PERPENDICULAR_PLANE);
// max allowed difference between the plane normal and the given axis
seg.setEpsAngle(sac_angle_threshold_);
seg.setAxis(vec);
seg.setMethodType(pcl::SAC_RANSAC);
seg.setDistanceThreshold(sac_distance_threshold_);
seg.setMaxIterations(1000);
seg.setInputCloud(det_points);
seg.setOptimizeCoefficients(sac_optimise_coefficients_);
seg.segment(*inlier_indices, *coefficients);
if (inlier_indices->indices.empty()) {
// if no inlier points don't try and extract
return;
}
// extract the planar points
pcl::ExtractIndices<pcl::PointXYZI> extract;
extract.setInputCloud(det_points);
extract.setIndices(inlier_indices);
extract.setNegative(false);
extract.filter(*inlier_pts);
return;
}
I would say the best way to do this is to disable setOptimizeCoefficients and then do that manually after seg.segment. You basically have to recreate these lines: https://github.com/PointCloudLibrary/pcl/blob/10235c9c1ad47989bdcfebe47f4a369871357e2a/segmentation/include/pcl/segmentation/impl/sac_segmentation.hpp#L115-L123 .
You can access the model via getModel() (https://pointclouds.org/documentation/classpcl_1_1_s_a_c_segmentation.html#ac7b9564ceba35754837b4848cf448d78).
Ultimately got it working, with advice from IBitMyBytes by setting seg.setOptimizeCoefficients(false); and then manually optimising after doing my own check:
// if we can, optimise model coefficients
if (sac_optimise_coefficients_ && inlier_indices->indices.size() > 4) {
pcl::SampleConsensusModel<pcl::PointXYZI>::Ptr model = seg.getModel();
Eigen::VectorXf coeff_refined;
Eigen::Vector4f coeff_raw(coefficients->values.data());
model->optimizeModelCoefficients(inlier_indices->indices,
coeff_raw, coeff_refined);
coefficients->values.resize(coeff_refined.size());
memcpy(&coefficients->values[0], &coeff_refined[0],
coeff_refined.size() * sizeof (float));
// Refine inliers
model->selectWithinDistance(coeff_refined, sac_distance_threshold_,
inlier_indices->indices);
}
I'm using reduceRegion to sum the number of water pixels determined by an NDWI. I want to do this on a collection of images to see change in water area over a period time.
The values are returned from the reduceRegion in the console and appear to be integers, but I am unable to extract them as such. This seems to be a common problem, but the solution is typically using getInfo function to bring these values over to client side. Unfortunately getInfo returns a null in this case.
The code below is for a collection of images OR a single image. The single image(image1) returns an image with an extra property(waterArea), and the mapped algorithm blows up(because the function is returning nulls).
I've also tried using getInfo on waterAg to potentially bring that list over to client side, however that returns the same List of objects that I don't want.
var image1 = ee.Image(leastcloud.first()).select(['B11','B8','B3'])
var stackarea = function(image){
var watermask = ee.Image(0)
var MNDWI = image.normalizedDifference(['B3','B11'])
watermask = watermask.where(MNDWI.gt(.31),1);
//sum of water pixels in an area
var sum = ee.Number(watermask.reduceRegion({
reducer: ee.Reducer.sum(),
geometry: geometry,
scale: 20,
maxPixels: 1e9
}));
var clientSide = sum.getInfo()
var setArea = image.set('waterArea', clientSide)
return setArea
};
var single = stackarea(image1)
print(single)
var watermapped = filterSpot.map(stackarea)
var waterAg = watermapped.aggregate_array('waterArea')
print(waterAg)
I'm really not sure how to extract these values as numbers...
I'm hoping to get a list of numbers so that I can concatenate that array to other arrays(date of image, cloud pixel %, etc.)
reduceRegion returns a dictionary object, not a number-like object. Therefore, in your stackarea function, clientSide variable is a dictionary (i.e. an object), not a number.
The number that you're after is stored in the dictionary returned by the reduceRegion function. You can get your hand on that number by using a get function on this dictionary object:
var sum = watermask.reduceRegion({
reducer: ee.Reducer.sum(),
geometry: geometry,
scale: 20,
maxPixels: 1e9
}).get('nd');
In this way, sum will be a server-side number that store the value you're after (There's no need to use ee.Number here as it does not help much).
In case you wonder about why using get('nd') but not get('somethingelse'), well it's the name of the band in your watermask image, and this name is the default band name given to the result of the normalizedDifference function.
And in my opinion, you don't even need to use the getInfo function which takes much more time to execute. Just delete the line var clientSide = sum.getInfo() and modify the next line into var setArea = image.set('waterArea', sum).
Hope this help.
Experienced programmer playing around with Gamemaker2 Studio.
Trying to draw some random squares on the screen using a 2D array to store the "map"
Step 1 : declare a 2D array MyMap[25,25] this works
Step 2 : Set 100 random locations in Map[]=1 this works
I get a crash when I try to look up the values I have stored in the array.
Its crashing with:
**Execution Error - Variable Index [3,14] out of range [26,14] **
So it looks like it is trying to read 26 element, when you can see from my code the for next loop only goes to 20 and the array bound is 25.
Oddly enough it does the first two loops just fine?
Looking like a bug, I've spent so much time trying to work it out, anyone got an idea what is going on?
var tx=0;
var ty=0;
var t=0;
MyMap[25,25]=99; **// Works**
for( t=1; t<100; t+=1 ) **// Works**
{
MyMap[random(20),random(15)]=1
}
for( tx=1; tx<20; tx+=1 )
{
for( ty=1; ty<15; ty+=1 )
{
show_debug_message(string(tx) + ":" + string(ty))
t = MyMap[tx,ty]; /// **<---- Crashes Here**
if t=1 then {draw_rectangle(tx*32,ty*32,tx*32+32,ty*32+32,false) }
}
}
The line MyMap[random(20),random(15)]=1 does not initialize values in the entire array, creating a sparse array(where some elements do not exist).
The line MyMap[25,25]=99;
Should read:
for( tx=1; tx<20; tx+=1 )
{
for( ty=1; ty<15; ty+=1 )
{
MyMap[tx,ty]=99;
}
}
This will pre-initialize the all of the array values to 99. Filling out the array.
Then you can randomly assign the ones. (You will probably get less than 100 ones the due to duplicates in the random function and the random returning zeros.)
You should have the above code in the Create Event, or in another single fire or controlled fire event, and move the loops for the draw into the Draw Event.
All draw calls should be in the Draw Event. If the entire block were in Draw, it would randomize the blocks each step.
I have a dataset I'm working with that is buildings and electrical power use over time.
There are two aggregations on these buildings that are simple sums across the entire timespan and I have those written. They end up looking like:
var reducer = reductio();
// How much energy is used in the whole system
reducer.value("energy").sum(function (d) {
return +d.Energy;
});
These work great.
The third aggregation, however, is giving me some trouble. I need to find the point that the sum of all the buildings is at its greatest. I need the max of the sum and the time it happened.
I wrote:
reducer.value("power").sum(function (d) {
return +d.Power;
}).max(function (d) {
return +d.Power;
}).aliasProp({
time: function (d, v) {
return v.Timestamp;
}
});
But, this is not necessarily the biggest power use. I'm pretty sure this returns the sum and the time when any individual building used the most power.
So if the power values were 1, 1, 1, 15. I would end up with 18, when there might be a different moment when the values were 5, 5, 5, 5 for a total of 20. The 20 is what I need.
I am at a loss for how to get the maximum of a sum. Any advice?
Just to restate: You are grouping on time, so your group keys are time periods of some sort. What you want is to find the time period (group) for which power use is greatest.
If I'm right that this is what you want, then you would not do this in your reducer, but rather by sorting the groups. You can order groups by using the group.order method: https://github.com/crossfilter/crossfilter/wiki/API-Reference#group_order
// During group setup
group.order(function(p) { return p.power.sum; })
// Later, when you want to grab the top power group
group.top(1)
Reductio's max aggregation should just give you the maximum value that occurs within the group. So given a group with values 1,1,1,15, you would get back the value 15. It sounds like that's not what you want.
Hopefully I understood properly. If not, please comment. If you can put together an example with toy data that is public and where you can tell me what you would like to see vs what you are getting, I should be able to help out.
Update based on example:
So, what you want (based on the description in the example) is to find the maximum power usage for any given time within the selected time period. So you would do the following:
var timeDim = buildings.dimension(function(d) { return d.Timestamp })
var timeGrp = timeDim.group().reduceSum(function(d) { return d.Power })
var maxResults = timeGrp.top(1)
Whenever you want to find the max power usage time for your current filter, just call timeGrp.top(1) and the key of that group will be the time with the maximum power.
Note: Don't filter on timeDim as the filters on a dimension are not applied to groups defined on that dimension.
Here's an updated JSFiddle that writes out the maximum group to the console: https://jsfiddle.net/esjewett/1o3robm3/1/