I have two sets of points in two separate tables like this :
t1 :
Point_1 |Lat | Long
..................
Point_n |Lat |Long
and
t2 :
Pt_1 |Lat | Long
..................
Pt_m |Lat |Long
with no relation between the two tables.
What would be the best way (least resources) to identify the top 3 closest points in t2 for each pt in t1, particulalrly when t1 and t2 are huge? Maybe Geohashing?
What I tried and seems to work fine with small datasets is :
t1
| extend blah=1
| join kind=fullouter (t2 |extend blah=1) on blah
| extend distance = geo_distance_2points(Long,Lat,Long1,Lat1)
|sort by spbldrom_code, distance asc
| extend rnk = row_number(1,point <> prev(point))
| where rnk<=3
|project point, pt, distance, rnk
Please pardon the sloppiness ; I'm learning .
Thank you!
Try reducing the data size on both sides of the join operator, by filtering out irrelevant or ill formatted rows and columns. Perhaps you can use geo_point_in_polygon\circle() to throw out irrelevant data.
Try using broadcast join or maybe shuffle join:
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/broadcastjoin
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/shufflequery
You can use s2\geohash\h3 hashing functions in two ways:
a. Per each table, combine nearby points into one representative point. The
idea is to use hash cell central point as a representative for all
points that reside in the cell. This will reduce tables sizes. Something like:
datatable(lng:real, lat:real)
[
10.1234, 53,
10.3579, 53,
10.6842, 53,
]
| summarize by hash = geo_point_to_s2cell(lng, lat, 8)
| project geo_s2cell_to_central_point(hash)
b. Calculate hash value for each point and join on the hash value. Something like:
let t1 =
datatable(lng:real, lat:real)
[
10.3579, 53,
10.6842, 53,
];
let t2 =
datatable(lng:real, lat:real)
[
10.1234, 53,
];
t1 | extend hash = geo_point_to_s2cell(lng, lat, 8)
| join kind=fullouter hint.strategy=broadcast (t2 | extend hash = geo_point_to_s2cell(lng, lat, 8)) on hash
Perhaps partition operator might also speed up the query:
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/partitionoperator
I found what I think to be a better way to do this and want to share it.
Firstly, the issue with tessellation / geo-hashing is this:
Let's assume you have two sets of points with coordinates in two tables T1 and T2 and what to calculate the closest point in T2 for each point in T1. Now let's assume you have a point in T1 very close to the border of a geo-hash cell, and another point in T2 close to the same border, but in the neighboring geo-hash cell. Using the join method based on hash id, the algorithm will never calculate the distance between these two points, although they are very close, so the end result will miss this pair.
A better way to do the join of the two tables to calculate inter-points distance is generating a join key based on truncated coordinates. So for each point in each table , we create this key based on the relevancy of interpoint distance (what is the max inter-point distance we care about).
Example : for a point with coordinates ( 45.1234; -120.5678 ) the join key could be 25.1-120.6 (truncation and concatenation). With this rounding and using the join method , we would capture everything in table 2 within app 15 km radius of that point in table 1. Going for 25-120 as the join key will capture everything within 150Km. This will reduce significantly the joined table and avoids the caveats of geo-hashing method.
At this point I'm better at writing prose than code :), however I hope what I described above it makes sense. It certainly works for my project while circumventing the resource problems (cpu/mem).
Happy you've found a way that works for you. Another option that you may try is also taking into account neighbor cells.
H3 hash has such capability: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/geo-h3cell-rings-function
Something like this:
let h3_resolution = 8;
let t1 = datatable(lng1:real, lat1:real)
[
40.75864778392896, -73.97856558479198,
40.74860253711237, -73.98577679198793,
40.741092676839024, -73.9902397446769,
];
let t2 = datatable(lng2:real, lat2:real)
[
40.75594965648444, -73.98157034840024,
40.766085141039774, -74.01798702196743
];
t1
| extend hash = geo_point_to_h3cell(lng1, lat1, h3_resolution)
| join kind = inner (
t2
| extend rings = geo_h3cell_rings(geo_point_to_h3cell(lng2, lat2, h3_resolution),1)
| project lng2, lat2, hash_array = array_concat(rings[0], rings[1])
| mv-expand hash_array to typeof(string)
) on $left.hash == $right.hash_array
| project-away hash, hash_array
| extend distance = geo_distance_2points(lng1, lat1, lng2, lat2)
| project p1 = tostring(pack_array(lng1, lat1)), p2 = pack_array(lng2, lat2), distance
| sort by distance asc
| summarize closest_3_points = make_list(p2, 3) by p1
Recently, I stumbled over the following problem in an embedded software project. I cannot yet figure out, under which conditions there is a unique solution and how it can be found.
Let us assume that we have a 16-bit MASK and a 16-bit COMPARE value and we want to set them, such that for a defined range of consecutive IDs {a, ..., b} (e.g. {0x78, 0x79, ..., 0x97}) those IDs satisfy bitwise
ID & MASK == COMPARE
while IDs outside that range do not satisfy the above equation.
As an example: If a=0x100 and b=0x1FF, the MASK is set as 0x700 and the COMPARE as 0x100.
I have the following questions:
What are the conditions for the minimal and maximal ID, so that uniquely defined MASK and COMPARE values exist?
How can they be calculated?
Looking forward to your answers!
The combination of MASK and COMPARE can be described as a string of 0 and 1 with wildcards (*) to indicate "don't care" positions. If there's a * left of a 0 or 1, the string does not describe a continuous range. Thus, the strings that represent ranges are exactly those where all the * are at the right end. The combinations of a and b that can be checked this way are then those where we can divide the 16 bits into a left part and a right part such that a and b are equal in the left part and the right part is all zero for a and all ones for b. We can test this as follows:
((((a ^ b) + 1) | a) & b) == a
I have 2 3D vectors. (objects with X, Y and Z float values)
In my diagram below, I would like to determine the length of the green line.
This is the distance along Vector 1 that Vector 2 is. Or, the distance from the origin to the end of a line on Vector 1 which is at 90' to Vector 1 and passes thorough the point at the end of Vector 2.
I am doing this in Unity3D so I have access to quite a few helper methods that enable me to get the length of a Vector3 and so on very easily.
The length is obviously
norm(v2)*cos(angle(v1,v2))
and since
cos(angle(v1,v2))=abs(dot(v1,v2))/norm(v1)/norm(v2)
the final formula is
abs(dot(v1,v2))/norm(v1)
One could also say that
e1 = v1/norm(v1)
is the unit vector in the direction of v1, and that the green vector is
dot(e1,v2)*e1
resulting in the same length formula.
This is projection of Vector2 onto Vector1 direction. The simplest way (I think) to find it - using scalar product
D = |V2| * DotProduct(V2, V1) / (|V2| * |V1|) = DotProduct(V2, V1) / |V1|
where |V1| is the length of V1 vector
Im not sure but I think this is what you wanted
Vector3 distance = Vector3.Lerp(Vector3.zero, vector_1, vector_2.sqrMagnitude / vector_1.sqrMagnitude);
http://docs.unity3d.com/ScriptReference/Vector3-sqrMagnitude.html
http://docs.unity3d.com/ScriptReference/Vector3.Lerp.html
I have a SQLite table with fields Latitude and Longitude (neg and pos numbers)
I have a latitude *delta* and a longitude *delta* of (for example '23') based on a Latitude of 50 and a longitude of 75 (again, for example)
How do I query the table for a list of all records which are + or - 23 latitudes and + or - 23 longitudes when there are both positive and negative latitude and longitude values in the table.
The lat/long fields are of type NUMERIC and are indexed ASC.
select [fields]
from [table]
where latitude of 50 -+23 and
longitude of 75 -+23
Lookup the delta and stating values first
Calculate the +/- values based on the delta and the starting values per-record
Use the "BETWEEN operator" to get the result (scroll down on that link to find the info)
If you're using an ORM between you and the database, or a programming language on top of the DB, you might be able to do steps 1 and 2 on the fly, thereby only executing one DB query in step 3.
I want to create a board class from canvas, which will
allow to track click position on it in coordinates like A2, where
A...D is Y coordinate in some scale
and 1...3 is X coordinate
For example
see image http://img.skitch.com/20091001-k6ybfehid6y8irad36tbsiif15.jpg
What I want to create is a kind of convertor from canvas localX and localY to my
new coordinates like A2.
I am thinking of implementing if condition this way
if (0.4 - x*size(from 1-3 here)/canvas.width <= X <= 0.4 + x*size(from 1-3 here)/canvas.width)
X = x;
This way I can assigned needed coordinates in X range. e.g. 1, 2 ,3 etc
But what to do with alphanumeric range. (if for example I want to make it extensible)...
Maybe there is a way to convert ASCII to char? Pls. suggest your solution
The same way as in JavaScript: fromCharCode. If y is an integer starting at 1 for A:
String.fromCharCode(64+y)+x
you can use the function fromCharCode in String class to do that.
for example: String.fromCharCode(ascii code);