WHAT I HAVE TO DO:
I'm building an application where I want to do a query in firebase and receive the users who are in a range of X kilometers.
But I want to limit this received documents and also to have a function for getting more documents on the scroll. I'm using a StreamBuilder->ListView.build in order to build my list. When I'm going to receive the end of the scrollable list I want to get more documents from Firestore (if I have any).
WHAT I'VE DONE ALREADY:
So far I found just 2 solutions for my problem:
Firestore Helpers
GeoFlutterFire
THE PROBLEM:
Both of those libraries are good with just one BIG problem, in both of this case I'm not able to limit the number of documents that I'm going to receive (I can JUST limit the original query for Firestore.instance.collection("MyColection")) but is not going to help me that much because I will have this case
Give me 10 documents from Firestore -> Give me the 5km range users ->
I will receive just 4 let's say.
What I want instead is to receive 10 documents from Firestore within a range of X.
Is there any solution for this problem? Or I need to implement my own library for this? Or is not possible even so?
GeoPoint userGeoPoint = user.geoPoint["geopoint"];
// distance in miles
double distance = 30;
double lat = 0.0144927536231884;
double lon = 0.0181818181818182;
double lowerLat = userGeoPoint.latitude - (lat * distance);
double lowerLon = userGeoPoint.longitude - (lon * distance);
double greaterLat = userGeoPint.latitude + (lat * distance);
double greaterLon = userGeoPint.longitude + (lon * distance);
GeoPoint lesserGeopoint = GeoPoint(lowerLat, lowerLon);
GeoPoint greaterGeopoint = GeoPoint(greaterLat, greaterLon);
Query query = Firestore.instance
.collection(path)
.where("geoPoint.geopoint", isGreaterThan: lesserGeopoint)
.where("geoPoint.geopoint", isLessThan: greaterGeopoint)
.limit(limit);
This worked for me. Hope it helps.
A bit late but I had the same problem and I decided to build an external server with distance ordering and limit functionality. Server stores objects that includes geopoint and Firestore document ID.
The client sends a request to the server with center point and radius.
The server returns a list of document IDs.
Fetch documents from Firestore based on provided IDs.
The best I came up with.
Related
Last year Firestore introduced count queries, which allows you to retrieve the number of results in a query/collection without actually reading the individual documents.
The documentation for this count feature mentions:
Aggregation queries rely on the existing index configuration that your queries already use, and scale proportionally to the number of index entries scanned. This means that aggregations of small- to medium-sized data sets perform within 20-40 ms, though latency increases with the number of items counted.
And:
If a count() aggregation cannot resolve within 60 seconds, it returns a DEADLINE_EXCEEDED error.
How many documents can Firestore actually count within that 1 minute timeout?
I created some collections with many documents in a test database, and then ran COUNT() queries against that.
The code to generate the minimal documents through the Node.js Admin SDK:
const db = getFirestore();
const col = db.collection("10m");
let count = 0;
const writer = db.bulkWriter();
while (count++ < 10_000_000) {
if (count % 1000 === 0) await writer.flush();
writer.create(col.doc(), {
index: count,
createdAt: FieldValue.serverTimestamp()
})
}
await writer.close();
Then I counted them with:
for (const name of ["1k", "10k", "1m", "10m"]) {
const start = Date.now();
const result = await getCountFromServer(collection(db, name));
console.log(`Collection '${name}' contains ${result.data().count} docs (counting took ${Date.now()-start}ms)`);
}
And the results I got were:
count
ms
1,000
120
10,000
236
100,000
401
1,000,000
1,814
10,000,000
16,565
I ran some additional tests with limits and conditions, and the results were always in line with the above for the number of results that were counted. So for example, counting 10% of the collection with 10m documents took about 1½ to 2 seconds.
So based on this, you can count up to around 40m documents before you reach the 60 second timeout. Honestly, given that you're charged 1 document read for every up to 1,000 documents counted, you'll probably want to switch over to stored counters well before that.
I'm trying to use HERE Map Attributes API to retrieve the addresses for some map area.
As far as I understood the layer I need is the POINT_ADDRESS layer (check the docs).
The problem is that the layer contains coordinates and identifiers only, but not the structured addresses:
[
{
"ADDRESS_POINT_ID": "334030589",
"LINK_ID": "1286528309",
"SIDE": "R",
"ADDRESSES": "ENGBN682",
"ADDRESS_TYPE": "1",
"BUILDING_NAMES": "ENGBNnull",
"DISPLAY_LAT": "-2751772",
"DISPLAY_LON": "15307500",
"ARRIVAL_LINK_ID": null,
"ARRIVAL_SIDE": null,
"LAT": "-2751797",
"LON": "15307488",
"TOPOLOGY_ID": "215602187",
"START_OFFSET": "8043"
},
...
]
So the question is: how to retrieve the structured address for every point from the POINT_ADDRESS layer? Should I use HERE Batch Geocoder API and perform reverse geocoding for these points? Or it's still possible to achieve that with Map Attributes API only?
On this link https://developer.here.com/documentation/content-map-attributes/dev_guide/topics/here-map-content.html are not all layers in the list.
Please see all layers on https://demo.support.here.com/pde .
In the weu region on https://demo.support.here.com/pde/layers?region=WEU&release=latest&url_root=pde.api.here.com
To retrieve the structured address you need additional layers ROAD_ADMIN_NAMES and ROAD_NAME
Then you request will be like: https://smap.hereapi.com/v8/maps/attributes.json?in=proximity:-27.51772,153.075001;r=100&layers=ROAD_NAME_FCn,ROAD_ADMIN_NAMES_FCn,POINT_ADDRESS&apikey=
Regarding docs on https://developer.here.com/documentation/content-map-attributes/api-reference.html
The parameter 'in' could be: proximity or bbox or corridor - for these in case for big areas you can get limitations due big data. Therefore some times it could be better use in=tile:...
Tile Ids you can get by these formulas:
tile size = 180° / 2^level [degree]
tileY = trunc((latitude + 90°) / tile size)
tileX = trunc((longitude + 180°) / tile size)
tileID = tileY * 2 * (2^level) + tileX
Or you can read this documentation https://demo.support.here.com/pde/indexes?region=WEU&release=latest&url_root=pde.api.here.com
How to get tileid by some index like:
https://smap.hereapi.com/v8/maps/index.json?layer=ROAD_GEOM_FCn&attributes=LINK_ID&values=1286528309,130742823
Then by tile request will be like:
https://smap.hereapi.com/v8/maps/attributes?apikey=&layers=POINT_ADDRESS,ROAD_NAME_FC4,ROAD_NAME_FC5,ROAD_ADMIN_NAMES_FC4,ROAD_ADMIN_NAMES_FC5&in=tile:46594870,11648411,46594870,11648411,46594870
According to this blog post, firebase array keys are created using a timestamp:
It does this by assigning a permanent, unique id based on the current timestamp (offset to match server time).
Is there a way to recover this timestamp for use later, given the key?
As I said in my comment, you should not rely on decoding the timestamp from the generated id. Instead of that, you should simply store it in a property in your Firebase.
That said, it turns out to be fairly easy to get the timestamp back:
// DO NOT USE THIS CODE IN PRODUCTION AS IT DEPENDS ON AN INTERNAL
// IMPLEMENTATION DETAIL OF FIREBASE
var PUSH_CHARS = "-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz";
function decode(id) {
id = id.substring(0,8);
var timestamp = 0;
for (var i=0; i < id.length; i++) {
var c = id.charAt(i);
timestamp = timestamp * 64 + PUSH_CHARS.indexOf(c);
}
return timestamp;
}
var key = prompt("Enter Firebase push ID");
if (key) {
var timestamp = decode(key);
console.log(timestamp+"\n"+new Date(timestamp));
alert(timestamp+"\n"+new Date(timestamp));
}
I'll repeat my comment, just in case somebody thinks it is a good idea to use this code for anything else than as an exercise in reverse engineering:
Even if you know how to retrieve the timestamp from the key, it would be a bad idea to do this in production code. The timestamp is used to generate a unique, chronologically ordered sequence. If somebody at Firebase figures out a more efficient way (whichever subjective definition of efficiency they happen to choose) to accomplish the same goal, they might change the algorithm for push. If your code needs a timestamp, you should add the timestamp to your data; not depend on it being part of your key.
Update
Firebase documented the algorithm behind Firebase push IDs. But the above advice remains: don't use this as an alternative to storing the date.
Here's a version of Frank's code re-written in Swift (4.2 at the time of writing.)
Just to be clear, my use case for this was to patch my old models with no timestamps (createdAt, updatedAt.) I could just throw in random dates in them just to save me some headaches. But then that wouldn't be relevant to their models. I knew that there's an element of time baked into these auto-ids based on what I've read from other articles.
let PUSH_CHARS = "-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"
func decode(autoId: String) -> TimeInterval {
let substring = autoId.substring(toIndex: 8)
var timestamp = 0
for i in 0..<substring.length {
let c = Character(substring[i])
timestamp = (timestamp * 64) + PUSH_CHARS.firstIndex(of: c)!.encodedOffset
}
return TimeInterval(exactly: timestamp)!
}
Grab the Playground-ready code here: https://gist.github.com/mkval/501c03cbb66cef12728ed1a19f8713f7.
And in python
PUSH_CHARS = "-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"
def get_timestamp_from_id(id):
timestr = id[0:8]
timestamp = 0
for idx, ch in enumerate(timestr):
timestamp = timestamp * 64 + PUSH_CHARS.index(ch)
return timestamp/1000
Since firebase not support any spatial indexing then I use geohash which someone advice here.
Geohash is using characters to find the nearby.
So let say I have w22qt4vhye1c w99qt4vdf4vc w22qt4vhzerct geohash store in firebase and I want to query only which geohash close to w22qt4.
How to do that?
I know firebase has startAt. I tried but not work.
UPDATE
Storing geohash to firebase
//Encode lat lng, result w22qt4vhye1c3
var geoHashLatLng = encodeGeoHash(lat,lng);
firebaseRef.child('/geohash/' + geoHashLatLng).set(id);
so in json
root
- geohash
- w22qt4vhye1c3
- id
- z99qt4vdf4vc
- w22qt4vhzerct
My question here.
I Just want to query geohash which start from characters w22qt4. Can we do that in firebase?
UPDATE 2
startAt() seems like not query with characters start with ...
Example: I have following gehash
geohash node
geohash
- a123
- a333
- b123
- c123
- d123
With the following code
var firebaseRef = new Firebase('https://test.firebaseio.com');
var query = firebaseRef.child('/geohash').startAt(null, 'a');
query.on('child_added', function(snapshot) {
console.log(snapshot.name());
});
Will get this results
startAt(null, 'a') //return a123, a333, b123, c123, d123
startAt(null, 'c123') //return c123, d123
startAt(null, 'd') //return d123
My expected results
startAt(null, 'a') //return a123, a333
startAt(null, 'c123') //return c123
startAt(null, 'd') //return d123
My guess, startAt() will query the 26 alphabet letters in sequence but not matching.
So, what can I do in firebase so I can get my expected results above?
In addition to Andrew's answer above, what you want to do to get the desired effect with geohashes (i.e. be able to do prefix queries that let you specify accuracy in the 'query', specifically, for example every 3 characters in the geohash gets you ~ +-80km) is to store the data in a more granular and discretized manner.
So, rather than how you're storing it now, you'll want to save the data by chopping the geohash string key into fragments (I've done it at the 3 character boundary, but you should choose an appropriate tokenization strategy that gives you the desired accuracy) and use each fragment as a child name, as such:
root
- geohash
- w22
- qt4
- vhy
- e1c
- w22qt4vhye1c3
- id
- vhz
- erc
- w22qt4vhzerct
- id
- z99
- qt4
- vdf
- z99qt4vdf4vc
- id
Then, as in your question, when you need to get all geohashes that start with w22qt4 you would do a query against:
firebaseRef.child('/geohash/w22/qt4/')
Which would then give you the vhy and vhz children which then have all the actual geohashes that you were interested in.
You can use a normal startAt query to find keys near the specified key. For example, in this case you'd:
var query = firebaseRef.child("geohash").startAt(null, 'c').limit(5);
query.on("child_added", ...);
This will give you the 5 elements after geoHashLatLng when sorted lexigraphically.
If you want to end your query at some specified key, you can do that as well. For example:
var query = firebaseRef.child("geohash").startAt(null, 'c').endAt('d');
Given a database that contains three fields:
Latitude
Longitude
Proximity
Where Lat and Long are GPS coordinates, and Proximity is (some unit - feet? Seconds? Minutes?)
And given the user's current GPS lat/long...
I want to write a SQL query that will retrieve all rows where the user is within "Proximity" of those rows.
And the trick: This has to work in SQLite, which only supports fairly primitive data types. No cheating and relying on SQL Server (or some other product that provides better geospace functions).
Any suggestions?
Here is a C-based custom function for sqlite [copied from links noted below]. This can be used within an iOS app. It assumes you have columns named latitude and longitude, and calculates the difference between them and any lat/long coordinates you provide. Excellent write-up, works as-is.
#define DEG2RAD(degrees) (degrees * 0.01745327) // degrees * pi over 180
static void distanceFunc(sqlite3_context *context, int argc, sqlite3_value **argv)
{
// check that we have four arguments (lat1, lon1, lat2, lon2)
assert(argc == 4);
// check that all four arguments are non-null
if (sqlite3_value_type(argv[0]) == SQLITE_NULL || sqlite3_value_type(argv[1]) == SQLITE_NULL ||
sqlite3_value_type(argv[2]) == SQLITE_NULL ||
sqlite3_value_type(argv[3]) == SQLITE_NULL) {
sqlite3_result_null(context);
return;
}
// get the four argument values
double lat1 = sqlite3_value_double(argv[0]);
double lon1 = sqlite3_value_double(argv[1]);
double lat2 = sqlite3_value_double(argv[2]);
double lon2 = sqlite3_value_double(argv[3]);
// convert lat1 and lat2 into radians now, to avoid doing it twice below
double lat1rad = DEG2RAD(lat1);
double lat2rad = DEG2RAD(lat2);
// apply the spherical law of cosines to our latitudes and longitudes, and set the result appropriately
// 6378.1 is the approximate radius of the earth in kilometres
sqlite3_result_double(context, acos(sin(lat1rad) * sin(lat2rad) + cos(lat1rad) * cos(lat2rad) * cos(DEG2RAD(lon2) - DEG2RAD(lon1))) *
6378.1);
}
This defines an SQL function distance(Latitude1, Longitude1,
Latitude2, Longitude2), which returns the distance (in kilometres)
between two points.
To use this function, add the code above ... and
then add this line immediately after you call sqlite3_open:
sqlite3_create_function(sqliteDatabasePtr, "distance", 4, SQLITE_UTF8, NULL, &distanceFunc, NULL, NULL);
…where sqliteDatabasePtr is the database pointer returned by your call
to sqlite3_open.
Assuming you have a table called Locations, with columns called
Latitude and Longitude (both of type double) containing values in
degrees, you can then use this function in your SQL like this:
SELECT * FROM Locations ORDER BY distance(Latitude, Longitude, 51.503357, -0.1199)
This example orders the locations in your database based on how far
away they are from the London Eye, which is at 51.503357, -0.1199.
EDIT :
Original link http://www.thismuchiknow.co.uk/?p=71 is dead, so as someone mentioned in comment, you can use this link : https://web.archive.org/web/20160808122817/http://www.thismuchiknow.co.uk/?p=71 to get that webpage
The Haversine formula is what you want. Using simple Euclidian distance formula isn't sufficient, because the curvature of the earth affects the distance between two points.
Creating a Store Locator with PHP, MySQL & Google Maps is an article about implementing this solution with MySQL, but SQLite doesn't support trig functions.
Check out the answers in Calculating Great-Circle Distance with SQLite here on Stack Overflow for further tips on extending SQLite functions so you can solve this.
You also know that there is a plugin for SQLite called Spatialite which has all the same functions as PostGIS and SQLServer? I assume you are trying to use SQLLite on the iPhone or in the browser or something. If not they I HIGHLY reccomend spatialite.