Finding a large database of locations around the world? - dictionary

I am creating a sample application for a client using mapbox and I need to huge dataset of locations that span accross the whole world in lt lng format.
E.g.
var addressPoints = [
[-37.8210922667, 175.2209316333, "2"],
[-37.8210819833, 175.2213903167, "3"],
[-37.8210881833, 175.2215004833, "3A"],
];
I was thinking of a list of hotels around the world or something? Or an extremely cleaver way of creating locations on the fly in Javascript that do not end up in the sea?
Any help would be great appreciated!
Thanks.

This is a fun question - I don't know of a huge data set but some ideas I had for generating one are:
Strava API
If you have a free API key, you can query Strava's segments end point and give it a lat/lng boundary, so you will get start and end points within those bounds (but limited to 10 results so you'd have to loop to get enough data and may exceed your API call limit, however if you just create it once it might not be so bad). You could also get a lot of points from a specific bike ride/run, but they'd all be close together which I don't think you want.
http://strava.github.io/api/v3/segments/#explore e.g.
Throttling
The default rate limit allows 600 requests every 15 minutes, with up
to 30,000 requests per day.
API call
curl -G https://www.strava.com/api/v3/segments/explore \
-H "Authorization: Bearer YOUR_TOKEN" \
-d bounds=37.821362,-122.505373,37.842038,-122.465977 \
Response
{
"segments": [
{
"id": 229781,
"name": "Hawk Hill",
"climb_category": 1,
"climb_category_desc": "4",
"avg_grade": 5.7,
"start_latlng": [
37.8331119,
-122.4834356
],
"end_latlng": [
37.8280722,
-122.4981393
],
"elev_difference": 152.8,
"distance": 2684.8,
"points": "}g|eFnm#n#Op#VJr#"
},
{
"id": 632535,
"name": "Hawk Hill Upper Conzelman to Summit",
"climb_category": 0,
"climb_category_desc": "NC",
"avg_grade": 8.10913,
"start_latlng": [
37.8334451,
-122.4941994
],
"end_latlng": [
37.8281297,
-122.4980005
],
"elev_difference": 67.29200000000003,
"distance": 829.834,
"points": "_j|eFvc#p#SbAu#h#Qn#?RTDH"
}
]
}
Google Geocoding API
This API will accept an address and give you a lat/lng (and more) but is throttled. You'd just need a list of addresses which would be pretty easy to get.
API https://developers.google.com/maps/documentation/geocoding/
Throttling
Users of the free API: 2,500 requests per 24 hour period. 5 requests
per second.
API call
https://maps.googleapis.com/maps/api/geocode/json?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA
Response
{
"results" : [
{
"address_components" : [
{
"long_name" : "1600",
"short_name" : "1600",
"types" : [ "street_number" ]
},
{
"long_name" : "Amphitheatre Pkwy",
"short_name" : "Amphitheatre Pkwy",
"types" : [ "route" ]
},
{
"long_name" : "Mountain View",
"short_name" : "Mountain View",
"types" : [ "locality", "political" ]
},
{
"long_name" : "Santa Clara",
"short_name" : "Santa Clara",
"types" : [ "administrative_area_level_2", "political" ]
},
{
"long_name" : "California",
"short_name" : "CA",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United States",
"short_name" : "US",
"types" : [ "country", "political" ]
},
{
"long_name" : "94043",
"short_name" : "94043",
"types" : [ "postal_code" ]
}
],
"formatted_address" : "1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA",
"geometry" : {
"location" : {
"lat" : 37.42291810,
"lng" : -122.08542120
},
"location_type" : "ROOFTOP",
"viewport" : {
"northeast" : {
"lat" : 37.42426708029149,
"lng" : -122.0840722197085
},
"southwest" : {
"lat" : 37.42156911970850,
"lng" : -122.0867701802915
}
}
},
"types" : [ "street_address" ]
}
],
"status" : "OK"
}

Related

JSONPath Export Key & Value

I am using JSONPath to filter out a json file, I am trying to retrieve both the Key + Value from the JSON array but I am lost at figuring how best to do this.
QUERY
$.phoneNumbers[*].[type,number]
ARRAY
{
"firstName": "John",
"lastName" : "doe",
"age" : 26,
"address" : {
"streetAddress": "naist street",
"city" : "Nara",
"postalCode" : "630-0192"
},
"phoneNumbers": [
{
"type" : "iPhone",
"number": "0123-4567-8888",
"address" : {
"streetAddress": "naist street",
"city" : "Nara",
"postalCode" : "630-0192"
}
},
{
"type" : "home",
"number": "0123-4567-8910",
"address" : {
"streetAddress": "naist street",
"city" : "Nara",
"postalCode" : "630-0192"
}
}
]
}
CURRENT OUTPUT
[
"iPhone",
"0123-4567-8888",
"home",
"0123-4567-8910"
]
Desired Outcome
[
{
"type" : "iPhone",
"number": "0123-4567-8888",
},
{
"type" : "home",
"number": "0123-4567-8910",
}
]
What you're looking for can't be done with JSON Path because you're looking to transform the data. JSON path is only going to give you the values in a flat list.
You might find some luck with JMES Path. It's pretty similar, but I'm less familiar with it.
You likely need to add quotes
$.phoneNumbers[*].['type','number']

Best way to get list of state-level places (google places api)

I'm currently working a project with the requirement to get a list of state-level places. I've tried the following two APIs but none of them can 100% meet the requirement:
Google Autocomplete API. In order to get rid off the detailed street-level places I passed types: ["(regions)"] which is OK from business' point of view (even though only state-level would be perfect, but city-level is still OK). However when user search for Washington, USA the API only returns 5 cities named Washington, and user have to search for WA, USA instead.
Google Places API. This API can successfully handle the Washington case above. However I couldn't find a way to restrict the API to get rid off the street-level places, unless I filter the returned places by the types attribute.
Does anyone know what is the best API to use for this scenario?
Doing a Text Search request with the search string as Washington, USA I get only 1 result which seems to be the WA state.
Request (use your own API key): https://maps.googleapis.com/maps/api/place/textsearch/json?query=Washington%2C%20USA&language=en&key=YOUR_API_KEY
Result:
{
"html_attributions" : [],
"results" : [
{
"formatted_address" : "Washington, USA",
"geometry" : {
"location" : {
"lat" : 47.7510741,
"lng" : -120.7401385
},
"viewport" : {
"northeast" : {
"lat" : 49.0024442,
"lng" : -116.91558
},
"southwest" : {
"lat" : 45.543541,
"lng" : -124.8489739
}
}
},
"icon" : "https://maps.gstatic.com/mapfiles/place_api/icons/v1/png_71/geocode-71.png",
"name" : "Washington",
"photos" : [
{
"height" : 2124,
"html_attributions" : [
"\u003ca href=\"https://maps.google.com/maps/contrib/101488181553592859672\"\u003eArko Bhattacharjee\u003c/a\u003e"
],
"photo_reference" : "ATtYBwLa-P4aH5Uh3rD2CKaZmwb8HyNPWZ1WZUnw_y4iaUouh4B6BKOwAo52kkiBNSfn3MJmkW6Ju50H_e7zhxs5m5oBJva-dywtTv2cF_OgX6COeUfnre_SVmXw1qA4Fof924hSOvutwFnjsO6MJ1R71ozhBfieWrtvN9dSwVbgQbvMu-3y",
"width" : 1440
}
],
"place_id" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"reference" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"types" : [ "administrative_area_level_1", "political" ]
}
],
"status" : "OK"
}
Double check this by doing a Place Details request with the returned place id ChIJ-bDD5__lhVQRuvNfbGh4QpQ:
Request (use your own API key): https://maps.googleapis.com/maps/api/place/details/json?place_id=ChIJ-bDD5__lhVQRuvNfbGh4QpQ&language=en&key=YOUR_API_KEY
Result:
{
"html_attributions" : [],
"result" : {
"address_components" : [
{
"long_name" : "Washington",
"short_name" : "WA",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United States",
"short_name" : "US",
"types" : [ "country", "political" ]
}
],
"adr_address" : "\u003cspan class=\"region\"\u003eWashington\u003c/span\u003e, \u003cspan class=\"country-name\"\u003eUSA\u003c/span\u003e",
"formatted_address" : "Washington, USA",
"geometry" : {
"location" : {
"lat" : 47.7510741,
"lng" : -120.7401386
},
"viewport" : {
"northeast" : {
"lat" : 49.00256875707851,
"lng" : -116.9155800245149
},
"southwest" : {
"lat" : 45.54354101516995,
"lng" : -124.8489739457119
}
}
},
"icon" : "https://maps.gstatic.com/mapfiles/place_api/icons/v1/png_71/geocode-71.png",
"name" : "Washington",
"photos" : [],
"place_id" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"reference" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"types" : [ "administrative_area_level_1", "political" ],
"url" : "https://maps.google.com/?q=Washington,+USA&ftid=0x5485e5ffe7c3b0f9:0x944278686c5ff3ba",
"utc_offset" : -480,
"website" : "http://www.access.wa.gov/"
},
"status" : "OK"
}

Decode Linkedin region codes

I've gathered json via Linkedin API.
Node:
[GetCompanyPageStatistics][followStatistics][regions]
stores information about followers localizations, for example:
"regions": {
"_total": 9,
"values": [
{
"entryKey": "pl-8172",
"entryValue": "18"
},
{
"entryKey": "pl-8355",
"entryValue": "2"
},
{
"entryKey": "pl-8218",
"entryValue": "1"
},
{
"entryKey": "de-4944",
"entryValue": "1"
}
]
}
Given values consit of country code and region code. Is there a way to decode the region? Linkedin API documentation isn't very helpful:
https://developer.linkedin.com/docs/reference/geography-codes
(there's only eu.pl for Poland without any information about region-specific codes)
Looks like there's a v2 api to get that data:
https://learn.microsoft.com/en-us/linkedin/shared/references/v2/standardized-data/locations/regions
E.g., hitting https://api.linkedin.com/v2/regions/8172?oauth2_access_token=XXX gets you:
{
"country": "urn:li:country:pl",
"name": {
"locale": {
"country": "US",
"language": "en"
},
"value": "Warsaw, Masovian District, Poland"
},
"id": 8172,
"$URN": "urn:li:region:8172",
"states": []
}

Elasticsearch geo_bbox with Google Places API viewport locations

Hopefully I can explain my issue.
I am using Google's Autocomplete to get a person destination. Once I have the destination I can use google's place api to look up the geometry viewport. But the viewport is from theNorth/East to South/West. My problem is with Elasticsearch the geo_bbox takes it from top/left to bottom/right. See examples below.
Google Place API Response:
{
"html_attributions" : [],
"result" : {
"address_components" : [
{
"long_name" : "Devon",
"short_name" : "Devon",
"types" : [ "administrative_area_level_2", "political" ]
},
{
"long_name" : "England",
"short_name" : "England",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United Kingdom",
"short_name" : "GB",
"types" : [ "country", "political" ]
}
],
"adr_address" : "\u003cspan class=\"region\"\u003eDevon\u003c/span\u003e, \u003cspan class=\"country-name\"\u003eUK\u003c/span\u003e",
"formatted_address" : "Devon, UK",
"geometry" : {
"location" : {
"lat" : 50.77721349999999,
"lng" : -3.9994610
},
"viewport" : {
"northeast" : {
"lat" : 51.24619840,
"lng" : -2.88664060
},
"southwest" : {
"lat" : 50.20189620,
"lng" : -4.68065630
}
}
},
"icon" : "http://maps.gstatic.com/mapfiles/place_api/icons/geocode-71.png",
"id" : "065fb13848aeae653fe25ee19d8a0a0289bd64dc",
"name" : "Devon",
"reference" : "CnRoAAAAN84gxrolBJLF_flpuwIyGum7pjln_KP9NJIHydIHMfbAo93nUdHSyYbv90GPUlW8Jx5D2ba-2fJHyzcHAdkZLkaJzSBEfj7dkR4NnBWzK9mFikZdyz3cee9IvOruHCZaicKfiXJP18jSfxXkvVv8gBIQrAopOdNbGQ6ovqEaEUqrxxoUaUYJEahZx7mTfJwEWOnxjTP3e9A",
"types" : [ "administrative_area_level_2", "political" ],
"url" : "https://maps.google.com/maps/place?q=Devon&ftid=0x486bf8abdac00247:0x12c6ba0cca9c58fd"
},
"status" : "OK"
}
Elastic Search Example
{
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"geo_bounding_box" : {
"pin.location" : {
"top_left" : {
"lat" : 40.73,
"lon" : -74.1
},
"bottom_right" : {
"lat" : 40.717,
"lon" : -73.99
}
}
}
}
}
}
Is there a better way to get more info from google or can elasticsearch work from alternate corners. Although the docs does not say this?
Oh dear how thick I am.
All I needed to do was swap the result's around for my elastic search. ie,
{
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"geo_bounding_box" : {
"pin.location" : {
"top_left" : {
"lat" : result.northeast.lat,
"lon" : result.southwest.lon
},
"bottom_right" : {
"lat" : result.southwest.lat,
"lon" : result.northwest.lon
}
}
}
}
}
}
If you know the North East and South West corners, you can calculate the North West (top left) and South East (bottom right) corners:
var NE = bounds.getNorthEast();
var SW = bounds.getSouthWest();
// calculated points
var NW = new google.maps.LatLng(NE.lat(), SW.lng());
var SE = new google.maps.LatLng(SW.lat(),NE.lng());

Google Geocoder Lookup get postal code by country and city

i'm currently using Google Geocoder API to determine location data for my web service.
It's important for me to get the postal code for a given country and city, but it's look like that google is limiting the result.
Here's the API call:
http://maps.googleapis.com/maps/api/geocode/json?address=deutschland,+saarlouis&language=de&sensor=false
The result is like that:
{
"results" : [
{
"address_components" : [
{
"long_name" : "Saarlouis",
"short_name" : "Saarlouis",
"types" : [ "locality", "political" ]
},
{
"long_name" : "Landkreis Saarlouis",
"short_name" : "Landkreis Saarlouis",
"types" : [ "administrative_area_level_3", "political" ]
},
{
"long_name" : "Saarland",
"short_name" : "SL",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "Deutschland",
"short_name" : "DE",
"types" : [ "country", "political" ]
}
],
"formatted_address" : "Saarlouis, Deutschland",
"geometry" : {
"bounds" : {
"northeast" : {
"lat" : 49.36187570,
"lng" : 6.815549499999999
},
"southwest" : {
"lat" : 49.26046930,
"lng" : 6.67501510
}
},
"location" : {
"lat" : 49.31346060,
"lng" : 6.752286499999999
},
"location_type" : "APPROXIMATE",
"viewport" : {
"northeast" : {
"lat" : 49.36187570,
"lng" : 6.815549499999999
},
"southwest" : {
"lat" : 49.26046930,
"lng" : 6.67501510
}
}
},
"types" : [ "locality", "political" ]
}
],
"status" : "OK"
}
Thx for help.
Regards
Geocoding does not give postcode. Goenames has a database for Germany. Other countries are also available.
The reason you don't get the postal code is the query is too broad. If you would add a street name to the query the result will contain a postal code. A solution to your problem if you don't have a street name or don't want to use it is to split the geocoding into two parts:
Step 1: Use the city to get the GPS coordinates
https://maps.googleapis.com/maps/api/geocode/json?address=amsterdam
Step 2: Use the the GPS coordidates to get the postal code
https://maps.googleapis.com/maps/api/geocode/json?latlng=52.3182742,4.7288558
(Note: Double of same question posted here, answered by MidnightMotion. Adding to make easier to lookup.
https://gis.stackexchange.com/questions/33966/google-geocoder-lookup-get-postal-code-by-country-and-city)

Resources