Elasticsearch geo_bbox with Google Places API viewport locations - google-maps-api-3

Hopefully I can explain my issue.
I am using Google's Autocomplete to get a person destination. Once I have the destination I can use google's place api to look up the geometry viewport. But the viewport is from theNorth/East to South/West. My problem is with Elasticsearch the geo_bbox takes it from top/left to bottom/right. See examples below.
Google Place API Response:
{
"html_attributions" : [],
"result" : {
"address_components" : [
{
"long_name" : "Devon",
"short_name" : "Devon",
"types" : [ "administrative_area_level_2", "political" ]
},
{
"long_name" : "England",
"short_name" : "England",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United Kingdom",
"short_name" : "GB",
"types" : [ "country", "political" ]
}
],
"adr_address" : "\u003cspan class=\"region\"\u003eDevon\u003c/span\u003e, \u003cspan class=\"country-name\"\u003eUK\u003c/span\u003e",
"formatted_address" : "Devon, UK",
"geometry" : {
"location" : {
"lat" : 50.77721349999999,
"lng" : -3.9994610
},
"viewport" : {
"northeast" : {
"lat" : 51.24619840,
"lng" : -2.88664060
},
"southwest" : {
"lat" : 50.20189620,
"lng" : -4.68065630
}
}
},
"icon" : "http://maps.gstatic.com/mapfiles/place_api/icons/geocode-71.png",
"id" : "065fb13848aeae653fe25ee19d8a0a0289bd64dc",
"name" : "Devon",
"reference" : "CnRoAAAAN84gxrolBJLF_flpuwIyGum7pjln_KP9NJIHydIHMfbAo93nUdHSyYbv90GPUlW8Jx5D2ba-2fJHyzcHAdkZLkaJzSBEfj7dkR4NnBWzK9mFikZdyz3cee9IvOruHCZaicKfiXJP18jSfxXkvVv8gBIQrAopOdNbGQ6ovqEaEUqrxxoUaUYJEahZx7mTfJwEWOnxjTP3e9A",
"types" : [ "administrative_area_level_2", "political" ],
"url" : "https://maps.google.com/maps/place?q=Devon&ftid=0x486bf8abdac00247:0x12c6ba0cca9c58fd"
},
"status" : "OK"
}
Elastic Search Example
{
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"geo_bounding_box" : {
"pin.location" : {
"top_left" : {
"lat" : 40.73,
"lon" : -74.1
},
"bottom_right" : {
"lat" : 40.717,
"lon" : -73.99
}
}
}
}
}
}
Is there a better way to get more info from google or can elasticsearch work from alternate corners. Although the docs does not say this?

Oh dear how thick I am.
All I needed to do was swap the result's around for my elastic search. ie,
{
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"geo_bounding_box" : {
"pin.location" : {
"top_left" : {
"lat" : result.northeast.lat,
"lon" : result.southwest.lon
},
"bottom_right" : {
"lat" : result.southwest.lat,
"lon" : result.northwest.lon
}
}
}
}
}
}

If you know the North East and South West corners, you can calculate the North West (top left) and South East (bottom right) corners:
var NE = bounds.getNorthEast();
var SW = bounds.getSouthWest();
// calculated points
var NW = new google.maps.LatLng(NE.lat(), SW.lng());
var SE = new google.maps.LatLng(SW.lat(),NE.lng());

Related

Best way to get list of state-level places (google places api)

I'm currently working a project with the requirement to get a list of state-level places. I've tried the following two APIs but none of them can 100% meet the requirement:
Google Autocomplete API. In order to get rid off the detailed street-level places I passed types: ["(regions)"] which is OK from business' point of view (even though only state-level would be perfect, but city-level is still OK). However when user search for Washington, USA the API only returns 5 cities named Washington, and user have to search for WA, USA instead.
Google Places API. This API can successfully handle the Washington case above. However I couldn't find a way to restrict the API to get rid off the street-level places, unless I filter the returned places by the types attribute.
Does anyone know what is the best API to use for this scenario?
Doing a Text Search request with the search string as Washington, USA I get only 1 result which seems to be the WA state.
Request (use your own API key): https://maps.googleapis.com/maps/api/place/textsearch/json?query=Washington%2C%20USA&language=en&key=YOUR_API_KEY
Result:
{
"html_attributions" : [],
"results" : [
{
"formatted_address" : "Washington, USA",
"geometry" : {
"location" : {
"lat" : 47.7510741,
"lng" : -120.7401385
},
"viewport" : {
"northeast" : {
"lat" : 49.0024442,
"lng" : -116.91558
},
"southwest" : {
"lat" : 45.543541,
"lng" : -124.8489739
}
}
},
"icon" : "https://maps.gstatic.com/mapfiles/place_api/icons/v1/png_71/geocode-71.png",
"name" : "Washington",
"photos" : [
{
"height" : 2124,
"html_attributions" : [
"\u003ca href=\"https://maps.google.com/maps/contrib/101488181553592859672\"\u003eArko Bhattacharjee\u003c/a\u003e"
],
"photo_reference" : "ATtYBwLa-P4aH5Uh3rD2CKaZmwb8HyNPWZ1WZUnw_y4iaUouh4B6BKOwAo52kkiBNSfn3MJmkW6Ju50H_e7zhxs5m5oBJva-dywtTv2cF_OgX6COeUfnre_SVmXw1qA4Fof924hSOvutwFnjsO6MJ1R71ozhBfieWrtvN9dSwVbgQbvMu-3y",
"width" : 1440
}
],
"place_id" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"reference" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"types" : [ "administrative_area_level_1", "political" ]
}
],
"status" : "OK"
}
Double check this by doing a Place Details request with the returned place id ChIJ-bDD5__lhVQRuvNfbGh4QpQ:
Request (use your own API key): https://maps.googleapis.com/maps/api/place/details/json?place_id=ChIJ-bDD5__lhVQRuvNfbGh4QpQ&language=en&key=YOUR_API_KEY
Result:
{
"html_attributions" : [],
"result" : {
"address_components" : [
{
"long_name" : "Washington",
"short_name" : "WA",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United States",
"short_name" : "US",
"types" : [ "country", "political" ]
}
],
"adr_address" : "\u003cspan class=\"region\"\u003eWashington\u003c/span\u003e, \u003cspan class=\"country-name\"\u003eUSA\u003c/span\u003e",
"formatted_address" : "Washington, USA",
"geometry" : {
"location" : {
"lat" : 47.7510741,
"lng" : -120.7401386
},
"viewport" : {
"northeast" : {
"lat" : 49.00256875707851,
"lng" : -116.9155800245149
},
"southwest" : {
"lat" : 45.54354101516995,
"lng" : -124.8489739457119
}
}
},
"icon" : "https://maps.gstatic.com/mapfiles/place_api/icons/v1/png_71/geocode-71.png",
"name" : "Washington",
"photos" : [],
"place_id" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"reference" : "ChIJ-bDD5__lhVQRuvNfbGh4QpQ",
"types" : [ "administrative_area_level_1", "political" ],
"url" : "https://maps.google.com/?q=Washington,+USA&ftid=0x5485e5ffe7c3b0f9:0x944278686c5ff3ba",
"utc_offset" : -480,
"website" : "http://www.access.wa.gov/"
},
"status" : "OK"
}

Elasticsearch low indexing speed

I have a blog that contains 14k posts and tried to add these posts to the elastic search index.
I indexed some of the posts, but it's extremely slow, and it will take about 6 hours to estimate. All the performance optimization tips from the official site I made. In my opinion, I removed the redundant data such as post meta. Can I increase indexing speed? Add the index configuration below:
{
"test-post-1" : {
"aliases" : { },
"mappings" : {
"date_detection" : false,
"properties" : {
"ID" : {
"type" : "long"
},
"guid" : {
"type" : "keyword"
},
"menu_order" : {
"type" : "long"
},
"permalink" : {
"type" : "keyword"
},
"post_content" : {
"type" : "text"
},
"post_date" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"post_excerpt" : {
"type" : "text"
},
"post_id" : {
"type" : "long"
},
"post_mime_type" : {
"type" : "keyword"
},
"post_modified" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss"
},
"post_name" : {
"type" : "text",
"fields" : {
"post_name" : {
"type" : "text"
},
"raw" : {
"type" : "keyword",
"ignore_above" : 10922
}
}
},
"post_parent" : {
"type" : "long"
},
"post_status" : {
"type" : "keyword"
},
"post_title" : {
"type" : "text",
"fields" : {
"post_title" : {
"type" : "text",
"analyzer" : "standard"
},
"raw" : {
"type" : "keyword",
"ignore_above" : 10922
},
"sortable" : {
"type" : "keyword",
"ignore_above" : 10922,
"normalizer" : "lowerasciinormalizer"
}
}
},
"post_type" : {
"type" : "text",
"fields" : {
"post_type" : {
"type" : "text"
},
"raw" : {
"type" : "keyword"
}
}
}
}
},
"settings" : {
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "5000"
},
"ignore_malformed" : "true"
},
"number_of_shards" : "1",
"provided_name" : "test-post-1",
"max_shingle_diff" : "8",
"max_result_window" : "1000000",
"creation_date" : "1582745447768",
"analysis" : {
"filter" : {
"shingle_filter" : {
"max_shingle_size" : "5",
"min_shingle_size" : "2",
"type" : "shingle"
},
"edge_ngram" : {
"min_gram" : "3",
"side" : "front",
"type" : "edgeNGram",
"max_gram" : "10"
},
"ewp_word_delimiter" : {
"type" : "word_delimiter",
"preserve_original" : "true"
},
"ewp_snowball" : {
"type" : "snowball",
"language" : "russian"
}
},
"normalizer" : {
"lowerasciinormalizer" : {
"filter" : [
"lowercase",
"asciifolding"
],
"type" : "custom"
}
},
"analyzer" : {
"ewp_lowercase" : {
"filter" : [
"lowercase"
],
"type" : "custom",
"tokenizer" : "keyword"
},
"shingle_analyzer" : {
"filter" : [
"lowercase",
"shingle_filter"
],
"type" : "custom",
"tokenizer" : "standard"
},
"default" : {
"filter" : [
"ewp_word_delimiter",
"lowercase",
"stop",
"ewp_snowball"
],
"char_filter" : [
"html_strip"
],
"language" : "russian",
"tokenizer" : "standard"
}
}
},
"number_of_replicas" : "1",
"uuid" : "cWGjSF4FQ1Or0A_0oSlA2g",
"version" : {
"created" : "7050299"
}
}
}
}
}
Wordpress version: 5.3.2
Elasticsearch version: 7.5.2
Enabled plugins: ElasticPress

Finding a large database of locations around the world?

I am creating a sample application for a client using mapbox and I need to huge dataset of locations that span accross the whole world in lt lng format.
E.g.
var addressPoints = [
[-37.8210922667, 175.2209316333, "2"],
[-37.8210819833, 175.2213903167, "3"],
[-37.8210881833, 175.2215004833, "3A"],
];
I was thinking of a list of hotels around the world or something? Or an extremely cleaver way of creating locations on the fly in Javascript that do not end up in the sea?
Any help would be great appreciated!
Thanks.
This is a fun question - I don't know of a huge data set but some ideas I had for generating one are:
Strava API
If you have a free API key, you can query Strava's segments end point and give it a lat/lng boundary, so you will get start and end points within those bounds (but limited to 10 results so you'd have to loop to get enough data and may exceed your API call limit, however if you just create it once it might not be so bad). You could also get a lot of points from a specific bike ride/run, but they'd all be close together which I don't think you want.
http://strava.github.io/api/v3/segments/#explore e.g.
Throttling
The default rate limit allows 600 requests every 15 minutes, with up
to 30,000 requests per day.
API call
curl -G https://www.strava.com/api/v3/segments/explore \
-H "Authorization: Bearer YOUR_TOKEN" \
-d bounds=37.821362,-122.505373,37.842038,-122.465977 \
Response
{
"segments": [
{
"id": 229781,
"name": "Hawk Hill",
"climb_category": 1,
"climb_category_desc": "4",
"avg_grade": 5.7,
"start_latlng": [
37.8331119,
-122.4834356
],
"end_latlng": [
37.8280722,
-122.4981393
],
"elev_difference": 152.8,
"distance": 2684.8,
"points": "}g|eFnm#n#Op#VJr#"
},
{
"id": 632535,
"name": "Hawk Hill Upper Conzelman to Summit",
"climb_category": 0,
"climb_category_desc": "NC",
"avg_grade": 8.10913,
"start_latlng": [
37.8334451,
-122.4941994
],
"end_latlng": [
37.8281297,
-122.4980005
],
"elev_difference": 67.29200000000003,
"distance": 829.834,
"points": "_j|eFvc#p#SbAu#h#Qn#?RTDH"
}
]
}
Google Geocoding API
This API will accept an address and give you a lat/lng (and more) but is throttled. You'd just need a list of addresses which would be pretty easy to get.
API https://developers.google.com/maps/documentation/geocoding/
Throttling
Users of the free API: 2,500 requests per 24 hour period. 5 requests
per second.
API call
https://maps.googleapis.com/maps/api/geocode/json?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA
Response
{
"results" : [
{
"address_components" : [
{
"long_name" : "1600",
"short_name" : "1600",
"types" : [ "street_number" ]
},
{
"long_name" : "Amphitheatre Pkwy",
"short_name" : "Amphitheatre Pkwy",
"types" : [ "route" ]
},
{
"long_name" : "Mountain View",
"short_name" : "Mountain View",
"types" : [ "locality", "political" ]
},
{
"long_name" : "Santa Clara",
"short_name" : "Santa Clara",
"types" : [ "administrative_area_level_2", "political" ]
},
{
"long_name" : "California",
"short_name" : "CA",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United States",
"short_name" : "US",
"types" : [ "country", "political" ]
},
{
"long_name" : "94043",
"short_name" : "94043",
"types" : [ "postal_code" ]
}
],
"formatted_address" : "1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA",
"geometry" : {
"location" : {
"lat" : 37.42291810,
"lng" : -122.08542120
},
"location_type" : "ROOFTOP",
"viewport" : {
"northeast" : {
"lat" : 37.42426708029149,
"lng" : -122.0840722197085
},
"southwest" : {
"lat" : 37.42156911970850,
"lng" : -122.0867701802915
}
}
},
"types" : [ "street_address" ]
}
],
"status" : "OK"
}

querying elasitcsearch parent child documents

We work with two types of documents on elastic search (ES): items and slots, where items are parents of slot documents.
We define the index with the following command:
curl -XPOST 'localhost:9200/items' -d #itemsdef.json
where itemsdef.json has the following definition
{
"mappings" : {
"item" : {
"properties" : {
"id" : {"type" : "long" },
"name" : {
"type" : "string",
"_analyzer" : "textIndexAnalyzer"
},
"location" : {"type" : "geo_point" },
}
}
},
"settings" : {
"analysis" : {
"analyzer" : {
"activityIndexAnalyzer" : {
"alias" : ["activityQueryAnalyzer"],
"type" : "custom",
"tokenizer" : "whitespace",
"filter" : ["trim", "lowercase", "asciifolding", "spanish_stop", "spanish_synonym"]
},
"textIndexAnalyzer" : {
"type" : "custom",
"tokenizer" : "whitespace",
"filter" : ["word_delimiter_impl", "trim", "lowercase", "asciifolding", "spanish_stop", "spanish_synonym"]
},
"textQueryAnalyzer" : {
"type" : "custom",
"tokenizer" : "whitespace",
"filter" : ["trim", "lowercase", "asciifolding", "spanish_stop"]
}
},
"filter" : {
"spanish_stop" : {
"type" : "stop",
"ignore_case" : true,
"enable_position_increments" : true,
"stopwords_path" : "analysis/spanish-stopwords.txt"
},
"spanish_synonym" : {
"type" : "synonym",
"synonyms_path" : "analysis/spanish-synonyms.txt"
},
"word_delimiter_impl" : {
"type" : "word_delimiter",
"generate_word_parts" : true,
"generate_number_parts" : true,
"catenate_words" : true,
"catenate_numbers" : true,
"split_on_case_change" : false
}
}
}
}
}
Then we add the child document definition using the following command:
curl -XPOST 'localhost:9200/items/slot/_mapping' -d #slotsdef.json
Where slotsdef.json has the following definition:
{
"slot" : {
"_parent" : {"type" : "item"},
"_routing" : {
"required" : true,
"path" : "parent_id"
},
"properties": {
"id" : { "type" : "long" },
"parent_id" : { "type" : "long" },
"activity" : {
"type" : "string",
"_analyzer" : "activityIndexAnalyzer"
},
"day" : { "type" : "integer" },
"start" : { "type" : "integer" },
"end" : { "type" : "integer" }
}
}
}
Finally we perform a bulk index with the following command:
curl -XPOST 'localhost:9200/items/_bulk' --data-binary #testbulk.json
Where testbulk.json holds the following data:
{"index":{"_type": "item", "_id":35}}
{"location":[40.4,-3.6],"id":35,"name":"A Name"}
{"index":{"_type":"slot","_id":126,"_parent":35}}
{"id":126,"start":1330,"day":1,"end":1730,"activity":"An Activity","parent_id":35}
I'm trying to make the following query: search for all items within a certain distance to a location that have children (slots) in the specified days and within certain start and end ranges.
An item with more slots fulfilling the condition should score higher.
I tried starting with existing samples but the docs are really scarce and its hard to move forward.
Clues?
I don't think there is a way to write an efficient query that would do something like this without moving location to slots. You can do something like this, but it can quite inefficient for some data:
{
"query": {
"top_children" : {
"type": "blog_tag",
"query" : {
"constant_score" : {
"query" : {
... your query for children goes here ...
}
}
},
"score" : "sum",
"factor" : 5,
"incremental_factor" : 2
}
},
"filter": {
"geo_distance" : {
"distance" : "200km",
"location" : {
"lat" : 40,
"lon" : -70
}
}
}
}
}
Basically, what this query is doing is this, it takes your range query or filter for children and whatever other conditions you need and wraps it into constant_score query to make sure that all children have score of 1.0. The top_children query collects all these children and accumulates their scores to the parents. And then filter filters out parents that are too far away.

Google Geocoder Lookup get postal code by country and city

i'm currently using Google Geocoder API to determine location data for my web service.
It's important for me to get the postal code for a given country and city, but it's look like that google is limiting the result.
Here's the API call:
http://maps.googleapis.com/maps/api/geocode/json?address=deutschland,+saarlouis&language=de&sensor=false
The result is like that:
{
"results" : [
{
"address_components" : [
{
"long_name" : "Saarlouis",
"short_name" : "Saarlouis",
"types" : [ "locality", "political" ]
},
{
"long_name" : "Landkreis Saarlouis",
"short_name" : "Landkreis Saarlouis",
"types" : [ "administrative_area_level_3", "political" ]
},
{
"long_name" : "Saarland",
"short_name" : "SL",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "Deutschland",
"short_name" : "DE",
"types" : [ "country", "political" ]
}
],
"formatted_address" : "Saarlouis, Deutschland",
"geometry" : {
"bounds" : {
"northeast" : {
"lat" : 49.36187570,
"lng" : 6.815549499999999
},
"southwest" : {
"lat" : 49.26046930,
"lng" : 6.67501510
}
},
"location" : {
"lat" : 49.31346060,
"lng" : 6.752286499999999
},
"location_type" : "APPROXIMATE",
"viewport" : {
"northeast" : {
"lat" : 49.36187570,
"lng" : 6.815549499999999
},
"southwest" : {
"lat" : 49.26046930,
"lng" : 6.67501510
}
}
},
"types" : [ "locality", "political" ]
}
],
"status" : "OK"
}
Thx for help.
Regards
Geocoding does not give postcode. Goenames has a database for Germany. Other countries are also available.
The reason you don't get the postal code is the query is too broad. If you would add a street name to the query the result will contain a postal code. A solution to your problem if you don't have a street name or don't want to use it is to split the geocoding into two parts:
Step 1: Use the city to get the GPS coordinates
https://maps.googleapis.com/maps/api/geocode/json?address=amsterdam
Step 2: Use the the GPS coordidates to get the postal code
https://maps.googleapis.com/maps/api/geocode/json?latlng=52.3182742,4.7288558
(Note: Double of same question posted here, answered by MidnightMotion. Adding to make easier to lookup.
https://gis.stackexchange.com/questions/33966/google-geocoder-lookup-get-postal-code-by-country-and-city)

Resources