I'm looking for some solution to get coordinate from address or any WITHOUT using an API.
Usually, we use the Google Maps API, but at this time we can't use an API over the internet because of our budget.
Is there any solution??
purpose : scrape some web site, such as the one below
sample web site : https://www.yelp.com/biz/honey-badger-brooklyn-2?osq=Restaurants
return : [ lat, lon ]
lang : python
framework : Scrapy
I'm not sure what you're looking for here - you need some mapping of [street address]->[lat,long], which generally comes in the form of a large database, for example the one Google Maps has in its back end.
There is no way to calculate the coordinates of an address without such a database, so you either need to get a database of your own (likely by buying existing data from another company), or use an API call to a service that accesses such a database for you.
Related
I have a published layer in Google Maps Engine that I am attempting to display using the Google Maps V3 API. In GME, the layer's "Shared with" access list includes my user and the "API access" access list also includes by user.
I am making the Maps V3 API call using MapsEngineLayer from the visualization library, and setting the appropriate auth token and layer asset id as part of the layerOptions.
The API call is failing with the error message:
Google Maps API V3 error: 403 (Forbidden access for too many pageviews)
The URL looks like:
https://earthbuilder.googleapis.com/my_gme_layer_asset_id-4/maproot/json?
output=jsonp&access_token=my_auth_token&callback=xdc._tsel5i
I have found some discussion threads related to "403" and "forbidden", but am having difficulty figuring out the meaning of (and solution to) the "too many pageviews" issue.
Any suggestions would be appreciated.
If you tried with #David solution and it doesnt work. It worths to check that you are not caching (or storing locally) the google maps js script. Google don't allow that. If you serve that file it will work for about 3 days and then stop working.
How many requests have you made so far today, there are usage limits on the Maps API that may be preventing you from requesting any further data.
https://developers.google.com/maps/faq#usagelimits
Although it does seem unlikely that you have hit their hard set 25,000 requests, you may want to make sure that you aren't accidentally DOSing them with http requests. That sort of thing will invariably burn through your limit, and potentially place your IP on a blacklist.
Also, you should check the Maps API Reference materials, i think you may be trying to use a deprecated API.
https://developers.google.com/maps/documentation/webservices
Your http request should look more like this:
http://maps.googleapis.com/maps/api/service/output?parameters
Where output is either json or xml.
edit: The Maps API Help page is located here.
I saw this previous post but I have not been able to adapt the answer to get my code to work.
I am trying to filter on the term bruins and need to reference cacert.pem since for authentication on my Windows machine. Lastly, I have written a function to parse each response (my.function) and need to include this as well.
postForm("https://stream.twitter.com/1/statuses/sample.json",
userpwd="user:pass",
cainfo = "cacert.pem",
a = "bruins",
write=my.function)
I am looking to stay completely within R and unfortunately need to use Windows.
Simply, how can I include the search term(s) that I want such that the response is filtered?
Thanks in advance.
Alright, so I've looked at what you're doing, and some of what you're working on may be helped by examining the Twitter API methods, although it can be difficult to figure out how to translate some of the examples into R (via the RCurl Package).
What you're currently trying is very close to what you need to do, you simply need to change two things.
First of all, you're querying the url for the random sample of statuses. This url returns a random sample of roughly 1% of all tweets.
If you're interested in collecting only tweets about specific keywords, you want to use the filter API url: "https://stream.twitter.com/1/statuses/filter.json"
After changing that, you simply need to change your parameter from "a" to "postfields", and the parameter you'd be passing would look like: "track=bruins"
Finally, you should use the getURL function, to open a continuous stream, so all tweets with your keywords can be collected, rather than using the postForm command (which I believe is intended for HTML forms).
so your final function call should look like the following:
getURL("https://stream.twitter.com/1/statuses/filter.json",
userpwd="Username:Password",
cainfo = "cacert.pem",
write=my.function,
postfields="track=bruins")
For manipulating twitter, use the twitteR package.
library(twitteR)
searchTwitter("bruins")
You can include other parameters (like cainfo) in the call to searchTwitter, and they should get passed getForm underneath.
I don't think the Streaming API is currently included in twitteR - the search api is different (it's backward looking, whereas streaming is "current looking").
From my understanding, streaming is quite different to how lots of APIs work typically work; rather than pulling data from a web service and having a defined object returned, you're setting up a "pipe" for Twitter to push data to you and you then listen for that response.
You also need to worry about OAuth I think (which twitteR does deal with).
Is there any reason that you want to keep in R? I've used python successfully with the Streaming API and a package called tweepy to write data to a MySQL database and then use R to query and analyse the data.
Last time I checked, twitteR did not talk to the streaming API. Moreover, as far as I know, very few publicly-available Twitter Streaming API connection libraries in any language honor Twitter's recommendations on reconnecting when Streaming disconnects / throws an error.
My recommendation is to access Streaming via a library that's actively maintained, write the re-connection protocol yourself if you have to, and persist the data into a database that handles JSON natively. I'm about to embark on a project of this nature and will be writing the collector in Perl, doing my own re-connect logic and persisting into either PostgreSQL or MongoDB. Most likely it will be MongoDB; PostgreSQL doesn't get native JSON till 9.2.
Late to the game, I know, but you'll want to use the "streamR" package for access to Twitter's streaming API.
I was trying to use Google Distance Matrix API JSON in my project, after formatting the url, for example:
http://maps.googleapis.com/maps/api/distancematrix/json?origins=Vancouver+BC|Seattle&destinations=San+Francisco|Victoria+BC&mode=bicycling&language=en&sensor=false
I was able to view the result in browser, but when I tried to make a http request to get the result, I will receive status : REQUEST_DENIED
In documentation:
REQUEST_DENIED indicates that the service denied use of the Distance
Matrix service by your application.
But I still don't know why my request is denied. More Info from google:
Use of the Distance Matrix API must relate to the display of
information on a Google Map; for example, to determine
origin-destination pairs that fall within a specific driving time from
one another, before requesting and displaying those destinations on a
map. Use of the service in an application that doesn't display a
Google map is prohibited.
If I can't get the result, how can I apply it on google map?
Anyone with same problem? I will post sample code later, to clarify my question more clearly.
I had a similar problem, but it was due to assuming that the sensor tag would default to false, but it is in fact required.
Google Distance Matrix API has a limit of elements set to 2500.
I highly doubt it that you would have gone over this limit, as you would have to have calculated the maximum 10 x 10 matrix 25 times over.
I am from South Africa, and it refused to work for any local addresses until I changed the transport mode to driving (from walking, I mean seriously, we have cars). Maybe that'll help
https://maps.googleapis.com/maps/api/distancematrix/json?origins=Seattle&destinations=San+Francisco&key=YOUR_API_KEY
Please make sure that you have created a credentials on Google Console which can be called as "API KEY" and you are using this API KEY in above request URL on the place of "YOUR_API_KEY"
I had the same problem earlier.
you need to check out with the Google Api key version and codes/scripts of version you are using.
I believe you need to get a Google Maps API key
You need to set an API KEY for using Google map services like this:
https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=Washington,DC&destinations=New+York+City,NY&key=**YOUR_API_KEY**
First thing is that you need is to have an API key assigned to your application. This is done by creating it in the Google API Manager.
I hope your have enabled distance matrix API from google console for your application.
please refer
https://support.google.com/googleapi/answer/6158841?hl=en
along with this you can restrict API_KEY for unauthorized use.
I need a webservice to get information of whole of the countries and their provinces location and area or population is anyone who knows this web service ?
why you need web service? I don't think so its exist.
what you need to do is, you can create table in your DB and add data, Its easy to find SQL insert queries to dump the data. please check these URLs
http://amrelgarhytech.blogspot.com/2008/08/list-of-countries-cities-languages.html
Country, state list for SQL Server
You can also use the great service from the chaps at http://www.geonames.org/
I did a small tool to generate an XML file based on the output of their webservices: https://github.com/yosoyadri/GeoNames-XML-Builder
All credit should really go to the C# library at http://geonamesdotorgdotnet.codeplex.com/ that I'm using to parse the response XML from the services.
You have an already made list of countries and provinces at https://raw.github.com/yosoyadri/GeoNames-XML-Builder/master/continents-countries-statesprovinces.xml
As Muhammad I think it is always good practice to keep your local copy of the data rather than querying the service everytime.
I am working on a project which requires a server side access to google map api. i want to calculate distance (actual distance, not straight line). google map api supports javascript and not asp.net. please give suggestions ...!
you specified google maps in your question - but have you looked at Virtual Earth? Specifically this routing with Virtual Earth Web Service example sounds exactly like what you want:
server-side access (just Add Service Reference inside visual studio)
actual distance (not straight line) since it is using a route
The concerns raised by others about T&Cs for 'internal/intranet use' apply to VE as well as Google I think - you'll have to read up about whether your application needs licensing or not.
p.s. if you did just want to calculate straight-line distance, I have instructions using SQL Server 2008; which also links to some straight c# code that does it too.
The Google API allows you to Geocode via a server side call:
http://code.google.com/apis/maps/documentation/services.html#Geocoding_Direct
This would allow you to get the longitude and latitude of the locations. You can then cache these and use them to calculate distance using the techniques CMS suggests.
You will need to be careful of the Google T&C's though as you are only allowed to store the geocoding data for use on a Google map which is publicly available.
You would probably also run into limitations on the number of requests you could make from a single IP.
However I think what you mean by non-straight line distance is distance taking into account roads and one way streets etc.
If this is the case I think a commercial service is your only option. Although theoretically you could do it all via screen scraping, I'm almost certain that this would break Google's T&C's.
The simplest solution would probably be just to embed a Google map on a page of your application and let the user calculate the distance. You could pre-fill the to and from fields if required.
Again if this is for an internal app i.e. Not publicly available "my understanding" of the Google T&C's would forbid this.
Use something like firebug or fiddler to look at the requests that are being sent to Google from javascript you should then beable to build the request using that information and an HTTPWebRequest in .net and retrieve the same information.
HTH
You can calculate the distance of two geographical coordinates (latitude, longitude) using the Great-circle distance algorithm.
Here you can find some other formulas for distance calculation.
Well, you've pretty much identified the key issue, the Gmaps API is a browser resident javascript API and there's not much getting around that. Most of the API is executed in the browser so there's not much network traffic to spy on.
As tsaunders mentions there is a geocoding API call that is restfully accessible, but it only does reverse/geocoding and if you have lat/lng's already you can use the calculations the rms suggested, but they are as tsaunders points out 'as the crow flies' distance.
If indeed you are looking for road taken distance, the API does do routing but you are back in the browser to get the start/end points from the user.
Perhaps you can be a little more specific about what you are trying to do and why you feel this requires you to to access the API from your server. My application for instance has features that gather information from the user and sends requests back to my server to work on, some of that data are processed by the Gmaps API first.
If I were to use a API platform, I certinaly would not use Google as the free one does not include advances Geocoding menaing the accuracy is poor. There is also no sla , support or rights of service.
The directions are poor, the coverage for Ireland and Geocoding is almost childlike and the privacy stinks. No professional business would use a google mapping solution.
They copy everyone else's idea, say they are there own and get loads of press (they only added tube stations in 2006) an dcyclc lanes (2010), viamichelin added these 2006 and Traffic in 2009 !
Any agency or developers looking for an API should stick to Bing or ViaMichelin for better customisation and user experience which is killer !