The below linked map contains a number of layers which I would like to be able to extract as polygons [if possible]. I've not previously done any web-scraping and realise that doing so in regards to the geographic data on this system represents a significant challenge.
Ideally I would only want to extract the data relating to the 'Shopping Local Centre' category.
Happy to try to use Phython or R to achieve such, just wondered if anyone had any ideas.....
Web scraping (using BeautifulSoup, for example) would get you the HTML objects from a webpage. You would need basic knowledge of Python for this.
Or you could avoid that by going this route:
With QGIS and Geofabrik you can gather retail location polygons and their attributes of a given area.
Use Geofabrik to download your area of interest in *.shp (shapefile) format. It looks like you're in Greater Manchester, so I navigated to the download page here (it's a 50MB file for the greater-manchester-latest-free.shp.zip).
Once you download that, open it in QGIS and you'll see in the attributes it has retail locations.
That site is using WMS to display the map (I work for the company that makes iShare) so there is no vector content for you to scrape as it works entirely with images.
The easiest way to get the data would be to ask the council to provide it, you might need to make it a freedom of information request but they should be happy to provide the data in a usable GIS format.
Related
I have been creating a twitter bot to post satellite images of random coordinates and I have found that the vast majority are of the ocean, so I would like to somehow make the bot detect that the entire photo is blue and repeat the process. The bot code is available here.
Thanks in advance.
As an alternative to employing machine vision to assess the content of the image to determine whether to sample again, you could simply sample from coordinates that are not the ocean! There are many files produced by geographers that have sought to map the boundaries of the world's continents.
For example, see here for code to do so. You'll need to understand projections and other features of Geographic Information Systems if you want to get extremely accurate, but you can get reasonably accurate using the polygons for oceans. If you REALLY wanted to get accurate, you could obtain a set of shapefiles that partition all possible coordinates (rather than a single "world" shapefile) and use the more-detailed mapping that may accompany those more-zoomed-in shapefiles.
Alternatively, you can implement the model here, provided in a guide to water detection in satellite imagery using Python.
I tried to use Web Scraper, but it only works for a few data entries not for hundreds of data points. Is there a way to scrape a large amount of data solely using Web Scraper or is there a better alternative like python? I intend to scrape information of the location name, address, rating number, and website. Thanks for any inputs!
Sitemap:
{"_id":"mymaps","startUrl":["https://www.google.com/maps/d/u/0/edit?....."],"selectors":[{"id":"activityelement","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.i4ewOd-TaUzNb-haAclf","multiple":true,"delay":"1000","clickElementSelector":"div.un1lmc-pbTTYe-ibnC6b","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"activityname","type":"SelectorText","parentSelectors":["activityelement"],"selector":"div.i4ewOd-TaUzNb-r4nke","multiple":false,"regex":"","delay":0},{"id":"activityrating","type":"SelectorText","parentSelectors":["activityelement"],"selector":"span.fO2voc-jRmmHf-LJTIlf-wcwwM-H6j5tf","multiple":false,"regex":"","delay":0},{"id":"activityaddress","type":"SelectorText","parentSelectors":["activityelement"],"selector":".OzwZjf-jRmmHf-MZArnb-KDwhZb div:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"activitywebsite","type":"SelectorLink","parentSelectors":["activityelement"],"selector":"div:nth-of-type(4) a","multiple":false,"delay":0}]}
Have you tried Octoparse?
for hundreds of data points
It does allow 10'000s of input URLs...
Try downloading the KML file from the three vertical dots at the top of the My Map, and use the 'Export as KML instead of KMZ' checkbox option...then you'll get all the geocoordinates in a human readable XML file.
I have a set of raster layers that cover the same place, over different years. I want to display it on my website using the timeslider so that I can clearly view the changes between rasters over different years. However, the example given by ESRI site is based on featurelayer, and I have a raster layer. How can I make the raster layer time-aware?
If you currently have multiple raster layers, you will need to convert them to a single layer that also happens to be time-aware.
Esri Support wrote a blog post about this -- back in 2011, but the principles should still work -- and a GIS StackExchange question covers publication of time-aware datasets as an image service. To summarize their workflow:
Create a new mosaic dataset or raster catalog. Add the individual rasters in.
Add a new field, of type date (e.g. year).
Populate year with values.
Enable time on the layer, and publish it as an image service.
The Esri JavaScript API example will help with implementation, once you've got everything configured and published.
I want to get all tourist attractions around a zip code in the US. How do I do that from OpenStreetMap api?
I have found this http://wiki.openstreetmap.org/wiki/Key:tourism
But I'm not sure how to use it.
http://nominatim.openstreetmap.org/search?q=135+pilkington+avenue,+birmingham&format=json&polygon=1&addressdetails=1&tourism=attraction
You can use overpass turbo e.g. with the query wizard to show for the current bounding box the available objects.
To do this for a zip code instead of a bounding box this is much more complicated even in a properly mapped region like Germany, because to my knowledge there are no relations (or very rarely) that map a zip code to a boundary polygon. You could obtain several addresses with zip codes and then create a kind of a boundary for it but that won't be very precise. Regarding US you also have probably less features like zip code mapped due to the Tiger import (less mapping activity), not sure.
I have created a google map displaying markers and the weather layer using Google Maps API V3.
The weather layer is great for displaying weather information at well-known locations; however, for my map I only care about the wind speed rather than the forecast (ex. sunny, rainy, etc). Is there any way to modify the weather layer so that it displays "wind: s 5m/s" directly on the map instead of the sun/rain icon?
I realize that you can click on the icon to get the wind information but given that we do not care about the actual weather, we do not want people to have to click to get that. Instead we want the wind displayed directly.
(If it is not possible, then any alternative suggestions would be greatly appreciated as well :) )
I made a Weather API available on mashape, and they have a ready to use simple PHP SDK.
This api is really simple to use because we use the cool standards that are available nowadays, like JSON and REST.
If you like it please give it a try on mashape
I've tried but haven't found a way to change how the weather icons look when they are loaded.
It is possible (as you may know) to create a custom overlay using discrete weather data items (like wind speed) when a weather icon is clicked but that wouldn't seem to solve your problem.
See" "google.maps.weather.WeatherConditions object specification" in the V3 API reference
Like many of these kinds of things, I imagine it is theoretically possible to scan the DOM, find the weather 'markers', extract the info you want and redraw them as you wish them to look, but you may (unsure about this) find that the weather icons are actually images and data like wind speed doesn't actually exist in the DOM until the weather icon is clicked: The discrete data may be the product of an ajax load in response to the click event. Even if you got a DOM crawler to work it could be broken by a new release of the API and you'd never know what hit you.
Best bet may be to submit a change request to the API developers asking for the ability to format the weather icons and specify their content.
or
Use some other online source of json or xml weather data and create your own weather markers, labels, or whatever on your map.
Sorry I can't provide a more constructive answer :-(