How to scraping/Extract data from ESRI Arcgis from website? - web-scraping

I was trying to extract data from an ESRI map embedded in a website. The objective would be by introducing geographic coordinates to be able to access the values ​​present on the map.
I leave here a print of the map and the respective address. I just cannot understand which method I should use since the map is embedded in the site. What processes should I use?
*for academic purposes
https://zonamentopf.portaldasfinancas.gov.pt/simulador/default.jsp

I must admit, I am not quite sure what you are after in terms of extracting data. I visited the site, and it is a pretty basic embedded web map created using the ArcGIS API for JavaScript, albeit wrapped in JSP. The web map is consuming some publicly accessible Esri services (World_Imagery, World_Boundaries_and_Places), and some non-public organizational services from www.portaldasfinancas.gov.pt.
It seems to me you are interested in geocoding. The pointer/marker on your map was looked up using Esri's ArcGIS Online Geocoding Service:
You have reached the home for the ArcGIS Online Geocoding Service. Use
it to turn addresses into coordinates, coordinates into addresses, or
to locate a point-of-interest.
If you are a developer, detailed documentation for using this service
in your application is available in the ArcGIS Online Geocoding
Service Help.
If you are an ArcGIS for Desktop user, you can use this service for
finding addresses interactively or geocoding a table of addresses.
Since I am not quite clear what you are trying to do, this is about all I can offer for now.

Note: Please respect data ownership. When in doubt, don't save a copy of someone else's data.
It sounds like you want to extract the polygon data in this map. The web app is making requests like this:
https://zonamentopf.portaldasfinancas.gov.pt/simulador/proxy.jsp?http://ags/arcgis/rest/services/SIMIMI/SIMIMI/MapServer/identify?f=json&tolerance=0&returnGeometry=false&imageDisplay=400%2C400%2C96&geometry=%7B%22x%22%3A-886651.3363331377%2C%22y%22%3A4443259.272690449%7D&geometryType=esriGeometryPoint&sr=102100&mapExtent=-887053.8250602124%2C4442583.282721534%2C-885178.7292100423%2C4443968.70385924&layers=all%3A0%2C1%2C2%2C3%2C4&callback=dojo.io.script.jsonp_dojoIoScript11._jsonpCallback
You can remove the callback parameter if you just want the data:
https://zonamentopf.portaldasfinancas.gov.pt/simulador/proxy.jsp?http://ags/arcgis/rest/services/SIMIMI/SIMIMI/MapServer/identify?f=json&tolerance=0&returnGeometry=false&imageDisplay=400%2C400%2C96&geometry=%7B%22x%22%3A-886651.3363331377%2C%22y%22%3A4443259.272690449%7D&geometryType=esriGeometryPoint&sr=102100&mapExtent=-887053.8250602124%2C4442583.282721534%2C-885178.7292100423%2C4443968.70385924&layers=all%3A0%2C1%2C2%2C3%2C4
However, if you just go to that link, you will get an error about a missing or incorrect referrer. To overcome that, you will need to add a header called Referer with value https://zonamentopf.portaldasfinancas.gov.pt/simulador/default.jsp. You can do that in a script or in an application like Postman, but you will not be able to do it in your own web app.
To view the available layers: https://zonamentopf.portaldasfinancas.gov.pt/simulador/proxy.jsp?http://ags/arcgis/rest/services/SIMIMI/SIMIMI/MapServer?f=json (with the Referer header I mentioned)
To view metadata about a single layer, use the following URL. Replace the 0 in /0/ with a different layer ID to get a different layer. https://zonamentopf.portaldasfinancas.gov.pt/simulador/proxy.jsp?http://ags/arcgis/rest/services/SIMIMI/SIMIMI/MapServer/0?f=json (with the Referer header I mentioned)
To query one of those layers to get the data, use the following URL to get everything, or add some query parameters if you want to filter the data. Replace the 0 in /0/ with a different layer ID to get a different layer. https://zonamentopf.portaldasfinancas.gov.pt/simulador/proxy.jsp?http://ags/arcgis/rest/services/SIMIMI/SIMIMI/MapServer/0/query?f=json&outFields=*&where=0%3D0 (with the Referer header I mentioned)
Related answer: https://stackoverflow.com/a/50213419/720773

Related

Mixing HERE content with other providers' map tiles

I am considering extending an application with HERE maps API calls (geocoding, routing, and maybe others). The application allows to select providers for different services and mix them. For example, using map tiles from one provider and showing search results from another one. I wonder, whether it is allowed to do so with HERE REST API call results. For example, show route polyline and use routing instructions from HERE and mix them with Mapbox GL tiles.
Question stems from https://legal.here.com/en-gb/terms/acceptable-use-policy, "Layering and Modifications" which seems to me prohibiting such use. In contrast, Is it possible to use here's Maps API combined with another base map (e.g., Bing, Google, OSM)? response states that it would be OK.
While I can make a separate profile that would enable HERE APIs only with HERE map tiles, I would prefer to allow users to mix providers as it is now. Currently, only HERE map tiles are supported, but those should allow me to layer other data on top, as far as I understand the allowed use terms.
In contrast to reply at Stack Overflow linked above, reply in Knowledge Base states that showing geocoding results on top of other providers tiles is prohibited.

How to use HERE Maps to develop custom routing algorithms?

I am trying to use HERE Maps to implement the eco-routing algorithm which we developed in our lab that requires real time traffic data as an input to the routing algorithm. So far, we were using INRIX dataset to verify our algorithms, but we decided to move to HERE Maps, and use the Traffic API to harness the power of real time traffic data.
Traffic API was very simple to use, and I managed to get the traffic data for a bounding box. It seems like that the traffic flow is passed for each TMC. My question is how to assign the traffic flow data to the links of the map using TMC. More specifically, I am wondering if there is a way to transform the TMC values to an adjacency matrix so that I can assign the traffics to each link and run my optimization algorithm over that graph? Also is it possible to overlay Open Street Maps (OSM) with the HERE Maps traffic data?
Thanks
You can associate the TMC data you get from the traffic data response to a particular link by using PDE layers LINK_TMC_FC1..5. You have look into our PDE documentation in developer site and tcs.ext.here.com/pde/maps?url_root=pde.api.here.com to see the list of layers provided.
Below is a sample PDE request for the Link TMC. Replace app id and code placeholders with actuals:
https://pde.api.here.com/1/tile.json?&layer=LINK_TMC_FC1&level=9&tilex=496&tiley=358&region=WEU&release=18131&url_root=pde.api.here.com&app_id={YOUR-APP-ID}&app_code={YOUR-APP-CODE}
For all your other questions on creating a matrix and OSM overlay, you have to write your own algorithm. There is no straight forward function provided by Here.

Google Tracks geofencing & visualisation API

I am developing a node.js application which (amongst other things) will recieve location information from remote users and allow them to interact with each other via the server.
I'm using the Google Tracks API because I like the idea of being able to track users when appropriate, set up geofencing to define my coverage areas and to visualise what's happening.
The Google tracks API documentation is reasonable, however I'm not sure how I would go about actually visualising the entities and geofencing I have setup on a map - this is not something that I can find covered elsewhere.
Ideally I would be able to simply embed a map into a webpage which could link with my Tracks API account and show all of the fencing and entities. Another nice feature would be the ability to 'draw' a geofence, is there anything out there which would allow this?
Thanks :-)
Tracks API does not currently offer any kind of server-side rendering for your Tracks data, so the best approach is to use the API to retrieve the crumbs (or just current location) and render them using polylines (or just markers) in the Google Maps API. You can similarly get all your geofences and render them using polygons.
Because this is done clientside, you'll probably want to limit your data to a reasonable number (depending on the browser/OS combination, something like O(thousands) of vertices).
All this assumes that your app meets the terms of service of the Maps API so check those out as well.

ASP.NET/Silverlight Location Points on Google/Bing Map

I realize that the question is pretty complicated and may require much research. Hope anybody can help me to get useful resources to achieve my goal.
I want to have a Google or Bing map on my ASP.NET 4 application (C#) to display all my logged in users as points on the map.
I understand that this involves five major problems
Get the location of the device (most likely standard laptop with IE9 browser) based on its unique IP address.
Integrate Google or Bing map with ASP.NET or Silverlight application.
Display the right portion of the map with the right Zooming depending on logged-in users locations.
Finally, mark the addresses as points on the map.
Note that the locations points should be dynamically reflected when any of the locations is changed.
The database is implemented using SQL SERVER 2005/2008R2
There are geo location services that can give you the latitude and longitude given an ip address.
As you mention you would be storing these in a database, getting all the current users would be simple database call.
Integrating a google map into a html page is very simple. You would only have to emit the necessary javascript from your page.
You should mark the points on the map first.
Google maps api has calls to fit the map to show all current points(fitBounds). I am guessing bing would have something similar.
To reflect the current points, you would have to fresh your locations from the database. I highly recommend an ajax call that returns json and using that to replot the points.

Using the Googlemaps Geocoder and Storing the Data

I am building an Events site in which the address of a given event will be stored to be used within a map on the page. Whilst viewing an Individual Events' page i would like to provide a list of similar events on side of the page. I would also like to rank this list by distance and display a small map with a marker on it for each item on the list.
I was thinking of Geocoding the Events address upon content input and then storing this data within the sites database. Doing this would allow me to calculate the distance of an event on the server and then provide an ordered list to display.
My question is this:
Would this violate the Googlemaps Terms of Service?
These terms:
"the geocoder for any purpose other than obtaining locations that will be displayed using the Google Maps APIs is a violation of the Terms of Service. You may use the HTTP geocoder to geocode addresses outside of your Google Maps API application so that they may be cached and later displayed using one of the Google Maps APIs, but locations obtained using the Geocoding Web Service may not be used by any other application, distributed by other means, or resold."
Suggest that storing the data is not a problem but i am unsure that the process of ordering the list by long lat would not result in a violation.
Any advice would be greatly appreciated.
As long as the map your refer to in your question is a Google Map and you are using the geocoded data to display features on a Google Map, you are fine. The client-side google.maps.Geocoderdev-guide is intended for dynamic requests that are based on real-time user interaction that drives a need to perform on-the-fly geocoding. But Google encourages you to consider using the Geocoding Web Servicedev-guide on the server-side, when you wish to pre-geocode static addresses or coordinates prior to using the response values on a Google Map. Storing those values is assumed.
It is common to perform some form of post-response/post-storage processing on those values, just like the ordering by distance that you describe, to fit whatever use case is needed to populate your map with the appropriate data. I understand you want to be careful to remain within the TOS, but what you describe sounds pretty standard.

Resources