I am trying to get the pop-up data of all the points present in this interactive map made with leaflet. As far as I could observe, the pop-up data appears only when the point marker is clicked. The points are located in the element <div class="leaflet-pane leaflet-marker-pane"> and pop-ups in <div class="leaflet-pane leaflet-popup-pane"></div>
Is there any way to use some web scraping tools to click on all the points and download the pop-ups?
Webmap: http://gis.sutran.gob.pe/alerta_sutran/
Open the Firefox Network Monitor and observe that a HTTP POST request is made to http://gis.sutran.gob.pe/alerta_sutran/script_cgm/carga_xlsx.php. To download, make use of curl:
curl 'http://gis.sutran.gob.pe/alerta_sutran/script_cgm/carga_xlsx.php' -X POST --data-raw 'tipo=MAPA'
Related
I would like to get the location (coordinates) in latitude and longitude from the clickable map on this web page and other similar pages. I am not sure how to go about this using rvest or a scraping package since the Google Maps link with coordinates as shown below only appears if I click on the Ver Mapa button in blue.
https://www.google.com/maps/embed/v1/search?q=-26.106150272877,-56.89951582606415&key=XXXXXXXXXXXXXX&zoom=14
under <div class="modal-body "id=map-canvas">
You are right, that the link is executed only when clicking the button. You also cannot access the link through rvest, because it is stored inside of a script. However, the script is loaded together with the HTML. Which means, it is accessible through the source-text of the website.
The website somehow had to know, where the coordinates are, to succesfully query google maps. Since you are only interested in the coordinates (I suppose), here is a code snippet that should get you started.
It downloads the html as text and through regex filters the relevant info from the stored google request.
You can play around with the regex here: https://regex101.com/r/XIagYl/1
What I did was just navigate to the source text of the website and ctrl-f "google.com/maps" and there the link was sitting in the script. :)
text <- httr::GET("https://clasipar.paraguay.com/inmuebles/propiedades-rurales/granja-sobre-asfalto-con-infraestructura-completa-2093966") %>% httr::content(as ="text")
stringr::str_extract_all(text,"(?<=q=).*(?=\\&key)")
Anybody know how to create a proper URL to link to here.com maps, from my site? We use an extensive amount of API calls to embed the maps inside our site, but I want to send emails to people with a 'Click Here for directions' link inside pointing to here.com driving directions with the address or coordinates built into the URL.
Example:
With Google Maps, all we would have to do is make a simple link like this:
https://www.google.com/maps?saddr=32.5589671,-97.2586011&daddr=32.7787855,-97.4495476
Just pass along the variables saddr and daddr within the URL. I've tried everything to build a proper URL for here.com maps and it doesn't process it properly.
My best attempt was:
https://www.here.com/directions/drive/32.2589671,-97.4486011/Weatherford,_Texas?x=ep&map=30.38114,-89.89326,6,normal
But it does not process this URL properly and now I'm pretty much stuck, since I have tried everything that I know to try.
Something like https://www.here.com/directions/drive/start:lat1,lng1/end:lat2,lng2 seems to work.
The text before the colon provides the text of the start/end location
There should be absolutely no spaces between the latitude and longitude
If no map parameter is added, the map view defaults to show the whole route on a street map
If a map parameter of the form ?map=lat,lng,zoom,maptype is added, the map initially shows the map view specified, then moves to show the whole route - like this satellite map link
I've checked that the first link (without the map parameter) also works on a Windows Phone to forward to the HERE Maps app if it is installed. I've not tested what happens if the Android or iOS HERE Maps apps are installed on mobile. If the HERE Maps app is not present on a mobile, the site should forward to the m.here.com mobile friendly site.
Other endpoints will vary by device - the following are fine on tablets:
Parisian Restaurants https://www.here.com/discover/restaurant?map=48.85,2.35,15,normal
New York Traffic https://www.here.com/traffic/explore?map=40.7127,-74,15,normal
In our site there is a portfolio page where thumbnail images are listed, when any one click on any image it will open in light box and on clicking the cross image the light box get close.
I have recorded this scenario through HTTP proxy , and found that under recording controller only upto portfolio page sampler is recorded but clicking on the thumbnail images events which i did is not recorded.
i then look into the view result tree listener and found that click on thumbnail images is listed there.
so what i want to know that how i can record opening images in light box in jmeter
If Light box opening is only client side (javascript) then what you're seeing is completely fine.
Jmeter is not a browser, it only records client to server traffic.
You should use "HTTP Request" sampler with direct url to images. If Jmeter doesn't record clicks anyway opening photos will load server so it's not reason to avoid them.
I'm currently creating an interactive web map using Google Maps Javascript API version 3. I'm not sure if this is possible but I would like to create a series of map layers consisting of different kml files that could be toggled from the side bar. I would also like to be able to access the individual points within the kmls so that when the user clicks on a particular point, photos, text etc. can be loaded outside the map window...
Any ideas on how to accomplish this? I've been searching for a while and only found ways of configuring the infowindow which I personally find unattractive...
There is a way to load KML into a google maps application natively, there is actually a pretty good example of how this is done here:
http://gmaps-samples-v3.googlecode.com/svn/trunk/layers/layers.html
One trick I will mention is that since the KML is downloaded to Google's servers in order to be rendered on the map, it must exist on a public facing web server somewhere, Google won't be able to access it from your local machine.
I found this tutorial which seems like it'll do the trick!
To create dynamic charts and graphs in ASP.Net, I have placed the (GDI+) code for each chart in separate page files – so the output from running a chart page by itself if to just display the chart onscreen.
The chart page contains the code to get the relevant data from SQL Server, as well as the chart generation code. The generated chart object is then saved to Response.OutputStream.
A content page where I want to display the chart provides various parameters to determine what data will be used to generate the chart (such as userID, project ID, selection from dropdown lists etc.).
To display the chart as part of a content page, I then place an image object in the page, collect the various chart parameters, and in code behind set the URL of the image object to be the chart page, plus all the chart parameters added to the query string.
This works fine, except for one thing: when I right click on the chart image as it appears in the content page in the browser, I get an option to “open image”, which displays a page with the chart all by itself (perhaps not surprising as it is generated with a chart page). However, the problem is that in this chart page, the full URL with the querystring is visible, which means that a user could change the values in the query string, and thereby generate a new chart. This is a big problem, as it might enable a user to generate a chart for data they should not have access to.
Is there a way to avoid this problem whilst still using this “chart page” approach to generating and displaying dynamic charts? Alternatively, should I rather save the dynamically generated images as files, either on disk or as SQL Server Filestream objects for example, and then reference these in the page?
By the way, I am aware of the various ASP.Net chart controls that are available. However, the charts I need to generate are highly customised to a very specialised application, so those controls will unfortunately not work in this instance.
I have the same concern for one of my previous projects. One cheap but not 100% fool-proof solution is to insert a http-referer check before you process the image. Images loaded from within you page will have the referrer URL of your domain, those entered directly via the browser won't have.
Of course it's very easy to hack by faking referrer url, still it is some form of deterrent. Will be glad if others can offer their solutions.
Another way is to add some sort of checksum params into your querystring. Only you know how to 'decrypt' and only generate image when the checksum is 'correct'.
First of all, I assume that your users are authenticated (logged in) to your website, so you know who they are and what data/charts they are supposed to have access to?
You need to check the incoming parameters in your image-generating code to see if the current user has access to the data he asks for. Typically you would query your user/privilege tables for this (or call the Membership API or whatever).
Trying to hide the URL (for example by doing a POST instead of a GET to the image-generating page) is just "security by obscurity", which is not a good idea.
By the way, your problem is called "URL tampering", a Google search for this should get you in the right direction.