Rvest html_nodes span once and Xpath - r

Question

I would like to collect parish.name from masstimes.org
I used selector gadget. The CSS selector 'span' as an XPath is shown below. Please report any bugs that you find with this converter. The result is //span
The data I would like to collect is here: <span once="parish.name" class="">Saint John Chrysostom [Ruthenian]</span>
I'm not sure what the html_nodes command should look like?
Thanks,

Answer 1

Javascript is required to display the results. So you won't scrape anything with Rvest. You should use RSelenium. As an alternative you can download the JSON loaded in the background to fetch the data.
First, you need to obtain the lat and long of the city you're looking for. The website uses Arcgis API to get them. For example, for New York the GET url is :
https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/findAddressCandidates?f=json&SingleLine=New%20York,%20NY,%20USA
Output :
{"spatialReference":{"wkid":4326,"latestWkid":4326},"candidates":[{"address":"New York","location":{"x":-74.007139999999936,"y":40.714550000000031},"score":100,"attributes":{},"extent":{"xmin":-74.257139999999936,"ymin":40.464550000000031,"xmax":-73.757139999999936,"ymax":40.964550000000031}}]}
From this output, lat is 40.715 (rounded with 3 digits) and long is -74.007. Use GET and content (as text) from httr package to load the file in R. And str_extract from stringr to extract these 2 values.
Example. I'm looking for Paris in France. We modify the URL (add Paris and FRA into it) to get the JSON, then store its content. We extract lat and long, then construct the URL with paste0.
data=GET("https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/findAddressCandidates?f=json&SingleLine=Paris,%20FRA")
parse=content(data,as="text")
lat = round(as.numeric(str_extract_all(parse,"\\d+\\.\\d+")[[1]][2]),digits = 3)
long = round(as.numeric(str_extract_all(parse,"\\d+\\.\\d+")[[1]][1]),digits = 3)
paste0("https://apiv4.updateparishdata.org/Churchs/?lat=",lat,"&long=",long,"&pg=1")
Output :
https://apiv4.updateparishdata.org/Churchs/?lat=48.857&long=2.341&pg=1
You can also manually lookup these values with your favorite search engine.
Once you get them, you can construct your request url. Like the following one :
https://apiv4.updateparishdata.org/Churchs/?lat=40.091&long=-82.95&pg=1
where lat and long is the value you've found and pg the page number (30 results per page).
Load the JSON in R with jsonlite(a df is created) and extract the column of interest :
library (jsonlite)
mydata <- fromJSON("https://apiv4.updateparishdata.org/Churchs/?lat=40.091&long=-82.95&pg=1")
mydata$name
Output :
[1] "Saint John Chrysostom [Ruthenian]" "Saint Elizabeth"
[3] "Saint Anthony" "St. Matthias"
[5] "Saint Paul" "Holy Resurrection [Melkite]"
[7] "St. Michael" "St. James the Less"
[9] "Our Lady of Peace" "Immaculate Conception"
[11] "Saints Augustine and Gabriel" "Saint Peter"
[13] "Holy Name" "Saint Timothy"
[15] "Ohio Dominican University" "St. Thomas More Newman Center"
[17] "Church of the Resurrection" "Saint Andrew Roman Catholic Church "
[19] "Saint Matthew" "Saint Joan of Arc Catholic Church"
[21] "St. Thomas the Apostle" "St. Agatha"
[23] "Sacred Heart Church" "Saint Dominic"
[25] "Saint John the Baptist " "Saint Francis of Assisi"
[27] "Saint Patrick" "Holy Spirit"
[29] "Saint Brendan" "Saint Christopher"