Scrape web page with splashr and load more button - r

Question

I'm try to scrape a few 1801 census pages with splashr that may have 0 to many Load More buttons (since 50 records are loaded at a time). This page should have 174.
url <- "https://digitalarkivet.no/en/census/district/tf01058443000001"
doc <- splash("localhost") %>% render_html(url, wait =3)
html_nodes(doc2, xpath="//h4[not(#class)]/a") %>% length()
[1] 50
I tried following the url by Load More, but that just gets the first 50 records again.
url2 <- html_nodes(doc, xpath="//div[#class='load-more']") %>% html_attr("data-url")
[1] "https://digitalarkivet.no/en/census/related/rural-residences/tf01058443000001?page=2"
Note that most districts have fewer than 50 records, so I don't need to click load more for every page.

Answer 1

Thx for trying the splashr package (I'm the author).
Thankfully, you won't need it in this case. The data load is done through XHR requests which we can mimic in R:
library(httr)
library(rvest)
census_page <- function(district, page=1L) {
GET(
url = "https://digitalarkivet.no",
path=sprintf("en/census/related/rural-residences/%s", district),
accept_json(),
add_headers(
`User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.45 Safari/537.36",
Referer = "https://digitalarkivet.no/en/census/district/tf01058443000001",
`X-Requested-With` = "XMLHttpRequest"
),
query = list(page=page)
) -> res
stop_for_status(res)
res <- content(res)
list(
divs = read_html(res$view),
next_page = parse_url(res$nextPage)$query$page
)
}
Now, just pass-in the district and page of data you want:
res <- census_page("tf01058443000001", 1)
And get the results:
str(res, 1)
## List of 2
## $ divs :List of 2
## ..- attr(*, "class")= chr [1:2] "xml_document" "xml_node"
## $ next_page: chr "2"
The function returns a list with:
divs which is the parsed content containing the <div>s of the info you want
next_page can be used to pass to another call of the function
I didn't try it through to the end (i.e. I don't know if there will always be a 'next page') and you will need to extract the data from the <div>s on your own, but this will help you avoid a third-party dependency.