Extract table from web page by page - r

Question

I have written a code for web scraping table from webpage. This code extracts table from page one (in url /page=0):
url <- "https://ss0.corp.com/auth/page=0"
login <- "john.johnson" (fake)
password <- "67HJL54GR" (fake)
res <- GET(url, authenticate(login, password))
content <- content(res, "text")
table <- fromJSON(content) %>%
as.data.farme()
I want to write a code to extract rows from table page by page and then to bind them. I do that, cause table is too large and i can't extract everything at once (it will brake the system). I don't know what how many pages there can be, it changes, so it must stop once the last page is collected. How could i do that?

Answer 1

I cannot test to guarantee this will work because the question is not reproducible, but you mainly need three steps:
Setup the url and credentials
url <- "http://someurl/auth/page="
login <- ""
password <- ""
Iterate over all (I'm assuming there are N) pages and store the result in a list. Note that we modify the url properly for each page.
tables <- lapply(1:N, function(page) {
# Create the proper url and make the request
this_url <- paste0(url, page)
res <- GET(this_url, authenticate(login, password))
# Extract the content just like you would in a single page
content <- content(res, "text")
table <- fromJSON(content) %>%
as.data.frame()
return(table)}
)
Aggregate all the tables in the list in a single complete table using rbind
complete <- do.call(rbind, tables)
I hope this helps at least giving a direction.