Rvest: Building a queue for several html-links - r

Question

I'm currently webscraping a news magazine, but unfortunately, I don't have a clue on how to establish a working queue. I can only scrape the content of all articles on one page, but I want a queue, that automatically does the same thing for the rest of articles.
library(rvest)
library(tidyverse)
library(data.table)
library(plyr)
library(writexl)
map_dfc(.x = c("em.entrylist__title", "time.entrylist__time"),
.f = function(x) {read_html("https://www.sueddeutsche.de/news/page/1?search=Corona&sort=date&all%5B%5D=dep&all%5B%5D=typ&all%5B%5D=sys&time=2020-07-19T00%3A00%2F2020-07-27T23%3A59&startDate=27.07.2020&endDate=01.08.2020") %>%
html_nodes(x) %>%
html_text()}) %>%
bind_cols(url = read_html("https://www.sueddeutsche.de/news/page/1?search=Corona&sort=date&all%5B%5D=dep&all%5B%5D=typ&all%5B%5D=sys&time=2020-07-19T00%3A00%2F2020-07-27T23%3A59&startDate=27.07.2020&endDate=01.08.2020") %>%
html_nodes("a.entrylist__link") %>%
html_attr("href")) %>%
setNames(nm = c("title", "time", "url")) -> temp
map_df(.x = temp$url[1:50],
.f = function(x){tibble(url = x,
text = read_html(x) %>%
html_nodes("#article-app-container > article > div.css-isuemq.e1lg1pmy0 > p:nth-child(n)") %>%
html_text() %>%
list
)}) %>%
unnest(text) -> foo
foo
X2 <- ddply(foo, .(url), summarize,
Xc=paste(text,collapse=","))
final <- merge(temp, X2, by="url")
In this case, I got 30 pages filled with articles, but my scripts only supports the scraping of one page.
The only thing that changes in between the pages is the page number (https://www.sueddeutsche.de/news/**page/1**?search=...)
If you could give me a hint on how to include all pages into the queue at once, I would be more than grateful. Thanks a lot :)

Answer 1

How would a queue in dataframe form work for you?
The following suggestion is held a little more generic, so it'll work beyond the specific use-case. You would be able to add more URLs to scrape as you go, but only new ones will be kept, due to dplyr::distinct.
(I've initiated the queue to hold the first 5 pages you want to scrape, you can add more right away or dynamically if you find links on the DOM...)
library(dplyr)
library(lubridate)
queue <- tibble(
url = paste0("https://www.sueddeutsche.de/news/page/", 1:5, "?search=Corona&sort=date&all%5B%5D=dep&all%5B%5D=typ&all%5B%5D=sys&time=2020-07-19T00%3A00%2F2020-07-27T23%3A59&startDate=27.07.2020&endDate=01.08.2020"),
scraped_time = lubridate::NA_POSIXct_
)
results <- list()
while(length(open_rows <- which(is.na(queue$scraped_time))) > 0) {
i <- open_rows[1]
url <- queue$url[i]
[...]
results[[url]] <- <YOUR SCRAPING RESULT>
queue$scraped_time[i] <- lubridate::now()
if (<MORE PAGES TO QUEUE>) {
queue <- queue %>%
tibble::add_row(url = c('www.spiegel.de', 'www.faz.de')) %>%
arrange(desc(scraped_time)) %>%
distinct(url, .keep_all = T)
}
}