Follow links in a loop with rvest - r

Question

I'm trying to learn the rvest package, but the documentation and the examples on the web are either very basic or very complex. I could not find how to use the follow_link function in a loop to browse some number of pages. Perhaps I did not understand its logic at all...
Here is a simplified example of my attempt:
library(rvest)
url <-
"https://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500"
s <- html_session(url)
liste <- list()
for (i in 1:2) {
data <-
s %>%
read_html() %>%
html_nodes("#mw-whatlinkshere-list li")
result <- c(liste, data)
s <- s %>%
follow_link(xpath = "//a[text()='next 500']/#href")
}
I've also tried to avoid the jump_link, like this : it's better, but I'm not sure is the best and fastest solution :
liste <- c()
while (!is.na(url)) {
data <-
url %>%
read_html() %>%
html_nodes("#mw-whatlinkshere-list li")
liste <- c(liste, data)
url <- url %>%
read_html() %>%
html_node(xpath = "//a[text()='next 500']") %>%
html_attr("href") %>%
paste0("https://www.wikidata.org", .)
print(url)
}
Any advice is welcome and would be appreciated.

Answer 1

Try this:
library(rvest)
url <- "https://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500"
s <- html_session(url)
liste <- list()
for (i in 1:2) {
data <-
s %>%
read_html() %>%
html_nodes("#mw-whatlinkshere-list li")
# There was a mistake here. You were overwriting your results
liste <- c(liste, data)
# Here you have to pass a 'a' tag, not a 'href' value. Besides,
# there is two 'next 500' tags. They are the same, but you have
# to pick one.
s <- s %>%
follow_link(xpath = "//a[text()='next 500'][1]")
}