How can I do a loop of url to scrape it? - r

Question

I want to screap a web site with 10 pages. I created a function to screap the web site, but I needed put all 10 urls, one after one, like this:
scraper <- function(link){
page = read_html(link)
titulo = page %>% html_nodes("h4 a") %>% html_text()
tipo = page %>% html_nodes("h4+ .row .col-md-4") %>% html_text()
data = page %>% html_nodes("p.col-md-6") %>% html_text()
protocolo = page %>% html_nodes(".row:nth-child(3) .col-md-4") %>% html_text()
situacao = page %>% html_nodes(".row~ .row+ .row p.col-md-4:nth-child(1)") %>% html_text()
regime = page %>% html_nodes("p.col-md-4:nth-child(2)") %>% html_text()
quorum = page %>% html_nodes(".col-md-4~ .col-md-4+ .col-md-4") %>% html_text()
autoria = page %>% html_nodes(".row:nth-child(5) .col-md-12") %>% html_text()
assunto = page %>% html_nodes(".row:nth-child(6) .col-md-12") %>% html_text()
result <- data.frame(titulo, tipo, data, protocolo, situacao, regime, quorum, autoria, assunto)
return(result)
}
link1 <- "https://santabarbara.siscam.com.br/Documentos/Pesquisa/74?Pesquisa=Simples&Pagina=1&Documento=117&Modulo=8&AnoInicial=2022"
result1 <- scraper(link1)
link2 <- "https://santabarbara.siscam.com.br/Documentos/Pesquisa/74?Pesquisa=Simples&Pagina=2&Documento=117&Modulo=8&AnoInicial=2022"
result2 <- scraper(link2)
How can I put all link jut once? Maybe doing a loop?

Answer 1

The urls for your pages differ only by the Pagina= part. Hence you could easily create loop over the pages and create the URLs dynamically for which I use a small custom function and purrr::map_df which will bind the results into one dataframe.
For the reprex I only loop over the first two pages:
library(rvest)
make_url <- function(page) {
paste0(
"https://santabarbara.siscam.com.br/Documentos/Pesquisa/74?Pesquisa=Simples&Pagina=",
page, "&Documento=117&Modulo=8&AnoInicial=2022"
)
}
result <- purrr::map_df(1:2, function(page) {
url <- make_url(page)
scraper(url)
}, .id = "page")
dim(result)
#> [1] 30 10
unique(result$page)
#> [1] "1" "2"