Create aggregate df out of webscraping multiple pages using Rvest and Glue - r

Question

I'm working on scraping data from a table on the following website
https://fantasy.nfl.com/research/scoringleaders?position=1&statCategory=stats&statSeason=2019&statType=weekStats&statWeek=1
I want to create a scrape that takes all 17 weeks, all four positions (qb,rb,wr,te) and takes the first 4 pages to get the first 100 rows (only 25 shown on a page at a time).
library(tidyverse)
library(rvest)
library(glue)
scrape_19 <- function(week, position, page) {
Sys.sleep(3)
cat(".")
url <- glue("https://fantasy.nfl.com/research/scoringleaders?{page}position={position}&sort=pts&statCategory=stats&statSeason=2019&statType=weekStats&statWeek={week}")
read_html(url) %>%
html_nodes("table") %>%
html_table(header = T) %>%
simplify() %>%
first() %>%
setNames(paste0(colnames(.), as.character(.[1,]))) %>%
slice(-1) %>%
list()
}
Here are all the iterations of each call in glue:
week = 1:17;
position = 1:4;
page = c("", "offset=26&", "offset=51&", "offset=76&")
The problem I run into is when I try to make one df with all the data for each week, position and page. Here is code that works for week and position but will not work for an additional nested df.
scaffold <- tibble(week = weeks,
position = list(positions)) %>% tidyr::unnest()
scaffold
tbl_data <- scaffold %>%
mutate(data = purrr::map2(week, position, ~scrape_19(.x, .y)[[1]]))
Basically, I need help in crafting the scaffold and turning that scaffold into the final total data set with all weeks, positions and pages.

Answer 1

Here is my attempt. I am not sure if glue() is the way to go. See below.
first_name <- c("Fred", "Ana", "Bob")
last_name <- c("JOhnson", "Trump")
glue('My name is {first_name} {last_name}.')
Error: Variables must be length 1 or 3
Your case is similar to this example. So I tried to create all possible links using loops with map(). Then, I checked if all URLs exist or not. I used map_dfr() in order to loop through all URLs and bind all data frames. In this process, I added week and position information as well. If position is 1, it is QB. If necessary, replace these numbers by yourself. Note that I scraped four URLs in this demonstration.
library(httr)
library(rvest)
library(tidyverse)
# Create all URLs.
# Create 4 base URLs
paste("https://fantasy.nfl.com/research/scoringleaders?",
c("", "offset=26&", "offset=51&", "offset=76&"),
"position={position}&sort=pts&statCategory=stats&statSeason=2019&statType=weekStats&statWeek={week}",
sep = "") -> mytemp
# For each base URL, create 4 URLs. (4 x 4 = 16 URLs)
map(.x = 1:4,
.f = function(x){gsub(x = mytemp, pattern = "\\{position\\}", replacement = x)}) %>%
unlist -> mytemp
# For each of the 16 URLs, create 17 URLs
map(.x = 1:17,
.f = function(x){gsub(x = mytemp, pattern = "\\{week\\}", replacement = x)}) %>%
unlist -> myurls
# Check if any URLs are invalid
sapply(myurls, url_success) %>% table
# TRUE
# 272
# Scrape the tables
map_dfr(.x = myurls[1:4],
.f = function(x){read_html(x) %>%
html_nodes("table") %>%
html_table() %>%
simplify() %>%
first() %>%
setNames(paste0(colnames(.), as.character(.[1,]))) %>%
slice(-1) %>%
mutate(position = str_extract(string = x, pattern = "(?<=position=)\\d+(?=&)"),
week = str_extract(string = x, pattern = "(?<=statWeek=)\\d+"))},
.id = "url") -> foo
url Rank Player Opp PassingYds PassingTD PassingInt RushingYds RushingTD ReceivingRec ReceivingYds
1 1 1 Lamar Jackson QB - BAL #MIA 324 5 - 6 - - -
2 1 2 Dak Prescott QB - DAL NYG 405 4 - 12 - - -
3 1 3 Deshaun Watson QB - HOU #NO 268 3 1 40 1 - -
4 1 4 Matthew Stafford QB - DET #ARI 385 3 - 22 - - -
5 1 5 Patrick Mahomes QB - KC #JAX 378 3 - 2 - - -
ReceivingTD RetTD MiscFumTD Misc2PT FumLost FantasyPoints position week
1 - - - - - 33.56 1 1
2 - - - - - 33.40 1 1
3 - - - - - 30.72 1 1
4 - - - - 1 27.60 1 1
5 - - - - - 27.32 1 1