Make a loop to scrape a website to create multiple dataframes - r

Question

I'm working on a project where I can see two ways to potentially solve my problem. I'm scraping a webpage by using a loop to save the each page locally as a HTML file. The problem I'm having is when I try to select on the files in my local folder they are basically blank pages. I'm not sure why. I've used this same code on other sites for this project with success.
This is the code I'm using.
#scrape playoff teams for multiple seasons and saved html to local folder
for(i in 2002:2021){
playoff_url <- read_html(paste0("https://www.espn.com/nfl/stats/player/_/season/",i,"/seasontype/3"))
playoff_stats <- playoff_url %>%
write_html(paste0("playoff",i,".HTML"))
}
My second option is to scrape individual seasons into a data frame, but I would like to do it in a loop, and to not have to run this code 20 different times. I also don't want to continually scrape data from the site every time I run the code. It doesn't matter if all the data is in 1 large data frame for all 20 seasons or 20 separate ones. I can export the code to a local file then import it when I need it.
#read in code for playoff QBs from ESPN and added year column
playoff_url <- read_html("https://www.espn.com/nfl/stats/player/_/season/2015/seasontype/3")
play_QB2015 <-playoff_url %>%
html_nodes("table") %>%
html_table()
#combine list from QB playoff data to convert to dataframe
play_QB2015 <- c(play_QB2015[[1]], play_QB2015[[2]])
# Convert list to dataframe using data.frame()
play_QB2015 <- data.frame(play_QB2015)
play_QB2015$Year = 2015

Answer 1

Not sure what happens to your files, but first downloading and storing with httr2 and then parsing saved files with rvest works fine for me (sorry for overused tidyverse ..) :
library(fs)
library(dplyr)
library(httr2)
library(rvest)
library(purrr)
library(stringr)
dest_dir <- path_temp("playoffs")
dir_create(dest_dir)
years <- 2002:2012
# collect all years to a list
playoff_lst <- map(
set_names(years),
~ {
dest_file <- path(dest_dir, str_glue("{.x}.html"))
# only download if local copy is not present
if (!file_exists(dest_file)){
request(str_glue("https://www.espn.com/nfl/stats/player/_/season/{.x}/seasontype/3")) %>%
req_perform(dest_file)
}
read_html(dest_file) %>%
html_elements("table") %>%
html_table() %>%
bind_cols()
}
)
Results:
names(playoff_lst)
#> [1] "2002" "2003" "2004" "2005" "2006" "2007" "2008" "2009" "2010" "2011"
#> [11] "2012"
head(playoff_lst$`2002`)
#> # A tibble: 6 × 17
#> RK Name POS GP CMP ATT `CMP%` YDS AVG `YDS/G` LNG TD
#> <int> <chr> <chr> <int> <int> <int> <dbl> <int> <dbl> <dbl> <int> <int>
#> 1 1 Rich Gan… QB 3 73 115 63.5 841 7.3 280. 50 7
#> 2 2 Brad Joh… QB 3 53 98 54.1 670 6.8 223. 71 5
#> 3 3 Tommy Ma… QB 2 51 89 57.3 633 7.1 316. 40 5
#> 4 4 Steve Mc… QB 2 48 80 60 532 6.7 266 39 3
#> 5 5 Jeff Gar… QB 2 49 85 57.6 524 6.2 262 76 3
#> 6 6 Donovan … QB 2 46 79 58.2 490 6.2 245 42 1
#> # … with 5 more variables: INT <int>, SACK <int>, SYL <int>, QBR <lgl>,
#> # RTG <dbl>
dir_tree(dest_dir)
#> ... /RtmpcjLFJe/playoffs
#> ├── 2002.html
#> ├── 2003.html
#> ├── 2004.html
#> ├── 2005.html
#> ├── 2006.html
#> ├── 2007.html
#> ├── 2008.html
#> ├── 2009.html
#> ├── 2010.html
#> ├── 2011.html
#> └── 2012.html
Created on 2023-02-16 with reprex v2.0.2