Attempting to scrape an "unscrapable" page? - r

Question

I'm attempting to build a simple scraper, iterating through a website to pull two pieces of information and build myself a little reference list.
This is what the url looks like: "https://www.mtgstocks.com/prints/[[n]]"
The two pieces of information are the card name (Forbidden Alchemy) and card set (Innistrad).
Pretty straightforward, yeah? I thought so.
I attempted to pass any relevant anchors — css or xpath  — to try and isolate the two variables, but was met with "{xml_nodeset (0)}".
Here's the code that I ran:
# return page info
page_html <- read_html(httr::GET("https://www.mtgstocks.com/prints/1"))
# extract item name
page_html %>%
html_nodes("h3") %>%
html_nodes("a") %>%
html_text()
# character(0)
I've scraped enough webpages to know that this information is being hidden, but I'm not exactly sure how. Would love help!

Answer 1

They are gathering the information from their API, which you can see in the network section of the developer tools. There is a ton of info of card that you can gather from that API - take a look.
library(tidyverse)
library(httr2)
get_card <- function(card_num) {
cat("Scraping card num:", card_num, "\n")
data <- str_c("https://api.mtgstocks.com/prints/", card_num) %>%
request() %>%
req_perform() %>%
resp_body_json(simplifyVector = TRUE)
tibble(
name = data %>%
pluck("name"),
set = data %>%
pluck("card_set") %>%
pluck("name")
)
}
get_card(1)
# A tibble: 1 × 2
name set
<chr> <chr>
1 Forbidden Alchemy Innistrad
Or get for a range of numbers. If the card does not exists, the code returns NA.
map_dfr(1:20, possibly(get_card, otherwise = tibble(
name = NA_character_,
set = NA_character_
)))
# A tibble: 20 × 2
name set
<chr> <chr>
1 Forbidden Alchemy Innistrad
2 NA NA
3 Fortress Crab Innistrad
4 Frightful Delusion Innistrad
5 Full Moon's Rise Innistrad
6 Furor of the Bitten Innistrad
7 Gallows Warden Innistrad
8 Galvanic Juggernaut Innistrad
9 Garruk Relentless Innistrad
10 Gatstaf Shepherd Innistrad
11 Gavony Township Innistrad
12 Geist of Saint Traft Innistrad
13 Geist-Honored Monk Innistrad
14 Geistcatcher's Rig Innistrad
15 Geistflame Innistrad
16 Ghost Quarter Innistrad
17 Ghostly Possession Innistrad
18 Ghoulcaller's Bell Innistrad
19 Ghoulcaller's Chant Innistrad
20 Ghoulraiser Innistrad