Applying a function to multiple lists - r

Question

I am doing research on U.S. Lobbying, who publishes their data as an open API that is very poorly integrated and only seems to allow 250 observations to be downloaded at one time. I would like to compile the whole data set into one data table but am struggling with the last step to do so. This is what I have thus far
base_url <- sample("https://lda.senate.gov/api/v1/contributions/?page=", 10, rep = TRUE) #Set the number between the commas as how many pages you want
numbers <- 1:10 #Set the second number as how many pages you want
pagesize <- sample("&page_size=250", 10, rep = TRUE) #Set the number between the commas as how many pages you want
pages <- data.frame(base_url, numbers, pagesize)
pages$numbers <- as.character(pages$numbers)
pages$url <- with(pages, paste0(base_url, numbers, pagesize)) # creates list of pages you want. the list is titled pages$url
for (i in 1:length(pages$url)) assign(pages$url[i], GET(pages$url[i])) # Creates all the base lists in need of extraction
The last two things I need to do are extract the data table from the created lists and then full join all of them. I know how to join all of them but extracting the data frames is proving to be challenging. basically, to all the created lists I need to apply the function fromJSON(rawToChar(list$content)). I have tried using lapply but have yet to figure it out. any help would be greatly welcomed!

Answer 1

When you were assigning GET(pages$url[i])) to your data frame you were coercing it to a character vector. Better to assign it to a list and keep it as a response:
library(httr)
library(jsonlite)
library(dplyr) # for bind_rows
page_content <- list()
for (i in 1:length(pages$url)) page_content[[i]] <- GET(pages$url[i]) # Creates all the base lists in need of extraction
Then you can use the code you had written - fromJSON(rawToChar()) - to extract it from raw bytes to characters:
results_list <- lapply(
page_content,
\(page) fromJSON(rawToChar(page[["content"]]))["results"][[1]]
)
results_table <- do.call(bind_rows, results_list)
dim(results_table) # 2500 27
names(results_table)
# [1] "url" "filing_uuid" "filing_type" "filing_type_display" "filing_year"
# [6] "filing_period" "filing_period_display" "filing_document_url" "filing_document_content_type" "filer_type"
# [11] "filer_type_display" "dt_posted" "contact_name" "comments" "address_1"
# [16] "address_2" "city" "state" "state_display" "zip"
# [21] "country" "country_display" "registrant" "lobbyist" "no_contributions"
# [26] "pacs" "contribution_items"