R JSONlite: How to tackle below error? - r

Question

See the below R code, I'm using JSONlite package to scrape data from a website:
library(jsonlite)
url <- "http://fantasy.premierleague.com/web/api/elements/"
seasonsdata <- data.frame(matrix(NA,nrow=1,ncol=20))
seasonsdata <- seasonsdata[-1,]
fetchData <- function(i) {res <- try(a <- fromJSON(paste0(url,i)))
if(!inherits(res,"try-error")) {b<-data.frame(a[1],a[20],a[21],as.data.frame(a$season_history))}}
seasonsdata <- lapply(1:696, fetchData)
seasonsdata <-do.call(rbind,lapply(seasonsdata,data.frame,stringsAsFactors=FALSE))
The code is working fine for 'i' till 10 at least, I'm getting the desired output. However, as I increase 'i' to 696, I'm getting the error:
Error in data.frame(a[1], a[20], a[21], as.data.frame(a$season_history)) :
arguments imply differing number of rows: 1, 0
Any advise?

Answer 1

If a$season_history is empty (page 57 is an example) then when you do data.frame(a[1], a[20], a[21], as.data.frame(a$season_history)) the first 3 elements have one row (they are scalars) and the last element has zero rows. In your function you can first check if a$season_history is there. If it it's not you can create a row of NAs in its places.
However there is another problem with your code you may not be aware of yet. Not every page to 696 exists and you get a 404 error when you try to pull the data from it. I added some steps to remove those pages before you do the final do.call(rbind, ...) step.
library(jsonlite)
url <- "http://fantasy.premierleague.com/web/api/elements/"
seasonsdata <- data.frame(matrix(NA, nrow = 1, ncol = 20))
seasonsdata <- seasonsdata[-1, ]
fetchData <- function(i) {
res <- try(a <- fromJSON(paste0(url, i)))
if (!inherits(res, "try-error")) {
if (nrow(as.data.frame(a$season_history)) == 0) {
b <- data.frame(a[1], a[20], a[21], as.data.frame(matrix(NA, ncol = 17)))
} else {
b <- data.frame(a[1], a[20], a[21], as.data.frame(a$season_history))
}
}
}
seasonsdata <- lapply(1:696, fetchData)
seasonsdata <- seasonsdata[!sapply(seasonsdata, is.null)]
seasonsdata <- seasonsdata[sapply(seasonsdata, is.data.frame)]
seasonsdata <- do.call(rbind,lapply(seasonsdata, data.frame, stringsAsFactors = FALSE))