Json to dataframe (empty observations, different lengths) - r

Question

Through an API I accessed weather information in Json format. I want to convert this data to a dataframe. The problem is that not for every date-city combination the api returns weather conditions, so a few rows are empty. Second , not every combination that does return gives the same aspects of the weather. My goal is to convert the Json to a dataframe, where rows that are empty are still showed in the dataframe (which does not happen when I unlist them) and the different aspects of the weather are properly showed under the right variable with NA values if there is no record for that particular variable. I've tried enlisting them and putting it into a dataframe, flattening the table etc (getting the error: arguments imply differing number of rows: 0, 1) . I've searched for this topic but none of them worked for my case (or maybe because I'm not that experienced I applied them wrong), but every tip is welcome!
The input looks like this:
reviewid dateofwriting lon lat
98338143 28-02-11 11,41693611 22,3193039
58929813 18-03-10 -3,7037902 40,4167754
65945346 31-05-10 -3,188267 55,953252
The output looks like this (the second observation returns 36 columns and the third one 38. the first entry is missing because there was no observation for that day and is not displayed)
enter image description here
[{},
{"daily":
[{"time":"2010-03-18",
"summary":"Partly cloudy throughout the day.",
"icon":"partly-cloudy-day",
"sunriseTime":"2010-03-18 07:22:51",
"sunsetTime":"2010-03-18 19:25:28",
"moonPhase":0.08,
"precipIntensity":0,
"precipIntensityMax":0,
"precipProbability":0,
"temperatureHigh":63.14,
"temperatureHighTime":1268928000,
"temperatureLow":45.16,
"temperatureLowTime":1268971200,
"apparentTemperatureHigh":63.14,
"apparentTemperatureHighTime":1268928000,
"apparentTemperatureLow":45.16,
"apparentTemperatureLowTime":1268971200,
"dewPoint":36.97,
"humidity":0.58,
"pressure":1025.96,
"windSpeed":1.24,
"windGust":7.87,
"windGustTime":1268866800,
"windBearing":48,
"cloudCover":0.54,
"uvIndex":5,
"uvIndexTime":1268913600,
"visibility":6.19,
"temperatureMin":43.97,
"temperatureMinTime":"2010-03-18 07:00:00",
"temperatureMax":63.14,
"temperatureMaxTime":"2010-03-18 17:00:00",
"apparentTemperatureMin":42.03,
"apparentTemperatureMinTime":"2010-03-18 08:00:00",
"apparentTemperatureMax":63.14,
"apparentTemperatureMaxTime":"2010-03-18 17:00:00"}]},
{"daily":
[{"time":"2010-05-30 01:00:00",
"summary":"Mostly cloudy until evening.",
"icon":"partly-cloudy-day",
"sunriseTime":"2010-05-30 05:38:39",
"sunsetTime":"2010-05-30 22:44:55",
"moonPhase":0.58,
"precipIntensity":0.0038,
"precipIntensityMax":0.0766,
"precipIntensityMaxTime”:"2010-05-30 04:00:00",
"precipProbability":1,
"precipType":"rain",
"temperatureHigh":58.99,
"temperatureHighTime":1275242400,
"temperatureLow":36.62,
"temperatureLowTime":1275278400,
"apparentTemperatureHigh":58.99,
"apparentTemperatureHighTime":1275242400,
"apparentTemperatureLow":36.62,
"apparentTemperatureLowTime":1275278400,
"dewPoint":43.61,
"humidity":0.76,
"pressure":1011.52,
"windSpeed":4.65,
"windGust":21.4,
"windGustTime":1275224400,
"windBearing":350,
"cloudCover":0.61,
"uvIndex":5,
"uvIndexTime":1275213600,
"visibility":5.85,
"temperatureMin":45.99,
"temperatureMinTime":"2010-05-30 07:00:00",
"temperatureMax":58.99,
"temperatureMaxTime":"2010-05-30 20:00:00",
"apparentTemperatureMin":43.31,
"apparentTemperatureMinTime":"2010-05-30 06:00:00",
"apparentTemperatureMax":58.99,
"apparentTemperatureMaxTime":"2010-05-30 20:00:00"}]}]
The goal is to add these rows to the input excel above.
icon sunrisetime sunsettime etc.
NA NA NA etc.
partly-cloudy-day 18-03-10 07:22 18-03-10 19:25 etc.
partly-cloudy-day 30-05-10 05:38 30-05-10 22:44 etc.

Answer 1

There is a problem dealing with the responses that return NULL. To simplify the issue, it is easier to remove these non responses and then parse the remaining JSON response. If desired, one can go back and add the empty rows for the non responses.
library(jsonlite)
library(dplyr)
#test<- result from converting the JSON response.
#vector of reviewid, used to make the initial request to the API
reviewid<-c(98338143, 58929813, 65945346)
#find only the responses that are not Null or blank
valid<-which(sapply(1:nrow(test), function(j) {length(test[[1]][[j]])}) >0)
NullResponses<-which(sapply(1:nrow(test), function(j) {length(test[[1]][[j]])}) == 0)
#create a list of data frames with the data from row of the response
dflist<-lapply( valid, function(j) {
temp<-t(as.matrix(unlist(test[j,])))
df<-data.frame(reviewid=reviewid[j], temp, stringsAsFactors = FALSE)
df
})
#bind the rows together.
answer<-bind_rows(dflist)