Need help navigating lists when converting JSON into dataframe/CSV - r

Question

I am trying to scrape a javascript rendered table and after trying both selenium and phantomJS I've decided that JSON would be the easiest way to do it. However I am quite new to R and not very good at handling lists, and because of that I cannot get my data into the table format I desire. I've looked at a number of solutions but for some reason they don't really work on the JSON I have.
The JSON data is rendered through this URL. And this is the actual website where the table is located.
What I've done so far is to try to parse the JSON into R and coerce it into a dataframe, based on what I've seen from most answers on stackoverflow.
library(httr)
library(jsonlite)
rf <- GET(url) #the entire URL is very long so I'm not putting it here
rfc <- content(rf)
Doing this returns me a large list of four elements, rfc. I then apply the following function.
library(httr)
library(jsonlite)
json_file <- lapply(rfc, function(x) {
x[sapply(x, is.null)] <- NA
unlist(x)
})
This returns me an error:
Error in x[sapply(x, is.null)] <- NA : invalid subscript type 'list'
Given that I only need the second element of the list, which is where the information is at, I attempt to subset it:
json_file <- lapply(rfc[2], function(x) {
x[sapply(x, is.null)] <- NA
unlist(x)
})
This returns me a large list, 12mb in size. When I try to coerce it to a dataframe using as.data.frame, R returns me 506472 observations of 1 variable. The different columns have all been squashed into one and the headers are gone.
Can anyone tell me how I should go about doing this? There's a free online JSON to CSV converter here that does exactly what I need beautifully. This is what it produces:
Unfortunately this is not a solution. Because I intend to run this in Shiny I want to do everything within R. Any help is appreciated, thanks.

Answer 1

You need to take the element rfc$data$DailyProductionAndFlowList, which itself is effectively a list of single-row data frames, and bind them together. You'll need to overwrite the NULL values first:
df <- do.call(rbind, lapply(rfc$data$DailyProductionAndFlowList, function(x) {
x[sapply(x, is.null)] <- "NULL"
as.data.frame(x, stringsAsFactors = FALSE)
}))
To show you that the result is sensible, I've put it in a tibble here for nicer printing:
as_tibble(df)
#> # A tibble: 3,997 x 11
#> GasDate FacilityId FacilityName LocationId LocationName Demand Supply
#> <chr> <int> <chr> <int> <chr> <dbl> <dbl>
#> 1 2020-0~ 520047 Eastern Gas~ 520008 Sydney 94.4 0
#> 2 2020-0~ 520047 Eastern Gas~ 520009 Canberra 16.5 0
#> 3 2020-0~ 520047 Eastern Gas~ 530015 Longford Hub 0 234.
#> 4 2020-0~ 520047 Eastern Gas~ 590011 Regional - ~ 22.4 0
#> 5 2020-0~ 520047 Eastern Gas~ 590012 Regional - ~ 2.68 19.4
#> 6 2020-0~ 520047 Eastern Gas~ 520008 Sydney 113. 0
#> 7 2020-0~ 520047 Eastern Gas~ 520009 Canberra 19.7 0
#> 8 2020-0~ 520047 Eastern Gas~ 530015 Longford Hub 0 225.
#> 9 2020-0~ 520047 Eastern Gas~ 590011 Regional - ~ 27.5 0
#> 10 2020-0~ 520047 Eastern Gas~ 590012 Regional - ~ 5.05 20.1
#> # ... with 3,987 more rows, and 4 more variables: TransferIn <dbl>,
#> # TransferOut <dbl>, HeldInStorage <chr>, LastUpdated <chr>

Answer 2

Another approach:
library( data.table )
library( rjson )
#location of data
json.url = "https://aemo.com.au/aemo/api/v1/GasBBReporting/DailyProductionAndFlow?FacilityIds=540093,580010,540101,544261,540047,530030,540083,540096,580020,540071,540077,520075,540059,520054,520090,540094,540098,540080,540090,540086,540050,540097,540055,520047,540089,540070,540092,530071,530042,540088,540075,544253,540061,530038,530039,530040,580100,580040,540064,530043,550050,550045,550046,550054,520053,530061,520060,580050,540084,530041,530044,580060,580070,540065,550052,530060,540058,540085,540102,540073,540057,540095,544260,540110,540040,540082,540072,540062,540103,550061,550060,540060,540066,540067,540076,540068,580210,570050,540051,532005,530110,540045,540046,540091,580030,540069,540087,580180,540074&FromGasDate=07/08/2020&ToGasDate=07/09/2020"
#retrieve lastest data
mydata <- data.table::rbindlist( rjson::fromJSON( file = json.url )$data$DailyProductionAndFlowList,
use.names = TRUE, fill = TRUE )