Row binding of arrays loaded from JSON sources in R - r

Question

I feel this must be an easy issue, but my search fu is failing me, so your assistance is very welcome, and apologies if it is indeed answered elsewhere.
I'm working with JSON data from a REST API (specifically GitHub data for pull requests) which contains nested arrays (in this case, the comments on the PRs, which then nest other things, like the data for the comment author). I use JSONlite::fromJSON to parse this, and I get a dataframe with nested sets of lists and data.frames. Here's a cut-down example of a single row (PR):
jsn = '[
{
"pr":123,
"comment_total":2,
"comments":[
{
"user":{"name":"Me Myself","username":"me"},
"body":"comment 1"
},
{
"user":{"name":"Me Myself","username":"me"},
"body":"comment 2"
}
]
}
]'
This represents a single pull request, which has two comments on it. If I load this with JSONLite, I get 1 row as expected:
> df = jsonlite::fromJSON(jsn)
> str(df)
'data.frame': 1 obs. of 3 variables:
$ pr : int 123
$ comment_total: int 2
$ comments :List of 1
..$ :'data.frame': 2 obs. of 2 variables:
.. ..$ user:'data.frame': 2 obs. of 2 variables:
.. .. ..$ name : chr "Me Myself" "Me Myself"
.. .. ..$ username: chr "me" "me"
.. ..$ body: chr "comment 1" "comment 2"
I'd like to unwrap the first level of this comments column, so that I get one row per PR comment, but I'm struggling to do so. What I'm aiming for is something like:
pr comment_total comments.user comments.body
1 123 2 <data.frame> comment 1
2 123 2 <data.frame> comment 2
I thought tidyr::unnest() would deal with this, but it doesn't seem to like the nested data.frames:
> unnest(df)
Error in bind_rows_(x, .id) :
Argument 1 can't be a list containing data frames
I also looked at purrr::map_dfr which outputs rows, but I can't seem to get that right either - I'm using it to access the data.frame directly, but it's still unhappy:
> map_dfr(df,.id="comments", `[[`,1)
Error in bind_rows_(x, .id) :
Argument 3 can't be a list containing data frames
I'm sure I'm missing something obvious, but I can't see it - someone enlighten me? Thanks!
EDIT: The code I'm using to get the data from GitHub looks like below - if there are better ways to query this, I'm interested.
library(httr)
base_url = 'https://api.github.com/repos/ansible/ansible'
# `pr` comes from a loop, e.g. pr = 38508
issue_url = paste0(base_url,'/issues/',pr,'/comments')
# api_user and api_key are my GitHub credentials
i_resp <- GET(issue_url, authenticate(api_user,api_key))
issue_comments = as.tibble(
jsonlite::fromJSON(
content(i_resp,as="text"),
flatten = TRUE
)
)

Answer 1

One way is to tell jsonlite to not be as helpful as it usually is and then work the unwinding on your own (much like XML processing, heavily nested output from custom API endpoints tend to need some domain/API knowledge to get the data into rectangles):
jsonlite::fromJSON(
txt = jsn,
simplifyVector = FALSE,
simplifyDataFrame = FALSE,
flatten = FALSE
) %>%
map_df(~{ # in case there is more than one
dplyr::as_data_frame(.x) %>%
mutate(
body = map_chr(comments, ~.x$body),
username = map_chr(comments, ~.x$user$name),
name = map_chr(comments, ~.x$user$name)
) %>%
select(-comments)
})
## # A tibble: 2 x 5
## pr comment_total body username name
## <int> <int> <chr> <chr> <chr>
## 1 123 2 comment 1 Me Myself Me Myself
## 2 123 2 comment 2 Me Myself Me Myself