Can I control how tidyjson joins nested arrays? - r

Question

It seems as though tidyjson uses an inner-join-like behaviour on nested arrays, thus dropping records with empty child arrays. Is there a way to get left-join-like behaviour instead, filling with NAs?
For example, these fake data have one record with a populated nested array (middles) and two records where middles is empty:
library(tidyjson)
people <- c('{"age": 32, "name": [{"first": "Bob", "last": "Smith", "middles":[{"middle1":"John", "middle2":"Rick"}]}]}',
'{"age": 54, "name": [{"first": "Susan", "last": "Doe", "middles":[]}]}',
'{"age": 18, "name": [{"first": "Ann", "last": "Jones", "middles":[]}]}')
From these data I wish to have a dataframe with all the parent records retained and missing child-array information filled with NAs (~ left join) as such:
# A tibble: 3 x 5
age first last middle1 middle2
<dbl> <chr> <chr> <chr> <chr>
1 32 Bob Smith John Rick
2 54 Susan Doe NA NA
3 18 Ann Jones NA NA
However, extracting a nested array with some empty child arrays causes loss of their parent's information (~ inner join):
people %>%
spread_all() %>%
enter_object("name") %>% gather_array() %>%
spread_all() %>% select(-document.id,-array.index) %>%
enter_object("middles") %>% gather_array %>%
spread_all() %>% select(-array.index) %>%
tbl_df()
# A tibble: 1 x 5
age first last middle1 middle2
<dbl> <chr> <chr> <chr> <chr>
1 32 Bob Smith John Rick
Is there a way to avoid this; i.e., to retain all the rows even when child arrays are empty?
A workaround but not a solution
A possible workaround is to literally do a left join, but this means duplicating the JSON read, which is not trivial given gigabytes of data.
wrap_dplyr_verb <- function(dplyr.verb) {
# Creates a tidyjson verb out of a dplyr verb
# https://github.com/colearendt/tidyjson/blob/master/R/tbl_json.R
function(.data, ...) {
# Check if reserved ..JSON name already in data.frame
if ("..JSON" %in% names(.data))
stop("'..JSON' in the column names of tbl_json object being filtered")
# Assign JSON to the data.frame so it is treated as any other column
.data$..JSON <- attr(.data, "JSON")
# Apply the transformation
y <- dplyr.verb(dplyr::as_tibble(.data), ...)
# Reconstruct tbl_json without ..JSON column
tbl_json(dplyr::select(y, -..JSON), y$..JSON)
}
}
left_join_json = wrap_dplyr_verb(left_join)
people %>%
spread_all() %>%
enter_object("name") %>% gather_array() %>%
spread_all() %>% select(-document.id,-array.index) %>%
left_join_json(
people %>%
spread_all() %>%
enter_object("name") %>% gather_array() %>%
spread_all() %>% select(-document.id,-array.index) %>%
enter_object("middles") %>% gather_array %>%
spread_all() %>% select(-array.index)
) %>%
tbl_df()
Joining, by = c("age", "first", "last")
# A tibble: 3 x 5
age first last middle1 middle2
<dbl> <chr> <chr> <chr> <chr>
1 32 Bob Smith John Rick
2 54 Susan Doe NA NA
3 18 Ann Jones NA NA

Answer 1

Thanks so much for asking this question! It's definitely an interesting use case and one that I think could be improved upon within tidyjson.
Of course, I forgot that I had already worked with you on this question in GitHub, so I implemented with a different pattern here, in case it's any help to you or others. Basically, the idea is to go "tall" instead of "wide" first. I have no idea what it does to execution time, and whether or not there are left joins hidden in here that hurt the timing.
library(tidyjson)
library(dplyr)
library(tidyr)
people <- c('{"age": 32, "name": [{"first": "Bob", "last": "Smith", "middles":[{"middle1":"John", "middle2":"Rick"}]}]}',
'{"age": 54, "name": [{"first": "Susan", "last": "Doe", "middles":[]}]}',
'{"age": 18, "name": [{"first": "Ann", "last": "Jones", "middles":[]}]}')
as_tbl_json(people) %>% spread_all() %>% enter_object("name") %>% gather_array("nameid") %>%
gather_object("key") %>%
{bind_rows(
filter(., key != "middles") %>% append_values_string("value"),
filter(., key == "middles") %>% gather_array("middleid") %>%
select(-key) %>%
gather_object("key") %>%
append_values_string("value")
)} %>%
# drop tbl_json
as_tibble() %>%
select(-document.id, -nameid, -middleid) %>%
# could also use tidyr::pivot_wider
tidyr::spread(key, value)
#> # A tibble: 3 x 5
#> age first last middle1 middle2
#> <dbl> <chr> <chr> <chr> <chr>
#> 1 18 Ann Jones <NA> <NA>
#> 2 32 Bob Smith John Rick
#> 3 54 Susan Doe <NA> <NA>
Created on 2020-06-28 by the reprex package (v0.3.0)