Nested Lists with different lengths into dataframe - r

Question

I am using R and the package mongolite to get data from a MongoDB. This results in data consisting of lots of nested lists that cannot be simplified into a data frame by using unlist, rbindlist, nor bind_rows from dplyr (at least I did not manage to do it).
After a lot of trial and error I found a way to do it using the package reshape2 with the function melt and using dplyr and tidyr to get it into the form I want it to be. However, the melting takes a lot of time (up to 15 mins per list, and I have 6 of them).
Do you have any ideas how I can make this faster? (Of course another possible solution would be to query the MongoDB correctly such that it does not result in lists but in something more like my target data frame).
The following code creates a dummy dataset with similar attributes, the target form of the dataset and my solution to get there.
Dummy Data:
dummy_data <- list(
list(actions = list(list(action_type = "link_clicks", value = 30),
list(action_type = "post_likes", value = 3)),
date = '2015-08-04'),
list(actions = list(list(action_type = "link_clicks", value = 10),
list(action_type = "post_likes", value = 2),
list(action_type = "page_engagement", value = 5)),
date = '2015-08-02')
)
Target Form:
final_data = data.frame(c(30, 10), c(3, 2), c(NA, 5), c('2015-08-04', '2015-08-02'))
names(final_data) = c('actions: link_clicks', 'actions: post_likes', 'actions: page_engagement', 'date')
final_data
Temporary solution
Solution <- reshape2::melt(dummy_data)
Solution <- Solution %>%
select(L1, L2, L3, L4, value) %>%
mutate(L4 = ifelse(is.na(L4), L2, L4)) %>%
spread(key = L4, value = value) %>%
mutate(L2 = ifelse(!is.na(action_type), paste0(L2, ": ", action_type), L2),
value = ifelse(!is.na(value), value, date)) %>%
select(L1, L2, value) %>%
spread(key = L2, value = value) %>%
select(-L1)
If you have any advice on the mongolite query here is the simplest kind of queries I use:
M_DB <- mongolite::mongo(collection = "name", url = "url")
M_DB_List <- M_DB$iterate()$batch(size = 100000)
Thanks a lot
**Edit: **
A more complex data structure as this is closer to my problem
dummy_data_complex <- list(
list(actions = list(list(action_type = "link_clicks", value = 30),
list(action_type = "post_likes", value = 3)),
date = '2015-08-04',
currency = 'EUR'),
list(actions = list(list(action_type = "link_clicks", value = 10),
list(action_type = "post_likes", value = 2),
list(action_type = "page_engagement", value = 5)),
date = '2015-08-02',
demographics = list(gender = "female",
list(actions = list(action_type = "link_clicks", value = 10)))
))

Answer 1

Here is an option with tidyverse
library(tidyverse)
dummy_data %>%
map_df(~ .x %>%
as_tibble(.) %>%
mutate(actions = map(actions, as_tibble)) %>%
unnest) %>%
group_by(date, action_type) %>%
mutate(n = row_number()) %>%
spread(action_type, value) %>%
select(-n)
# A tibble: 2 x 4
# Groups: date [2]
# date link_clicks page_engagement post_likes
#* <chr> <dbl> <dbl> <dbl>
#1 2015-08-02 10.0 5.00 2.00
#2 2015-08-04 30.0 NA 3.00

Answer 2

I was able to find a fast solution for my problem.
It was solved by using another query as SymbolixAU suggested.
Instead of using iterate() I used find() which then resulted in a dataframe with nested lists inside it.
From this point on I was easily able to get to my target using tidyr::unnest()
Thanks for your help.