How to merge 2 JSON files including objects and arrays using jq? - jq

Question

I'm using jq to try and merge 2 json files into one unique file.
The result is close to what I was looking for, but not just right.
File 1:
{
"series": "Harry Potter Movie Series",
"writer": "J.K. Rowling",
"movies": [
{
"title": "Harry Potter and the Philosopher's Stone",
"actors": [
{
"names": [
"Emma Watson",
"Other actor"
],
"other": "Some value"
}
]
},
{
"title": "Harry Potter and the Chamber of Secrets",
"actors": [
{
"names": [
"Emma Watson"
],
"other": "Some value"
}
]
}
]
}
File 2:
{
"series": "Harry Potter Movie Series",
"producer": "David Heyman",
"movies": [
{
"title": "Harry Potter and the Philosopher's Stone",
"year": "2001"
},
{
"title": "Harry Potter and the Chamber of Secrets",
"year": "2002"
}
]
}
Expected result:
{
"series": "Harry Potter Movie Series",
"writer": "J.K. Rowling",
"movies": [
{
"title": "Harry Potter and the Philosopher's Stone",
"year": "2001",
"actors": [
{
"names": [
"Emma Watson",
"Other actor"
],
"other": "Some value"
}
]
},
{
"title": "Harry Potter and the Chamber of Secrets",
"year": "2001",
"actors": [
{
"names": [
"Emma Watson"
],
"other": "Some value"
}
]
}
],
"producer": "David Heyman"
}
Best result I've got so far (only arrays with actors are missing):
{
"series": "Harry Potter Movie Series",
"writer": "J.K. Rowling",
"movies": [
{
"title": "Harry Potter and the Philosopher's Stone",
"year": "2001"
},
{
"title": "Harry Potter and the Chamber of Secrets",
"year": "2002"
}
],
"producer": "David Heyman"
}
Using one of the commands below:
jq -s '.[0] * .[1]' file1 file2
jq --slurp 'add' file1 file2
jq '. * input' file1 file2
If I switch order of files I either end up losing 'actors' from file1 or 'year' from file2.
How it should work:
the elements in file 2 will be leading and should replace the matching elements in file 1.
the elements in file 1 that doesn't exist in file 2 (like writer and movies[].actors elements) shouldn't be deleted
the elements in file 2 that doesn't exist yet in file 1 will be added (like producer and movies[].year).
a title is unique and should by default not occur more then once, but if it does remove the duplicates.
I would assume there is a solution to get these movies arrays perfectly merged with jq.

Answer 1

You are looking for a solution that "merges" objects and arrays. For the former you have already found + (or add) for a top-level merge, and * for a recursive merge, but merging arrays (namely the two .movies fields) needs more specification from your end as there is no canonical solution for that.
In a comment you state
.movies[0] always correspond to the same movie in both files
This enables you to use transpose to align the items from both arrays, and then apply object-merging on each pair of corresponding items. If you want to merge deeper arrays as well (e.g. .movies[].actors or .movies[].actors[].names) you need to extend this approach accordingly. Here's a solution using plain add for the merging of the array items as well as of the other top-level fields:
jq -s 'add + {movies: map(.movies) | transpose | map(add)}' file1 file2
{
"series": "Harry Potter Movie Series",
"writer": "J.K. Rowling",
"movies": [
{
"title": "Harry Potter and the Philosopher's Stone",
"actors": [
{
"names": [
"Emma Watson",
"Other actor"
],
"other": "Some value"
}
],
"year": "2001"
},
{
"title": "Harry Potter and the Chamber of Secrets",
"actors": [
{
"names": [
"Emma Watson"
],
"other": "Some value"
}
],
"year": "2002"
}
],
"producer": "David Heyman"
}
Demo