I have multiple files containing both A array elements at the top level, and files containing A array elements nested in B elements. Is it possible to extract the data in a single jq line?
File 1:
{ "A" : [
{ "x" : " y" }
]
}
File 2:
{ "B" : [
{ "A" : [
{ "x" : "y" }
] }
] }
I have tried the following command
jq -r 'select(.A[] != null or .B[].A[] != null) | .A[] | .x'
without expected results.
It should be straightforward with using getpath/1. Dynamically identify all paths where the leaf value is A
getpath(paths | select(.[-1] == "A")) | .[].x
jqplay - Demo 1, Demo 2
or a hacky-way to rely on the fact that A is an array and x will always be one of the leaf paths
getpath(paths | select(.[-3] == "A" and .[-1] == "x"))
To visualize how the solution works, run the command paths on your input JSON, which breaks down the original JSON to all possible root-leaf paths. getpath/1 works by getting the value at a given path, where the last element in the array i.e. -1 is the leaf path x and 2 places before last is A.
Another option would be to use the .[]? (see the manual) which works
like .[], but no errors will be output if . is not an array or object.
jq -r '., .B[]? | .A[]?.x'
Demo
Related
I have a large file of records that contain fields that look something like this:
{
"id": "1000001",
"updatedDate": "2018-12-21T01:52:00Z",
"createdDate": "1993-11-30T02:59:25Z",
"varFields": [
{
"fieldTag": "b",
"content": "1000679727"
},
{
"fieldTag": "v",
"content": "v.1"
}
}
I need to extract the .content element along with other things, but only when the fieldTag associated with it is "v". Only some records contain a fieldTag "v".
When I try to parse using
(.varFields[] |select(.fieldTag=="v") | "\(.content)") // ""
it works fine so long as v is present. However, when it is not present, I get
jq: error (at <stdin>:353953): Cannot iterate over null (null)
I tried to get rid of the error with multiple variations, including things to the effect of
(select((.varFields[] |select(.fieldTag=="v") | .content) != null) | .varFields[] |select(.fieldTag=="v") | "\(.content)") // ""
but I'm still getting the same error. What am I missing?
Take a look at the error suppression operator ? that works a bit like the new ?. nullable chaining operator in Javascript.
The ? operator, used as EXP?, is shorthand for try EXP.
Example:
jq '[.[]|(.a)?]'
Input [{}, true, {"a":1}]
Output [null, 1]
They have a slightly simpler demonstrable example of this at https://jqplay.org/jq?q=%5B.%5B%5D%7C(.a)%3F%5D&j=%5B%7B%7D%2C%20true%2C%20%7B%22a%22%3A1%7D%5D and the try-catch operator is similar if all you need is custom error handling (or just error ignoring...).
I have a custom event with a json (string) property called EventInfo. Sometimes this property will be larger than the 150 character limit set on event properties, so I have to split it into multiple properties, ie EventInfo0, EventInfo1, ect.
For example (shortened for simplicity)
EventInfo0: [{ "label" : "likeButton", "stat],
EventInfo1: [us" : "success" }]
I found out how to look at EventInfo as a json in app insights like:
customEvents
| where name == "people"
| extend Properties = todynamic(tostring(customDimensions.Properties))
| extend type=parsejson(Properties.['EventInfo'])
| mvexpand type
| project type, type.label, type.status]
Is there a way I can concatenate EventInfo0 and EventInfo1 to create the full json string, and query that like above?
According to the documentation, the 150 character limit is on the key, and not on the entire payload. So splitting as you're doing it may not actually be required.
https://learn.microsoft.com/en-us/azure/azure-monitor/app/data-model-event-telemetry#custom-properties
that said, to answer your questions - while it's not efficient to do this at query time, the following could work:
datatable(ei0:string, ei1:string)
[
'[{ "label" : "likeButton", "stat]', '[us" : "success" }]',
'[{ "lab]', '[el" : "bar", "hello": "world" }]'
]
| project properties = parse_json(strcat(substring(ei0, 1, strlen(ei0) - 2), substring(ei1, 1, strlen(ei1) - 2)))
| project properties.label
properties_label
----------------
likeButton
bar
I realize there are several other JmesPath join questions here, but I'm having trouble with a separate problem that I haven't found any examples for, where I need to concatenate (ie, join) a set of JSON values that have dynamically-named keys into a single element.
If I start with the following JSON data structure:
{
"data": [
{
"secmeetingdays":
{
"dayset_01":
{
"day_01": "M",
"day_02": "W",
"day_03": "F"
},
"dayset_02":
{
"day_01": "T",
"day_02": "TH"
}
},
}]
}
I would like to end up with something like this:
[
[
"M,W,F"
],
[
"T,TH"
]
]
I've started the query to flatten the data down, but am completely stuck with the join syntax. Nothing I try seems to be working.
Attempt 1: data[].secmeetingdays | [0].*.*
[
[
"M",
"W",
"F"
],
[
"T",
"TH"
]
]
Almost, but not quite there.
Attempt 2: data[].secmeetingdays | [0].*.* | {join(',',#)}
fails
Attempt 3: data[].secmeetingdays | [0].*.*.join(',',#)
fails
Attempt 4: data[].secmeetingdays | {join(',',#[0].*.*)}
fails
I tried avoiding 2 flattens to have some reference to grab onto inside the join.
Attempt 4 data[].secmeetingdays | [0].* | join(',',#[]).
fails
Attempt 6 data[].secmeetingdays | [0].*.* | #.join(',',[]) Gives a result, but it's not what I want:
"M,W,F,T,TH"
Update:
Attempt 7 data[].secmeetingdays[].*.* | [].join(',',#) gets me a lot closer but is also not exactly what I need:
[
"M,W,F",
"T,TH"
]
I might be able to work with this solution, but will leave this open in case someone has the accurate answer to the question.
The example here https://jmespath.org/ has a join, but it is only on a single list of items. How can I join the sub-arrays without affecting the structure of the parents?
data[*].secmeetingdays.values(#)[].values(#).join(',', #).to_array(#)
Gives you the example desired output but I see no benefit to wrapping each single string in an extra array.
data[].secmeetingdays.values(#) | [*][*].values(#).join(',', #)
Produces more logical output (to me) because it gives an array of daysets for each item in the data array:
[
[
"M,W,F",
"T,TH"
]
]
Note that the proper way to deal with such data is to write a script that iterates the objects, parses the keys and guarantees ordered output after sorting the items. JSON parsers have no obligation to keep object properties ordered the same as they were stored/read, so blindly converting to an array as above is not certain to be the order you desire. Using key names to store order is superfluous. Chronologically ordered data should be stored as arrays like so:
{
"data": [
{
"secmeetingdays": [
[
"M",
"W",
"F"
],
[
"T",
"TH"
]
]
}
]
}
[[0].title,[1].title].join(',', #).to_array(#)
RESULT: ["some1,some2"]
[[0].title,[1].title].join(',', #)
RESULT: "some1,some2"
[[0].title,[1].title]
RESULT: ["some1,some2"]
I have a file named param1.txt which contains certain variables. I have another file as source1.txt which contains place holders. I want to replace the place holders with the values of the variables that I get from the parameter file.
I have basically hard coded the script where the variable names in the parameter.txt file is known before hand. I want to know a dynamic solution to the problem where the variable names will not be known beforehand. In other words, is there any way to find out the variable names in a file using the source command in UNIX?
Here is my script and the files.
Script:
#!/bin/bash
source /root/parameters/param1.txt
sed "s/{DB_NAME}/$DB_NAME/gI;
s/{PLANT_NAME}/$PLANT_NAME/gI" \
/root/sources/source1.txt >
/root/parameters/Output.txt`
param1.txt:
PLANT_NAME=abc
DB_NAME=gef
source1.txt:
kdashkdhkasdkj {PLANT_NAME}
jhdbjhasdjdhas kashdkahdk asdkhakdshk
hfkahfkajdfk ljsadjalsdj {PLANT_NAME}
{DB_NAME}
I cannot comment since I don't have enough points.
But is it correct that this is what you're looking for:
How to reference a file for variables using Bash?
Your problem statement isn't very clear to me. Perhaps you can simplify your problem and desired state.
Don't understand why you try to source param1.txt.
You can try with this awk :
awk '
NR == FNR {
a[$1] = $2
next
}
{
for ( i = 1 ; i <= NF ; i++ ) {
b = $i
gsub ( "^{|}$" , "" , b )
if ( b in a )
sub ( "{" b "}" , a[b] , $i )
}
} 1' FS='=' param1.txt FS=" " source1.txt
I have the following json data:
{"jsonrpc":"2.0","result":[],"id":1}
{"jsonrpc":"2.0","result":[{"hostmacroid":"2392","hostid":"10953","macro":"{$GATEWAY}","value":"10.25.230.1"}],"id":1}
{"jsonrpc":"2.0","result":[{"hostmacroid":"1893","hostid":"12093","macro":"{$GATEWAY}","value":"10.38.118.1"}],"id":1}
{"jsonrpc":"2.0","result":[{"hostmacroid":"2400","hostid":"14471","macro":"{$GATEWAY}","value":"10.25.230.1"}],"id":1}
{"jsonrpc":"2.0","result":[{"hostmacroid":"799","hostid":"10798","macro":"{$GATEWAY}","value":"10.36.136.1"}],"id":1}
{"jsonrpc":"2.0","result":[],"id":1}
{"jsonrpc":"2.0","result":[{"hostmacroid":"1433","hostid":"10857","macro":"{$GATEWAY}","value":"10.38.24.129"}],"id":1}
{"jsonrpc":"2.0","result":[{"hostmacroid":"842","hostid":"13159","macro":"{$GATEWAY}","value":"10.38.113.1"}],"id":1}
{"jsonrpc":"2.0","result":[],"id":1}
I am trying to extract the value of the "value" field from each line. jq -r '.result[].value' <jsonfile> works perfectly but it does not take into account the JSON lines where there is no "value" field. I would like it to print an empty line for them. Is this possible with jq?
You can use this:
jq -r '.result[].value // "" ' a.json
This uses the or operator //. If .result[].value is present, the value will get printed, otherwise an empty line gets printed.
This would work:
jq -r '.result | if length > 0 then .[0].value else "" end'
Since false // X and null // X produce X, .result[].value // "" may not be what you want in all cases.
To achieve the stated goal as I understand it, you could use the following filter:
.result[] | if has("value") then .value else "" end