How to check for an optional field in jq? - jq

I have this jq filter:
some_command | jq -r '.elements[] | select(.state=="LIVE" and .group == "some_text" and .someFlag == false) | .name'
someFlag is an optional field. Hence, when it is absent, the expression doesn't show any result. I want to check for:
if someFlag is present, pass the check only if it has the false value
if someFlag is not present, treat it as false
How can I do that?

I used the alternative operator, //:
(.someFlag // false) == false)
So, if .someFlag isn't there, it is treated as false.
The whole expression is:
some_command | jq -r '.elements[] | select(.state=="LIVE" and .group == "some_text" and (.someFlag // false) == false)) | .name'
From jq documentation:
Alternative operator //:
A filter of the form a // b produces the same
results as a, if a produces results other than false and null.
Otherwise, a // b produces the same results as b.
This is useful for providing defaults: .foo // 1 will evaluate to 1 if
there’s no .foo element in the input. It’s similar to how or is
sometimes used in Python (jq’s or operator is reserved for strictly
Boolean operations).

In the given context, the direct translation of:
if someFlag is present, pass the check only if it has the false value
if someFlag is not present, treat it as false
to jq is:
if has("someflag") then .someflag == false else false end
Adjusted filter
.elements[]
| select(.state == "LIVE"
and .group == "some_text"
and (if has("someFlag")
then .someFlag == false
else false
end))
| .name
[This response has been updated in accordance with the update to the Q.]

Related

Comparison operator "==" used in unix isn't running the desired condition when successful

In my unix script, I'm using a comparison operator "==" to check if the particular count is zero or not. When it satisfies the condition, it should perform further operations. However though the count is matching, it's not running further lines in the loop.
Example:
count=`grep "$name-" names.txt | wc -l`
if [ "$count" == 0 ]
then
#do further operations
...
...
else
echo "Condition is not satisfied"
fi
However even though the count is zero, it doesn't seem to accept it and executes only the else loop.
Could you please help me here?
Use -eq for numerical comparison
count=`grep "$name-" names.txt | wc -l`
if [ "$count" -eq 0 ]
then
...
fi
= and == compare strings (and your string contains spaces).

Check if element is member of array [duplicate]

I have an array and I need to check if elements exists in that array or to get that element from the array using
jq, fruit.json:
{
"fruit": [
"apple",
"orange",
"pomegranate",
"apricot",
"mango"
]
}
cat fruit.json | jq '.fruit .apple'
does not work
The semantics of 'contains' is not straightforward at all. In general, it would be better to use 'index' to test if an array has a specific value, e.g.
.fruit | index( "orange" )
However, if the item of interest is itself an array, the general form:
ARRAY | index( [ITEM] )
should be used, e.g.:
[1, [2], 3] | index( [[2]] ) #=> 1
IN/1
If your jq has IN/1 then a better solution is to use it:
.fruit as $f | "orange" | IN($f[])
If your jq has first/1 (as does jq 1.5), then here is a fast definition of IN/1 to use:
def IN(s): first((s == .) // empty) // false;
any(_;_)
Another efficient alternative that is sometimes more convenient is to use any/2, e.g.
any(.fruit[]; . == "orange")
or equivalently:
any(.fruit[] == "orange"; .)
To have jq return success if the array fruit contains "apple", and error otherwise:
jq -e '.fruit|any(. == "apple")' fruit.json >/dev/null
To output the element(s) found, change to
jq -e '.fruit[]|select(. == "apple")' fruit.json
If searching for a fixed string, this isn't very relevant, but it might be if the select expression might match different values, e.g. if it's a regexp.
To output only distinct values, pass the results to unique.
jq '[.fruit[]|select(match("^app"))]|unique' fruit.json
will search for all fruits starting with app, and output unique values. (Note that the original expression had to be wrapped in [] in order to be passed to unique.)
[WARNING: SEE THE COMMENTS AND ALTERNATIVE ANSWERS.]
cat fruit.json | jq '.fruit | contains(["orange"])'
For future visitors, if you happen to have the array in a variable and want to check the input against it, and you have jq 1.5 (without IN), your best option is index but with a second variable:
.inputField as $inputValue | $storedArray|index($inputValue)
This is functionally equivalent to .inputField | IN($storedArray[]).
Expanding on the answers here, If you need to filter the array of fruit against another array of fruit, you could do something like this:
cat food.json | jq '[.fruit[] as $fruits | (["banana", "apple"] | contains([$fruits])) as $results | $fruits | select($results)]'
This will return an array only containing "apple" in the above sample json.
This modified sample did worked here:
jq -r '.fruit | index( "orange" )' fruit.json | tail -n 1
It gets only the last line of the output.
If it exist, it returns 0.
If don't, it returns null.

jq select elements with array not containing string

Now, this is somewhat similar to jq: select only an array which contains element A but not element B but it somehow doesn't work for me (which is likely my fault)... ;-)
So here's what we have:
[ {
"employeeType": "student",
"cn": "dc8aff1",
"uid": "dc8aff1",
"ou": [
"4210910",
"4210910 #Abg",
"4210910 Abgang",
"4240115",
"4240115 5",
"4240115 5\/5"
]
},
{
"employeeType": "student",
"cn": "160f656",
"uid": "160f656",
"ou": [
"4210910",
"4210910 3",
"4210910 3a"
] } ]
I'd like to select all elements where ou does not contain a specific string, say "4210910 3a" or - which would be even better - where ou does not contain any member of a given list of strings.
When it comes to possibly changing inputs, you should make it a parameter to your filter, rather than hardcoding it in. Also, using contains might not work for you in general. It runs the filter recursively so even substrings will match which might not be preferred.
For example:
["10", "20", "30", "40", "50"] | contains(["0"])
is true
I would write it like this:
$ jq --argjson ex '["4210910 3a"]' 'map(select(all(.ou[]; $ex[]!=.)))' input.json
This response addresses the case where .ou is an array and we are given another array of forbidden strings.
For clarity, let's define a filter, intersectq(a;b), that will return true iff the arrays have an element in common:
def intersectq(a;b):
any(a[]; . as $x | any( b[]; . == $x) );
This is effectively a loop-within-a-loop, but because of the semantics of any/2, the computation will stop once a match has been found.(*)
Assuming $ex is the list of exceptions, then the filter we could use to solve the problem would be:
map(select(intersectq(.ou; $ex) | not))
For example, we could use an invocation along the lines suggested by Jeff:
$ jq --argjson ex '["4210910 3a"]' -f myfilter.jq input.json
Now you might ask: why use the any-within-any double loop rather than .[]-within-all double loop? The answer is efficiency, as can be seen using debug:
$ jq -n '[1,2,3] as $a | [1,1] as $b | all( $a[]; ($b[] | debug) != .)'
["DEBUG:",1]
["DEBUG:",1]
false
$ jq -n '[1,2,3] as $a | [1,1] as $b | all( $a[]; . as $x | all( $b[]; debug | $x != .))'
["DEBUG:",1]
false
(*) Footnote
Of course intersectq/2 as defined here is still O(m*n) and thus inefficient, but the main point of this post is to highlight the drawback of the .[]-within-all double loop.
Here is a solution that checks the .ou member of each element of the input using foreach and contains.
["4210910 3a"] as $list # adjust as necessary
| .[]
| foreach $list[] as $e (
.; .; if .ou | contains([$e]) then . else empty end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
["4210910 3a"] as $list
| .[]
| $list[] as $e
| if .ou | contains([$e]) then . else empty end

r check if string contains special characters

I am checking if a string contains any special characters. This is what I have, and its not working,
if(grepl('^\\[:punct:]', val))
So if anybody can tell me what I am missing, that will be helpful.
Special characters
~ ` ! ## $ % ^ & * | : ; , ." |
As #thelatemail pointed out in the comments you can use:
grepl('[^[:punct:]]', val)
which will result in TRUE or FALSE for each value in your vector. You can add sum() to the beginning of the statement to get the total number of these cases.
You can also use:
grepl('[^[:alnum:]]', val)
which will check for any value that is not a letter or a number.

How to check if the variable value in AWK script is null or empty?

I am using AWK script to process some logs.
At one place I need to check if the variable value is null or empty to make some decision.
Any Idea how to achieve the same?
awk '
{
{
split($i, keyVal, "#")
key=keyVal[1];
val=keyVal[2];
if(val ~ /^ *$/)
val="Y";
}
}
' File
I have tried with
1) if(val == "")
2) if(val ~ /^ *$/)
not working in both cases.
The comparison with "" should have worked, so that's a bit odd
As one more alternative, you could use the length() function, if zero, your variable is null/empty. E.g.,
if (length(val) == 0)
Also, perhaps the built-in variable NF (number of fields) could come in handy? Since we don't have access to your input data it's hard to say though, but another possibility.
You can directly use the variable without comparison, an empty/null/zero value is considered false, everything else is true.
See here :
# setting default tag if not provided
if (! tag) {
tag="default-tag"
}
So this script will have the variable tag with the value default-tag except if the user call it like this :
$ awk -v tag=custom-tag -f script.awk targetFile
This is true as of :
GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.4, GNU MP 6.1.0)
It works just fine for me
$ awk 'BEGIN{if(val==""){print "null or empty"}}'
null or empty
You can't differentiate between variable being empty and null, when you access "unset" variable, awk just initializes it with default value(here it is "" - empty string). You can use some sort of workaround, for example, setting val_accessed variable to 0 and then to 1 when you access it. Or more simple approach(somewhat "hackish") setting val to "unitialized"(or to some other value which can't appear when running your program).
PS: your script looks strange for me, what are the nested brackets for?
I accidentally discovered this less-used function specific in gawk that could help differentiate :
****** gawk-only ******
BEGIN {
$0 = "abc"
print NF, $0
test_function()
test_function($(NF + 1))
test_function("")
test_function($0)
}
function test_function(_) { print typeof(_) }
1 abc
untyped
unassigned
string
string
So it seems, for non-numeric-like data :
absolutely no input to function at all : untyped
non-existent or empty field, including $0 : unassigned
any non-numeric-appearing string, including "" : string
Here's the chaotic part - numeric data :
strangely enough, for absolutely identical input, only differing between using $0 vs. $1 in function call, you frequently get a different value for typeof()
even a combination of both leading and trailing spaces doesn't prevent gawk from identifying it as strnum
[123]:NF:1
$0 = number:123 $1 = strnum:123 +$1 = number:123
[ 456.33]:NF:1
$0 = string: 456.33 $1 = strnum:456.33 +$1 = number:456.33000
[ 19683 ]:NF:1
$0 = string: 19683 $1 = strnum:19683 +$1 = number:19683
[-20.08554]:NF:1
$0 = number:-20.08554 $1 = strnum:-20.08554 +$1 = number:-20.08554
+/- inf/nan (same for all 4):
[-nan]:NF:1
$0 = string:-nan $1 = strnum:-nan +$1 = number:-nan
this one is a string because it was made from sprintf() :
[0x10FFFF]:NF:1
$0 = string:0x10FFFF $1 = string:0x10FFFF +$1 = number:0
using -n / --non-decimal-data flag, all stays same except
[0x10FFFF]:NF:1
$0 = string:0x10FFFF $1 = strnum:0x10FFFF +$1 = number:1114111
Long story short, if you want your gawk function to be able to differentiate between
empty-string input (""), versus
actually no input at all
e.g. when original intention is to directly apply changes to $0
then typeof(x) == "untyped" seems to be the most reliable indicator.
It gets worse when null-string padding versus a non-empty string of all zeros ::
function __(_) { return (!_) ":" (!+_) }
function ___(_) { return (_ == "") }
function ____(_) { return (!_) ":" (!""_) }
$0--->[ "000" ] | __(""$0)-->{ !(""$0) : !+(""$0) }-->[ 0:1 ]
___($0)-->{ $0=="" }-->[ 0 ] | ____($0)-->{ ! $0 : (!""$0) }-->[ 1:1000 ]
$0--->[ "" ] | __(""$0)-->{ !(""$0) : !+(""$0) }-->[ 1:1 ]
___($0)-->{ $0=="" }-->[ 1 ] | ____($0)-->{ ! $0 : (!""$0) }-->[ 1:1 ]
$0--->[ " -0.0 -0" ] | __(""$0)-->{ !(""$0) : !+(""$0) }-->[ 0:1 ]
___($0)-->{ $0=="" }-->[ 0 ] | ____($0)-->{ ! $0 : (!""$0) }-->[ 0:1 -0.0 -0 ]
$0--->[ " 0x5" ] | __(""$0)-->{ !(""$0) : !+(""$0) }-->[ 0:1 ]
___($0)-->{ $0=="" }-->[ 0 ] | ____($0)-->{ ! $0 : (!""$0) }-->[ 0:1 0x5 ]

Resources