I'm trying to use jq to filter my results when the value contains quote literals so my data looks like:
{"key": "site=\"abc\""}
I want to filter using contains (or some other method) for where site=abc but not site=abc123
current code that gets abc and abc123:
jq -c '.textPayload | select(contains("abc"))' test.json
I attempted to try to escape using \ but it looks like it doesn't work in the contains method?
Consider:
$ echo '{"key": "site=\"abc\""}' | jq 'select(.key | contains("\"abc\""))'
{
"key": "site=\"abc\""
}
$ echo '{"key": "site=\"abc\""}' | jq 'select(.key | index("\"abc\""))'
{
"key": "site=\"abc\""
}
$ echo '{"key": "site=\"abc\""}' | jq 'select(.key | test("\"abc\""))'
{
"key": "site=\"abc\""
}
So it's unclear what the difficulty is.
Related
given this input
{
"properties": {
"hello": [
"saysay"
],
"test": [
"1"
]
},
"uri": "uri"
}
I've managed to check whether hello exists using this command
curl url | jq 'has("hello")'
Now how to check if the value of hello equals to saysay when it is a string variable ?
if hardcoded, below command return true which is expected
curl url | jq -r '.properties.hello[]=="saysay"'
when i try this one, I've got the following error : (23) Failed writing body
curl url | jq -r '.properties.$variable[]=="saysay"'
You can use --arg to set a jq variable from the command line.
field=hello
curl url | jq --arg property "$field" '.properties[$property][] == "saysay"'
I have limited experience with jq and am having an issue doing a a select contains for a string in a boolean. This is my json and am looking to get back just tdonn.
[
"user",
"admin"
]
[
[
"tdonn",
true
]
]
Here is what im trying. I have tried many different ways too.
jq -e -r '.results[] | .series[] | select(.values[] | contains("tdon"))[]'
With the sample JSON shown in a comment, the following filter would produce the result shown:
.results[] | .series[][] | flatten[] | select(contains("tdon")?)
Output with -r option
tdonn
You might like to consider:
jq '.. | strings | select(contains("tdon"))'
I was checking the jq tutorial at https://programminghistorian.org/en/lessons/json-and-jq
It makes some json reshaping, extracting some data from a json file, found at https://programminghistorian.org/assets/jq_twitter.json
At some point it makes a group_by, grouping data with the same user, extracting some user data and adding its corresponding tweet ids with the command
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]}'
so far, so good... the response looks like this (just a part is extracted):
{
"user_id": 18270633,
"user_name": "ahhthatswhy",
"user_followers": 559,
"tweet_ids": [
501064204661850100
]
}
{
"user_id": 27202261,
"user_name": "Dushan41",
"user_followers": 1201,
"tweet_ids": [
619172281751711700,
619172321564098600
]
}
{
"user_id": 2500422674,
"user_name": "pecanEgba74318",
"user_followers": 17,
"tweet_ids": [
619172331592773600
]
}
But then I would like to add a {"multiple_tweets": true} to all the objects that have more than one tweet_ids.
If I plainly pipe, like this, it works fine:
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | (select(.tweet_ids | length > 1) .multiple_tweets = true)'
a part of the result:
{
"user_id": 1653718716,
"user_name": "OAnnie8",
"user_followers": 315,
"tweet_ids": [
501064215160172540
]
}
{
"user_id": 356854246,
"user_name": "DrJLMooreIII",
"user_followers": 4888,
"tweet_ids": [
501064202904404000,
501064231387947000
],
"multiple_tweets": true
}
{
"user_id": 117155917,
"user_name": "rebekahwsm",
"user_followers": 5069,
"tweet_ids": [
501064233186893800
]
}
But if (for whatever reason, in this example is not really needed, in fact I was doing it just to understand the update-assignment) I want to use the |= operator,
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I get the error ' jq: error (at :30259): Invalid path expression with result {"user_id":1330235048,"use... '
Now the thing that I really can't understand. If instead of using the operator |= directly, I pipe through the identity operator first, it works fine.
What is the reason of this behaviour? Why does |.|= behave differently than |= ?
Why does this change anything?
jq -s '. | group_by(.user) | .[] | {user_id: .[0].user.id, user_name: .[0].user.screen_name, user_followers: .[0].user.followers_count, tweet_ids: [.[].id]} | . |= (select(.tweet_ids | length > 1) .multiple_tweets = true)'
I guess I'm still not understanding how the |= operator really works.
Thank you for your help.
JQ manual explains that behavior as follows:
The left-hand side can be any general path expression; see path().
Note that the left-hand side of |= refers to a value in .. Thus $var.foo |= . + 1 won't work as expected ($var.foo is not a valid or useful path expression in .); use $var | .foo |= . + 1 instead.
Since the underlying builtin (_modify) is implemented using setpath, getpath, and delpaths; the LHS of |= must be a valid path expression that can be represented as an array; in other words, path(LHS) must not fail. See below examples.
$ jq -n 'path(1)'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 |= . + 1'
jq: error (at <unknown>): Invalid path expression with result 1
$ jq -n '1 | path(.)'
[]
$ jq -n '1 | . |= . + 1'
2
I am trying write a shell script that will get some json from URL and parse the json and extract fields.
This is what is done so far.
#!/bin/bash
token=$(http POST :3000/signin/frontm user:='{"email": "sourav#frontm.com", "password": "Hello_789"}' | jq -r '.data.id_token')
cred=$(http POST :3000/auth provider_name:frontm token:$token user:=#/tmp/user.json | jq '{ creds: .creds, userUuid: .user.userId }')
echo $cred
access=$(jq -r "'$cred'")
echo $access
So the output from echo $cred is a json:
Eg:
{ "creds": { "accessKeyId": "ASIAJPM3RDAZXEORAQ5Q", "secretAccessK
ey": "krg5GbU6gtQV+a5pz4ChL+ECVJm+wKogjglXOqr6", "sessionToken": "Ag
oGb3JpZ2luEAYaCXVzLWVhc3QtMSKAAmhOg7fedV+sBw+8c45HL9naPjqbC0bwaBxq
mQ9Kuvnirob8KtTcsiBkJA/OfCTpYNUFaXXYfUPvbmW5UveDJd+32Cb5Ce+3lAOkkL
aZyWJgvhM1u53WNuMekhcZX7SnlCcaO4e/A9TR74qMOsVptonw5jFB5zjbEI4hFsVX
UHXtkYMYpSyG+2P2LxWRqTg4XKcg2vT+qrLtiXu3XNK70wuCe0/L4/HjjzlLvChmhe
TRs8u8ZRcJvSim/j1sLqe85Sl1qrFv/7msCaxUa3gZ3dOcfHliH64+8NHfS1tkaVkS
iM2x4wxTdZI/SafduFDvGCsltxe9p5zQD0Jb1Qe02ccqpgUIWxAAGgw3NzE5NTYwMD
EyNDciDOQZkq8t+c7WatNLHyqDBahqpQwxpGsYODIC1Db/M4+PXmuYMdYKLwjv3Df2
JeTMw2RT1h8M0IOOPvyBWetwB42HLhv5AobIMkNVSw6tpGyZC/bLMGJatptB0hVMBg
/80VnI7pTPiSjb/LG46bbwlbJevPoorCEEqMZ3MlAJ2Xt2hMmA+sHBRRvV1hlkMnS8
NW6w9xApSGrD001zdfFkmBbHw+c4vmX+TMT7Bw0bHQZ5FQSpEBOw9M5sNOIoa+G/pP
p4WoHiYfGHzaXGQe9Iac07Fy36W/WRebZapvF7TWoIpBjAV+IrQKP3ShJdBi3Oa6py
lGUQysPa3EN0AF/gDuTsdz7TDsErzzUERfQHksK495poG92YoG2/ir8yqTQtUDvshO
7U4SbFpUrozCYT6vp7++BWnpe+miIRCvjy2spqBqv2RY6lhgC6QPfS/365T+QbSTMc
R+ZNes0gX/QrEG4q1sMoxyTltL4sXS2Dz9UXywPkg78AWCOr34ii72m/67Gqe1P3KA
vBe9xF9Hem4H1WbYAqBN76ppyJyG17qK8b2/r71c8rdY+1gYcskV1vUfTQUVCGE0y2
JXKV2UMFOwoTzy6SFIGcuTeOAHiYPgTkMZ6X7hNjf56ihzBIbhSHaST8U4eNBka8j8
Y949ilJwz9QO0l1kwdb2+fQSMblHgeYvF1P8HxBSpRA28gKkkXMf73Zk27I3O2DRGb
lcXS4tKRvan4ASTi4qkdrvVwMT5mwJI4mGIJZSiMJqPxjVh5E9OicFbIOCRcbcIRDE
mj5t9EvaSbIm4ELBMuyoFjmKJmesE03uFRcHkEXkPBxhkJbQwkJeUxHll5kR1IYzvA
K2A2EiZqjkhiSJC4NRekEuM+5WowwuWw1wU=" }, "userUuid": "mugqRKHmTPxk
obBAtwTmKk" }
So basically I am stuck here .. how do i parse this json in $cred further and basically want to get access to say accessKeyId using jq further?
I wonder if the variable $cred really holds a string formated in 67 columns, but if that so, tr might help to remove the newline and extract the accessKeyId using jq:
echo "$cred" | tr -d '\n' | jq -r '.creds.accessKeyId'
I am trying to use jq 1.5 to develop a script that can take one or more user inputs that represent a key and recursively remove them from JSON input.
The JSON I am referencing is here:
https://github.com/EmersonElectricCo/fsf/blob/master/docs/Test.json
My script, which seems to work pretty well, is as follows.
def post_recurse(f):
def r:
(f | select(. != null) | r), .;
r;
def post_recurse:
post_recurse(.[]?);
(post_recurse | objects) |= del(.META_BASIC_INFO)
However, I would like to replace META_BASIC_INFO with one or more user inputs. How would I go about accomplishing this? I presume with --arg from the command line, but I am unclear on how to incorporate this into my .jq script?
I've tried replacing del(.META_BASIC_INFO) with del(.$module) and invoking with cat test.json | ./jq -f fsf_key_filter.jq --arg module META_BASIC_INFO to test but this does not work.
Any guideance on this is greatly appreciated!
ANSWER:
Based on a couple of suggestions I was able to arrive to the following that works and users JQ.
Innvocation:
cat test.json | jq --argjson delete '["META_BASIC_INFO","SCAN_YARA"]' -f fsf_module_filter.jq
Code:
def post_recurse(f):
def r:
(f | select(. != null) | r), .;
r;
def post_recurse:
post_recurse(.[]?);
(post_recurse | objects) |= reduce $delete[] as $d (.; delpaths([[ $d ]]))
It seems the name module is a keyword in 1.5 so $module will result in a syntax error. You should use a different name. There are other builtins to do recursion for you, consider using them instead of churning out your own.
$ jq '(.. | objects | select(has($a))) |= del(.[$a])' --arg a "META_BASIC_INFO" Test.json
You could also use delpaths/1. For example:
$ jq -n '{"a":1, "b": 1} | delpaths([["a"]])'
{
"b": 1
}
That is, modifying your program so that the last line reads like this:
(post_recurse | objects) |= delpaths([[ $delete ]] )
you would invoke jq like so:
$ jq --arg delete "META_BASIC_INFO" -f delete.jq input.json
(One cannot use --arg module ... as "$module" has some kind of reserved status.)
Here's a "one-line" solution using walk/1:
jq --arg d "META_BASIC_INFO" 'walk(if type == "object" then del(.[$d]) else . end)' input.json
If walk/1 is not in your jq, here is its definition:
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
If you want to recursively delete a bunch of key-value pairs, then here's one approach using --argjson:
rdelete.jq:
def rdelete(key):
walk(if type == "object" then del(.[key]) else . end);
reduce $strings[] as $s (.; rdelete($s))
Invocation:
$ jq --argjson strings '["a","b"]' -f rdelete.jq input.json