in AWS CLI, the command aws quicksight describe-data-set blah blah returns a json document with the following troublesome syntax:
{
"Status": 200,
"DataSet": {
"Arn": "arn:aws:quicksight:<region>:<acct>:dataset/b7c87122-e180-47a9-a8a4-19f171e13fc8",
"DataSetId": "b7c87122-e180-47a9-a8a4-19f171e13fc8",
"Name": "MyName",
"CreatedTime": "2022-08-16T12:01:54.948000-05:00",
"LastUpdatedTime": "2022-08-19T08:47:55.553000-05:00",
"PhysicalTableMap": {
"6fac5dee-3691-4ddd-ba7a-0667168bb80c": {
"CustomSql": {
"DataSourceArn": "arn:aws:quicksight:<region>:<acct>:datasource/46f83f8b-181e-4575-8d61-84c50125f3aa",
I need to address that DataSetArn, but the key "6fac5dee-3691-4ddd-ba7a-0667168bb80c" is unknown to me at runtime. How do I address it?
I tried:
jq -r '.DataSet.PhysicalTableMap.*.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.\*.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.?.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.\?.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.%.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.\%.CustomSql.DataSourceArn'
All return an error similar to:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.DataSet.PhysicalTableMap.\?.CustomSql.DataSourceArn
jq: 1 compile error
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
I'm a noob, I know I'm guessing here. Does anyone have any insight on this?
Something like this:
jq -r '.DataSet.PhysicalTableMap[].CustomSql.DataSourceArn'
The part .DataSet.PhysicalTableMap returns the object as one result, the following filter [] takes that object and returns each value as one result. The following filters take each of these separate results and refines more stuff.
Note: If the object is the top-level item then the syntax is .[] .
Related
As in Passing bash variable to jq, we should be able to use a JQ variable as $VAR in a jq expression.
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json
)
SO to extract project_id from the json file sample.json.
{
"dev": {
"gcp": {
"project_id": "forecast-dev-1234",
"project_number": "123456789",
"endpoint_id": "6837352639743655936"
}
}
}
Run the JQ expression using a variable but did not work.
# TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '.$TARGET.gcp.project_id' sample.json
-----
jq: error: syntax error, unexpected '
.$TARGET.gcp.project_id
(Unix shell quoting issues?) at <top-level>, line 1:
.$TARGET.gcp.project_id
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$TARGET.gcp.project_id
jq: 2 compile errors
Please help understand why and how to use the variable to form an expression to extract project_id.
JQ Manual does not provide clear explanation for variable and ---arg. Is there a good resource that clearly explain JQ variable and how to use it?
Another way to set the exit status is with the halt_error builtin function.
--arg name value:
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".
Workaround
Using interpolation.
$ TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '."\($TARGET)".gcp.project_id' sample_interpolation.json
-----
forecast-dev-1234
Version
jq --version
---
jq-1.64l
You're using variables fine, the problem is that object-identifier syntax doesn't allow general expressions. It's a shorthand syntax for when the key you're looking up is a fixed identifier-like string, like .foo or .project_id. As noted in the manual, you can use the more general generic object index filter for arbitrary keys including those that are calculated by some expression, such as .[$TARGET]:
$ TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '.[$TARGET].gcp.project_id' sample.json
forecast-dev-1234
❯ jq --version
jq-1.6
I'm using .jq file as a filter like following, it works:
❯ cat jq/script.jq
def fi(v):
v | tostring |
if test("\\.") then
"float"
else
"integer"
end;
def estype(v):
if type=="number" then
fi(v)
else
type
end;
def esprop(v):
if type=="object" then
{"properties": v | with_entries(.value |= esprop(.))}
else
{"type": estype(v)}
end;
with_entries(.value |= esprop(.))
❯ cat test.json | jq -f jq/script.jq
...(omit results)
But when I use it as library, it throw an error:
# comment the last filter, except the definitions of functions
❯ cat jq/script.jq
def fi(v):
v | tostring |
if test("\\.") then
"float"
else
"integer"
end;
def estype(v):
if type=="number" then
fi(v)
else
type
end;
def esprop(v):
if type=="object" then
{"properties": v | with_entries(.value |= esprop(.))}
else
{"type": estype(v)}
end;
# with_entries(.value |= esprop(.))
❯ cat test.json | jq -L jq/script.jq 'import script;'
jq: error: syntax error, unexpected IDENT, expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
import script;
jq: 1 compile error
What it means and how could I debug and fix this?
Are .jq files as a filter or a library has different syntax(doesn't seems like that)?
1a. What does it mean?
syntax error, unexpected IDENT, expecting FORMAT or QQSTRING_START
This means the parser found an identifier where it was expecting a string. (FORMAT is the token for a formatter like #csv or #text, while QQSTRING_START is the token for a string, like "script". In practice it's useless to use a formatter here since it won't let you use a non-constant string, but the parser doesn't know that.)
1b. How to debug and fix this?
Probably easiest to refer back to the manual. It says that the form expect for "import" is
import RelativePathString as NAME;
and the form expected for "include" is
include RelativePathString;
It lacks examples to make this 100% clear, but "RelativePathString" is a placeholder - it needs to be a literal string. Try one of these:
cat test.json | jq -L jq 'include "script"; with_entries(.value |= esprop(.))'
cat test.json | jq -L jq 'import "script" as script; with_entries(.value |= script::esprop(.))'
Note that the library path should be the directory containing your script, and the difference between include and import.
2. Do .jq files used as a filter or a library have a different syntax?
They use the same syntax. The problem was with the import statement, not with the script file.
I am using kislyuk/yq - The more often talked about version, which is a wrapper over jq, written in Python using the PyYAML library for YAML parsing
The version is yq 2.12.2
My jq is jq-1.6
I'm using ubuntu and bash scripts to do my parsing.
I wrote this as bash
alias=alias1
token=abc
yq -y -i ".tokens += { $alias: { value: $token }}" /root/.github.yml
I get the following error
jq: error: abc/0 is not defined at <top-level>, line 1:
.tokens += { alias1: { value: abc }}
I don't get it. Why would there be a /0 at the end?
The problem is abc is not interpreted as a literal string, when the double quotes are expanded by the shell. The underlying jq wrapper tries to match with abc as a standard built-in or a user-defined function which it was not able to resolve to, hence the error.
A JSON string (needed for jq) type needs to be quoted with ".." to be consistent with the JSON grammar. One way would be to pass the arg via command line with the --arg support
yq -y -i --arg t "$token" --arg a "$alias" '.tokens += { ($a): { value: $t } }' /root/.github.yml
Or have a quoting mess like below, which I don't recommend at all
yq -y -i '.tokens += { "'"$alias"'": { value: "'"$token"'" }}' /root/.github.yml
I am trying to query json using jq in bash scripting
file customers.json contains the following
[
{
"_id": "5968dd23fc13ae04d9000001",
"product_name": "sildenafil citrate",
"supplier": "Wisozk Inc",
"quantity": 262,
"unit_cost": "$1047"
},
{
"_id": "5968dd23fc13ae04d9000002",
"product_name": "Mountain Juniperus ashei",
"supplier": "Keebler-Hilpert",
"quantity": 292,
"unit_cost": "$874"
},
{
"_id": "5968dd23fc13ae04d9000003",
"product_name": "Dextromathorphan HBr",
"supplier": "Schmitt-Weissnat",
"quantity": 211,
"unit_cost": "$2053"
}
]
when i run following bash script
key="supplier"
value="Wisozk Inc"
jq ".[] | select(.$key==$value)" customers.json
It throws the following error
jq: error: syntax error, unexpected IDENT, expecting ';' or ')' (Unix shell quoting issues?) at <top-level>, line 1:
.[] | select(.supplier == Wisozk Inc)
jq: 1 compile error
I think the space between "Wisozk" and "Inc" is the problem, what to do?
Do not let your shell variables to be interpolated in jq under double quotes. The error is because of spaces in the value field, the variable $value undergoes word-splitting by and the filter expression of jq gets two words Wisozk and Inc instead of "Wisozk Inc".
Just pass your variables to jq's context using the --arg field and let it deal with it.
jq --arg k supplier --arg v "Wisozk Inc" 'map(select(.[$k] == $v))' json
This question is in parts duplicate to both the below questions, but not as a whole.
passing arguments to jq filter
Passing bash variable to jq
Let's pretend I'm running something like this:
jq -nr --arg target /tmp \
'(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"' \
| sh
Everything is fine unless I forgot to pass variable $target:
$ jq -nr '(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"'
jq: error: $target is not defined at <top-level>, line 1:
(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"
jq: 1 compile error
How can I catch this and use default value?
I've tried:
$target?
($target)?
try $target catch null
$target? // null
But it seems to be parsing-time error, which obviously can't be caught at runtime. Have I've missed any dynamic syntax?
I've found that command-line arguments can be found in $ARGS.name, but there are two drawbacks:
This was introduced in version 1.6, but I have 1.5 on CentOS 7.
It doesn't catch locally defined variables.
Assuming you need to do something more useful with jq than write 'Hello World' over a text file. I propose the following,
Maybe we can learn some programming tips from Jesus:
"Give to Caesar what belongs to Caesar, and give to God what belongs to God"
Suppose that Caesar is bash shell and God is jq, bash is appropriate to work and test the existence of files, directories and environment variables, jq is appropriate to process information in json format.
#!/bin/bash
dest_folder=$1
#if param1 is not given, then the default is /tmp:
if [ -z $dest_folder ]; then dest_folder=/tmp ; fi
echo destination folder: $dest_folder
#check if destination folder exists
if [ ! -d $dest_folder ]
then
echo "_err_ folder not found"
exit 1
fi
jq -nr --arg target $dest_folder '(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"' | sh
#if the file is succesfully created, return 0, if not return 1
if [ -e "$dest_folder/sample.txt" ]
then
echo "_suc_ file was created ok"
exit 0
else
echo "_err_ when creating file"
exit 1
fi
Now you can include this script as a step in a more complex batch, because it is congruent with linux style, returning 0 on success.