As in Passing bash variable to jq, we should be able to use a JQ variable as $VAR in a jq expression.
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json
)
SO to extract project_id from the json file sample.json.
{
"dev": {
"gcp": {
"project_id": "forecast-dev-1234",
"project_number": "123456789",
"endpoint_id": "6837352639743655936"
}
}
}
Run the JQ expression using a variable but did not work.
# TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '.$TARGET.gcp.project_id' sample.json
-----
jq: error: syntax error, unexpected '
.$TARGET.gcp.project_id
(Unix shell quoting issues?) at <top-level>, line 1:
.$TARGET.gcp.project_id
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$TARGET.gcp.project_id
jq: 2 compile errors
Please help understand why and how to use the variable to form an expression to extract project_id.
JQ Manual does not provide clear explanation for variable and ---arg. Is there a good resource that clearly explain JQ variable and how to use it?
Another way to set the exit status is with the halt_error builtin function.
--arg name value:
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".
Workaround
Using interpolation.
$ TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '."\($TARGET)".gcp.project_id' sample_interpolation.json
-----
forecast-dev-1234
Version
jq --version
---
jq-1.64l
You're using variables fine, the problem is that object-identifier syntax doesn't allow general expressions. It's a shorthand syntax for when the key you're looking up is a fixed identifier-like string, like .foo or .project_id. As noted in the manual, you can use the more general generic object index filter for arbitrary keys including those that are calculated by some expression, such as .[$TARGET]:
$ TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '.[$TARGET].gcp.project_id' sample.json
forecast-dev-1234
Related
in AWS CLI, the command aws quicksight describe-data-set blah blah returns a json document with the following troublesome syntax:
{
"Status": 200,
"DataSet": {
"Arn": "arn:aws:quicksight:<region>:<acct>:dataset/b7c87122-e180-47a9-a8a4-19f171e13fc8",
"DataSetId": "b7c87122-e180-47a9-a8a4-19f171e13fc8",
"Name": "MyName",
"CreatedTime": "2022-08-16T12:01:54.948000-05:00",
"LastUpdatedTime": "2022-08-19T08:47:55.553000-05:00",
"PhysicalTableMap": {
"6fac5dee-3691-4ddd-ba7a-0667168bb80c": {
"CustomSql": {
"DataSourceArn": "arn:aws:quicksight:<region>:<acct>:datasource/46f83f8b-181e-4575-8d61-84c50125f3aa",
I need to address that DataSetArn, but the key "6fac5dee-3691-4ddd-ba7a-0667168bb80c" is unknown to me at runtime. How do I address it?
I tried:
jq -r '.DataSet.PhysicalTableMap.*.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.\*.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.?.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.\?.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.%.CustomSql.DataSourceArn'
jq -r '.DataSet.PhysicalTableMap.\%.CustomSql.DataSourceArn'
All return an error similar to:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.DataSet.PhysicalTableMap.\?.CustomSql.DataSourceArn
jq: 1 compile error
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
I'm a noob, I know I'm guessing here. Does anyone have any insight on this?
Something like this:
jq -r '.DataSet.PhysicalTableMap[].CustomSql.DataSourceArn'
The part .DataSet.PhysicalTableMap returns the object as one result, the following filter [] takes that object and returns each value as one result. The following filters take each of these separate results and refines more stuff.
Note: If the object is the top-level item then the syntax is .[] .
❯ jq --version
jq-1.6
I'm using .jq file as a filter like following, it works:
❯ cat jq/script.jq
def fi(v):
v | tostring |
if test("\\.") then
"float"
else
"integer"
end;
def estype(v):
if type=="number" then
fi(v)
else
type
end;
def esprop(v):
if type=="object" then
{"properties": v | with_entries(.value |= esprop(.))}
else
{"type": estype(v)}
end;
with_entries(.value |= esprop(.))
❯ cat test.json | jq -f jq/script.jq
...(omit results)
But when I use it as library, it throw an error:
# comment the last filter, except the definitions of functions
❯ cat jq/script.jq
def fi(v):
v | tostring |
if test("\\.") then
"float"
else
"integer"
end;
def estype(v):
if type=="number" then
fi(v)
else
type
end;
def esprop(v):
if type=="object" then
{"properties": v | with_entries(.value |= esprop(.))}
else
{"type": estype(v)}
end;
# with_entries(.value |= esprop(.))
❯ cat test.json | jq -L jq/script.jq 'import script;'
jq: error: syntax error, unexpected IDENT, expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
import script;
jq: 1 compile error
What it means and how could I debug and fix this?
Are .jq files as a filter or a library has different syntax(doesn't seems like that)?
1a. What does it mean?
syntax error, unexpected IDENT, expecting FORMAT or QQSTRING_START
This means the parser found an identifier where it was expecting a string. (FORMAT is the token for a formatter like #csv or #text, while QQSTRING_START is the token for a string, like "script". In practice it's useless to use a formatter here since it won't let you use a non-constant string, but the parser doesn't know that.)
1b. How to debug and fix this?
Probably easiest to refer back to the manual. It says that the form expect for "import" is
import RelativePathString as NAME;
and the form expected for "include" is
include RelativePathString;
It lacks examples to make this 100% clear, but "RelativePathString" is a placeholder - it needs to be a literal string. Try one of these:
cat test.json | jq -L jq 'include "script"; with_entries(.value |= esprop(.))'
cat test.json | jq -L jq 'import "script" as script; with_entries(.value |= script::esprop(.))'
Note that the library path should be the directory containing your script, and the difference between include and import.
2. Do .jq files used as a filter or a library have a different syntax?
They use the same syntax. The problem was with the import statement, not with the script file.
I am using kislyuk/yq - The more often talked about version, which is a wrapper over jq, written in Python using the PyYAML library for YAML parsing
The version is yq 2.12.2
My jq is jq-1.6
I'm using ubuntu and bash scripts to do my parsing.
I wrote this as bash
alias=alias1
token=abc
yq -y -i ".tokens += { $alias: { value: $token }}" /root/.github.yml
I get the following error
jq: error: abc/0 is not defined at <top-level>, line 1:
.tokens += { alias1: { value: abc }}
I don't get it. Why would there be a /0 at the end?
The problem is abc is not interpreted as a literal string, when the double quotes are expanded by the shell. The underlying jq wrapper tries to match with abc as a standard built-in or a user-defined function which it was not able to resolve to, hence the error.
A JSON string (needed for jq) type needs to be quoted with ".." to be consistent with the JSON grammar. One way would be to pass the arg via command line with the --arg support
yq -y -i --arg t "$token" --arg a "$alias" '.tokens += { ($a): { value: $t } }' /root/.github.yml
Or have a quoting mess like below, which I don't recommend at all
yq -y -i '.tokens += { "'"$alias"'": { value: "'"$token"'" }}' /root/.github.yml
The command below is returning an error (jq version: 1.6):
$ jq --arg b bar . <<< '{ "foo": $b }'
parse error: Invalid numeric literal at line 1, column 12
Expected output:
{
"foo": "bar"
}
The jq 1.6 manual describes the --arg option thusly:
--arg name value: This option passes a
value to the jq program as a predefined variable. If you run jq with
--arg foo bar, then $foo is available in
the program and has the value "bar". Note that
value will be treated as a string, so --arg foo
123 will bind $foo to "123".
Named arguments are also available to the jq program as
$ARGS.named.
My usage appears correct. What's going on here?
My variable call was not within the jq program
The here-string I'm passing into jq
{ "foo": $b }
is not "the jq program" mentioned in the manual's --arg description. The lone . was the entire program, and did not use the variable $b.
I was trying to construct JSON from scratch by passing in my pattern on stdin. Instead, I should have provided the --null-input option, and replaced the . with the pattern I was attempting to pass in.
Description of --null-input
--null-input/-n:
Don't read any input at all! Instead, the filter is run once
using null as the input. This is useful when using jq as
a simple calculator or to construct JSON data from scratch.
Here's the correct invocation:
$ jq --arg b bar --null-input '{ "foo": $b }'
{
"foo": "bar"
}
I would like to pass an argument without quotes (JQ arg has double quotes by default) since it should be used as a filter. For e.g.
propt='.properties'
final=($(jq -r -c --arg p $propt '$p' sample.json))
echo $final
sample.json
{
"type": "object",
"description": "Contains information",
"properties": {
"type": {
"description": "Type"
}
}
}
So ultimately it prints out .properties instead of the expected {"type":{"description":"Type"}}
I use a bash shell for this purpose.
Please let me know what I am doing wrong.
If I understand you correctly, you're getting sidetracked by thinking you need to set up a variable in jq, instead of just letting the shell do an expansion:
% foo='.properties'
% jq -r -c "$foo" sample.json
output:
{"type":{"description":"Type"}}
Note the double quotes on $foo to still allow the shell to expand the variable to .properties. That said you could unsafely use: jq -r -c $foo sample.json
You can't use --arg in that way. The value of a --arg is a string, not a jq filter expression. If you do --arg p .properties, then $p will contain the string ".properties", it won't be evaluated as a program. Find a different way to do what you want, perhaps by defining a function.
For example, if you prefixed your program with def p: .properties; then you could use .|p in your program in the way that you're using $p now, and it would access the .properties of whatever value is in context.
Since jq does not have an “eval” function, the appropriate way to specify a path programmatically in jq is using a JSON array in conjunction with jq’s getpath and setpath built-ins, as appropriate.
Thus in your case you could use the -—argjson command-line option to pass in the path of interest, e.g.
-—argson p '["properties"]'
and your jq program would use getpath($p).
Needless to say, this approach works for arbitrarily nested paths.