I would like to pass an argument without quotes (JQ arg has double quotes by default) since it should be used as a filter. For e.g.
propt='.properties'
final=($(jq -r -c --arg p $propt '$p' sample.json))
echo $final
sample.json
{
"type": "object",
"description": "Contains information",
"properties": {
"type": {
"description": "Type"
}
}
}
So ultimately it prints out .properties instead of the expected {"type":{"description":"Type"}}
I use a bash shell for this purpose.
Please let me know what I am doing wrong.
If I understand you correctly, you're getting sidetracked by thinking you need to set up a variable in jq, instead of just letting the shell do an expansion:
% foo='.properties'
% jq -r -c "$foo" sample.json
output:
{"type":{"description":"Type"}}
Note the double quotes on $foo to still allow the shell to expand the variable to .properties. That said you could unsafely use: jq -r -c $foo sample.json
You can't use --arg in that way. The value of a --arg is a string, not a jq filter expression. If you do --arg p .properties, then $p will contain the string ".properties", it won't be evaluated as a program. Find a different way to do what you want, perhaps by defining a function.
For example, if you prefixed your program with def p: .properties; then you could use .|p in your program in the way that you're using $p now, and it would access the .properties of whatever value is in context.
Since jq does not have an “eval” function, the appropriate way to specify a path programmatically in jq is using a JSON array in conjunction with jq’s getpath and setpath built-ins, as appropriate.
Thus in your case you could use the -—argjson command-line option to pass in the path of interest, e.g.
-—argson p '["properties"]'
and your jq program would use getpath($p).
Needless to say, this approach works for arbitrarily nested paths.
Related
As in Passing bash variable to jq, we should be able to use a JQ variable as $VAR in a jq expression.
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json
)
SO to extract project_id from the json file sample.json.
{
"dev": {
"gcp": {
"project_id": "forecast-dev-1234",
"project_number": "123456789",
"endpoint_id": "6837352639743655936"
}
}
}
Run the JQ expression using a variable but did not work.
# TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '.$TARGET.gcp.project_id' sample.json
-----
jq: error: syntax error, unexpected '
.$TARGET.gcp.project_id
(Unix shell quoting issues?) at <top-level>, line 1:
.$TARGET.gcp.project_id
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$TARGET.gcp.project_id
jq: 2 compile errors
Please help understand why and how to use the variable to form an expression to extract project_id.
JQ Manual does not provide clear explanation for variable and ---arg. Is there a good resource that clearly explain JQ variable and how to use it?
Another way to set the exit status is with the halt_error builtin function.
--arg name value:
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".
Workaround
Using interpolation.
$ TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '."\($TARGET)".gcp.project_id' sample_interpolation.json
-----
forecast-dev-1234
Version
jq --version
---
jq-1.64l
You're using variables fine, the problem is that object-identifier syntax doesn't allow general expressions. It's a shorthand syntax for when the key you're looking up is a fixed identifier-like string, like .foo or .project_id. As noted in the manual, you can use the more general generic object index filter for arbitrary keys including those that are calculated by some expression, such as .[$TARGET]:
$ TARGET=dev
$ jq -r --arg TARGET "${TARGET}" '.[$TARGET].gcp.project_id' sample.json
forecast-dev-1234
I am using kislyuk/yq - The more often talked about version, which is a wrapper over jq, written in Python using the PyYAML library for YAML parsing
The version is yq 2.12.2
My jq is jq-1.6
I'm using ubuntu and bash scripts to do my parsing.
I wrote this as bash
alias=alias1
token=abc
yq -y -i ".tokens += { $alias: { value: $token }}" /root/.github.yml
I get the following error
jq: error: abc/0 is not defined at <top-level>, line 1:
.tokens += { alias1: { value: abc }}
I don't get it. Why would there be a /0 at the end?
The problem is abc is not interpreted as a literal string, when the double quotes are expanded by the shell. The underlying jq wrapper tries to match with abc as a standard built-in or a user-defined function which it was not able to resolve to, hence the error.
A JSON string (needed for jq) type needs to be quoted with ".." to be consistent with the JSON grammar. One way would be to pass the arg via command line with the --arg support
yq -y -i --arg t "$token" --arg a "$alias" '.tokens += { ($a): { value: $t } }' /root/.github.yml
Or have a quoting mess like below, which I don't recommend at all
yq -y -i '.tokens += { "'"$alias"'": { value: "'"$token"'" }}' /root/.github.yml
The command below is returning an error (jq version: 1.6):
$ jq --arg b bar . <<< '{ "foo": $b }'
parse error: Invalid numeric literal at line 1, column 12
Expected output:
{
"foo": "bar"
}
The jq 1.6 manual describes the --arg option thusly:
--arg name value: This option passes a
value to the jq program as a predefined variable. If you run jq with
--arg foo bar, then $foo is available in
the program and has the value "bar". Note that
value will be treated as a string, so --arg foo
123 will bind $foo to "123".
Named arguments are also available to the jq program as
$ARGS.named.
My usage appears correct. What's going on here?
My variable call was not within the jq program
The here-string I'm passing into jq
{ "foo": $b }
is not "the jq program" mentioned in the manual's --arg description. The lone . was the entire program, and did not use the variable $b.
I was trying to construct JSON from scratch by passing in my pattern on stdin. Instead, I should have provided the --null-input option, and replaced the . with the pattern I was attempting to pass in.
Description of --null-input
--null-input/-n:
Don't read any input at all! Instead, the filter is run once
using null as the input. This is useful when using jq as
a simple calculator or to construct JSON data from scratch.
Here's the correct invocation:
$ jq --arg b bar --null-input '{ "foo": $b }'
{
"foo": "bar"
}
How do you pass arguments to custom zsh functions?
For instance:
function kill_port_proc(port) {
lsof -i tcp:<port interpolated here>| grep LISTEN | awk '{print $2}'
}
I'm seeing so many examples online with ZSH functions, but there barely anything on passing arguments and interpolating them.
When defining a function, you cannot specify required arguments. That's why using both the function keyword and parens () seems useless to me.
To get the passed arguments, use positional parameters.
The positional parameters provide access to the command-line arguments of a shell function, shell script, or the shell itself; [...]
The parameter n, where n is a number, is the nth positional parameter. The parameter $0 is a special case [...]
About the $0 positional parameter:
The name used to invoke the current shell, or as set by the -c command line option upon invocation.
If the FUNCTION_ARGZERO option is set, $0 is set upon entry to a shell function to the name of the function, and upon entry to a sourced script to the name of the script, and reset to its previous value when the function or script returns.
Using your example:
function kill_port_proc {
lsof -i tcp:"$1" | grep LISTEN | awk '{print $2}'
}
Personaly, I like to document the function by, at least, adding the function signature prior to the definition.
Then, I declare local parameters for each arguments and readonly parameters when I want to protect them from unexpected modification.
If the argument is mandatory, I use a special parameter expansion form:
${name?word}
${name:?word}
In the first form, if name is set, or in the second form if name is both set and non-null, then substitute its value;
otherwise, print word and exit from the shell. Interactive shells instead return to the prompt.
If word is omitted, then a standard message is printed.
How I would write your example:
# kill_port_proc <port>
function kill_port_proc {
readonly port=${1:?"The port must be specified."}
lsof -i tcp:"$port" | grep LISTEN | awk '{print $2}'
}
my_function() {
if [ $# -lt 2 ]
then
echo "Usage: $funcstack[1] <first-argument> <second-argument>"
return
fi
echo "First argument: $1"
echo "Second argument: $2"
}
Usage
$ my_function
Usage: my_function <first-argument> <second-argument>
$ my_function foo
Usage: my_function <first-argument> <second-argument>
$ my_function foo bar
First argument: foo
Second argument: bar
Sorry if this is included somewhere, looked for a good 30-60 minutes for something along these lines. I am sure I just missed something! Total jq nub!
Basically I am trying to do a pick operation that is dynamic. My thought process was to do something like this:
pickJSON() {
getSomeJSON | jq -r --arg PICK "$1" '{ $PICK }'
}
pickJSON "foo, bar"
but this produces
{ "PICK": "foo, bar" }
Is there a way to essentially ask it to expand shell-style?
Desired Result:
pickJSON() {
getSomeJSON | jq -r --arg PICK "$1" '{ $PICK }'
# perhaps something like...
# getSomeJSON | jq -r --arg PICK "$1" '{ ...$PICK }'
}
pickJSON "foo, bar"
{ "foo": "foovalue", "bar": "barvalue" }
Note that I am new to jq and i just simplified what i am doing - if the syntax is broken that is why :-D my actual implementaiton has a few pipes in there and it does work if i dont try to pick the values out of it.
After a fairly long experimentation phase trying to make this work, I finally came up with what seems like a feasible and reliable solution without the extremely unsettling flaws that could come from utilizing eval.
To better highlight the overall final solution, I am providing a bit more of the handling that I am currently working with below:
Goal
Grab a secret from AWS Secrets Manager
Parse the returned JSON, which looks like this:
{
"ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyTestDatabaseSecret-a1b2c3",
"Name": "MyTestDatabaseSecret",
"VersionId": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE",
"SecretString": "{\n \"username\":\"david\",\n \"password\":\"BnQw&XDWgaEeT9XGTT29\"\n}\n",
"VersionStages": [
"AWSPREVIOUS"
],
"CreatedDate": 1523477145.713
}
Run some modifications on the JSON string received and pick only the statically requested keys from the secret
Set and export those values as environment variables
Script
# Capture a AWS Secret from secretsmanager, parse the JSON and expand the given
# variables within it to pick them from the secret and return given portion of
# the secret requested.
# #note similar to _.pick(obj, ["foo", "bar"])
getKeysFromSecret() {
aws secretsmanager get-secret-value --secret-id "$1" \
| jq -r '.SecretString | fromjson' \
| jq -r "{ $2 }"
}
# Uses `getKeysFromSecret` to capture the requested keys from the secret
# then transforms the JSON into a string that we can read and loop through
# to set each resulting value as an exported environment variable.
#
## Transformation Flow:
# { "foo": "bar", "baz": "qux" }
# -->
# foo=bar
# baz=qux
# -->
# export foo=bar
# export baz=qux
exportVariablesInSecret() {
while IFS== read -r key value; do
if [ -n "$value" ]; then
export "${key}"="${value}";
fi
done < <(getKeysFromSecret "$1" "$2" | jq -r 'to_entries | .[] | .key + "=" + .value')
}
Example JSON
{
...othervalues
"SecretString": "{\"foo\": \"bar\", \"baz\": \"qux\"}"
}
Example Usage
exportVariablesInSecret MY_SECRET "foo, bar"
echo $foo
# bar
Some Notes / Context
This is meant to set a given set of values as variables so that we aren't just setting an entire arbitrary JSON object as variables that could possibly cause issues / shadowing if someone adds a value like "path" to a secret
A critical goal was to absolutely never use eval to prevent possible injection situations. Far too easy to inject things otherwise.
Happy to see if anyone has a nicer way of accomplishing this. I saw many people recommending the use of declare but that sets the var to the local function scope only so its essentially useless.
Thanks to #cas https://unix.stackexchange.com/a/413917/308550 for getting me on the right track!