I'm trying to print out a field called "end" from a json file using jq but am running into the following error:
$ echo '{"start": 10, "end": 20}` > /tmp/out.json
$ jq .start /tmp/out.json
10
$ jq .end /tmp/out.json
error: syntax error, unexpected end, expecting $end
.end
^^^
1 compile error
This issue (https://github.com/stedolan/jq/issues/256) suggests using .["end"] as the selector but that doesn't seem to work either.
$ jq .["end"] /tmp/out.json
error: syntax error, unexpected end
.[end]
^^^
1 compile error
Any ideas?
This was fixed in more recent versions of jq. I can do this:
$ jq --version
jq-1.6-1-g2f2d05b
$ jq .end <<< '{"start": 10, "end": 20}'
20
Your second attempt failed because the shell removes the double quotes. You have to protect them by quoting the whole thing:
jq '.["end"]'
The relevant issue that describes your initial problem is Reserved words should not generate errors when used as object keys; the fix was in this commit, and it looks like it was in jq since version 1.5rc2.
Related
I am having problem to validate json string.
i am using below code
if jq -e . >/dev/null 2>&1 <<<"$json_string"; then
echo "Parsed JSON successfully and got something other than false/null"
else
echo "Failed to parse JSON, or got false/null"
fi
This does not work for json_string={"fruit":{"name":"app. this still shows Parsed JSON successfully and got something other than false/null where as the json string is incomplete.
Apparently it is one of the issues in jq-1.5. Un-terminated objects/arrays, without a corresponding close character, are being treated as valid objects and are accepted by the parser. Can reproduce in jq-1.5, but fixed in jq-1.6
On jq-1.6
jq -e . <<< '{"fruit":{"name":"app'
parse error: Unfinished string at EOF at line 2, column 0
echo $?
4
minimal reproducible example below, which again is handled well in 1.6 but doesn't throw an error in 1.5
jq -e . <<< '{'
parse error: Unfinished JSON term at EOF at line 2, column 0
jq -e . <<< '['
parse error: Unfinished JSON term at EOF at line 2, column 0
Suggest upgrading to jq-1.6 to make this work!
I have a json output, representing a linux command in one of it's values:
... ,"proc.cmdline":"sh -c pgrep -fl \"unicorn.* worker\[.*?\]\"", ...
In some cases, the command contains a backslash, so the outputing json will contain a backslash too.
I need to parse the output with jq, but it fails with an error:
parse error: Invalid escape at line 1, column 373
It refers to this: \[
However, this is a part of the command, so it is expected to be there.
If a manually edit the line, converting \[ to \\[, then it passes. However the resulting output contains both backslashes:
...
"proc.cmdline": "sh -c pgrep -fl \"unicorn.* worker\\[.*?\\]\"",
...
Now, I can't be there to manually edit every time. This output is produced automatically by another software, and I need to parse it with jq every time it comes in.
Also, even if I was able to edit every \[ to \\[, (like by using something like sed) the output becomes a lie, the second \ is fake.
Any ideas on how to work around this?
EDIT: here is the full json for reference (received raw by the output of the program I'm using (falco)):
{"priority":"Debug","rule":"Run shell untrusted","time":"2019-05-15T07:32:36.597411997Z", "output_fields": {"evt.time":1557905556597411997,"proc.aname[2]":"gitlab-mon","proc.aname[3]":"runsv","proc.aname[4]":"runsvdir","proc.aname[5]":"wrapper","proc.aname[6]":"docker-containe","proc.aname[7]":"docker-containe","proc.cmdline":"sh -c pgrep -fl \"unicorn.* worker\[.*?\]\"","proc.name":"sh","proc.pcmdline":"reactor.rb:249 ","proc.pname":"reactor.rb:249","user.name":null}}
JSON standard is quite explicit about which characters have to be escaped, and [ is not one of them (though reverse solidus - \ is). So it's your script / software generating JSON violates the JSON standard - you can validate it on any of well-known online JSON validators, e.g., like this one: https://jsoncompare.com/#!/simple/ - it will produce the error too.
If you cannot enhance/fix your script generating that JSON, then you'd need to ensure you double quote those non-compliant quotations before passing to JSON processor: e.g.
... | sed -E 's/\\([][])/\\\\\1/g' | ...
You'll need to fix whatever is generating that "json" string. Use something that produces compliant json.
If that's not an option for you, then you will have to modify it so that it is valid json. Fortunately jq can handle that. Read it in raw, fix the string then parse it.
Assuming we just need to fix the \[ and \] sequence:
$ ... | jq -R 'gsub("\\\\(?<c>[[\\]])"; "\\\\\(.c)") | fromjson | "your filter"'
Remember, "sh -c pgrep -fl \"unicorn.* worker\\[.*?\\]\"" is a string with escapes... it represents the value:
sh -c pgrep -fl "unicorn.* worker\[.*?\]"
So it's absolutely correct to have "both backslashes."
I am using rscript to run some expressions but I'm having an issue with some cases with dashes. A simple example would be:
$ rscript -e '-1'
ERROR: option '-e' requires a non-empty argument
Adding parenthesis works out (rscript -e (-1)) but I'm not always sure that they will be properly parenthesized.
In the documentation it says
When using -e options be aware of the quoting rules in the shell used
So I tried using different quoting rules for bash, escaping the dashes or using single quotes but it still doesn't work.
$ rscript -e "\-1"
Error: unexpected input in "\"
Execution halted
Is there something I'm missing?
You misunderstand one part here. "Expression" is something R can parse, ie:
$ R --slave -e '1+1'
[1] 2
$
What you hit with -1 is a corner case. You can do
$ R --slave -e 'a <- -1; a'
[1] -1
$
or
$ R --slave -e 'print(-1)'
[1] -1
$
For actual argument parsing do you want an package like docopt (which I like and use a lot), or getopt (which I used before) or optparse. All are on CRAN.
I actually use jq (1.5) with Windows 10 to Format different json files. I tried today to move the filters to a filter file to cut the length of my cmd commands.
I copied the filter directly from the command with all Quotations but i received an Syntax error. I tried to remove the qotations or Change them to ' but i still receive the Syntax error:
jq: error: syntax error, unexpected IDENT, expecting $end (Windows cmd shell quoting issues?) at <top-level>, line 1:
[.cruises[] | { nid: .cruise_nid, shipcategory: .ship_category, ship: .ship_title, company: .company_title, includeflight: .includes_flight, nights, waypoints: .waypoint_cities, title: .route_title}] C:\import\dreamlines_cruises.json > C:\Import\import_cruises.json
Any tips?
Regards Timo
Your jq filter as given (i.e. without quotation marks) looks fine, so let's assume you have successfully placed the text (hopefully formatted for readability :-) in a file, say format.jq
Then you would run something like this:
jq -f format.jq dreamlines_cruises.json
I am trying to write a script to output lines which fulfill a certain criteria into a new .txt file, trying to combined unix and awk
been googling but keep getting this error:syntax error near unexpected token `done'
Filename="bishan"
file="659.A"
while IFS= read line
do
cat $Filename.txt | awk '{ otherSubNo = substr($0,73,100);gsub(/
/,"",otherSubNo); if(length(otherSubNo)>8){ print "Subscriber Number is
",": ",substr($0,1,20)," Other Subscriber Number is ", " :
",substr($0,73,100) }}'| wc -l >> $Filename.txt
done <"$file"
example of 659.A is as follows:
This is the first line of the 659.a file:
6581264562 201611050021000000002239442239460000000019010000010081866368
00C0525016104677451 100C 0 0000
0111000 000000000000000000006598540021 01010000000000659619778001010000
000000659854000300000000000000000000 004700001
Please help, I have been googling about this but no avail
I was able to reproduce the specified error, albeit only with close approximation, by typing the script in notepad (windows) and testing it in cygwin.
script.sh:
while read myline
do
echo $myline
done
In ksh:
~> /usr/bin/ksh ./script.sh
: not found
./script.sh[7]: syntax error: 'done' unexpected
In bash:
~> /usr/bin/bash ./script.sh
./script.sh: line 2: $'\r': command not found
./script.sh: line 6: syntax error near unexpected token `done'
./script.sh: line 6: `done'
The said error (at least, in my case) is because of the CRLF characters. When I copy-paste the code to cygwin, the CRLF turns to LF (along with all invisible control characters that get lost), thus making the error disappear.