❯ jq --version
jq-1.6
I'm using .jq file as a filter like following, it works:
❯ cat jq/script.jq
def fi(v):
v | tostring |
if test("\\.") then
"float"
else
"integer"
end;
def estype(v):
if type=="number" then
fi(v)
else
type
end;
def esprop(v):
if type=="object" then
{"properties": v | with_entries(.value |= esprop(.))}
else
{"type": estype(v)}
end;
with_entries(.value |= esprop(.))
❯ cat test.json | jq -f jq/script.jq
...(omit results)
But when I use it as library, it throw an error:
# comment the last filter, except the definitions of functions
❯ cat jq/script.jq
def fi(v):
v | tostring |
if test("\\.") then
"float"
else
"integer"
end;
def estype(v):
if type=="number" then
fi(v)
else
type
end;
def esprop(v):
if type=="object" then
{"properties": v | with_entries(.value |= esprop(.))}
else
{"type": estype(v)}
end;
# with_entries(.value |= esprop(.))
❯ cat test.json | jq -L jq/script.jq 'import script;'
jq: error: syntax error, unexpected IDENT, expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
import script;
jq: 1 compile error
What it means and how could I debug and fix this?
Are .jq files as a filter or a library has different syntax(doesn't seems like that)?
1a. What does it mean?
syntax error, unexpected IDENT, expecting FORMAT or QQSTRING_START
This means the parser found an identifier where it was expecting a string. (FORMAT is the token for a formatter like #csv or #text, while QQSTRING_START is the token for a string, like "script". In practice it's useless to use a formatter here since it won't let you use a non-constant string, but the parser doesn't know that.)
1b. How to debug and fix this?
Probably easiest to refer back to the manual. It says that the form expect for "import" is
import RelativePathString as NAME;
and the form expected for "include" is
include RelativePathString;
It lacks examples to make this 100% clear, but "RelativePathString" is a placeholder - it needs to be a literal string. Try one of these:
cat test.json | jq -L jq 'include "script"; with_entries(.value |= esprop(.))'
cat test.json | jq -L jq 'import "script" as script; with_entries(.value |= script::esprop(.))'
Note that the library path should be the directory containing your script, and the difference between include and import.
2. Do .jq files used as a filter or a library have a different syntax?
They use the same syntax. The problem was with the import statement, not with the script file.
I have error with scan function, why?
https://jqplay.org/s/E-0qbbzRPS
I need do this without -r
There are two issues with your filter. Firstly, you need to separate parameters to a function with semicolon ;, not comma ,:
scan("([0-9A-Za-z_]+) == '([0-9A-Za-z_]+)"; "g")
Secondly, scan with two parameters is not implemented (in contradiction to the manual).
jq: error: scan/2 is not defined at <top-level>, line 1:
But as you are using scan, your regex will match multiple occurrences anyway, so you may as well just drop it :
.spec.selector | [scan("([0-9A-Za-z_]+) == '([0-9A-Za-z_]+)") | {(.[0]): .[1]}]
[
{
"app": "nginx"
}
]
Demo
I did and try but not able to rectify
opal#opal-Inspiron-15-3567:~/PRABHAT/unix$ bash valcode.sh
valcode.sh: line 5: unexpected EOF while looking for matching ``' valcode.sh: line 19: syntax error: unexpected end of file
IFS="|"
while echo "Enter deparment code:" ; do
read dcode
set -- `grep "^$dcode" <<-limit
01|accounts|6123
02 | admin | 5423
03 | marketing |6521
04 | personnel |2365
05 | production | 9876
06 | sales | 1006
limit'
case $# in
3) echo "deparment name : $2\nEmp-id of head of dept :$3\n"
shift 3 ;;
*) echo "Invalid code" ; continue
esac
done
the output is not coming as per desire
On line 4 you write `grep but the backtick ` is unmatched. Backticks always come in pairs so the interpreter keeps going looking for the match. Eventually it reached the end of the file without finding it and gives up.
Adding the matching backtick (at the end of the line?) will solve this problem.
I have a log file which logs cat and sub cat names that failed with message error. My goal is to find the most occurring categories.
e.g. log.:
Mon, 26 Nov 2018 07:51:07 +0100 | 164: [ERROR ***] Category ID not found for 'mcat-name1' 'subcat-name1' ref: '073'
Mon, 26 Nov 2018 07:51:08 +0100 | 278: [ERROR ***] Category ID not found for 'mcat-name2' 'subcat-name2' ref: '020'
Now I want to identify the top 10 categories that failed.
Using sed:
sed -e 's/\s/\n/g' < file.log | grep ERROR | sort | uniq -c | sort -nr | head -10
I am getting 1636 [ERROR
While I was looking for a list of categories sorting after amount of occurrenxe. e.g.
139 category1
23 category 2
...
You say you want to make a counting using sed, but actually, you are having an entire pipeline with sed, grep, sort, uniq and head. Generally, when this happens, your problem is screaming for awk:
awk 'BEGIN{FS="\047"; PROCINFO["sorted_in"]="#val_num_asc"}
/\[ERROR /{c[$2]++}
END{for(i in c) { print c[i],i; if(++j == 10) exit } }' file
The above solution is a GNU awk solution as it makes use of non-POSIX compliant features such as the sorting of the array traversal (PROCINFO). The field separator is set to the <single quote> (') which has octal value \047 as it assumes that the category name is between single quotes.
If you are not using GNU awk, you could use sort and head or do the sorting yourself. One way is:
awk 'BEGIN{FS="\047"; n=10 }
/\[ERROR /{ c[$2]++ }
END {
for (l in c) {
for (i=1;i<=n;++i) {
if (c[l] > c[s[i]]) {
for(j=n;j>i;--j) s[j]=s[j-1];
s[i]=l
break
}
}
}
for (i=1;i<=n;++i) {
if (s[i]=="") break
print c[s[i]], s[i]
}
}' file
or just do:
awk 'BEGIN{FS="\047"}
/\[ERROR /{c[$2]++}
END{for(i in c) { print c[i],i; if(++j == 10) exit } }' file \
| sort -nr | head -10
You got 1636 [ERROR because you change the space character into a newline character, then you grep the word ERROR, then you count.
This :
sed -e 's/\s/\n/g' < file.log | grep ERROR
Gives you this :
[ERROR
[ERROR
[ERROR
[ERROR
[ERROR
[ERROR
... (1630 more)
You need to grep first then sed (pretty sure you can do better with sed but I'm just talking about the logic behind the commands) :
grep ERROR file.log | sed -e 's/\s/\n/g' | sort | uniq -c | sort -nr | head -10
This may not be the best solution as it counts the word ERROR and other useless words, but you didn't give us a lot of information on the input file.
Assuming 'Bulgari' is an example of a category you want to extract, try
sed -n "s/.*ERROR.*\] Category '\([^']*\)'.*/\1/p" file.log |
sort | uniq -c | sort -rn | head -n 10
The sed command finds lines which match a fairly complex regular expression and captures part of the line, then replaces the match with the captured substring, and prints it (the -n option disables the default print action, so we only print the extracted lines). The rest is basically identical to what you already had.
In the regex, we look for (beginning of line followed by) anything (except a newline) followed by ERROR and later on followed by ] Category ' and then a string which doesn't contain a single quote, then the closing single quote followed by anything. The lots of "anything (except newline)" are required in order to replace the entire line with just the captured string from inside the single quotes. The backslashed parentheses are what capture an expression; google for "backref" for the full scoop.
Your original attempt would only extract the actual ERROR strings, because you replaced all the surrounding spaces with newlines (assuming vaguely that your sed accepts the Perl \s shorthand, which isn't standard in sed, and that \n gets interpreted as a literal newline in the replacement, which also isn't entirely standard or portable).
The way to go is to select the erred categories and replace the whole line with only the Category name using sed.
Give a try to this:
sed -e "s/^.* [[]ERROR .*[]] Category '\([^']*\)' .*$/\1/g" file.log | sort | uniq -c | sort -nr | head -16
^ is the start of the line
\( ... \) : the char sequence enclosed in this escaped parenthesis can be referred with \1 for the first pair appearing in the regex, \2 for the second pair etc.
$ is the end of the line.
The sed selects a line which contains [ERROR and some chars until a ], folled with the word Category, and then after the (space) char, any sequence of chars, up to the next space char, is selected with a pair of escaped parenthesis, followed with any sequence of chars up to the end of the line. If a such a line is found, it is replaced with the char sequence after Category.
Using Perl
> cat merlin.txt
Mon, 26 Nov 2018 07:51:07 +0100 | 164: [ERROR ***] Category ID not found for 'mcat-name1' 'subcat-name1' ref: '073'
Mon, 26 Nov 2018 07:51:08 +0100 | 278: [ERROR ***] Category ID not found for 'mcat-name2' 'subcat-name2' ref: '020'
Mon, 26 Nov 2018 07:51:21 +0100 | 1232: [ERROR ***] Category ID not found for 'make' 'model' ref: '228239'
> perl -ne ' { s/(.*)Category.*for(.+)ref.*/\2/g and s/(\047\S+\047)/$kv{$1}++/ge if /ERROR/} END { foreach (sort keys %kv) { print "$_ $kv{$_}\n" } } ' merlin.txt | sort -nr
'subcat-name2' 1
'subcat-name1' 1
'model' 1
'mcat-name2' 1
'mcat-name1' 1
'make' 1
>
I am very new to RNAseq analysis. I am trying to run HTSeq using Galaxy as I am unfamiliar with coding and how to do this with Python.
I get the following error:
Use of uninitialized value $input_file in concatenation (.) or string at /mnt/galaxyTools/shed_tools/toolshed.g2.bx.psu.edu/repos/fcaramia/edger/86292c2b0ba9/edger/htseq.pl line 57.
Usage: file [-bchikLlNnprsvz0] [--apple] [--mime-encoding] [--mime-type]
[-e testname] [-F separator] [-f namefile] [-m magicfiles] file ...
file -C [-m magicfiles]
file [--help]
Use of uninitialized value $input_file in concatenation (.) or string at /mnt/galaxyTools/shed_tools/toolshed.g2.bx.psu.edu/repos/fcaramia/edger/86292c2b0ba9/edger/htseq.pl line 62.
This is followed by a long list of where the error is found. For example:
Error occured when reading first line of sam file.
Error:
[Exception type: StopIteration, raised in count.py:81]
Use of uninitialized value $sample in hash element at /mnt/galaxyTools/shed_tools/toolshed.g2.bx.psu.edu/repos/fcaramia/edger/86292c2b0ba9/edger/htseq.pl line 67.
Use of uninitialized value $files[0] in string eq at /mnt/galaxyTools/shed_tools/toolshed.g2.bx.psu.edu/repos/fcaramia/edger/86292c2b0ba9/edger/htseq.pl line 78.