I did and try but not able to rectify
opal#opal-Inspiron-15-3567:~/PRABHAT/unix$ bash valcode.sh
valcode.sh: line 5: unexpected EOF while looking for matching ``' valcode.sh: line 19: syntax error: unexpected end of file
IFS="|"
while echo "Enter deparment code:" ; do
read dcode
set -- `grep "^$dcode" <<-limit
01|accounts|6123
02 | admin | 5423
03 | marketing |6521
04 | personnel |2365
05 | production | 9876
06 | sales | 1006
limit'
case $# in
3) echo "deparment name : $2\nEmp-id of head of dept :$3\n"
shift 3 ;;
*) echo "Invalid code" ; continue
esac
done
the output is not coming as per desire
On line 4 you write `grep but the backtick ` is unmatched. Backticks always come in pairs so the interpreter keeps going looking for the match. Eventually it reached the end of the file without finding it and gives up.
Adding the matching backtick (at the end of the line?) will solve this problem.
Related
As the tile says, I wonder when jq exits with code 1. In its manual, it says -e sets the exit status of jq to 0 if the last output values was neither false nor null, 1 if the last output value was either false or null. Not clear what it means by the last output values of false or null? what if I don't use -e?
The following is intended to supplement the information given about the -e option elsewhere on this page.
Assuming -e has NOT been specified, the return code is:
5 if a call to error/1 or halt_error/0 causes program termination
an integral value (*) depending on N if halt_error(N) causes program termination, where N is a number; in particular, if N is non-negative, then the status is set to (N%256).
Otherwise, but still assuming -e has NOT been specified, the return code is:
2 if a parsing error reading input causes program termination
3 or 4 if a syntax error in the jq program causes program termination
0 on normal termination.
(*) Specifically:
N % 256 | if . < 0 then 256+. else . end
This answer is based on current jq 1.6 source code from https://github.com/stedolan/jq
With --exit-status (-e), there are 6 possible exit codes:
0: jq output something and last line was neither false nor null
1: last line output was false or null
2: usage problem or system error
3: jq program compile error
4: jq didn't ouput anything at all
5: unknown (unexpected) error: any error other than 2 and 3
Without --exit-status (-e), 0 just means that jq ran successfully. Additionally, exit status 1 and 4 disappear and 0 is returned instead.
Here is (Unix Bourne shell) some ways to get 1 as an exit value:
$ echo false | jq -e .
false
$ echo '{ "foo": false }' | jq -e .foo
false
$ echo null | jq -e .
null
$ echo '{ "foo": null }' | jq -e .foo
null
$ echo '{ }' | jq -e .foo
null
$ echo '{ "foo": false }' | jq -e '.bar?'
null
Here is how to get 4:
$ echo 'false' | jq -e '.foo?'
And (I'm sure you want to know) here is one way to get 5:
$ echo false | jq .foo
jq: error (at <stdin>:1): Cannot index boolean with string "foo"
I have a log file which logs cat and sub cat names that failed with message error. My goal is to find the most occurring categories.
e.g. log.:
Mon, 26 Nov 2018 07:51:07 +0100 | 164: [ERROR ***] Category ID not found for 'mcat-name1' 'subcat-name1' ref: '073'
Mon, 26 Nov 2018 07:51:08 +0100 | 278: [ERROR ***] Category ID not found for 'mcat-name2' 'subcat-name2' ref: '020'
Now I want to identify the top 10 categories that failed.
Using sed:
sed -e 's/\s/\n/g' < file.log | grep ERROR | sort | uniq -c | sort -nr | head -10
I am getting 1636 [ERROR
While I was looking for a list of categories sorting after amount of occurrenxe. e.g.
139 category1
23 category 2
...
You say you want to make a counting using sed, but actually, you are having an entire pipeline with sed, grep, sort, uniq and head. Generally, when this happens, your problem is screaming for awk:
awk 'BEGIN{FS="\047"; PROCINFO["sorted_in"]="#val_num_asc"}
/\[ERROR /{c[$2]++}
END{for(i in c) { print c[i],i; if(++j == 10) exit } }' file
The above solution is a GNU awk solution as it makes use of non-POSIX compliant features such as the sorting of the array traversal (PROCINFO). The field separator is set to the <single quote> (') which has octal value \047 as it assumes that the category name is between single quotes.
If you are not using GNU awk, you could use sort and head or do the sorting yourself. One way is:
awk 'BEGIN{FS="\047"; n=10 }
/\[ERROR /{ c[$2]++ }
END {
for (l in c) {
for (i=1;i<=n;++i) {
if (c[l] > c[s[i]]) {
for(j=n;j>i;--j) s[j]=s[j-1];
s[i]=l
break
}
}
}
for (i=1;i<=n;++i) {
if (s[i]=="") break
print c[s[i]], s[i]
}
}' file
or just do:
awk 'BEGIN{FS="\047"}
/\[ERROR /{c[$2]++}
END{for(i in c) { print c[i],i; if(++j == 10) exit } }' file \
| sort -nr | head -10
You got 1636 [ERROR because you change the space character into a newline character, then you grep the word ERROR, then you count.
This :
sed -e 's/\s/\n/g' < file.log | grep ERROR
Gives you this :
[ERROR
[ERROR
[ERROR
[ERROR
[ERROR
[ERROR
... (1630 more)
You need to grep first then sed (pretty sure you can do better with sed but I'm just talking about the logic behind the commands) :
grep ERROR file.log | sed -e 's/\s/\n/g' | sort | uniq -c | sort -nr | head -10
This may not be the best solution as it counts the word ERROR and other useless words, but you didn't give us a lot of information on the input file.
Assuming 'Bulgari' is an example of a category you want to extract, try
sed -n "s/.*ERROR.*\] Category '\([^']*\)'.*/\1/p" file.log |
sort | uniq -c | sort -rn | head -n 10
The sed command finds lines which match a fairly complex regular expression and captures part of the line, then replaces the match with the captured substring, and prints it (the -n option disables the default print action, so we only print the extracted lines). The rest is basically identical to what you already had.
In the regex, we look for (beginning of line followed by) anything (except a newline) followed by ERROR and later on followed by ] Category ' and then a string which doesn't contain a single quote, then the closing single quote followed by anything. The lots of "anything (except newline)" are required in order to replace the entire line with just the captured string from inside the single quotes. The backslashed parentheses are what capture an expression; google for "backref" for the full scoop.
Your original attempt would only extract the actual ERROR strings, because you replaced all the surrounding spaces with newlines (assuming vaguely that your sed accepts the Perl \s shorthand, which isn't standard in sed, and that \n gets interpreted as a literal newline in the replacement, which also isn't entirely standard or portable).
The way to go is to select the erred categories and replace the whole line with only the Category name using sed.
Give a try to this:
sed -e "s/^.* [[]ERROR .*[]] Category '\([^']*\)' .*$/\1/g" file.log | sort | uniq -c | sort -nr | head -16
^ is the start of the line
\( ... \) : the char sequence enclosed in this escaped parenthesis can be referred with \1 for the first pair appearing in the regex, \2 for the second pair etc.
$ is the end of the line.
The sed selects a line which contains [ERROR and some chars until a ], folled with the word Category, and then after the (space) char, any sequence of chars, up to the next space char, is selected with a pair of escaped parenthesis, followed with any sequence of chars up to the end of the line. If a such a line is found, it is replaced with the char sequence after Category.
Using Perl
> cat merlin.txt
Mon, 26 Nov 2018 07:51:07 +0100 | 164: [ERROR ***] Category ID not found for 'mcat-name1' 'subcat-name1' ref: '073'
Mon, 26 Nov 2018 07:51:08 +0100 | 278: [ERROR ***] Category ID not found for 'mcat-name2' 'subcat-name2' ref: '020'
Mon, 26 Nov 2018 07:51:21 +0100 | 1232: [ERROR ***] Category ID not found for 'make' 'model' ref: '228239'
> perl -ne ' { s/(.*)Category.*for(.+)ref.*/\2/g and s/(\047\S+\047)/$kv{$1}++/ge if /ERROR/} END { foreach (sort keys %kv) { print "$_ $kv{$_}\n" } } ' merlin.txt | sort -nr
'subcat-name2' 1
'subcat-name1' 1
'model' 1
'mcat-name2' 1
'mcat-name1' 1
'make' 1
>
I am using jq to search for specific results in a large file. I do not care for duplicate entries matching this specific condition, and it takes a while to process the whole file. What I would like to do is print some details about the first match and then terminate the jq command on the file to save time.
I.e.
jq '. | if ... then "print something; exit jq" else ... end'
I looked into http://stedolan.github.io/jq/manual/?#Breakingoutofcontrolstructures but this didn't quite seem to apply
EDIT:
The file I am parsing contains multiple json objects, one after another. They are not in an array.
Here is an approach which uses a recent version of first/1 (currently in master)
def first(g): label $out | g | ., break $out;
first(inputs | if .=="100" then . else empty end)
Example:
$ seq 1000000000 | jq -M -Rn -f filter.jq
Output (followed by immediate termination)
"100"
Here I use seq in lieu of a large JSON dataset.
To do what is requested is possible using features that were added after the release of jq 1.4. The following uses foreach and inputs:
label $top
| foreach inputs as $line
# state: true means found; false means not yet found
(false;
if . then break $top
else if $line | tostring | test("goodbye") then true else false end
end;
if . then $line else empty end
)
Example:
$ cat << EOF | jq -n -f exit.jq
1
"goodbye"
3
4
EOF
Result:
"goodbye"
You can accomplish it using halt and the inputs builtin:
jq -n 'inputs | if ... then "something", halt else ... end'
Will print "something" and terminate gracefully when the condition matches.
For this to work (i.e. terminate when condition is true), jq needs the -n parameter. See this issue
I am using UNIX Korn shell. I am trying to create a program in Unix that will search a text file which is shown below:
Last Name:First Name:City:State:Class:Semester Enrolled:Year First Enrolled
Gilman:Randy:Manhattan:KS:Junior:Spring:2010
Denton:Jacob:Rochester:NY:Senoir:Fall:2009
Goodman:Joshua:Warren:MI:Freshman:Summer:2014
Davidson:Blair:Snohomish:WA:Sophmore:Fall:2013
Anderson:Neo:Seattle:WA:Senoir:Spring:2008
Beckman:John:Ft. Polk:LA:Freshman:Spring:2014
Then take that line and cut it out and display it vertically. So if I search Gilman. It would produce:
Gilman
Randy
Manhattan
KS
Junior
Spring
2010
However included in this I should also be able to produce the following layout:
Last Name: Gilman
First Name: Randy
City: Manhattan
State: KS
Class: Junior
Semester Enrolled: Spring
Year First Enrolled: 2010
Now I have figured out most of it which I will display in my code below:
cat<<MENULIST
A - Add Student Information
D - Delete Student Information
M - Modify Student Information
I - Inquiry on a Student
X - Exit
MENULIST
echo -en '\n'
echo -en '\n'
echo " Pleasa choose one of the following: "
#take input from operation
read choice
case $choice in
a|A) ;;
d|D) ;;
m|M) ;;
i|I)
#Create Inguiry Message
clear
echo " Record Inquiry "
echo -en '\n'
echo -en '\n'
echo "What is the last name of the person"
#Gather search parameter
read last_name
grep -i "$last_name" $1 | cut -f 1-7 -d ':' ;;
x|X) echo "The program is ending" ; exit 0;;
*) echo -en '\n' ;echo "Not an acceptable entry." ;echo "Please press enter to try again"; read JUNK | tee -a $2 ; continue
esac
I am just really having trouble on redirecting input to the proper output. There is much more to the program but this is the relevant portion for this problem. Any help would be appreciated.
EDIT:
Ok the final answer is shown below:
while IFS=: read c1 c2 c3 c4 c5 c6 c7 rest ; do
case "$c1" in
"$last_name" )
echo -e "Last Name: $c1\nFirst Name: $c2\nCity: $c3\nState: $c4\nClass: $c5\nSemester Enrolled: $c6\nYear First Enrolled: $c7\n\n"
;;
esac
done < $1
This performed the intended outcome correctly thank you everyone for your help. I still have some other questions but I will make a new topic for that so it doesn't get to jumbled together.
Replace your line
grep -i "$last_name" $1 | cut -f 1-7 -d ':' ;;
with
awk -F: -vnameMatch="$last_name" \
'$1==nameMatch{
printf("LastName:%s\nFirstName:%s\nCity:%s\nState:%s\nClass:%s\nSemester Enrolled:%s\nYear First Enrolled:%s\n\n", \
$1, $2, $3, $4, $5, $6, $7)
}' $1
;;
It's pretty much the same idea in ksh.
while IFS=: read c1 c2 c3 c4 c5 c6 c7 rest ; do
case "$c1" in
"$last_name" )
printf "LastName:%s\nFirstName:%s\nCity:%s\nState:%s\nClass:%s\nSemester Enrolled:%s\nYear First Enrolled:%s\n\n",
"$c1" "$c2" "$c3" "$c4" "$c5" "$c6" "$c7"
;;
esac
done < $1
I think I've got all the syntax right, but don't have the energy tonight to test :-/ ... If you can use this, and there are problems, post a comment and I'll clean it up.
IHTH.
EDIT:
#Case statement for conducting an inquiry of the file
i|I)
#Create Inguiry Message
clear
echo " Record Inquiry "
echo -en '\n'
echo -en '\n'
echo "What is the last name of the person:"
#Gather search parameter
read last_name
while IFS=: read c1 c2 c3 c4 c5 c6 c7 rest ; do
case "$c1" in
"$last_name" )
printf "Last Name:%s\nFirst Name:%s\nCity:%s\nState:%s\nClass:%s\nSemester Enrolled:%s\nYear First Enrolled:%s\n\n", "$c1", "$c2", "$c3", "$c4", "$c5", "$c6", "$c7"
;;
esac
done < $2
#Case statement for ending the program
x|X) echo "The program is ending" ; exit 0;;
The error message is:
./asg7s: line 26: syntax error at line 93: `)' unexpected
Line 93 is
x|X) echo "The program is ending" ; exit 0;;
Kind of weird because I didn't mess with that part of the program so I know it has to be something in the portion I changed.
RLG
Adding something extra so I can save RLG edit without other's "approval"
This awk should do:
last_name="Gilman"
awk -F: -v name="$last_name" 'NR==1 {for(i=1;i<=NF;i++) h[i]=$i} $1==name {for(i=1;i<=NF;i++) print h[i]FS,$i}' file
Last Name: Gilman
First Name: Randy
City: Manhattan
State: KS
Class: Junior
Semester Enrolled: Spring
Year First Enrolled: 2010
It stores the first line in array h (header).
Then if it finds the pattern, print out array and data.
Here I post an alternative to a shell script, in perl:
perl -F':' -lne '
BEGIN { $name = pop; }
$. == 1 and do { #header = #F; next; };
next if m/^\s*$/;
if ( $F[0] eq $name ) {
for ( $i = 0; $i < #F; $i++ ) {
printf qq|%s: %s\n|, $header[$i], $F[$i];
}
exit 0;
}
' infile Gilman
It uses -F swith to split fields with colon, -l removes last newline character and -n opens input file and process it line by line.
The script accepts two arguments, but I extract the last one before processing assuming it's the name you want to search.
First line is saved in an array called #header and for next lines it compares first field, and in a match, prints each header followed by each field of current line and aborts the program.
It yields:
Last Name: Gilman
First Name: Randy
City: Manhattan
State: KS
Class: Junior
Semester Enrolled: Spring
Year First Enrolled: 2010
I have a file containing many columns of text, including a timestamp along the lines of Fri Jan 02 18:23 and I need to convert that date into MM/DD/YYYY HH:MM format.
I have been trying to use the standard `date' tool with awk getline to do the conversion, but I can't quite figure out how to pass the fields into the 'date' command in the format it expects (quoted with " or 's,) as getline needs the command string enclosed in quotes too.
Something like "date -d '$1 $2 $3 $4' +'%D %H:%M'" | getline var
Now that I think about it, I guess what I'm really asking is how to embed awk variables into a string.
If you're using gawk, you don't need the external date which can be expensive to call repeatedly:
awk '
BEGIN{
m=split("Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec",d,"|")
for(o=1;o<=m;o++){
months[d[o]]=sprintf("%02d",o)
}
format = "%m/%d/%Y %H:%M"
}
{
split($4,time,":")
date = (strftime("%Y") " " months[$2] " " $3 " " time[1] " " time[2] " 0")
print strftime(format, mktime(date))
}'
Thanks to ghostdog74 for the months array from this answer.
you can try this. Assuming just the date you specified is in the file
awk '
{
cmd ="date \"+%m/%d/%Y %H:%M\" -d \""$1" "$2" "$3" "$4"\""
cmd | getline var
print var
close(cmd)
}' file
output
$ ./shell.sh
01/02/2010 18:23
and if you are not using GNU tools, like if you are in Solaris for example, use nawk
nawk 'BEGIN{
m=split("Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec",d,"|")
for(o=1;o<=m;o++){
months[d[o]]=sprintf("%02d",o)
}
cmd="date +%Y"
cmd|getline yr
close(cmd)
}
{
day=$3
mth=months[$2]
print mth"/"day"/"yr" "$4
} ' file
I had a similar issue converting a date from RRDTool databases using rrdfetch but prefer one liners that I've been using since Apollo computer days.
Data looked like this:
localTemp rs1Temp rs2Temp thermostatMode
1547123400: 5.2788174937e+00 4.7788174937e+00 -8.7777777778e+00 2.0000000000e+00
1547123460: 5.1687014581e+00 4.7777777778e+00 -8.7777777778e+00 2.0000000000e+00
One liner:
rrdtool fetch -s -14400 thermostatDaily.rrd MAX | sed s/://g | awk '{print "echo ""\`date -r" $1,"\`" " " $2 }' | sh
Result:
Thu Jan 10 07:25:00 EST 2019 5.3373432378e+00
Thu Jan 10 07:26:00 EST 2019 5.2788174937e+00
On the face of it this doesn't look very efficient to me but this kind of methodology has always proven to be fairly low overhead under most circumstances even for very large files on very low power computer (like 25Mhz NeXT Machines). Yes Mhz.
Sed deletes the colon, awk is used to print the other various commands of interest including just echoing the awk variables and sh or bash executes the resulting string.
For methodology or large files or streams I just head the first few lines and gradually build up the one liner. Throw away code.