unix script run search in parallel - unix

I have a script that searches files for a phrase in a number of different folders, then shows to output.
The trouble is, it does each search sequentially and takes a long time. I would like to make the searches run without waiting for the previous one to finish.
zipped_folders=("/extlogs/archive/rsyslog/folder1/"
"/extlogs/archive/rsyslog/folder2/")
folders=("/extlogs/rsyslog/Folder1/"
"/extlogs/rsyslog/Folder2/")
portal=0
mobile=0
email=0
if [ "$#" -ne 1 ]; then
echo "Incorrect Argument: logcount 201602"
exit 1
fi
for i in "${zipped_folders[#]}"
do
#echo $i"syslog-"$1*".log.gz"
((portal+=$(nohup gunzip -c $i"syslog-"$1*".log.gz" | grep -i "search1" | grep -v "Search1" | wc -l &)))
((mobile+=$(nohup gunzip -c $i"syslog-"$1*".log.gz" | grep -i "Search2" | wc -l &)))
((email+=$(nohup gunzip -c $i"syslog-"$1*".log.gz" | grep -i "search3" | grep -v "ActiveSync" | wc -l &)))
done
for i in "${folders[#]}"
do
((portal+=$(nohup cat $i"syslog-"$1*".log"| grep -i "search4"| grep -v "exsearch4" | wc -l &)))
((mobile+=$(nohup cat $i"syslog-"$1*".log" | grep -i "search5" | wc -l &)))
((email+=$(nohup cat $i"syslog-"$1*".log" | grep -i "search6" | grep -v "ActiveSync" | wc -l &)))
done
echo "Portal: " $portal
echo "Mobile: " $mobile
echo "Email: " $email
exit 1


You can use xargs.
find ${topdir} -name '*.gz' | xargs -n1 -P${PARALLEL_JOBS} -I {} bash -c "/usr/bin/grep 'criteria' {}"

Related

Not able to read file content with sed command

I am trying to read the below file line by line to perform the below operations
Extract the name of the file/directory alone and assign it one variable,
Extract the permission available in the line and add comma between the permission. Then assign it to another variable,
At last applying setfacl logic as shown in the output section.
File
# file: /disk1/script_1/ user::rwx group::r-x group:service:r-x mask::r-x other::r-x
# file: /disk1/script_1//hello.txt user::rw- group::r-- other::r--
# file: /disk1/script_1//bkp_10.txt user::rwx group::r-x other::r-x
Code
input="bkp_23.txt"
while IFS= read -r line;
do
echo $line
file_name=`sed -e 's/# file:\(.*\)/\1/g' "$line" | awk '{print $1}'`
echo $file_name
file_perm=`sed -e 's/# file:\(.*\)/\1/g' "$line" | awk '{$1=""}{print}' | tr ' ' ',' | awk
'{sub(",","")}1'`
echo $file_perm
echo "setfacl -m "$file_perm" "$file_name" executing"
done <"$input"
Output
setfacl -m user::rwx,group::r-x,group:service:r-x,mask::r-x,other::r-x /disk1/script_1/
setfacl -m user::rw-,group::r--,other::r-- /disk1/script_1//hello.txt
setfacl -m user::rwx,group::r-x,other::r-x /disk1/script_1//bkp_10.txt
Error
sed: can't read # file: /disk1/script_1/ user::rwx group::r-x group:service:r-x mask::r-x other::r-x: No such file or directory

$ cat input
# file: /disk1/script_1/ user::rwx group::r-x group:service:r-x mask::r-x other::r-x
# file: /disk1/script_1//hello.txt user::rw- group::r-- other::r--
# file: /disk1/script_1//bkp_10.txt user::rwx group::r-x other::r-x
$ while read _ _ path perms; do perms="$(echo "$perms" | tr -s ' ' ,)"; echo path="$path", perms="$perms"; done < input
path=/disk1/script_1/, perms=user::rwx,group::r-x,group:service:r-x,mask::r-x,other::r-x
path=/disk1/script_1//hello.txt, perms=user::rw-,group::r--,other::r--
path=/disk1/script_1//bkp_10.txt, perms=user::rwx,group::r-x,other::r-x

Try to echo the line content along with sed logic like this
file_name=$(echo "$line" | sed 's/# file:\(.*\)/\1/g' | awk '{print $1}')
file_perm=$(echo "$line" | sed -e 's/# file:\(.*\)/\1/g' | awk '{$1=""}{print}' | tr ' ' ',' | awk '{sub(",","")}1')

While loop skips the first line in output

I'm using the below command in Terminal on a Mac to read a file of email addresses and convert them to a MD5 hash.
tr -d " " < em.txt | tr '[:upper:]' '[:lower:]' | while read line; do
(echo -n $line | md5); done | awk '{print $1}' > hashes1.txt
This produces a file of hashes that are 1 row shorter than the original input file. But I can't figure out why.
This code does a few things, below.
Converts an email address to all lower case
Converts the email address to a MD5 Hash
Outputs a list of new email addresses to a hashes1.txt file
Thanks in advance!

Your tr command is wrong : it should be :
tr -d " " < em.txt |
tr '[[:upper:]]' '[[:lower:]]' |
while IFS= read -r line; do
echo -n "$line" | md5 | awk '{print $1}' >> hashes1.txt
done
or
while IFS= read -r line; do
echo -n "$line" | md5 | awk '{print $1}' >> hashes1.txt
done < <(tr -d " " < em.txt | tr '[[:upper:]]' '[[:lower:]]')
Changed the file feeding place too.
And ensure your file don't have strange characters with
od -c file
if yes, install dos2unix, then :
dos2unix file
or using perl :
perl -i -pe 's/\r//g' file

how to improve the below space issue coding

while i ran the below script it returns nothing... expect the prompt as the output..
#!/bin/sh
cd /dla
op=`df -k /dla/ |awk '{print $5}' |grep '%' |cut -d '%' -f1`
if[ $op -ge 80 ]
then
echo ' dla is more tha 80% - Purge started'
find /dla/ -name '20[0-9][0-9]*' -type d -print |sort |cut -d '/' -f7 |grep 20 | sort | uniq -c |head -20 > /tmp/file_list.dat
for i in cat /tmp/file_list.dat |awk '{print $#}'
do
find /dla -name $i -type d -exec rm -rf [] \;
newop='df -k /dla/ |awk '{print $5}' |grep '%' |cut -d '%' -f1'
if [$newop -le 80]
then
echo 'dla is 80% -Purge stopped'
exit 0
fi
done
else
echo 'dla is less than 80% - No Purge Required'
fi

Error while running command from cron (tcsh)

When I'm running this command from shell(tcsh), it executes perfectly-
cal | tail -6 | sed -e 's/^.\{3\}//' -e 's/.\{3\}$//' | tr -s '[:blank:]' '\n' | head -21 | tail -20 | tr -s '\n' ' ' | grep -w `date "+%e"` ; /usr/bin/bash -lc "if [ "$?" == 0 ] ; then echo xyz ; fi"
But when I put the exact same thing in a crontab, I get this error mail from my machine-
Subject: Output from "cron" command
Content-Length: 244
Your "cron" job on uatserver
cal | tail -6 | sed -e 's/^.\{3\}//' -e 's/.\{3\}$//' | tr -s '[:blank:]' '\n' | head -21 | tail -20 | tr -s '\n' ' ' | grep -w `date "+
produced the following output:
Usage: grep -hblcnsviw pattern file . . .
I'm sure that even my crontab commands are executed using tcsh as it is set to be the default.
p.s- My machine:
SunOS uatserver 5.10 Generic_127112-11 i86pc i386 i86pc

Your problem is that the PATH variable is not the same. Solaris has different flavors of grep
examples:
/usr/bin/grep
/usr/xpg4/bin/grep
You crontab ran /usr/bin/grep instead of /usr/xpg4/bin/grep. The two versions of grep have some different options.

Unix script to delete file if it contains single line

Consider I have a file abcde.txt which may contain one or more lines of text. I want a script that will DELETE the file if it contains single line.
Something like, if 'wc -l abscde.txt' = 1 then rm abscde.txt
My system : Solaris

Here's a simple bash script:
#!/bin/bash
LINECOUNT=`wc -l abscde.txt | cut -f1 -d' '`
if [[ $LINECOUNT == 1 ]]; then
rm -f abscde.txt
fi

delifsingleline () {
if [ $(cat $1 | wc -l) = "1" ]
then
echo "Deleting $1"
echo "rm $1"
fi
}
Lightly tested on zsh. Should work on bash as well.

This is (mostly) just a reformat of Ben's answer:
wc -l $PATH | grep '^1 ' > /dev/null && rm -f $PATH

Resources