Extract arguments - unix

I am working on automating CUPS (Create user provided services) in Cloud Foundry. I have a cups.sh file which contains the corresponding cf cups commands to be executed for a particular application. Below is sample:
cf cups service-A -p '{"uri": "https://sample uri"}'
cf cups service-B -p '{"uri": "https://sample uri","id": "abcd","token": "xyz"}'
I am trying to write a script will perform below use case:
Parse the cups.sh, line by line and extract the service name (e.g. service-A) and the argument following -p (e.g: '{"uri": "https://sample uri","id": "abcd","token": "xyz"}').
I am currently using below script:
File=cups.sh
sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' "$File" | while read line
do
temp=$(echo $line | cut -d' ' -f4)
echo $temp
done
This is not accurate as it returns me "https://sample uri"}'. Is there a more accurate way of extracting the argument -p and using it for further operations?

Extract what we need:
$> sed -n -r '/^\s*cf\s+cups\s+.+.*-p\s+'"'"'.+'"'"'/{s/\s*cf\s+cups\s+(.+)\s.*-p\s+('"'"'.+'"'"').*$/\1 \2/;p}' cups.sh
service-A '{"uri": "https://sample uri"}'
service-B '{"uri": "https://sample uri","id": "abcd","token": "xyz"}'
Now, form commands to push all results into 2 arrays (_srv and _cmd):
$> sed -n -r '/^\s*cf\s+cups\s+.+.*-p\s+'"'"'.+'"'"'/{s/\s*cf\s+cups\s+(.+)\s.*-p\s+('"'"'.+'"'"').*$/_srv+=('"'"'\1'"'"') _cmd+=(\2)/;p}' cups.sh
_srv+=('service-A') _cmd+=('{"uri": "https://sample uri"}')
_srv+=('service-B') _cmd+=('{"uri": "https://sample uri","id": "abcd","token": "xyz"}')
Finally, put everything in a bash file
#!/bin/bash
_fil=cups.sh
_srv=()
_cmd=()
eval `sed -n -r '/^\s*cf\s+cups\s+.+.*-p\s+'"'"'.+'"'"'/{s/\s*cf\s+cups\s+(.+)\s.*-p\s+('"'"'.+'"'"').*$/_srv+=('"'"'\1'"'"') _cmd+=(\2)/;p}' "$_fil"`
# test
_len=${#_srv[#]}
for (( i=-1;++i<_len; )); do
echo ${_srv[$i]} ${_cmd[$i]}
done
^\s*cf\s+cups\s+: search for lines starting with cf cups ...
\s+(.+)\s: extract the third column (delimited by \s (spaces)) as \1
-p\s+('"'"'.+'"'"'): extract the stuff between '' right after -p as \2 ('' included)

Is this what you are looking for?
$ # only lines with successful substitutions will be printed
$ sed -n 's/.*service-A -p //p' ip.txt
'{"uri": "https://sample uri"}'
$ sed -n 's/.*service-B -p //p' ip.txt
'{"uri": "https://sample uri","id": "abcd","token": "xyz"}'
$ # to save results in variable
$ a=$(sed -n 's/.*service-A -p //p' ip.txt)
$ echo "$a"
'{"uri": "https://sample uri"}'
With awk
$ awk -F"'" -v sq="'" '/service-A/{print sq $2 sq}' ip.txt
'{"uri": "https://sample uri"}'
To pass search term as variable
$ st='service-A'
$ sed -n 's/.*'"$st"' -p //p' ip.txt
'{"uri": "https://sample uri"}'
$ awk -F"'" -v s="$st" -v sq="'" '$0 ~ s{print sq $2 sq}' ip.txt
'{"uri": "https://sample uri"}'

Related

Not able to read file content with sed command

I am trying to read the below file line by line to perform the below operations
Extract the name of the file/directory alone and assign it one variable,
Extract the permission available in the line and add comma between the permission. Then assign it to another variable,
At last applying setfacl logic as shown in the output section.
File
# file: /disk1/script_1/ user::rwx group::r-x group:service:r-x mask::r-x other::r-x
# file: /disk1/script_1//hello.txt user::rw- group::r-- other::r--
# file: /disk1/script_1//bkp_10.txt user::rwx group::r-x other::r-x
Code
input="bkp_23.txt"
while IFS= read -r line;
do
echo $line
file_name=`sed -e 's/# file:\(.*\)/\1/g' "$line" | awk '{print $1}'`
echo $file_name
file_perm=`sed -e 's/# file:\(.*\)/\1/g' "$line" | awk '{$1=""}{print}' | tr ' ' ',' | awk
'{sub(",","")}1'`
echo $file_perm
echo "setfacl -m "$file_perm" "$file_name" executing"
done <"$input"
Output
setfacl -m user::rwx,group::r-x,group:service:r-x,mask::r-x,other::r-x /disk1/script_1/
setfacl -m user::rw-,group::r--,other::r-- /disk1/script_1//hello.txt
setfacl -m user::rwx,group::r-x,other::r-x /disk1/script_1//bkp_10.txt
Error
sed: can't read # file: /disk1/script_1/ user::rwx group::r-x group:service:r-x mask::r-x other::r-x: No such file or directory
$ cat input
# file: /disk1/script_1/ user::rwx group::r-x group:service:r-x mask::r-x other::r-x
# file: /disk1/script_1//hello.txt user::rw- group::r-- other::r--
# file: /disk1/script_1//bkp_10.txt user::rwx group::r-x other::r-x
$ while read _ _ path perms; do perms="$(echo "$perms" | tr -s ' ' ,)"; echo path="$path", perms="$perms"; done < input
path=/disk1/script_1/, perms=user::rwx,group::r-x,group:service:r-x,mask::r-x,other::r-x
path=/disk1/script_1//hello.txt, perms=user::rw-,group::r--,other::r--
path=/disk1/script_1//bkp_10.txt, perms=user::rwx,group::r-x,other::r-x
Try to echo the line content along with sed logic like this
file_name=$(echo "$line" | sed 's/# file:\(.*\)/\1/g' | awk '{print $1}')
file_perm=$(echo "$line" | sed -e 's/# file:\(.*\)/\1/g' | awk '{$1=""}{print}' | tr ' ' ',' | awk '{sub(",","")}1')

How to use variable to search and replace using sed or awk?

I am trying to find and replace a value using variable with the help of sed and awk but not getting any output -
#cat temp.txt
this is Testing of date : 2016-11-25 07:20:10
It is printing the variable but not working in gsub function -
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{print srch,repl}' temp.txt
2016-11-25 07:20:10 [25/Nov/16:07:20:10]
I tried below awk command -
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{gsub("srch","repl",$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub(srch,repl,$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub("$srch","$repl",$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub(srch,repl,$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{print gsub(srch,repl,$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub("srch","repl",$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub("$srch","$repl",$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub("$srch","$repl",$0)}' temp.txt
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub(srch,repl,$0)}' temp.txt
#var1="2016-11-25 07:20:10"
#var2="[25/Nov/16:07:20:10]"
#echo $var1 $var4
2011-11-25 07:20:10 [25/Nov/11:07:20:10]
#awk -v srch="$var1" -v repl="$var4" '{ gsub(srch,repl,$0)}' temp.txt
With sed command -
#echo $var1 $var4
2011-11-25 07:20:10 [25/Nov/11:07:20:10]
sed 's/"$var1"/"$var4"/' temp.txt
sed 's/$var1/'"${var4}"'/' temp.txt
sed 's|$var1|'"${var4}"'|' temp.txt
sed 's/\$var1/${var4}/' temp.txt
sed 's/\$var1/$var4/' temp.txt
sed "s/"$var1"/"$var4"/" temp.txt
sed 's/'$var1'/'$var4'/' temp.txt
sed 's/'$var1'/'$var4\/' temp.txt
sed -e "s/${var1}/${var4}/' temp.txt
sed -e "s/${var1}/${var4}/" temp.txt
sed "s/$var1/$var4/" temp.txt
sed 's/'"$var1"'/'"$var4"'/' temp.txt
sed 's/'"$var1"'/'$var4'/' temp.txt
Not sure what i am missing.
Expected output -
#this is Testing of date : [25/Nov/11:07:20:10]
this one works, but you need to add print or nothing will be printed:
awk -v srch="2016-11-25 07:20:10" -v repl="[25/Nov/16:07:20:10]" '{ gsub(srch,repl,$0); print}' temp.txt
result:
this is Testing of date : [25/Nov/16:07:20:10]
with sed you can do it as well (as long as you use double quotes, or variables won't be evaluated), but since replacement string contains slashes, you have to change sed expression delimiter or you get a sed parsing error, I chose #
$ var1="2016-11-25 07:20:10"; var4="[25/Nov/11:07:20:10]"
$ sed "s#$var1#$var4#" temp.txt
result:
this is Testing of date : [25/Nov/11:07:20:10]
It would be:
var1="2016-11-25 07:20:10"
var2="[25/Nov/16:07:20:10]"
awk -v srch="$var1" -v repl="$var2" '{gsub(srch,repl)} 1' temp.txt
sed 's#'"$var1"'#'"$var2"'#g' temp.txt
You should also read Is it possible to escape regex metacharacters reliably with sed as the above would fail if var1 contained RE metacharacters or var2 contained backreferences.
At a bit of a tangent - I would tackle it in perl and parse your date. (I know you didn't ask about perl, but I think it's a good fit for this - it's installed in most of the places you have sed or awk available)
It would work a bit like this:
#!/usr/bin/env perl
use strict;
use warnings;
use Time::Piece;
#iterate __DATA__ special filehandle
while ( <DATA> ) {
#extract time via regex
my ( $time_str ) = m/([\d\-]+\s+[\d\:]+)$/;
next unless $time_str;
#parse it into a Time::Piece object
my $timestamp = Time::Piece -> strptime ("$time_str", "%Y-%m-%d %H:%M:%S");
#print it using a new format string
my $new_time_str = $timestamp -> strftime("%d/%b/%y:%H:%M:%S");
#replace old with new in line
s/$time_str/\[$new_time_str\]/;
#print current line
print;
}
__DATA__
this is Testing of date : 2016-11-25 07:20:10
And for the sake of a command line solution, that condenses into:
perl -pe '($ts)=m/([\d\-]+\s+[\d\:]+)$/;s{$ts}{Time::Piece->strptime($ts,"%Y-%m-%d %H:%M:%S")->strftime("[%d/%b/%y:%H:%M:%S]")}e;'
You can pipe into this, or specify a filename as an argument, just like you would with sed or awk.

unix script run search in parallel

I have a script that searches files for a phrase in a number of different folders, then shows to output.
The trouble is, it does each search sequentially and takes a long time. I would like to make the searches run without waiting for the previous one to finish.
zipped_folders=("/extlogs/archive/rsyslog/folder1/"
"/extlogs/archive/rsyslog/folder2/")
folders=("/extlogs/rsyslog/Folder1/"
"/extlogs/rsyslog/Folder2/")
portal=0
mobile=0
email=0
if [ "$#" -ne 1 ]; then
echo "Incorrect Argument: logcount 201602"
exit 1
fi
for i in "${zipped_folders[#]}"
do
#echo $i"syslog-"$1*".log.gz"
((portal+=$(nohup gunzip -c $i"syslog-"$1*".log.gz" | grep -i "search1" | grep -v "Search1" | wc -l &)))
((mobile+=$(nohup gunzip -c $i"syslog-"$1*".log.gz" | grep -i "Search2" | wc -l &)))
((email+=$(nohup gunzip -c $i"syslog-"$1*".log.gz" | grep -i "search3" | grep -v "ActiveSync" | wc -l &)))
done
for i in "${folders[#]}"
do
((portal+=$(nohup cat $i"syslog-"$1*".log"| grep -i "search4"| grep -v "exsearch4" | wc -l &)))
((mobile+=$(nohup cat $i"syslog-"$1*".log" | grep -i "search5" | wc -l &)))
((email+=$(nohup cat $i"syslog-"$1*".log" | grep -i "search6" | grep -v "ActiveSync" | wc -l &)))
done
echo "Portal: " $portal
echo "Mobile: " $mobile
echo "Email: " $email
exit 1
You can use xargs.
find ${topdir} -name '*.gz' | xargs -n1 -P${PARALLEL_JOBS} -I {} bash -c "/usr/bin/grep 'criteria' {}"

Using sed to enclose matches in double quotes

I'm trying to extract headers from emails and create a JSON fragment from them. I'm using sed to pull out the keys and values, but it's failing to put the trailing quote on each of the lines:
$ cat email1 | grep -i -e "^subject:" -e "^from:" -e "^to:" | \
sed -n 's/\^([^:]*\):[ ]*\(.*\)$/"\1":"\2"/gp'
"From":"Blah Blech <blah.blech#blahblech.com>
"To":"foo#bar.com
"Subject":"Yeah
I don't understand why the replacement pattern isn't working.
awk to the rescue!
$ awk -F": *" -vOFS=":" -vq="\"" 'tolower($0)~/^from|to|subject/
{print q$1q,q$2q}' email1
which combines cat or grep steps as well.
Stripping the carriage returns as #tripleee suggested fixed the issue with sed (using ctrl-v ctrl-m to capture the literal carriage return):
$ cat email1 | tr -d '^M' | grep -i -e "^subject:" -e "^from:" -e "^to:" | \
sed -n 's/^\([^:]*\):[ ]*\(.*\)$/"\1":"\2"/gp'
"From":"Blah Blech <blah.blech#blahblech.com>"
"To":"foo#bar.com"
"Subject":"Yeah"

Printing only the first field in a string

I have a date as 12/12/2013 14:32 I want to convert it into only 12/12/2013.
The string can be 1/1/2013 12:32 or 1/10/2013 23:41 I need only the date part.
You can do this easily with a variety of Unix tools:
$ cut -d' ' -f1 <<< "12/12/2013 14:32"
12/12/2013
$ awk '{print $1}' <<< "12/12/2013 14:32"
12/12/2013
$ sed 's/ .*//' <<< "12/12/2013 14:32"
12/12/2013
$ grep -o "^\S\+" <<< "12/12/2013 14:32"
12/12/2013
$ perl -lane 'print $F[0]' <<< "12/12/2013 14:32"
12/12/2013
$ echo "12/12/2013 14:32" | awk '{print $1}'
12/12/2013
print $1 --> Prints first column of the supplied string. 12/12/2013
print $2 --> Prints second column of the supplied string. 14:32
By default, awk treats the space character as the delimiter.
If your date string is stored in a variable, then you don't need to run an external program like cut, awk or sed, because modern shells like bash can perform string manipulation directly which is more efficient.
For example, in bash:
$ s="1/10/2013 23:41"
$ echo "${s% *}"
1/10/2013

Resources