i have command
ccv
this will output as below
Your Project : gdgdd750V64OG , Building Block : cgd9gdd .
if i do
ccv | awk '{ print $9}'
cgd9gdd
now i am tring to create a log file by running a build process
& tee log_`date +%Y%m%d%H%M%S`_`ccv | awk '{ print $9}'`
but this is not creating the log corrrectly.
instaed have same logs with thier name as each and every field of the output of ccv as above.that is "Your Project : gdgdd750V64OG , Building Block : cgd9gdd ."for each and every word including . , there is a file and all files are same copies.
is there anything wrong with the log file creation?
& tee log_$(date +%Y%m%d%H%M%S)_$(ccv | awk '{ print $9'})
Related
I have a file named as TEST123_20112020.csv
I have a constant path as processed/daily/ where the file will be transferred daily.
I need to know the Unix command which will create the directory according to the year and month of the file respectively and will transfer that file as well to that directory.
For eg:
File Name : TEST123_20112020.csv
Directory : processed/daily/2020/Nov
File Name : TEST123_15092022.csv
Directory : processed/daily/2020/Sep
You can achieve this with find piped through to awk and so:
find . -name "TEST123.*csv" | awk -F[_\.] '{ yr=substr($2,length($2)-3);yr1=substr($2,length($2)-1);dat1=substr($2,1,4)""yr1;"date -d \""dat1"\" +%b" | getline mon;print "mkdir processed/daily/"yr"/"mon }'
Take the output of the find command, set the field delimiter to _ and . in awk with -F Then set the variable yr to the full year using awk's substr function. Set the two digit year (yr1) to be used with the date command later. Set a date command compliant date (dat1) utilising yr1 and then use this with a date command and awk's get line to read this into a variable mon. The variable mon with contain the short form month name as required. Finally print this month along with the year to form a mkdir command.
Once you have verified that this command is as expected (this is important!) you can substitute the final print command for awk's system function to actually execute the command and so:
find . -name "TEST123.*csv" | awk -F[_\.] '{ yr=substr($2,length($2)-3);yr1=substr($2,length($2)-1);dat1=substr($2,1,4)""yr1;"date -d \""dat1"\" +%b" | getline mon;system("mkdir processed/daily/"yr"/"mon) }'
My shell script fails on this portion
dt=$(date +%Y%m%d)
syntax error at line 35: `dt=$' unexpected
It's a bit strange as I'm using the same to get the datetime and don't get any problem
timestamp=$(date +%Y%m%d%H%M%S)
20170911105251
I've checked in Notepad++ as well that I don't have any dirty characters as suggested by #Frank below
Below is the sample script I have. Please note I have omitted some portions that are confidential. Basically the script transfers files from Server A to B based on the dt specified. If no date parameter is specified during execution, it takes the current date.
if [ -z "$1" ]
then
dt=$(date +%Y%m%d)
else
dt=$1
fi
export rundate
timestamp=$(date +%Y%m%d%H%M%S)
logfile=${log_path}/${batch}-${dt}-${timestamp}.log
mail_file=${log_path}/${batch}-mail-${timestamp}.txt
export mail_file
filecount=$(wc -l ${batch_filelist} | cut -d " " -f 1)
sftp ${dest} <<EOF >> $logfile
cd $destdir
lcd $sourcedir
put -p *.txt
exit
EOF
mail -s "SFTP Done (dt:$dt)" $(cat $email_accounts) < $mail_file
Becarful , when you copy text from anywhere and paste in file (i think you did it) , it can add "dirty" characters .
If you have a text editor like notepad++ , by clicking on :
It will show all characters .
Via notepad++ you can change the file format from windows to unix , by right click on bottom-right in page :
Save the file and send again in to your machine .
The syntax seems correct so it should work .
Anyway i mentioned notepad++ but you can do this with other text editors.
dt=$(date +'%Y%m%d') and not dt=$(dt +'%Y%m%d')
A working example below :
!/usr/bin/bash
if [ -z $1 ]
then
dt=$(date +'%Y%m%d')
else
dt=$1
fi
echo $dt
Result :
xxxxxxxxxxxxxx:/pathtosh> prova.sh
20170911
xxxxxxxxxxxxxx:/pathtosh> prova.sh 1
1
I have a question on creating file name on basis of output we get on running a command. Below is the example
Have 2 records like below
cat test1.txt
Unable to find Solution
Can you please help
And I am running below command to get the last word from the first line and i want to have file name to be that name(Last word name)
cat test1.txt | head -1 | awk '{ print $NF }'
Solution
Can you please help me to get the file name as a last word name
When you want to redirect your output to a file and have the filename from the first line of your output, you can use
outfile=""
your_command | while read line; do
if [ -z "${outfile}" ]; then
outfile=$(echo "${line}"| awk '{ print $NF }')
fi
echo "${line}" >> ${outfile}
done
I would like to know how to search certain pattern of files (GunZip Files) in all Sub Directories ( Month wise / Date wise - Sub Directories created).
And then, execute a script on the found files. Also need to populate FILENAME along with output for tracking purpose and further analysis on that particular files.
Step1: For example: currently searching files on this pattern TT_DETAIL*.gz.
find /cygdrive/c/Test/ -name TT_DETAIL*.gz
output#1:
/cygdrive/c/Test/Feb2014/TT_DETAIL_20141115.csv.gz
/cygdrive/c/Test/Jan2014/TT_DETAIL_20141110.csv.gz
/cygdrive/c//Test/Mar2014/TT_DETAIL_20141120.csv.gz
Step2:
zcat TT_DETAIL*.gz | awk 'BEGIN { FS=OFS=","} { if ($11=="10") print $2,$3,$6,$10,$11,$17}' >Op_TT_Detail.txt
cat Op_TT_Detail.txt
ZZZ,AAA,ECH,1,10,XXX
ZZZ,BBB,ECH,1,10,XXX
ZZZ,CCC,ECH,1,10,XXX
ZZZ,DDD,ECH,1,10,XXX
Thanks fedorqui for below script is working fine without FILENAME.
while IFS= read -r file
do
awk 'BEGIN { FS=OFS=","} { if ($11=="10") print $2,$3,$6,$10,$11,$17}' <(zcat "$file") >>Op_TT_Detail.txt
done < <(find /cygdrive/c/Test/ -name TT_DETAIL*.gz)
Have tried below command to populate FILENAME along with output for tracking purpose :
while IFS= read -r file
do
awk 'BEGIN { FS=OFS=","} { if ($11=="10") print $2,$3,$6,$10,$11,$17,FILENAME}' <(zcat "$file") >>Op_TT_Detail.txt
done < <(find /cygdrive/c/Test/ -name TT_DETAIL*.gz)
Desired Output:
ZZZ,AAA,ECH,1,10,XXX,/cygdrive/c/Test/Feb2014/TT_DETAIL_20141115.csv.gz
ZZZ,BBB,ECH,1,10,XXX,/cygdrive/c/Test/Feb2014/TT_DETAIL_20141115.csv.gz
ZZZ,CCC,ECH,1,10,XXX,/cygdrive/c//Test/Mar2014/TT_DETAIL_20141120.csv.gz
ZZZ,DDD,ECH,1,10,XXX,/cygdrive/c//Test/Mar2014/TT_DETAIL_20141120.csv.gz
Since FILENAME is not working for *.gz files , should I write" find /cygdrive/c/Test/ -name TT_DETAIL*.gz " into another output file
then call that output file into script , I don't have a write access for source files located server.
Looking for your suggestions !!!
Nice to see you are using the snippet I wrote in the previous question!
I would use this:
while IFS= read -r file
do
awk -v file="$file" 'BEGIN { FS=OFS=","} \
{ if ($11=="10") print $2,$3,$6,$10,$11,$17, file}' \
<(zcat "$file") >>Op_TT_Detail.txt
done < <(find /cygdrive/c/Test/ -name TT_DETAIL*.gz)
That is, with -v file="$file" you give the file name as a variable to awk. And then you use it in your print command.
Would like to split a csv file according to the 2nd "field". For instance the csv file contains:
Input.csv :: /c/Test/
aaa,K1,ppp
ddd,M3,ppp
bbb,K1,ppp
ccc,L2,ppp
This file would be split into three separate files according to second field.
First file: /c/Test/K1/K1.csv
aaa,K1,ppp
bbb,K1,ppp
Second file: /c/Test/M3/M3.csv
ddd,M3,ppp
Third file: /c/Test/L2/L2.csv
ccc,L2,ppp
Tried the below command to split file based on 2nd column and working fine, however the splitted files on the same directory
Like: /c/Test/K1.csv and /c/Test/M3.csv etc ..
awk -F, '{ print > $2".csv"}' Input.csv
Have tried the below command is not working to create subdirectory and incomplete , please help ..
awk -F, 'BEGIN { system("mkdir -p $2") } { print > $10".csv"}' Input.csv
awk -F, '{ print >/system("mkdir -p $2")/ $2".txt"}' Input.csv
awk -F, '{ system("mkdir -p "$2); print > $2"/"$2".csv"}' Input.csv
Assuming your Input.csv contains:
aaa,K1,ppp
ddd,M3,ppp
bbb,K1,ppp
ccc,L2,ppp
And this file is in /c/Test/ and you want directories to be created in /c/Test/.
The main difference with your attempt is system("mkdir -p "$2) i.e put $2 outside of quotes. This will concatenate "mkdir -p " and the value of $2. When you put it inside quotes it becomes literal $2 and the value is not available to mkdir command.
After the directory is created, it prints output to the desired file which has the path $2"/"$2".csv"
You could do this in bash using read:
#!/bin/bash
while IFS="," read a b c; do
mkdir -p "$b" && echo "$a,$b,$c" >> "$b/$b.csv"
done < Input.csv
read splits the input line on the input field separator (IFS). It makes the directory based on the name of the second column and echos the line to the relevant file.
Or if you can use bash arrays:
#!/bin/bash
(
IFS=","
while read -a l
do
mkdir -p "${l[1]}" && echo "${l[*]}" >> "${l[1]}/${l[1]}.csv"
done < Input.csv
)
The use of ( ) means that the original value of $IFS is preserved once the script has run.