I have a question on creating file name on basis of output we get on running a command. Below is the example
Have 2 records like below
cat test1.txt
Unable to find Solution
Can you please help
And I am running below command to get the last word from the first line and i want to have file name to be that name(Last word name)
cat test1.txt | head -1 | awk '{ print $NF }'
Solution
Can you please help me to get the file name as a last word name
When you want to redirect your output to a file and have the filename from the first line of your output, you can use
outfile=""
your_command | while read line; do
if [ -z "${outfile}" ]; then
outfile=$(echo "${line}"| awk '{ print $NF }')
fi
echo "${line}" >> ${outfile}
done
Related
I have a sed command that is capturing a single line with sometext. The line in the file it is capturing ends with a linefeed. I am trying to utilize this variable in a pipeline, however, when I attempt to echo, or use it with other commands requiring an input, the result is a blank. Ex:
sed '1,1!d' somefile.txt | echo "$1", I know the variable itself is not empty as I can replace echo "$1" with cat $1 and see the correct printout.
edit - I have tried piping to a tr -d and removing the newline. I have confirmed the newline character is gone, yet echos still show blank. Cats do not.
edit 2 - I piped the variable into an if statement ... | if [[ -z $1 ]]; then cat $1; fi it hits the if, is determined to be empty, so runs the cat, which prints a non-empty line to console. If the variable is empty why is cat still printing out information?
What is causing this inconsistency and how can I solve my problem? The ultimate goal is to run the output of one sed, through another to replace specific lines in a target file.
sed '1,1!d' somefile.txt | sed '2,1s/.*/'$1'/' targetfile.txt
Contents of somefile.txt:
these
are
words
Contents of targetfile.txt:
The next line should say these
This line should say these
The previous line should say these
Output of echo after sed:
<empty>
Output of cat after sed:
these
Output of 2nd sed, using input from 1st:
The next line should say these
the previous line should say these
You are confused about arguments and input data. Look at this:
$ echo "$1"
$ echo "foo" | if [[ -z $1 ]]; then cat $1; fi
foo
The first argument to my shell, $1 is empty so if [[ -z $1 ]] succeeds. The reason that cat $1 produces output is that you have a fundamental shell programming error in that statement - you aren't quoting your variable, $1. The correct syntax isn't cat $1, it's cat "$1". Look at the difference:
$ echo "foo" | if [[ -z $1 ]]; then cat "$1"; fi
cat: '': No such file or directory
We can simplify the code to make what's happening clearer:
$ echo "foo" | cat $1
foo
$ echo "foo" | cat "$1"
cat: '': No such file or directory
The reason that echo "foo" | cat $1 produces output is that the unquoted $1 is expanded by the shell to nothing before cat is called so that statement is equivalent to just echo "foo" | cat and so cat just copies the input coming in from the pipe to it's output.
On the other hand echo "foo" | cat "$1" generates an error because the shell expands "$1" to the null string before cat is called and so it's then asking cat to open a file named <null> and that of course does not exist, hence the error.
Always quote your shell variables unless you have a specific reason not to and fully understand all of the implications. Read a shell man page and/or google that if you're not sure what those implications are.
wrt another part of your code you have:
sed '1,1!d' somefile.txt | echo "$1"
but, unlike cat, echo neither reads it's input from a pipe nor from a file name passed as an argument. The input to echo is just the list of string arguments you provide it so while echo "foo" | cat will cause cat to read the input stream containing foo and output it, echo "foo" | echo will produce no output because echo isn't designed to read input from a pipe and so it'll just print a null string since you gave it no arguments.
It's not clear what you're really trying to accomplish but I think you might want to replace the 2nd line of targetfile.txt with the first line of somefile.txt. If so that's just:
awk '
NR==FNR { if (NR==1) new=$0; next }
FNR==2 { $0 = new }
{ print }
' somefile.txt targetfile.txt
Do not try to use sed to do it or you'll find yourself in escaping/quoting hell because, unlike awk, sed does not understand literal strings, see Is it possible to escape regex metacharacters reliably with sed.
You appear to want to extract the first line from file1 and use it to replace the second line in file2.
At the moment, you are extracting that value from the first file with your first sed but sending it to the second sed on its stdin rather than as a parameter ($1).
Your description is confusing so I will use this as file1:
File 1 Line 1
File 1 Line 2
File 1 Line 3
And this as file2:
File 2 Line 1
File 2 Line 2
File 2 Line 3
There are many ways to do this.
Method 1
# Extract line1 from file1
extracted=$(sed '1!d' file1)
# Replace line 2 in file2 with extracted value
sed "2s/.*/$extracted/" file2
Not sure why I feel like a dentist now :-)
If you want to put it all on one line, as some folks like to do:
x=$(sed '1!d' file1) && sed "2s/.*/$x/" file2
Method 2
This one is a bit tricky. It uses the first sed to write a script for the second sed:
sed 's/^/2s|.*|/;s/$/|/;q' file1 | sed -f /dev/stdin file2
If you look at the first sed on its own you will see it is generating a script for the second one:
sed 's/^/2s|.*|/;s/$/|/;q' file1
2s|.*|File 1 Line 1|
If you look at the second sed, you will see it is executing a script passed on its standard input:
sed -f /dev/stdin ...
Method 3
Easier still is awk:
awk 'FNR==NR{if(NR==1)saved=$0;next} FNR==2{$0=saved}1' file1 file2
File 2 Line 1
File 1 Line 1
File 2 Line 3
All you need to notice is that I am passing 2 files to awk and that FNR==NR inside the script means that awk is currently processing the first file, because FNR is the line number in the current file and NR is the total number of lines awk has processed from all files so far. So, when processing the second file, NR is greater than FNR by the number of lines in the first file.
I have a file that contains dates and lat longs and I want to convert the dates that are in UTC to unix time. I decided to use the unix date function within an awk script to do this. I tried:
awk 'BEGIN {t=$3; c="date -j -f %Y%j%H%M%S "t" +%s"; c|getline; close( c ); print $1, $2, c; }' test.txt
where t is the third row in my file and is in the format 2014182120311. I am new to awk/scripting and not sure if it is possible to imbed an awk variable into a unix command inside an awk line.
When I run the script I get the error:
usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
So I think I am not defining "t" properly. Any help is much appreciated!
On a side note- I have tried mktime and the other awk time functions but they do not work for me I believe because I do not have gawk.
Do you not have the option of installing gawk? You'll thank yourself later (e.g. see Converting list of dates from a file to timestamp with bash).
The BEGIN section is executed before any file is opened, so $3 (the 3rd field of the current record read from the current file) has no meaning in that section and so t=$3 just sets the variable t to the null string.
You don't want to print the command, c, you want to print the result of executing the command as read by getline but when you use undirected getline it overwrites $0, $1, etc. so you should save the result in a var (see http://awk.info/?tip/getline).
You probably want something like this:
awk '{
cmd = "date -j -f %Y%j%H%M%S " $3 " +%s"
if ( (cmd | getline time) > 0 ) {
print $1, $2, time
}
close(cmd)
}' test.txt
but if not, post some sample input and expected output so we can help you.
I need to find files where a specific string appears twice or more.
For example, for three files:
File 1:
Hello World!
File 2:
Hello World!
Hello !
File 3:
Hello World!
Hello
Hello Again.
--
I want to grep Hello and only get files 2 & 3.
What about this:
grep -o -c Hello * | awk -F: '{if ($2 > 1){print $1}}'
Since the question is tagged grep, here is a solution using only that utility and bash (no awk required):
#!/bin/bash
for file in *
do
if [ "$(grep -c "Hello" "${file}")" -gt 1 ]
then
echo "${file}"
fi
done
Can be a one-liner:
for file in *; do if [ "$(grep -c "Hello" "${file}")" -gt 1 ]; then echo "${file}"; fi; done
Explanation
You can modify the for file in * statement with whatever shell expansion you want to get all the data files.
grep -c returns the number of lines that match the pattern, with multiple matches on a line still counting for just one matched line.
if [ ... -gt 1 ] test that more than one line is matched in the file. If so:
echo ${file} print the file name.
This awk will print the file name of all files with 2 or more Hello
awk 'FNR==1 {if (a>1) print f;a=0} /Hello/ {a++} {f=FILENAME} END {if (a>1) print f}' *
file2
file3
What you need is a grep that can recognise patterns across line endings ("hello" followed by anything (possibly even line endings), followed by "hello")
As grep processes your files line by line, it is (by itself) not the right tool for the job - unless you manage to cram the whole file into one single line.
Now, that is easy, for example using the tr command, replacing line endings by spaces:
if cat $file | tr '\n' ' ' | grep -q 'hello.*hello'
then
echo "$file matches"
fi
This is quite efficient, even on large files with many (say 100000) lines, and can be made even more efficient by calling grep with --max-count=1 , making it stop the search after a match has been found. It doesn't matter whether the two hellos are on the same line or not.
After reading your question, I think you also want to find the case hello hello in one line. ( find files where a specific string appears twice or more.) so I come up with this one-liner:
awk -v p="hello" 'FNR==1{x=0}{x+=gsub(p,p);if(x>1){print FILENAME;nextfile}}' *
in the above line, p is the pattern you want to search
it will print the filename if the file contains the pattern two or more times. no matter they are in same or different lines
during the processing, after checking some line, if we had already found two or more pattern, print the filename and stop processing current file, take the next input file, if there still are. This is helpful if you have big files.
A little test:
kent$ head f*
==> f <==
hello hello world
==> f2 <==
hello
==> f3 <==
hello
hello
SK-Arch 22:27:00 /tmp/test
kent$ awk -v p="hello" 'FNR==1{x=0}{x+=gsub(p,p);if(x>1){print FILENAME;nextfile}}' f*
f
f3
Another way:
grep Hello * | cut -d: -f1 | uniq -d
Grep for lines containing 'Hello'; keep only the file names; print only the duplicates.
grep -c Hello * | egrep -v ':[01]$' | sed 's/:[0-9]*$//'
Piping to a scripting language might be overkill, but it's oftentimes much easier than just using awk
grep -rnc "Hello" . | ruby -ne 'file, count = $_.split(":"); puts "#{file}: #{count}" if count&.to_i >= 2'
So for your input, we get
$ grep -rnc "Hello" . | ruby -ne 'file, count = $_.split(":"); puts "#{file}: #{count}" if count&.to_i >= 2'
./2: 2
./3: 3
Or to omit the count
grep -rnc "Hello" . | ruby -ne 'file, _ = $_.split(":"); puts file if count&.to_i >= 2'
Would like to split a csv file according to the 2nd "field". For instance the csv file contains:
Input.csv :: /c/Test/
aaa,K1,ppp
ddd,M3,ppp
bbb,K1,ppp
ccc,L2,ppp
This file would be split into three separate files according to second field.
First file: /c/Test/K1/K1.csv
aaa,K1,ppp
bbb,K1,ppp
Second file: /c/Test/M3/M3.csv
ddd,M3,ppp
Third file: /c/Test/L2/L2.csv
ccc,L2,ppp
Tried the below command to split file based on 2nd column and working fine, however the splitted files on the same directory
Like: /c/Test/K1.csv and /c/Test/M3.csv etc ..
awk -F, '{ print > $2".csv"}' Input.csv
Have tried the below command is not working to create subdirectory and incomplete , please help ..
awk -F, 'BEGIN { system("mkdir -p $2") } { print > $10".csv"}' Input.csv
awk -F, '{ print >/system("mkdir -p $2")/ $2".txt"}' Input.csv
awk -F, '{ system("mkdir -p "$2); print > $2"/"$2".csv"}' Input.csv
Assuming your Input.csv contains:
aaa,K1,ppp
ddd,M3,ppp
bbb,K1,ppp
ccc,L2,ppp
And this file is in /c/Test/ and you want directories to be created in /c/Test/.
The main difference with your attempt is system("mkdir -p "$2) i.e put $2 outside of quotes. This will concatenate "mkdir -p " and the value of $2. When you put it inside quotes it becomes literal $2 and the value is not available to mkdir command.
After the directory is created, it prints output to the desired file which has the path $2"/"$2".csv"
You could do this in bash using read:
#!/bin/bash
while IFS="," read a b c; do
mkdir -p "$b" && echo "$a,$b,$c" >> "$b/$b.csv"
done < Input.csv
read splits the input line on the input field separator (IFS). It makes the directory based on the name of the second column and echos the line to the relevant file.
Or if you can use bash arrays:
#!/bin/bash
(
IFS=","
while read -a l
do
mkdir -p "${l[1]}" && echo "${l[*]}" >> "${l[1]}/${l[1]}.csv"
done < Input.csv
)
The use of ( ) means that the original value of $IFS is preserved once the script has run.
i have command
ccv
this will output as below
Your Project : gdgdd750V64OG , Building Block : cgd9gdd .
if i do
ccv | awk '{ print $9}'
cgd9gdd
now i am tring to create a log file by running a build process
& tee log_`date +%Y%m%d%H%M%S`_`ccv | awk '{ print $9}'`
but this is not creating the log corrrectly.
instaed have same logs with thier name as each and every field of the output of ccv as above.that is "Your Project : gdgdd750V64OG , Building Block : cgd9gdd ."for each and every word including . , there is a file and all files are same copies.
is there anything wrong with the log file creation?
& tee log_$(date +%Y%m%d%H%M%S)_$(ccv | awk '{ print $9'})