Count file line along with file name in unix - unix

I have 3 files at temp dir as below :
test1.txt -- It has 4 lines
test2.txt -- It has 5 lines
test3.txt -- It has 6 lines
Need to count the lines in the file along with the name to a separate file (LIST.txt), like below
'test1.txt','4'
'test2.txt','5'
'test3.txt','6'
Code Tried :
FileDir=/temp/test*.txt
for file in ${FileDir}
do
filename=$(basename $file) & awk 'END {print NR-1}' ${file} >> /temp/LIST.txt
done
This is not giving me the name, it only gives me the line counts.
Also, how to get the output of those 2 commands separated by ',' ?

Perhaps this would suit?
FileDir=/temp/test*.txt
for file in ${FileDir}
do
awk 'END{print FILENAME "," NR}' "$file"
done > LIST.txt
cat LIST.txt
/temp/test1.txt,4
/temp/test2.txt,5
/temp/test3.txt,6
Remove "/temp/" and include single quotes:
cd /temp
FileDir=test*.txt
for file in ${FileDir}
do
awk 'END{q="\047"; print q FILENAME q "," q NR q}' "$file"
done > ../LIST.txt
cd ../
cat LIST.txt
'test1.txt','4'
'test2.txt','5'
'test3.txt','6'
An alternative approach:
FileDir=/temp/test*.txt
for file in ${FileDir}
do
awk 'END{q="\047"; n = split(FILENAME, a, "/"); print q a[n] q "," q NR q}' "$file"
done > LIST.txt
cat LIST.txt
'test1.txt','4'
'test2.txt','5'
'test3.txt','6'

Related

Transposing multiple columns in multiple rows keeping one column fixed in Unix

I have one file that looks like below
1234|A|B|C|10|11|12
2345|F|G|H|13|14|15
3456|K|L|M|16|17|18
I want the output as
1234|A
1234|B
1234|C
2345|F
2345|G
2345|H
3456|K
3456|L
3456|M
I have tried with the below script.
awk -F"|" '{print $1","$2","$3","$4"}' file.dat | awk -F"," '{OFS=RS;$1=$1}1'
But the output is generated as below.
1234
A
B
C
2345
F
G
H
3456
K
L
M
Any help is appreciated.
What about a single simple awk process such as this:
$ awk -F\| '{print $1 "|" $2 "\n" $1 "|" $3 "\n" $1 "|" $4}' file.dat
1234|A
1234|B
1234|C
2345|F
2345|G
2345|H
3456|K
3456|L
3456|M
No messing with RS and OFS.
If you want to do this dynamically, then you could pass in the number of fields that you want, and then use a loop starting from the second field.
In the script, you might first check if the number of fields is equal or greater than the number you pass into the script (in this case n=4)
awk -F\| -v n=4 '
NF >= n {
for(i=2; i<=n; i++) print $1 "|" $i
}
' file
Output
1234|A
1234|B
1234|C
2345|F
2345|G
2345|H
3456|K
3456|L
3456|M
# perl -lne'($a,#b)=((split/\|/)[0..3]);foreach (#b){print join"|",$a,$_}' file.dat
1234|A
1234|B
1234|C
2345|F
2345|G
2345|H
3456|K
3456|L
3456|M

Finding amount of sequence matches per line

I'm looking to use GREP or something similiar to find the total matches of a 5 letter sequence (AATTC) in every line of a file, and then print the result in a new file. For example:
File 1:
GGGGGAATTCGAATTC
GGGGGAATTCGGGGGG
GGGGGAATTCCAATTC
Then in another file it prints the matches line by line
File 2:
2
1
2
Awk solution:
awk '{ print gsub(/AATTC/,"") }' file1 > file2
The gsub() function returns the number of substitutions made
$ cat file2
2
1
2
If you have to use grep, then put that in a while loop,
$ while read -r line; do grep -o 'AATTC'<<<"$line"|wc -l >> file2 ; done < file1
$ cat file2
2
1
2
Another way: using perl.
$ perl -ne 'print s/AATTC/x/g ."\n"' file1 > file2

pattern match and create multiple files LINUX

I have a pipe delimited file with over 20M rows. In 4th column I have a date field. I have to take the partial value (YYYYMM) from the date field and write the matching data to a new file appending it to file name. Thanks for all your inputs.
Inputfile.txt
XX|1234|PROCEDURES|20160101|RC
XY|1634|PROCEDURES|20160115|RC
XM|1245|CODES|20170124|RC
XZ|1256|CODES|20170228|RC
OutputFile_201601.txt
XX|1234|PROCEDURES|20160101|RC
XY|1634|PROCEDURES|20160115|RC
OutputFile_201701.txt
XM|1245|CODES|20170124|RC
OutputFile_201702.txt
XZ|1256|CODES|20170228|RC
Using awk:
$ awk -F\| '{f="outputfile_" substr($4,1,6) ".txt"; print >> f ; close (f)}' file
$ ls outputfile_201*
outputfile_201601.txt outputfile_201701.txt outputfile_201702.txt
Explained:
$ awk -F\| ' # pipe as delimiter
{
f="outputfile_" substr($4,1,6) ".txt" # form output filename
print >> f # append record to file
close(f) # close output file
}' file

While read line, awk $line and write to variable

I am trying to split a file into different smaller files depending on the value of the fifth field. A very nice way to do this was already suggested and also here.
However, I am trying to incorporate this into a .sh script for qsub, without much success.
The problem is that in the section where the file to which output the line is specified,
i.e., f = "Alignments_" $5 ".sam" print > f
, I need to pass a variable declared earlier in the script, which specifies the directory where the file should be written. I need to do this with a variable which is built for each task when I send out the array job for multiple files.
So say $output_path = ./Sample1
I need to write something like
f = $output_path "/Alignments_" $5 ".sam" print > f
But it does not seem to like having a $variable that is not a $field belonging to awk. I don't even think it likes having two "strings" before and after the $5.
The error I get back is that it takes the first line of the file to be split (little.sam) and tries to name f like that, followed by /Alignments_" $5 ".sam" (those last three put together correctly). It says, naturally, that it is too big a name.
How can I write this so it works?
Thanks!
awk -F '[:\t]' ' # read the list of numbers in Tile_Number_List
FNR == NR {
num[$1]
next
}
# process each line of the .BAM file
# any lines with an "unknown" $5 will be ignored
$5 in num {
f = "Alignments_" $5 ".sam" print > f
} ' Tile_Number_List.txt little.sam
UPDATE, AFTER ADDING -V TO AWK AND DECLARING THE VARIABLE OPATH
input=$1
outputBase=${input%.bam}
mkdir -v $outputBase\_TEST
newdir=$outputBase\_TEST
samtools view -h $input | awk 'NR >= 18' | awk -F '[\t:]' -v opath="$newdir" '
FNR == NR {
num[$1]
next
}
$5 in num {
f = newdir"/Alignments_"$5".sam";
print > f
} ' Tile_Number_List.txt -
mkdir: created directory little_TEST'
awk: cmd. line:10: (FILENAME=- FNR=1) fatal: can't redirect to `/Alignments_1101.sam' (Permission denied)
awk variables are like C variables - just reference them by name to get their value, no need to stick a "$" in front of them like you do with shell variables:
awk -F '[:\t]' ' # read the list of numbers in Tile_Number_List
FNR == NR {
num[$1]
next
}
# process each line of the .BAM file
# any lines with an "unknown" $5 will be ignored
$5 in num {
output_path = "./Sample1/"
f = output_path "Alignments_" $5 ".sam"
print > f
} ' Tile_Number_List.txt little.sam
To pass the value of the shell variable such as $output_path to awk you need to use the -v option.
$ output_path=./Sample1/
$ awk -F '[:\t]' -v opath="$ouput_path" '
# read the list of numbers in Tile_Number_List
FNR == NR {
num[$1]
next
}
# process each line of the .BAM file
# any lines with an "unknown" $5 will be ignored
$5 in num {
f = opath"Alignments_"$5".sam"
print > f
} ' Tile_Number_List.txt little.sam
Also you still have the error from your previous question left in your script
EDIT:
The awk variable created with -v is obase but you use newdir what you want is:
input=$1
outputBase=${input%.bam}
mkdir -v $outputBase\_TEST
newdir=$outputBase\_TEST
samtools view -h "$input" | awk -F '[\t:]' -v opath="$newdir" '
FNR == NR && NR >= 18 {
num[$1]
next
}
$5 in num {
f = opath"/Alignments_"$5".sam" # <-- opath is the awk variable not newdir
print > f
}' Tile_Number_List.txt -
You should also move NR >= 18 into the second awk script.

How to change a file format from rows to columns?

file format change : top to bottom <> left to right
input file format:
100
150
200
300
500
output file format should be:
100,150,200,300,500
I need to apply this in reverse, too.
Just replace the linefeeds by comma:
$ tr '\n' ',' < input.txt > output.txt
and reverse
$ tr ',' '\n' < input.txt > output.txt
#!/bin/sh
i=0
while read line ; do
i=`expr $i + 1`
if [ $i -eq 1 ] ; then
echo -e "$line\c"
else
echo -e ",$line\c"
fi
done < filename
echo
Use this shell script to convert the \n as ,
The drawback in the tr command is end of the line one comma will be there to overcome
that use this script.

Resources