I have file as
1|dev|Smith|78|minus
1|ana|jhon|23|plus
1|ana|peter|22|plus
2|dev|dash|45|minus
2|dev||44|plus
I want output as, against uniq value of column 1 and 2 print multiple values of column 3 and 5
1|dev|Smith|minus
1|ana|jhon;peter|plus;plus
2|dev|dash;|minus;plus
I can accumulate multiple records into 1 just for one column , I want to do it for 2 column in one command
awk -F"|" '{if(a[$1"|"$2])a[$1"|"$2]=a[$1"|"$2]";"$5; else
a[$1"|"$2]=$5;}END{for (i in a)print i, a[i];}' OFS="|" input.txt > output.txt
It is giving output as
2|dev|minus;plus
1|ana|plus;plus
1|dev|minus
If datamash is okay
$ # -g 1,2 tells to group by 1st and 2nd column
$ # collapse 3 collapse 5 tells to combine those column values
$ datamash -t'|' -g 1,2 collapse 3 collapse 5 < ip.txt
1|dev|Smith|minus
1|ana|jhon,peter|plus,plus
2|dev|dash,|minus,plus
$ # easy to change , to ; if input file doesn't contain ,
$ datamash -t'|' -g 1,2 collapse 3 collapse 5 < ip.txt | tr ',' ';'
1|dev|Smith|minus
1|ana|jhon;peter|plus;plus
2|dev|dash;|minus;plus
In awk, not the usual way, but first setting $3|$5 and then adding outwards like <-;$3|$5;-> to $3;$3|$5;$5, that's why ;dash instead of dash;:
$ awk '
BEGIN { FS=OFS="|" }
{
a[$1 OFS $2]=$3(a[$1 OFS $2]?";"a[$1 OFS $2]";":"|")$5
}
END {
for(i in a)
print i,a[i]
}' file
2|dev|;dash|minus;plus
1|ana|peter;jhon|plus;plus
1|dev|Smith|minus
The proper awk way would probably be closer to:
$ awk '
BEGIN { FS=OFS="|" }
{
i=$1 OFS $2
a[i] = a[i] ( a[i]=="" || $3=="" ? "" : ";" ) $3
b[i] = b[i] ( b[i]=="" || $5=="" ? "" : ";" ) $5
}
END {
for(i in a)
print i,a[i],b[i]
}' file
2|dev|dash|minus;plus
1|ana|jhon;peter|plus;plus
1|dev|Smith|minus
Related
I have a file that looks like this:
cat f1.csv:
col1,col2,col3
AK136742,BC051226,996
AK161599,Gm15417,4490
AK161599,Gm15417,6915
AK161599,Zbtb7b,1339
AK161599,Zbtb7b,1475
AK161599,Zbtb7b,1514
What I want to do is to keep one of the otherwise-duplicated rows if they have a greater number on col3. So if the col1 and col2 are the same then keep the row if has the greater number on the col3.
So the desired output should be:
col1,col2,col3
AK136742,BC051226,996
AK161599,Gm15417,6915
AK161599,Zbtb7b,1514
I used the command below but it does not solve the problem:
cat f1.csv | sort -rnk3 | awk '!x[$3]++'
Any help is appreciated - thanks!
with your shown samples, please try following.
awk '
BEGIN{
FS=OFS=","
}
{ ind = $1 FS $2 }
FNR==1{
print
next
}
{
arr[ind]=(arr[ind]>$NF?arr[ind]:$NF)
}
END{
for(i in arr){
print i,arr[i]
}
}
' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section of this program from here.
FS=OFS="," ##Setting FS, OFS as comma here.
}
{ ind = $1 FS $2 } ##Setting ind as 1st and 2nd field value here.
FNR==1{ ##Checking if its first line.
print ##Then print it.
next ##next will skip all further statements from here.
}
{
arr[ind]=(arr[ind]>$NF?arr[ind]:$NF) ##Creating arr with index of ind and keeping only higher value after each line comparison of last field.
}
END{ ##Starting END block of this program from here.
for(i in arr){ ##Starting a for loop here.
print i,arr[i] ##Printing index and array arr value here.
}
}
' Input_file ##Mentioning Input_file name here.
$ head -n 1 f1.csv; { tail -n +2 f1.csv | sort -t, -k1,2 -k3rn | awk -F, '!seen[$1,$2]++'; }
col1,col2,col3
AK136742,BC051226,996
AK161599,Gm15417,6915
AK161599,Zbtb7b,1514
or to avoid naming the input file twice (e.g. so it'll work if the input is a pipe):
$ awk '{print (NR>1) "," $0}' f1.csv | sort -t, -k1,1n -k1,2 -k3rn | cut -d',' -f2- | awk -F, '!seen[$1,$2]++'
col1,col2,col3
AK136742,BC051226,996
AK161599,Gm15417,4490
AK161599,Zbtb7b,1339
The answers provided seem a little complicated to me. Here's an answer all in awk:
#! /usr/bin/awk -f
NR == 1 {
heading = $0
next
}
{
key = $1 "," $2
if( values[key] < $3 ) {
values[key] = $3
}
}
END {
print heading
for( k in values ) {
print k "," values[k] | "sort -t, -k1,2"
}
}
$ ./max.awk -F, max.dat
col1,col2,col3
AK136742,BC051226,996
AK161599,Gm15417,6915
AK161599,Zbtb7b,1514
Using sort, you need
sort -t, -k3,3nr file.csv | sort -t, -su -k1,2
The first sort sorts the input numerically by the 3rd column in the descending order. The second sort is stable -s (not all sort implementations support that) and uniques the output by the first two columns, thus leaving the maximum for each combination.
I ignored the header line.
This might be simple - I have a file as below:
df.csv
col1,col2,col3,col4,col5
A,2,5,7,9
B,6,10,2,3
C,3,4,6,8
I want to perform max(col2,col4) - min(col3,col5) but I get an error using max and min in awk and write the result in a new column. So the desired output should look like:
col1,col2,col3,col4,col5,New_col
A,2,5,7,9,2
B,6,10,2,3,3
C,3,4,6,8,2
I used the code below but is does not work - how can I solve this?
awk -F, '{print $1,$2,$3,$4,$5,$(max($7,$9)-min($8,$10))}'
Thank you.
$ cat tst.awk
BEGIN { FS=OFS="," }
{ print $0, (NR>1 ? max($2,$4) - min($3,$5) : "New_col") }
function max(a,b) {return (a>b ? a : b)}
function min(a,b) {return (a<b ? a : b)}
$ awk -f tst.awk file
col1,col2,col3,col4,col5,New_col
A,2,5,7,9,2
B,6,10,2,3,3
C,3,4,6,8,2
If your actual "which is larger" calculation is more involved than just using >, e.g. if you were comparing dates in some non-alphabetic format or peoples names where you have to compare the surname before the forename and handle titles, etc., then you'd write the functions as:
function max(a,b) {
# some algorithm to compare the 2 strings
}
function min(a,b) {return (max(a,b) == a ? b : a)}
You may use this awk:
awk 'BEGIN{FS=OFS=","} NR==1 {print $0, "New_col"; next} {print $0, ($2 > $4 ? $2 : $4) - ($3 < $5 ? $3 : $5)}' df.csv
col1,col2,col3,col4,col5,New_col
A,2,5,7,9,2
B,6,10,2,3,3
C,3,4,6,8,2
A more readable version:
awk '
BEGIN { FS = OFS = "," }
NR == 1 {
print $0, "New_col"
next
}
{
print $0, ($2 > $4 ? $2 : $4) - ($3 < $5 ? $3 : $5)
}' df.csv
get an error using max and min in awk and write the result in a new column.
No such function are available in awk but for two values you might harness ternary operator, so in place of
max($7,$9)
try
($7>$9)?$7:$9
and in place of
min($8,$10)
try
($8<$10)?$8:$10
Above exploit ?: which might be explained as check?valueiftrue:valueiffalse, simple example, let file.txt content be
100,100
100,300
300,100
300,300
then
awk 'BEGIN{FS=","}{print ($1>$2)?$1:$2}' file.txt
output
100
300
300
300
Also are you sure about 1st $ in $(max($7,$9)-min($8,$10))? By doing so you instructed awk to get value of n-th column, where n is result of computation inside (...).
for below scenario i need ur help.
File 1: Is a config file
a|b|c|d|e|f|g
File 2: Input file
a|c|d|g
I have to compare "File 1" and "File 2" and print the following from file 2
a||c|d|||g
So basically I need to compare both the records and for the matching records i have to print it from file 2 and when not matching i have to put NULLS.
I think I can interpret what you're asking: you want to "remap" File2 onto the headers of File1.
$ cat File1
first|last|email|phone|address|colour|size
$ cat File2
first|last|phone|size
Glenn|Jackman|555-555-1212|L
$ awk -F '|' '
NR == FNR {
n = NF
for (i=1; i<=NF; i++) head[i] = $i
print
next
}
FNR == 1 {
for (i=1; i<=NF; i++) f2head[i] = $i
next
}
{
for (i=1; i<=NF; i++) data[f2head[i]] = $i
for (i=1; i<=n; i++)
printf "%s%s", data[head[i]], (i<n ? FS : RS)
}
' File1 File2
first|last|email|phone|address|colour|size
Glenn|Jackman||555-555-1212|||L
From the first block, the head array will be:
{1:"first",2:"last",3:"email",4:"phone",5:"address",6:"colour",7:"size"}
From the second block, the f2head array will be:
{1:"first",2:"last",3:"phone",4:"size"}
In the third block, the data array will be
{"first":"Glenn","last":"Jackman","phone":"555-555-1212","size":"L"}
another awk
$ awk 'BEGIN {FS=OFS="|"};
NR==1 {n=split($0,ht);
for(i=1;i<=n;i++) h[ht[i]]=i; next}
NR==2 {n2=split($0,h2)}
{split($0,t); $0="";
for(i=1;i<=n2;i++) $(h[h2[i]])=t[i]}1' file1 file2
seems cryptic but in essence does the following:
find the index of header2 (h2) values in header1 (h) and set the corresponding field ($(h[h2[i]])) in from file2 (t[i]).
I am trying to split a file into different smaller files depending on the value of the fifth field. A very nice way to do this was already suggested and also here.
However, I am trying to incorporate this into a .sh script for qsub, without much success.
The problem is that in the section where the file to which output the line is specified,
i.e., f = "Alignments_" $5 ".sam" print > f
, I need to pass a variable declared earlier in the script, which specifies the directory where the file should be written. I need to do this with a variable which is built for each task when I send out the array job for multiple files.
So say $output_path = ./Sample1
I need to write something like
f = $output_path "/Alignments_" $5 ".sam" print > f
But it does not seem to like having a $variable that is not a $field belonging to awk. I don't even think it likes having two "strings" before and after the $5.
The error I get back is that it takes the first line of the file to be split (little.sam) and tries to name f like that, followed by /Alignments_" $5 ".sam" (those last three put together correctly). It says, naturally, that it is too big a name.
How can I write this so it works?
Thanks!
awk -F '[:\t]' ' # read the list of numbers in Tile_Number_List
FNR == NR {
num[$1]
next
}
# process each line of the .BAM file
# any lines with an "unknown" $5 will be ignored
$5 in num {
f = "Alignments_" $5 ".sam" print > f
} ' Tile_Number_List.txt little.sam
UPDATE, AFTER ADDING -V TO AWK AND DECLARING THE VARIABLE OPATH
input=$1
outputBase=${input%.bam}
mkdir -v $outputBase\_TEST
newdir=$outputBase\_TEST
samtools view -h $input | awk 'NR >= 18' | awk -F '[\t:]' -v opath="$newdir" '
FNR == NR {
num[$1]
next
}
$5 in num {
f = newdir"/Alignments_"$5".sam";
print > f
} ' Tile_Number_List.txt -
mkdir: created directory little_TEST'
awk: cmd. line:10: (FILENAME=- FNR=1) fatal: can't redirect to `/Alignments_1101.sam' (Permission denied)
awk variables are like C variables - just reference them by name to get their value, no need to stick a "$" in front of them like you do with shell variables:
awk -F '[:\t]' ' # read the list of numbers in Tile_Number_List
FNR == NR {
num[$1]
next
}
# process each line of the .BAM file
# any lines with an "unknown" $5 will be ignored
$5 in num {
output_path = "./Sample1/"
f = output_path "Alignments_" $5 ".sam"
print > f
} ' Tile_Number_List.txt little.sam
To pass the value of the shell variable such as $output_path to awk you need to use the -v option.
$ output_path=./Sample1/
$ awk -F '[:\t]' -v opath="$ouput_path" '
# read the list of numbers in Tile_Number_List
FNR == NR {
num[$1]
next
}
# process each line of the .BAM file
# any lines with an "unknown" $5 will be ignored
$5 in num {
f = opath"Alignments_"$5".sam"
print > f
} ' Tile_Number_List.txt little.sam
Also you still have the error from your previous question left in your script
EDIT:
The awk variable created with -v is obase but you use newdir what you want is:
input=$1
outputBase=${input%.bam}
mkdir -v $outputBase\_TEST
newdir=$outputBase\_TEST
samtools view -h "$input" | awk -F '[\t:]' -v opath="$newdir" '
FNR == NR && NR >= 18 {
num[$1]
next
}
$5 in num {
f = opath"/Alignments_"$5".sam" # <-- opath is the awk variable not newdir
print > f
}' Tile_Number_List.txt -
You should also move NR >= 18 into the second awk script.
How can I print every second row as tab delimited second column like below. thanx in advance.
input
wex
2
cr_1.b
4
output
wex 2
cr_1.b 4
Here's another option that doesn't depend on the length of lines:
awk '{ if (NR % 2 == 1) tmp=$0; else print tmp, $0; }' <filename>
If you really want a tab separator, use printf "%s\t%s\n",tmp,$0; instead.
Assuming you have no blank lines in your input file, this should do the trick:
awk 'length(f) > 0 { print f $0; f = "" } length(f) == 0 { f = $0 }' file