I wanted to drop first 3 column;
This is my data;
DETAIL 02032017
Name Gender State School Class
A M Melaka SS D
B M Johor BB E
C F Pahang AA F
EOF 3
I want my data like this:
DETAIL 02032017
School Class
SS D
BB E
AA F
EOF 3
This is my current command that I get mycommandoutput:
awk -v date="$(date +"%d%m%Y")" -F\| 'NR==1 {h=$0; next}
{file="TEST_"$1"_"$2"_"date".csv";
print (a[file]++?"": "DETAIL"date"" ORS h ORS) $0 > file} END{for(file in a) print "EOF " a[file] > file}' testing.csv
Can anyone help me?
Thank you :)
I want to remove first three column
If you just want to remove the first three columns, you can just set them to empty strings, leaving alone those that don't have three columns, something like:
awk 'NF>=3 {$1=""; $2=""; $3=""; print; next}{print}'
That has the potentially annoying habit of still having the field separators between those empty fields but, since modifying columns will reformat the line anyway, I assume that's okay:
DETAIL 02032017
School Class
SS D
BB E
AA F
EOF 3
If awk is the only tool being used to process them, the spacing won't matter. If you do want to preserve formatting (meaning that the columns are at very specific locations on the line), you can just get a substring of the entire line:
awk '{if (NF>=3) {$0 = substr($0,25)}; print}'
Since that doesn't modify individual fields, it won't trigger a recalculation of the line that would change its format:
DETAIL 02032017
School Class
SS D
BB E
AA F
EOF 3
Related
Say I have 6 different columns in a text file (as shown below)
A1 B1 C1 D1 E1 F1
1 G PP GG HH GG
z T CC GG FF JJ
I would like to extract columns first, second and fourth columns as A1_B1_D1 collapsed together and the third column separated by tab.
So the result would be:
A1_B1_D1 C1
1_G_GG PP
z_T_GG CC
I tried
cut -f 1,2,4 -d$'\t' 3, but is just not what I want.
If you need to maintain your column alignment, you can check the length of the combination of fields 1, 2 and 4 and add one or two tab characters as necessary,
awk '{
printf (length($1"_"$2"_"$4) >= 8) ? "%s_%s_%s\t%s\n" : "%s_%s_%s\t\t%s\n",
$1,$2,$4,$3
}' file
Example Output
A1_B1_D1 C1
1_G_GG PP
z_T_GG CC
Could you please try following.
awk '
BEGIN{
OFS="\t"
}
{
print $1"_"$2"_"$4,$3
}
' Input_file
I've tried RavinderSingh13 code and it has the same output as mine but I don't quite know the difference, anyways, here it is:
awk -F ' ' '{print $1"_"$2"_"$4"\t"$3}' /path/to/file
This might work for you (GNU sed):
sed 's/^(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+.*/\1_\2_\4\t\3/' -E file
Use pattern matching and back references.
\S+ means one or more non-white space characters.
\s+ means one or more white space characters.
\t represents a tab.
Another awk and using column -t for formatting.
$ cat cols_345.txt
A1 B1 C1 D1 E1 F1
1 G PP GG HH GG
z T CC GG FF JJ
$ awk -v OFS="_" '{ $3="\t"$3; print $1,$2,$4 $3 } ' cols_345.txt | column -t
A1_B1_D1 C1
1_G_GG PP
z_T_GG CC
$
I have a simple text file, delimited by multiple spaces, and with a different number of columns (6 or 5).
What I am trying to do is, for the rows with more than 5 columns, combine the 2 last columns in one, doing:
cat data.txt | awk '{if(NF>5) print $1,$2,$3,$4,$5"_"$6; else print $0} OFS="," ' > data.csv
The problem is that the OFS is not working for the else statement.
Example - input:
a d e t er ap
b q j n mm
Output that I am getting:
a,d,e,t,er_ap
b q j n mm
Desirable output:
a,d,e,t,er_ap
b,q,j,n,mm
Any suggestions?
Set your OFS in the BEGIN block so that it's a comma before any processing happens. Also when you do print $0 without manipulating the line in any way, awk will just spit out the line as-is with whatever delimiters are in place in the source file. Personally I think that's dumb, but that's awk. As a workaround, just set one column equal to itself, then print:
awk 'BEGIN{OFS=","}{if(NF>5) print $1,$2,$3,$4,$5"_"$6; else {$1=$1;print $0}}' data.txt
If you anticipate more than 6 columns you can just have it toss underscores for all of them after column 5 with some printf trickery too
awk '{for (i=1;i<=NF;i++){printf (i==NF)?"%s\n":(i>=5)?"%s_":"%s,", $i}}' data.txt
count how many times A and B shows up in this example file:
Ex:
1,2,3,A
2,3,1,A
3,1,2,A
1,2,3,B
1,3,2,B
Expected Output should be:
A 3
B 2
So far I have:
grep -cw "*A" <file>
with output:
3
Which only displays the number of occurrences.
Try this:
mayankp#mayank:~/$ cat t1.txt
1,2,3,A
2,3,1,A
3,1,2,A
1,2,3,B
1,3,2,B
mayankp#mayank:~/$ awk -F, 'NR{arr[$4]++}END{for (a in arr) print a, arr[a]}' t1.txt
A 3
B 2
Let me know if this helps.
I recently used the awk command to remove duplicate lines, and spaces between lines but I am not getting the desired output file.
Input file:
a b
a b
c d
c d
e f
e f
Desired output:(I wanted to remove duplicate lines and all spaces in between lines)
a b
c d
e f
I used the following code:
awk '!x[$0]++' input file > output file
And got this output:
a b
c d
e f
The space between the first line and all the rest is still in the output file.
Help please and thank you.
awk 'NF && !seen[$0]++' inputfile.txt > outputfile.txt
NF removes white lines or lines containing only tabs or whitespaces.
!seen[$0]++ removes duplicates.
If the original line order of the input is important, then the following will not work for you. If you don't care about the order, then read on.
For me, awk is not the best tool for this problem.
Since you are trying to use awk, I assume you are in a unix-like environment, so:
When I hear "eliminate blank lines" I think "grep".
When I hear "eliminate duplicate lines" I think "uniq" (which requires sort, though not in your example since it is already sorted.)
So, given a file 'in.txt' that duplicates your example, the following produces the desired output.
grep -v "^[[:space:]]*$" in.txt | uniq
Now, if your real data is not sorted, that won't work. Instead use:
grep -v "^[[:space:]]*$" in.txt | sort -u
Your output may be in a different order than the input in this case.
cat test
a b
a b
c d
c d
e f
e f
awk '$0 !~ /^[[:space:]]*$/' test
a b
a b
c d
c d
e f
e f
Similar question to many previous ones (including mine) but I can't find the solution. This is purely a syntax error and I cannot figure out how to make it work.
I have two files in Unix. In file1 I have 5 columns and about 6000 rows. I am trying to match rows in file2 to rows in file1 IF column 1 matches exactly AND if the value in row 5 of file1 is less than 0.00000005 for said row.
file1:
SNPs Context Intergenic Risk Allele Frequency p-Value
rs9747992 Intergenic 1 0.086 2.00E-07
rs2059865 Intron 0 0.235 3.00E-07
rs117020818 Intergenic 1 0.046 7.00E-07
rs1074145 Intergenic 1 0.162 4.00E-09
file2:
snpid hg18chr bp a1 a2 zscore pval CEUmaf
rs3131972 1 742584 A G 0.289 0.7726 .
rs3131969 1 744045 A G 0.393 0.6946 .
rs3131967 1 744197 T C 0.443 0.658 .
rs1048488 1 750775 T C -0.289 0.7726 .
I can do the first part BUT it keeps outputting a file that is larger than the first two. I am unsure if this is a real result file or just full of duplicates? I also cannot do the 'less than' command. I have tried putting it into the command as a second pattern and also piping it, as below:
awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a) {print $0}}' file1 file2 > output | awk '{if (a[$5] < 0.00000005)}'
and
awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a && $5 < 0.00000005)} {print $0}}' file1 file2 > output
Both times it's giving me the same size file which is much larger than either file1 or file2. If you want examples of the tables please just say.
Tentative solution:
A tentative solution I am using is to just make a new file containing only lines from file1 which have that <0.00000005 value. This works though I would like to know my original answer for posterity.
awk '$5<=0.00000005' file1 > file11
Per my comments above, if you're using file2 as a filter list, you need to load it into the a[] array.
I've made up a small sample of how that works, the test for $28 < .000005 should be easy to add as you have it in your code.
With file data1
1 2 3 4 5 6 7
2 3 4 5 6 7 8
4 5 8 7 8 9 10
and file searchList
3
Then
awk 'FNR==NR{a[$0]=$0;next}
FNR!=NR{ if ($2 in a) print $0}
#dbg END{for (x in a) print "x="x " a[x]=" a[x]
}' searchList data1
gives output
2 3 4 5 6 7 8
edit Per our conversation in comments, my best guess without seeing your required output would be
I've added an extra record in file1 so there can be match
rs3131972 Intergenic 1 0.086 2.00E-07
awk '( FNR==NR && (sprintf("%.07f",$5) < .000000005) ) {
a[$1]=$0
#dbg print "a["$1"]="a[$1]
next
}
FNR!=NR{
#dbg print "$1="$1
if ($1 in a)print "Matched:" $0
}' file1 file2
The output is now
Matched:rs3131972 1 742584 A G 0.289 0.7726 .
IHTH
Shellter's answer is good. Mine is more about what you did wrong. Your first attempt
> awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a) {print $0}}
' file1 file2 > output | awk '{if (a[$5] < 0.00000005)}'
fails because your pipeline is wrong. You need to pipe awk | awk > output not awk >output | awk. The latter will receive no input and produce no output from the last step of the pipeline. Also, the second Awk instance has no knowledge of the variables you used in the first.
Furthermore, you seem to have a recurring problem with spurious braces in Awk. The general syntax is awk "condition1 { action1 } condition2 { action2 }..." where you can omit a condition to do an action unconditionally, or omit the action part (with the braces) to perform the default action { print $0 }. But here, you have only an action, which is however actually a condition, with no side effects such as printing anything. You want to remove the braces and the if wrapper.
So you need
awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a) {print $0}}' file1 file2 |
awk '$5 < 0.00000005' >output
which (in accordance with the rules for omitting a condition or an action, and with some refactoring) can be much simplified to
awk 'FNR==NR{a[$1]=$0;next}
$1 in a' file1 file2 |
awk '$5 < 0.00000005' >output
Your second attempt is closer;
> awk 'FNR==NR{a[$1]=$0;next}
{if ($1 in a && $5 < 0.00000005)} {print $0}}' file1 file2 > output
but again, you have too many brackets. The closing brace after the if ruins it all! So you have effectively "if (condition)" then nothing (maybe this should be a syntax error!), followed by a new block with an unconditional print. But overall, this is much better.
awk 'FNR==NR{a[$1]=$0;next}
{if ($1 in a && $5 < 0.00000005) print $0}' file1 file2 > output
which of course can be simplified to
awk 'FNR==NR{a[$1]=$0;next}
($1 in a) && $5 < 0.00000005' file1 file2 > output
Answer that worked based on Shellters assistance.
awk -F $'\t' 'NR==FNR{if ($5 < 0.00000005){a[$1]=$0}} NR!=FNR{if ($1 in a) print $0}' file1 file2 > output
Thanks