I have a file that is sorted like the following:
2 Good
2 Hello
3 Goodbye
3 Begin
3 Yes
3 No
I want to search for the highest value in the file and display what is one the line?
3 Goodbye
3 begin
3 Yes
3 No
How would I do this?
awk to the rescue!
$ awk 'FNR==NR{if(max<$1) max=$1; next} $1==max' file{,}
3 Goodbye
3 Begin
3 Yes
3 No
double-pass, find the maximum and filter out the rest.
cat file.txt | sort -r | awk '{if ($1>=prev) {print $0; prev=$1}}'
3 Yes
3 No
3 Goodbye
3 Begin
Assuming file.txt contains
2 Good
2 Hello
3 Goodbye
3 Begin
3 Yes
3 No
First get the highest value in the file into a variable. Considering the file is already sorted, pickup the last line in the file. Then parse out the number using awk.
highest=`tail -1 file.list|awk '{print $1}'`
Then grep the file using that value.
grep "^${highest} " file.list
This should do the job. I am only using awk as required in the question:
awk 'BEGIN {v=0} {l = l "\n" $0} {if ($1>v) {l = $0; v = $1}} END {print l}' file.txt
The variable v is initialized (before parsing the file) to 0. Then each line is read and kept in memory; if the first field ($1) is greater than v, then update v and empty what is in l. At the end, just print the content of l.
It's easier than you think.
awk '/^3/' file
3 Goodbye
3 Begin
3 Yes
3 No
Related
I have a text file:
head train_test_split.txt
1 0
2 1
3 0
4 1
5 1
What I want to do is save the first column values for which second column value is 1 to file train.txt.
So, the corresponding first column value for second column value with 1 are: 2,4,5. So, in my train.txt file i want:
2
4
5
How can I do this easily unix?
You can use awk for this:
awk '$2 == 1 { print $1 }' inputfile
That is,
$2 == 1 is a filter,
matching lines where the 2nd column is 1,
and print $1 means to print the first column.
In Perl:
$ perl -lane 'print "$F[0]" if $F[1]==1' file
Or GNU grep:
$ grep -oP '^(\S+)(?=[ \t]+1$)' file
But awk is the best. Use awk...
Hi I have a Unix command that produces a list of ip addresses along with other columns information . i want to add something to the command so that it displays it as a set of 3 lines then a space or ---- and then the next 3 lines and so on.
how can I achieve this ?
for example:
1.2.3.4 xy
1.3.5.7 ab
1.25.7.9 cd
-------------
1.25.7.8 kl
1.3.4.5 mn
1.25.7.8 op
-------------
1.24.5.6 la
1.3.4.5 ka
1.25.7.8 xz
You can use awk to print an extra line after every 40 lines:
awk '{ print $0; if(++i % 40 == 0) printf("-------------\n") }' file
Change % 40 to % 3 to print that extra line after every 3 lines.
$ seq 9 | awk 'NR>1 && (NR%3)==1{print "---"} 1'
1
2
3
---
4
5
6
---
7
8
9
To convert rows into tab-delimited, it's easy
cat input.txt | tr "\n" " "
But I have a long file with 84046468 lines. I wish to convert this into a file with 1910147 rows and 44 tab-delimited columns. The first column is a text string such as chrXX_12345_+ and the other 43 columns are numerical strings. Is there a way to perform this transformation?
There are NAs present, so I guess sed and substituting "\n" for "\t" if the string preceding is a number doesn't work.
sample input.txt
chr10_1000103_+
0.932203
0.956522
1
0.972973
1
0.941176
1
0.923077
1
1
0.909091
0.9
1
0.916667
0.8
1
1
0.941176
0.904762
1
1
1
0.979592
0.93617
0.934783
1
0.941176
1
1
0.928571
NA
1
1
1
0.941176
1
0.875
0.972973
1
1
NA
0.823529
0.51366
chr10_1000104_-
0.952381
1
1
0.973684
sample output.txt
chr10_1000103_+ 0.932203 (numbers all tab-delimited)
chr10_1000104_- etc
(sorry alot of numbers to type manually)
sed '
# use a delimiter
s/^/M/
:Next
# put a counter
s/^/i/
# test counter
/^\(i\)\{44\}/ !{
$ !{
# not 44 line or end of file, add the next line
N
# loop
b Next
}
}
# remove marker and counter
s/^i*M//
# replace new line by tab
s/\n/ /g' YourFile
some limite if more than 255 tab on sed (so 44 is ok)
Here's the right approach using 4 columns instead of 44:
$ cat file
chr10_1000103_+
0.932203
0.956522
1
chr10_1000104_-
0.952381
1
1
$ awk '{printf "%s%s", $0, (NR%4?"\t":"\n")}' file
chr10_1000103_+ 0.932203 0.956522 1
chr10_1000104_- 0.952381 1 1
Just change 4 to 44 for your real input.
If you are seeing control-Ms in your output it's because they are present in your input so use dos2unix or similar to remove them before running the tool or with GNU awk you could just set -v RS='\n\r'.
When posting questions it's important to make it as clear, simple, and brief as possible so that as many people as possible will be interested in helping you.
BTW, cat input.txt | tr "\n" " " is a UUOC and should just be tr "\n" " " < input.txt
Not the best solution, but should work:
line="nonempty"; while [ ! -z "$line" ]; do for i in $(seq 44); do read line; echo -n "$line "; done; echo; done < input.txt
If there is an empty line in the file, it will terminate. For a more permanent solution I'd try perl.
edit:
If you are concerned with efficiency, just use awk.
awk '{ printf "%s\t", $1 } NR%44==0{ print "" }' < input.txt
You may want to strip the trailing tab character with | sed 's/\t$//' or make the awk script more complicated.
This might work for you (GNU sed):
sed '/^chr/!{H;$!d};x;s/\n/\t/gp;d' file
If a line does not begin with chr append it to the hold space and then delete it unless it is the last. If the line does start chr or it is the last line, then swap to the hold space and replace all newlines by tabs and print out the result.
N.B. the start of the next line will be left untouched in the pattern space which becomes the new hold space.
Similar question to many previous ones (including mine) but I can't find the solution. This is purely a syntax error and I cannot figure out how to make it work.
I have two files in Unix. In file1 I have 5 columns and about 6000 rows. I am trying to match rows in file2 to rows in file1 IF column 1 matches exactly AND if the value in row 5 of file1 is less than 0.00000005 for said row.
file1:
SNPs Context Intergenic Risk Allele Frequency p-Value
rs9747992 Intergenic 1 0.086 2.00E-07
rs2059865 Intron 0 0.235 3.00E-07
rs117020818 Intergenic 1 0.046 7.00E-07
rs1074145 Intergenic 1 0.162 4.00E-09
file2:
snpid hg18chr bp a1 a2 zscore pval CEUmaf
rs3131972 1 742584 A G 0.289 0.7726 .
rs3131969 1 744045 A G 0.393 0.6946 .
rs3131967 1 744197 T C 0.443 0.658 .
rs1048488 1 750775 T C -0.289 0.7726 .
I can do the first part BUT it keeps outputting a file that is larger than the first two. I am unsure if this is a real result file or just full of duplicates? I also cannot do the 'less than' command. I have tried putting it into the command as a second pattern and also piping it, as below:
awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a) {print $0}}' file1 file2 > output | awk '{if (a[$5] < 0.00000005)}'
and
awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a && $5 < 0.00000005)} {print $0}}' file1 file2 > output
Both times it's giving me the same size file which is much larger than either file1 or file2. If you want examples of the tables please just say.
Tentative solution:
A tentative solution I am using is to just make a new file containing only lines from file1 which have that <0.00000005 value. This works though I would like to know my original answer for posterity.
awk '$5<=0.00000005' file1 > file11
Per my comments above, if you're using file2 as a filter list, you need to load it into the a[] array.
I've made up a small sample of how that works, the test for $28 < .000005 should be easy to add as you have it in your code.
With file data1
1 2 3 4 5 6 7
2 3 4 5 6 7 8
4 5 8 7 8 9 10
and file searchList
3
Then
awk 'FNR==NR{a[$0]=$0;next}
FNR!=NR{ if ($2 in a) print $0}
#dbg END{for (x in a) print "x="x " a[x]=" a[x]
}' searchList data1
gives output
2 3 4 5 6 7 8
edit Per our conversation in comments, my best guess without seeing your required output would be
I've added an extra record in file1 so there can be match
rs3131972 Intergenic 1 0.086 2.00E-07
awk '( FNR==NR && (sprintf("%.07f",$5) < .000000005) ) {
a[$1]=$0
#dbg print "a["$1"]="a[$1]
next
}
FNR!=NR{
#dbg print "$1="$1
if ($1 in a)print "Matched:" $0
}' file1 file2
The output is now
Matched:rs3131972 1 742584 A G 0.289 0.7726 .
IHTH
Shellter's answer is good. Mine is more about what you did wrong. Your first attempt
> awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a) {print $0}}
' file1 file2 > output | awk '{if (a[$5] < 0.00000005)}'
fails because your pipeline is wrong. You need to pipe awk | awk > output not awk >output | awk. The latter will receive no input and produce no output from the last step of the pipeline. Also, the second Awk instance has no knowledge of the variables you used in the first.
Furthermore, you seem to have a recurring problem with spurious braces in Awk. The general syntax is awk "condition1 { action1 } condition2 { action2 }..." where you can omit a condition to do an action unconditionally, or omit the action part (with the braces) to perform the default action { print $0 }. But here, you have only an action, which is however actually a condition, with no side effects such as printing anything. You want to remove the braces and the if wrapper.
So you need
awk 'FNR==NR{a[$1]=$0;next}{if ($1 in a) {print $0}}' file1 file2 |
awk '$5 < 0.00000005' >output
which (in accordance with the rules for omitting a condition or an action, and with some refactoring) can be much simplified to
awk 'FNR==NR{a[$1]=$0;next}
$1 in a' file1 file2 |
awk '$5 < 0.00000005' >output
Your second attempt is closer;
> awk 'FNR==NR{a[$1]=$0;next}
{if ($1 in a && $5 < 0.00000005)} {print $0}}' file1 file2 > output
but again, you have too many brackets. The closing brace after the if ruins it all! So you have effectively "if (condition)" then nothing (maybe this should be a syntax error!), followed by a new block with an unconditional print. But overall, this is much better.
awk 'FNR==NR{a[$1]=$0;next}
{if ($1 in a && $5 < 0.00000005) print $0}' file1 file2 > output
which of course can be simplified to
awk 'FNR==NR{a[$1]=$0;next}
($1 in a) && $5 < 0.00000005' file1 file2 > output
Answer that worked based on Shellters assistance.
awk -F $'\t' 'NR==FNR{if ($5 < 0.00000005){a[$1]=$0}} NR!=FNR{if ($1 in a) print $0}' file1 file2 > output
Thanks
I know how to get a range of lines by using awk and sed.
I also do know how to print out every nth line using awk and sed.
However, I don't know how to combined the two.
For example, I have a file with 1780000 lines.
For every 17800th line, I would like to print 17800th line plus the two after that.
So if I have a file with 1780000 lines and it starts from 1 and ends at 1780000, this will print:
1
2
3
17800
17801
17802
35600
35601
35602
# ... and so on.
Does anyone know how to get a range of line every nth interval using awk, sed, or other unix command?
Using GNU sed:
sed -n '0~17800{N;N;p}' input
Meaning,
For every 17800th line: 0~17800
Read two lines: {N;N;
And print these out: p}
We can also add the first three lines:
sed -n -e '1,3p' -e '0~17800{N;N;p}' input
Using Awk, this would be simpler:
awk 'NR%17800<3 || NR==3 {print}' input
$ cat file
1
2
3
4
5
6
7
8
9
10
$ awk '!(NR%3)' file
3
6
9
$ awk -v intvl=3 -v delta=2 '!(NR%intvl){print "-----"; c=delta} c&&c--' file
-----
3
4
-----
6
7
-----
9
10
$ awk -v intvl=4 -v delta=2 '!(NR%intvl){print "-----"; c=delta} c&&c--' file
-----
4
5
-----
8
9
$ awk -v intvl=4 -v delta=3 '!(NR%intvl){print "-----"; c=delta} c&&c--' file
-----
4
5
6
-----
8
9
10
seq -f %.0f 1780000 | awk 'NR < 4 || NR % 17800 < 3' | head
output:
1
2
3
17800
17801
17802
35600
35601
35602
53400
Explanation
The NR < 4 is for the first 3 lines because the requirement For every 17800th line, print 17800th line plus the two after that. doesn't fit the output you gave.
Here I use head for reducing the output size and you should remove it in your use case.
For GNU seq, you don't need -f %.0f.