Is there any faster way to truncate column in Unix - unix

Question

I want to truncate 4th column of TSV file to given length in Unix. File has records in few millions and is of size 8GB.
I am trying this but it seems to be kind of slow.
awk -F"\t" '{s=substr($4,0,256); print $1"\t"$2"\t"$3"\t"s"\t"$5"\t"$6"\t"$7}' file > newFile
Is there any faster alternatives for same?
Thanks

Answer 1

Your command could be written a little more nicely (assuming you are re-building the record), which may give some performance increases:
awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,256) }' file > newFile
If you have access to a multi-core machine (which you probably do), you can use GNU parallel. You may want to vary the number of cores you use (I've set 4 here) and the block size that's fed to awk (I've set this to two megabytes)...
< file parallel -j 4 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' > newFile
Here's some testing I did on my system using a 2.7G file with 100 million lines and a block size of 2M:
time awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' file >/dev/null
Results:
real 1m59.313s
user 1m57.120s
sys 0m2.190s
With one core:
time < file parallel -j 1 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' >/dev/null
Results:
real 2m28.270s
user 4m3.070s
sys 0m41.560s
With four cores:
time < file parallel -j 4 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' >/dev/null
Results:
real 0m54.329s
user 2m41.550s
sys 0m31.460s
With twelve cores:
time < file parallel -j 12 --pipe --block 2M -q awk 'BEGIN { FS=OFS="\t" } { $4 = substr($4,0,2) }' >/dev/null
Results:
real 0m36.581s
user 2m24.370s
sys 0m32.230s

Answer 2

I’ll assume that your file has exactly one space character between fields and no whitespace at the beginning of the line.  If that is wrong, this can be enhanced. 
Otherwise, this should work:
sed 's/^\([^ ]* [^ ]* [^ ]* [^ ]\{1,256\}\)[^ ]* /\1 /'
I haven’t actually tested it with 256-character-long data (I tested it with \{1,2\} and I have no idea how its speed compares to that of awk.  BTW, on some versions, you might need to leave off the backslashes from the curly braces and use just {1,256}.

Answer 3

If Scott or Steve's solutions are still too slow, it may be time to break out the C. Run as ./a.out < file > newFile. Test on a small file with some long fields first; I am not 100% sure I have the math right.
#include <stdio.h>
int
main(void)
{
int field = 1;
int character = 0;
int c;
while ((c = getchar()) != EOF)
{
switch (c)
{
case '\n':
field = 1;
character = 0;
break;
case '\t':
character = 0;
field++;
break;
default:
character++;
break;
}
if (field != 4 || character < 256)
putchar(c);
}
if (ferror(stdout) || fflush(stdout) || fclose(stdout))
{
perror("write");
return 1;
}
return 0;
}