I am new to writing shell scripts
I am trying to write an AWK command which does exactly the below
cut --complement -c $IGNORE_RANGE file.txt > tmp
$IGNORE_RANGE can be of any value say, 1-5 or 5-10 etc
i cannot use cut since i am in AIX and AIX does not support --complement, is there any way to achieve this using AWK command
Example:
file.txt
abcdef
123456
Output
cut --complement -c 1-2 file.txt > tmp
cdef
3456
cut --complement -c 4-5 file.txt > tmp
abcf
1236
cut --complement -c 1-5 file.txt > tmp
f
6
Could you please try following, written and tested with shown samples. We have range variable of awk which should be in start_of_position-end_of_position and we could pass it as per need.
awk -v range="4-5" '
BEGIN{
split(range,array,"-")
}
{
print substr($0,1,array[1]-1) substr($0,array[2]+1)
}
' Input_file
OR to make it more clear in understanding wise try following:
awk -v range="4-5" '
BEGIN{
split(range,array,"-")
start=array[1]
end=array[2]
}
{
print substr($0,1,start-1) substr($0,end+1)
}
' Input_file
Explanation: Adding detailed explanation for above.
awk -v range="4-5" ' ##Starting awk program from here creating range variable which has range value of positions which we do not want to print in lines.
BEGIN{ ##Starting BEGIN section of this program from here.
split(range,array,"-") ##Splitting range variable into array with delimiter of - here.
start=array[1] ##Assigning 1st element of array to start variable here.
end=array[2] ##Assigning 2nd element of array to end variable here.
}
{
print substr($0,1,start-1) substr($0,end+1) ##Printing sub-string of current line from 1 to till value of start-1 and then printing from end+1 which basically means will skip that range of characters which OP does not want to print.
}
' Input_file ##Mentioning Input_file name here.
You can do this in awk:
awk -v st=1 -v en=2 '{print substr($0, 1, st-1) substr($0, en+1)}' file
cdef
3456
Or:
awk -v st=4 -v en=5 '{print substr($0, 1, st-1) substr($0, en+1)}' file
abcf
1236
Related
I have to read ids from one file and search it in second xml file, if found write that entire line to third file. file1 is 111 MB, file2 is 40 GB
File1.xml
id1
id2
id5
File2.xml
<employees>
<employee><id>id1</id><name>test1</name></employee>
<employee><id>id2</id><name>test2</name></employee>
<employee><id>id3</id><name>test3</name></employee>
<employee><id>id4</id><name>test4</name></employee>
<employee><id>id5</id><name>test5</name></employee>
<employee><id>id6</id><name>test6</name></employee>
</employees>
File3.xml : result
<employee><id>id1</id><name>test1</name></employee>
<employee><id>id2</id><name>test2</name></employee>
<employee><id>id5</id><name>test5</name></employee>
i tried it with grep
grep -i -f file1.xml file2.xml >> file3.xml
but its giving memory exhausted error.
Another way i tried it with loop and awk command.
#while read -r id;do
#awk -v pat="$id" '$0~pat' file2.xml >> file3.xml
#done < file1.xml
its also taking too much time.
What could be the best optimal solution for this.
With your shown samples, please try following awk code. Written and tested in GNU awk.
awk -v FPAT='<id>[^<]*</id>' '
FNR==NR{
arr["<id>"$0"</id>"]
next
}
($1 in arr)
' file1.xml file2.xml
Explanation: Adding detailed explanation for above.
awk -v FPAT='<id>[^<]*</id>' ' ##Starting awk program and setting FPAT to <id>[^<]*<\\/id>
FNR==NR{ ##Checking condition which will be TRUE when file1.xml is being read.
arr["<id>"$0"</id>"] ##Creating an array arr which has index of <id> $0 </id> here.
next ##next will skip all further statements from here.
}
($1 in arr) ##Checking condition if $1 is present in arr then print that line.
' file1.xml file2.xml ##Mentioning Input_file names here.
This should work in any awk version:
awk 'FNR == NR {
seen["<id>" $1 "</id>"]
next
}
match($0, /<id>[^<]*<\/id>/) && substr($0, RSTART, RLENGTH) in seen
' file1 file2
<employee><id>id1</id><name>test1</name></employee>
<employee><id>id2</id><name>test2</name></employee>
<employee><id>id5</id><name>test5</name></employee>
I wanted to merge a string ABC to a line that I have extracted from a file.
The following command is used to extract the lines 20-25 in file_ABC, take only the first column, which is then transposed to become a row (or line).
sed -n '20,25p' < file_ABC | awk '{print $1}' | paste -s
This is the result:
2727778 14734 0 0 0 2713044
I would like to add at the first position of this line the string ABC.
ABC 2727778 14734 0 0 0 2713044
Any suggestion on how to do that?
A quick hack would be to use something like
printf 'ABC\t%s\n' "$(sed -n '20,25p' < file_ABC | awk '{print $1}' | paste -s)"
You could modify your initial command instead to use awk for everything, though:
awk '
BEGIN {printf "ABC"}
NR>=20 && NR<=25 {printf "\t%s", $1}
END {print ""}
' file_ABC
This might work for you (GNU sed):
sed '20,25{s/\s.*//;H};$!d;x;s/^/ABC/;s/\n/ /g' file
Gather up the first column fields by appending them to the hold space for rows 20 to 25 only. At the end of the file prepend ABC and replace the introduced newlines by spaces.
For fun, bash only
filename=file_ABC
words=("${filename##*_}")
i=0
while read -r word rest_of_line; do
((++i < 20 )) && continue
(( i > 25 )) && break
words+=("$word")
done < "$filename"
join() { local IFS=$1; shift; echo "$*"; }
join $'\t' "${words[#]}"
But this will be much slower than a single awk call.
if you want to keep all in one script
$ awk 'BEGIN {line="ABC"}
NR>=20 && NR<=25 {line=line FS $1}
NR==25 {print line; exit}' file
improved version as suggested by #EdMorton
$awk 'NR>=20 {line=line OFS $1}
NR==25 {print "ABC" line; exit}' file
Below are the full file names.
qwertyuiop.abcdefgh.1234567890.txt
qwertyuiop.1234567890.txt
trying to use
awk -F'.' '{print $1}'
How can i use awk command to extract below output.
qwertyuiop.abcdefgh
qwertyuiop
Edit
i have a list of files in a directory
i am trying to extract time,size,owner,filename into seperate variables.
for filenames.
NAME=$(ls -lrt /tmp/qwertyuiop.1234567890.txt | awk -F'/' '{print $3}' | awk -F'.' '{print $1}')
$ echo $NAME
qwertyuiop
$
NAME=$(ls -lrt /tmp/qwertyuiop.abcdefgh.1234567890.txt | awk -F'/' '{print $3}' | awk -F'.' '{print $1}')
$ echo $NAME
qwertyuiop
$
expected
qwertyuiop.abcdefgh
With GNU awk and other versions that allow manipulation of NF
$ awk -F. -v OFS=. '{NF-=2} 1' ip.txt
qwertyuiop.abcdefgh
qwertyuiop
NF-=2 will effectively delete last two fields
1 is an awk idiom to print contents of $0
Note that this assumes there are at least two fields in every line, otherwise you'd get an error
Similar concept with perl, prints empty line if number of fields in the line is less than 3
$ perl -F'\.' -lane 'print join ".", #F[0..$#F-2]' ip.txt
qwertyuiop.abcdefgh
qwertyuiop
With sed, you can preserve lines if number of fields is less than 3
$ sed 's/\.[^.]*\.[^.]*$//' ip.txt
qwertyuiop.abcdefgh
qwertyuiop
EDIT: Taking inspiration from Sundeep sir's solution and adding this following too in this mix.
awk 'BEGIN{FS=OFS="."} {$(NF-1)=$NF="";sub(/\.+$/,"")} 1' Input_file
Could you please try following.
awk -F'.' '{for(i=(NF-1);i<=NF;i++){$i=""};sub(/\.+$/,"")} 1' OFS="." Input_file
OR
awk 'BEGIN{FS=OFS="."} {for(i=(NF-1);i<=NF;i++){$i=""};sub(/\.+$/,"")} 1' Input_file
Explanation: Adding explanation for above code too here.
awk '
BEGIN{ ##Mentioning BEGIN section of awk program here.
FS=OFS="." ##Setting FS and OFS variables for awk to DOT here as per OPs sample Input_file.
} ##Closing BEGIN section here.
{
for(i=(NF-1);i<=NF;i++){ ##Starting for loop from i value from (NF-1) to NF for all lines.
$i="" ##Setting value if respective field to NULL.
} ##Closing for loop block here.
sub(/\.+$/,"") ##Substituting all DOTs till end of line with NULL in current line.
}
1 ##Mentioning 1 here to print edited/non-edited current line here.
' Input_file ##Mentioning Input_file name here.
I'm trying to find a position of a string
awk -F : '{if ( $0 ~ /Red Car/) print $0}' /var/lab/lab2/rusiuot/stud2001 | tail -l
and somehow I need to find a line position of Red Car. It is possible to do that using awk or grep?
You can do
awk '/Red Car/ {print NR}' /var/lab/lab2/rusiuot/stud2001
This will print the line number for the line with Red Car
If you like the line number to be printed at end of the file:
awk '/Red Car/ {a[NR]} 1; END {print "\nlines with pattern";for (i in a) printf "%s ",i;print ""}' file
Try something like:
grep -n "Red Car" /var/lab/lab2/rusiuot/stud2001 | cut -d":" -f 1
-n option will display the line number along with line where pattern is found.
I have a function
xyz()
{
x=$1*2
echo x
}
then I want to use it to replace a particular column in a csv file by awk.
File input.csv:
abc,2,something
def,3,something1
I want output like:
abc,4,somthing
def,6,something1
Command used:
cat input.csv|awk -F, -v v="'"`xyz "$2""'" 'BEGIN {FS=","; OFS=","} {$2=v1; print $0}'
Open file input.csv, calling function xyz by passing file 2nd filed as argument and result is stored back to position 2 of file, but is not working!
If I put constant in place of $2 while calling function it works:
Please help me to do this.
cat input.csv|awk -F, -v v="'"`xyz "14""'" 'BEGIN {FS=","; OFS=","} {$2=v1; print $0}'
This above line of code is working properly by calling the xyz function and putting the result back to 2nd column of file input.csv, but with only 14*2, as 14 is taken as constant.
There's a back-quote missing from your command line, and a UUOC (Useless Use of Cat), and a mismatch between variable v on the command line and v1 in the awk program:
cat input.csv|awk -F, -v v="'"`xyz "$2""'" 'BEGIN {FS=","; OFS=","} {$2=v1; print $0}'
^ Here ^ Here ^ Here
That should be written using $(…) instead:
awk -F, -v v="'$(xyz "$2")'" 'BEGIN {FS=","; OFS=","} {$2=v; print $0}' input.csv
This leaves you with a problem, though; the function xyz is invoked once by the shell before you start your awk script running, and is never invoked by awk. You simply can't do it that way. However, you can define your function in awk (and on the fly):
awk -F, 'BEGIN { FS = ","; OFS = "," }
function xyz(a) { return a * 2 }
{ $2 = xyz($2); print $0 }' \
input.csv
For your two-line input file, it produces your desired output.