using sed -n with variables - unix
I am having a log file a.log and i need to extract a piece of information from it.
To locate the start and end line numbers of the pattern i am using the following.
start=$(sed -n '/1112/=' file9 | head -1)
end=$(sed -n '/true/=' file9 | head -1)
i need to use the variables (start,end) in the following command:
sed -n '16q;12,15p' orig-data-file > new-file
so that the above command appears something like:
sed -n '($end+1)q;$start,$end'p orig-data-file > new-file
I am unable to replace the line numbers with the variables. Please suggest the correct syntax.
Thanks,
Rosy
When I realized how to do it, I was looking for anyway to get line number into a file containing the requested info, and display the file from that line to EOF.
So, this was my way.
with
PATTERN="pattern"
INPUT_FILE="file1"
OUTPUT_FILE="file2"
line number of first match of $PATTERN into $INPUT_FILE can be retrieved with
LINE=`grep -n ${PATTERN} ${INPUT_FILE} | awk -F':' '{ print $1 }' | head -n 1`
and the outfile will be the text from that $LINE to EOF. This way:
sed -n ${LINE},\$p ${INPUT_FILE} > ${OUTPUT_FILE}
The point here, is the way how can variables be used with command sed -n:
first witout using variables
sed -n 'N,$p' <file name>
using variables
LINE=<N>; sed -n ${LINE},\$p <file name>
Remove the single quotes thus. Single quotes turn off the shell parsing of the string. You need shell parsing to do the variable string replacements.
sed -n '('$end'+1)q;'$start','$end''p orig-data-file > new-file
Related
Unix command to parse string
I'm trying to figure out a command to parse the following file content: Operation=GET Type=HOME Counters=CacheHit=0,Exception=1,Validated=0 I need to extract Exception=1 into its own line. I'm fiddling with awk, sed and grep but not making much progress. Does anyone have any tips on using any unix command to perform this? Thanks
Since your file is close to bash syntax, there is a fun little trick you can do to make bash itself parse the file. First, use some program like tr to transform the input into a something bash can parse, and then "source" that, which will create shell variables you can expand later to get the values. source <(tr , $'\n' < file_name_goes_here) echo $Exception
Many ways to do this. Here is one assuming the file is called "file.txt". Grab the line you want, replace everything from the start of the line up to Except with just Except, then pull out the first field using comma as the delimiter.
$ grep Exception file.txt | sed 's/.*Except/Except/g' | cut -d, -f 1
Exception=1
If you wanted to use gawk:
$ grep Exception file.txt | sed 's/.*Except/Except/g' | gawk -F, '{print $1}'
Exception=1
or just using grep and sed:
$ grep Exception file.txt | sed 's/.*\(Exception=[0-9]*\).*/\1/g'
Exception=1
or as #sheltter reminded me:
$ egrep -o "Exception=[0-9]+" file.txt
Exception=1
No need to use a mix of commands.
awk -F, 'NR==2 {print RS$1}' RS="Exception" file
Exception=1
Here we split the line by the keyword we look for RS="Exception"
If the line has two record (only when keyword is found), then
print first field, separated using command, with Record selector.
PS This only works if you have one Exception field
How to use sed to extract text between two bar signs (i.e. '|')?
I would like to extract text that falls between two | signs in a file with multiple lines. For instance, I want to extract P16 from sp|P16|SM2. I have found a possible answer here. However, I cannot apply the answer to my case. I am using the following: sed -n '/|/,/|/ p' filename or this by escaping the | sign: sed -n '/\|/,/\|/ p' filename But what I receive as result are all the lines in the file unchanged even though I am using -n to suppress automatic printing of pattern space. Any ideas what I am missing? [EDIT]: I can get the desired result using the following. However, I would like an explanation why the above mentioned is not working: sed 's/^sp|//' filename | sed 's/|.*//'
the tool for this task is cut $ echo "sp|P16|SM2" | cut -d'|' -f2 P16
awk is better choice for column based data:
awk -F'|' '{print $2}'
will give you P16
sed one-liner:
The following sed one-liner will only leave the 2nd column for you:
kent$ echo "sp|P16|SM2"|sed 's/[^|]*|//;s/|[^|]*//'
P16
Or using grouping:
kent$ echo "sp|P16|SM2"|sed 's/.*|\([^|]*\)|.*/\1/'
P16
Short explanation why your two commands didn't work:
1) sed -n '/|/,/|/ p' filename
This sed will print lines between two lines which containing |
2) sed -n '/\|/,/\|/ p' filename
Sed takes BRE as default. If you escape the |, you gave them special meaning, the logical OR. again, the /pat1/,/pat2/ address was wrong usage for your case, it checks lines, not within a line.
redirecting in a shell script
I'm trying to write a script to swap out text in a file: sed s/foo/bar/g myFile.txt > myFile.txt.updated mv myFile.txt.updated myFile.txt I evoke the sed program, which swaps out text in myFile.txt and redirects the changed lines of text to a second file. mv then moves .updated txt file to myFile.txt, overwriting it. That command works in the shell. I wrote: #!/bin/sh #First, I set up some descriptive variables for the arguments initialString="$1" shift desiredChange="$1" shift document="$1" #Then, I evoke sed on these (more readable) parameters updatedDocument=`sed s/$initialString/$desiredChange/g $document` #I want to make sure that was done properly echo updated document is $updatedDocument #then I move the output in to the new text document mv $updatedDocument $document I get the error: mv: target `myFile.txt' is not a directory I understand that it thinks my new file's name is the first word of the string that was sed's output. I don't know how to correct that. I've been trying since 7am and every quotation, creating a temporary file to store the output in (disastrous results), IFS...everything so far gives me more and more unhelpful errors. I need to clear my head and I need your help. How can I fix this?
Maybe try echo $updatedDocument > $document
Change
updatedDocument=`sed s/$initialString/$desiredChange/g $document`
to
updatedDocument=${document}.txt
sed s/$initialString/$desiredChange/g $document
Backticks will actually put the entire piped output of the sed command into your variable value.
An even faster way would be to not use updatedDocument or mv at all by doing an in-place sed:
sed -i s/$initialString/$desiredChange/g $document
The -i flag tells sed to do the replacement in-place. This basically means creating a temp file for the output and replacing your original file with the temp file once it is done, pretty much exactly as you are doing.
#!/bin/sh
#First, I set up some descriptive variables for the arguments
echo "$1" | sed #translation of special regex char like . * \ / ? | read -r initialString
echo "$2" | sed 's|[\&/]|\\&|g' | read -r desiredChange
document="$3"
#Then, I evoke sed
sed "s/${initialString}/${desiredChange}/g" ${document} | tee ${document}
don't forget that initialString and desiredChange are pattern interpreted as regex, so a trnaslation is certainly needed
sed #translation of special regex char like . * \ / ? is to replace by the correct sed (discuss on several post on the site)
How to remove blank lines from a Unix file
I need to remove all the blank lines from an input file and write into an output file. Here is my data as below. 11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321 11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003 11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032 11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001 11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701 11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383 11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746
sed -i '/^$/d' foo This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original: sed '/^$/d' foo > foo.tmp mv foo.tmp foo If you also want to remove lines consisting only of whitespace (not just empty lines) then use: sed -i '/^[[:space:]]*$/d' foo Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too: sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo
awk 'NF' filename awk 'NF > 0' filename sed -i '/^$/d' filename awk '!/^$/' filename awk '/./' filename The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.
Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($): grep -v '^$' infile.txt > outfile.txt If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways: grep -P -v '^\s*$' infile.txt > outfile.txt or, without Perl regular expressions: grep -v '^[[:space:]]*$' infile.txt > outfile.txt
sed -e '/^ *$/d' input > output Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option. If I execute the above command still I have blank lines in my output file. What could be the reason? There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then: sed -e 's/ *$//' -e '/^ *$/d' input > output The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs. Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return: tr -d '\015' < input | sed -e 's/ *$//' -e '/^ *$/d' > output If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against: head -n 2 input | od -c Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.) You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with: echo "Hello World" | sed 's/[[:space:]][[:space:]]*/ /' If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.
grep . file grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.
with awk awk 'NF > 0' filename
To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it: cat file.txt | perl -lane "print if /\S/" Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do. Cheers
You can sed's -i option to edit in-place without using temporary file: sed -i '/^$/d' file
Inserting text to a file with Sed within shell Script
I tried to insert a text to the first line
of a file using sed. I do this inside a sh
script.
But why it hangs at the line of sed execution?
#! /bin/sh
# Command to execute
# ./mybashcode.sh test.nbq
nbqfile=$1
nbqbase=$(basename $nbqfile nbq)
taglistfiletemp="${nbqbase}taglist_temp"
taglistfile="${nbqbase}taglist"
./myccode $nbqfile |
sort |
uniq -c |
awk '{print $2}' > $taglistfiletemp
noftags=$(wc -l $taglistfiletemp | awk '{print $1}')
echo $noftags
# We want to append output of noftags
# to the first line of taglistfile
sed '1i\
$noftags' > $taglistfile
# why it hangs here
# the content of taglistfile is NIL
I'm not sure what you are trying to do with sed but it needs two inputs, the script (usually a search/replace) and the data you want to perform it on. If you only specify one it assumes it has got the regular expression and waits for data on stdin. As you haven't supplied anything on stdin it'll hang indefinitely. In addition, you have '$noftags' rather than "$noftags". The prior will output $noftags and the latter the contents of the variable, as single quotes do not allow variable expansion.
Have I got something wrong here? Or, all you want to do is insert some text at the start of another file? # $NewInitialText # $fileToInsertInto echo $NewInitialText > temp.file.txt cat $fileToInsertInto >> temp.file.txt mv temp.file.txt $fileToInsertInto Is that easier done than sed? -- Pun intended I guess.
it hangs because you forget to supply sed with the input file. .... ... sed -i.bak "1i $noftags" $taglistfile ...