I want to delete one or more specific line numbers from a file. How would I do this using sed?
If you want to delete lines from 5 through 10 and line 12th:
sed -e '5,10d;12d' file
This will print the results to the screen. If you want to save the results to the same file:
sed -i.bak -e '5,10d;12d' file
This will store the unmodified file as file.bak, and delete the given lines.
Note: Line numbers start at 1. The first line of the file is 1, not 0.
You can delete a particular single line with its line number by
sed -i '33d' file
This will delete the line on 33 line number and save the updated file.
and awk as well
awk 'NR!~/^(5|10|25)$/' file
$ cat foo
1
2
3
4
5
$ sed -e '2d;4d' foo
1
3
5
$
This is very often a symptom of an antipattern. The tool which produced the line numbers may well be replaced with one which deletes the lines right away. For example;
grep -nh error logfile | cut -d: -f1 | deletelines logfile
(where deletelines is the utility you are imagining you need) is the same as
grep -v error logfile
Having said that, if you are in a situation where you genuinely need to perform this task, you can generate a simple sed script from the file of line numbers. Humorously (but perhaps slightly confusingly) you can do this with sed.
sed 's%$%d%' linenumbers
This accepts a file of line numbers, one per line, and produces, on standard output, the same line numbers with d appended after each. This is a valid sed script, which we can save to a file, or (on some platforms) pipe to another sed instance:
sed 's%$%d%' linenumbers | sed -f - logfile
On some platforms, sed -f does not understand the option argument - to mean standard input, so you have to redirect the script to a temporary file, and clean it up when you are done, or maybe replace the lone dash with /dev/stdin or /proc/$pid/fd/1 if your OS (or shell) has that.
As always, you can add -i before the -f option to have sed edit the target file in place, instead of producing the result on standard output. On *BSDish platforms (including OSX) you need to supply an explicit argument to -i as well; a common idiom is to supply an empty argument; -i ''.
The shortest, deleting the first line in sed
sed -i '1d' file
As Brian states here, <address><command> is used, <address> is <1> and <command> <d>.
I would like to propose a generalization with awk.
When the file is made by blocks of a fixed size
and the lines to delete are repeated for each block,
awk can work fine in such a way
awk '{nl=((NR-1)%2000)+1; if ( (nl<714) || ((nl>1025)&&(nl<1029)) ) print $0}'
OriginFile.dat > MyOutputCuttedFile.dat
In this example the size for the block is 2000 and I want to print the lines [1..713] and [1026..1029].
NR is the variable used by awk to store the current line number.
% gives the remainder (or modulus) of the division of two integers;
nl=((NR-1)%BLOCKSIZE)+1 Here we write in the variable nl the line number inside the current block. (see below)
|| and && are the logical operator OR and AND.
print $0 writes the full line
Why ((NR-1)%BLOCKSIZE)+1:
(NR-1) We need a shift of one because 1%3=1, 2%3=2, but 3%3=0.
+1 We add again 1 because we want to restore the desired order.
+-----+------+----------+------------+
| NR | NR%3 | (NR-1)%3 | (NR-1)%3+1 |
+-----+------+----------+------------+
| 1 | 1 | 0 | 1 |
| 2 | 2 | 1 | 2 |
| 3 | 0 | 2 | 3 |
| 4 | 1 | 0 | 1 |
+-----+------+----------+------------+
cat -b /etc/passwd | sed -E 's/^( )+(<line_number>)(\t)(.*)/--removed---/g;s/^( )+([0-9]+)(\t)//g'
cat -b -> print lines with numbers
s/^( )+(<line_number>)(\t)(.*)//g -> replace line number to null (remove line)
s/^( )+([0-9]+)(\t)//g #remove numbers the cat printed
Related
I have a sed command that is capturing a single line with sometext. The line in the file it is capturing ends with a linefeed. I am trying to utilize this variable in a pipeline, however, when I attempt to echo, or use it with other commands requiring an input, the result is a blank. Ex:
sed '1,1!d' somefile.txt | echo "$1", I know the variable itself is not empty as I can replace echo "$1" with cat $1 and see the correct printout.
edit - I have tried piping to a tr -d and removing the newline. I have confirmed the newline character is gone, yet echos still show blank. Cats do not.
edit 2 - I piped the variable into an if statement ... | if [[ -z $1 ]]; then cat $1; fi it hits the if, is determined to be empty, so runs the cat, which prints a non-empty line to console. If the variable is empty why is cat still printing out information?
What is causing this inconsistency and how can I solve my problem? The ultimate goal is to run the output of one sed, through another to replace specific lines in a target file.
sed '1,1!d' somefile.txt | sed '2,1s/.*/'$1'/' targetfile.txt
Contents of somefile.txt:
these
are
words
Contents of targetfile.txt:
The next line should say these
This line should say these
The previous line should say these
Output of echo after sed:
<empty>
Output of cat after sed:
these
Output of 2nd sed, using input from 1st:
The next line should say these
the previous line should say these
You are confused about arguments and input data. Look at this:
$ echo "$1"
$ echo "foo" | if [[ -z $1 ]]; then cat $1; fi
foo
The first argument to my shell, $1 is empty so if [[ -z $1 ]] succeeds. The reason that cat $1 produces output is that you have a fundamental shell programming error in that statement - you aren't quoting your variable, $1. The correct syntax isn't cat $1, it's cat "$1". Look at the difference:
$ echo "foo" | if [[ -z $1 ]]; then cat "$1"; fi
cat: '': No such file or directory
We can simplify the code to make what's happening clearer:
$ echo "foo" | cat $1
foo
$ echo "foo" | cat "$1"
cat: '': No such file or directory
The reason that echo "foo" | cat $1 produces output is that the unquoted $1 is expanded by the shell to nothing before cat is called so that statement is equivalent to just echo "foo" | cat and so cat just copies the input coming in from the pipe to it's output.
On the other hand echo "foo" | cat "$1" generates an error because the shell expands "$1" to the null string before cat is called and so it's then asking cat to open a file named <null> and that of course does not exist, hence the error.
Always quote your shell variables unless you have a specific reason not to and fully understand all of the implications. Read a shell man page and/or google that if you're not sure what those implications are.
wrt another part of your code you have:
sed '1,1!d' somefile.txt | echo "$1"
but, unlike cat, echo neither reads it's input from a pipe nor from a file name passed as an argument. The input to echo is just the list of string arguments you provide it so while echo "foo" | cat will cause cat to read the input stream containing foo and output it, echo "foo" | echo will produce no output because echo isn't designed to read input from a pipe and so it'll just print a null string since you gave it no arguments.
It's not clear what you're really trying to accomplish but I think you might want to replace the 2nd line of targetfile.txt with the first line of somefile.txt. If so that's just:
awk '
NR==FNR { if (NR==1) new=$0; next }
FNR==2 { $0 = new }
{ print }
' somefile.txt targetfile.txt
Do not try to use sed to do it or you'll find yourself in escaping/quoting hell because, unlike awk, sed does not understand literal strings, see Is it possible to escape regex metacharacters reliably with sed.
You appear to want to extract the first line from file1 and use it to replace the second line in file2.
At the moment, you are extracting that value from the first file with your first sed but sending it to the second sed on its stdin rather than as a parameter ($1).
Your description is confusing so I will use this as file1:
File 1 Line 1
File 1 Line 2
File 1 Line 3
And this as file2:
File 2 Line 1
File 2 Line 2
File 2 Line 3
There are many ways to do this.
Method 1
# Extract line1 from file1
extracted=$(sed '1!d' file1)
# Replace line 2 in file2 with extracted value
sed "2s/.*/$extracted/" file2
Not sure why I feel like a dentist now :-)
If you want to put it all on one line, as some folks like to do:
x=$(sed '1!d' file1) && sed "2s/.*/$x/" file2
Method 2
This one is a bit tricky. It uses the first sed to write a script for the second sed:
sed 's/^/2s|.*|/;s/$/|/;q' file1 | sed -f /dev/stdin file2
If you look at the first sed on its own you will see it is generating a script for the second one:
sed 's/^/2s|.*|/;s/$/|/;q' file1
2s|.*|File 1 Line 1|
If you look at the second sed, you will see it is executing a script passed on its standard input:
sed -f /dev/stdin ...
Method 3
Easier still is awk:
awk 'FNR==NR{if(NR==1)saved=$0;next} FNR==2{$0=saved}1' file1 file2
File 2 Line 1
File 1 Line 1
File 2 Line 3
All you need to notice is that I am passing 2 files to awk and that FNR==NR inside the script means that awk is currently processing the first file, because FNR is the line number in the current file and NR is the total number of lines awk has processed from all files so far. So, when processing the second file, NR is greater than FNR by the number of lines in the first file.
I have a file that has the following as the last three lines. I want to retrieve the penultimate line, i.e. 100.000;8438; 06:46:12.
.
.
.
99.900; 8423; 06:44:41
100.000;8438; 06:46:12
Number of patterns: 8438
I don't know the line number. How can I retrieve it using a shell script? Thanks in advance for your help.
Try this:
tail -2 yourfile | head -1
A short sed one-liner inspired by https://stackoverflow.com/a/7671772/5287901
sed -n 'x;$p'
Explanation:
-n quiet mode: dont automatically print the pattern space
x: exchange the pattern space and the hold space (hold space now store the current line, and pattern space the previous line, if any)
$: on the last line, p: print the pattern space (the previous line, which in this case is the penultimate line).
Use this
tail -2 <filename> | head -1
ed and sed can do it as well.
str='
99.900; 8423; 06:44:41
100.000;8438; 06:46:12
Number of patterns: 8438
'
printf '%s' "$str" | sed -n -e '${x;1!p;};h' # print last line but one
printf '%s\n' H '$-1p' q | ed -s <(printf '%s' "$str") # same
printf '%s\n' H '$-2,$-1p' q | ed -s <(printf '%s' "$str") # print last line but two
From: Useful sed one-liners by Eric Pement
# print the next-to-the-last line of a file
sed -e '$!{h;d;}' -e x # for 1-line files, print blank line
sed -e '1{$q;}' -e '$!{h;d;}' -e x # for 1-line files, print the line
sed -e '1{$d;}' -e '$!{h;d;}' -e x # for 1-line files, print nothing
You don't need all of them, just pick one.
tail +2 <filename>
This prints from second line to last line.
To clarify what has already been said:
ec2thisandthat | sort -k 5 | grep 2012- | awk '{print $2}' | tail -2 | head -1
snap-e8317883
snap-9c7227f7
snap-5402553f
snap-3e7b2c55
snap-246b3c4f
snap-546a3d3f
snap-2ad48241
snap-d00150bb
returns
snap-2ad48241
tac <file> | sed -n '2p'
I have a requirement in unix to replace an occurrence of word with a space.
My File looks something like below. I need to replace |NA| with a space
File Format
1234|NA|NA|abcd|xyz
2345|NA|NA|NA|lmn
456|NA|abcd|xya|ggh
Expected Output
1234| | |abcd|xyz
2345| | | |lmn
456| |abcd|xya|ggh
I am using the following command but it only replaces the very first occurrence
sed 's/|NA|| |/g'
While the g modifier does make "global" replacements, the replacements must be non-overlapping. When overlapping replacements are required, one must loop:
$ sed ':a; s/|NA|/| |/g; ta' file.txt
1234| | |abcd|xyz
2345| | | |lmn
456| |abcd|xya|ggh
The above was tested on GNU sed. For BSD (OSX) sed (Hat tip: Jonathan Leffler), the label a must occur only at the end of a command string:
sed -e ':a' -e ' s/|NA|/| |/g; ta' file.txt
How it works
:a creates a label a.
s/|NA|/| |/g performs the substitution that you want but only for non-overlapping instances of |NA|.
ta tells sed to jump to label a if the preceding substitution command resulted in any changes to the line. In this way, the substitution command is repeated as many times as necessary to replace every occurrence of |NA|.
Just use awk for clarity, simplicity, portability, extensibility, etc., etc.:
$ awk '{while(gsub(/\|NA\|/,"| |"));}1' file
1234| | |abcd|xyz
2345| | | |lmn
456| |abcd|xya|ggh
First time through the loop the gsub() replaces all odd-numbered occurrences of the regexp and the 2nd time through it replaces any that are left. It will work as-is with any awk on any UNIX system.
Failed utterly trying to escape the vertical bar. Then made an attempt without
having the vertical bar involved and it worked! Missed also that the
replacement is only one space, now corrected. This way the field is easily
expandable by adding space.
awk '{gsub(/NA/," ")}1' file
1234| | |abcd|xyz
2345| | | |lmn
456| |abcd|xya|ggh
I have a formatted list of processes (top output) and I'd like to remove unnecessary information. How can I remove for example the second word+whitespace of each line.
Example:
1 a hello
2 b hi
3 c ahoi
Id like to delete a b and c.
You can use cut command.
cut -d' ' -f2 --complement file
--complement does the inverse. i.e. with -f2 second field was choosen. And with --complement if prints all fields except the second. This is useful when you have variable number of fields.
GNU's cut has the option --complement. In case, --complement is not available then, the following does the same:
cut -d' ' -f1,3- file
Meaning: print first field and then print from 3rd to the end i.e. Excludes second field and prints the rest.
Edit:
If you prefer awk you can do: awk {$2=""; print $0}' file
This sets the second to empty and prints the whole line (one-by-one).
Using sed to substitute the second column:
sed -r 's/(\w+\s+)\w+\s+(.*)/\1\2/' file
1 hello
2 hi
3 ahoi
Explanation:
(\w+\s+) # Capture the first word and trailing whitespace
\w+\s+ # Match the second word and trailing whitespace
(.*) # Capture everything else on the line
\1\2 # Replace with the captured groups
Notes: Use the -i option to save the results back to the file, -r is for extended regular expressions, check the man as it could be -E depending on implementation.
Or use awk to only print the specified columns:
$ awk '{print $1, $3}' file
1 hello
2 hi
3 ahoi
Both solutions have there merits, the awk solution is nice for a small fixed number of columns but you need to use a temp file to store the changes awk '{print $1, $3}' file > tmp; mv tmp file where as the sed solution is more flexible as columns aren't an issue and the -i option does the edit in place.
One way using sed:
sed 's/ [^ ]*//' file
Results:
1 hello
2 hi
3 ahoi
Using Bash:
$ while read f1 f2 f3
> do
> echo $f1 $f3
> done < file
1 hello
2 hi
3 ahoi
This might work for you (GNU sed):
sed -r 's/\S+\s+//2' file
I tried to insert a text to the first line
of a file using sed. I do this inside a sh
script.
But why it hangs at the line of sed execution?
#! /bin/sh
# Command to execute
# ./mybashcode.sh test.nbq
nbqfile=$1
nbqbase=$(basename $nbqfile nbq)
taglistfiletemp="${nbqbase}taglist_temp"
taglistfile="${nbqbase}taglist"
./myccode $nbqfile |
sort |
uniq -c |
awk '{print $2}' > $taglistfiletemp
noftags=$(wc -l $taglistfiletemp | awk '{print $1}')
echo $noftags
# We want to append output of noftags
# to the first line of taglistfile
sed '1i\
$noftags' > $taglistfile
# why it hangs here
# the content of taglistfile is NIL
I'm not sure what you are trying to do with sed but it needs two inputs, the script (usually a search/replace) and the data you want to perform it on. If you only specify one it assumes it has got the regular expression and waits for data on stdin. As you haven't supplied anything on stdin it'll hang indefinitely.
In addition, you have '$noftags' rather than "$noftags". The prior will output $noftags and the latter the contents of the variable, as single quotes do not allow variable expansion.
Have I got something wrong here?
Or, all you want to do is insert some text at the start of another file?
# $NewInitialText
# $fileToInsertInto
echo $NewInitialText > temp.file.txt
cat $fileToInsertInto >> temp.file.txt
mv temp.file.txt $fileToInsertInto
Is that easier done than sed? -- Pun intended I guess.
it hangs because you forget to supply sed with the input file.
....
...
sed -i.bak "1i $noftags" $taglistfile
...