Why sed removes last line? - unix
$ cat file.txt
one
two
three
$ cat file.txt | sed "s/one/1/"
1
two
Where is the word "three"?
UPDATED:
There is no line after the word "three".
As Ivan suggested, your text file is missing the end of line (EOL) marker on the final line. Since that's not present, three is printed out by sed but then immediately over-written by your prompt. You can see it if you force an extra line to be printed.
sed 's/one/1/' file.txt && echo
This is a common problem since people incorrectly think of the EOL as an indication that there's a following line (which is why it's commonly called a "newline") and not as an indication that the current line has ended.
Using comments from other posts:
older versions of sed do not process the last line of a file if no EOL or "new line" is present.
echo can be used to add a new line
Then, to solve the problem you can re-order the commands:
( cat file.txt && echo ) | sed 's/one/1/'
I guess there is no new line character after last line. sed didn't find line separator after last line and ignore it.
Update
I suggest you to rewrite this in perl (if you have it installed):
cat file.txt | perl -pe 's/one/1/'
Instead of cat'ing the file and piping into sed, run sed with the file name as an argument after the substitution string, like so:
sed "s/one/1/" file.txt
When I did it this way, I got the "three" immediately following by the prompt:
1
two
three$
A google search shows that the man page for some versions of sed (not the GNU or BSD versions, which work as you'd expect) indicate that it won't process an incomplete line (one that's not newline-terminated) at the end of a file. The solution is to ensure your files end with a newline, install GNU sed, or use awk or perl instead.
here's an awk solution
awk '{gsub("one","1")}1' file.txt
Related
Delete Special Word Using sed
I would like to use sed to remove all occurances of this line if and only if it is this <ab></ab> If this line, I would not want to delete it <ab>keyword</ab> My attempt that's not working: sed '/<ab></ab>/d' Thanks for any insight. I'm not sure what's wrong as I should not have to escape anything? I'm using a shell script named temp to execute this. My command is this: cat foobar.html | ./temp This is my temp shell script: #!/bin/sh sed -e '/td/!d' | sed '/<ab></ab>/d'
It looks like we have a couple of problems here. The first is with the / in the close-tag. sed uses this to delimit different parts of the command. Fortunately, all we have to do is escape it with \. Try: sed '/<ab><\/ab>/d' Here's an example on my machine: $ cat test <ab></ab> <ab></ab> <ab>test</ab> $ sed '/<ab><\/ab>/d' test <ab>test</ab> $ The other problem is that I'm not sure what the purpose of sed -e '/td/!d' is. In it's default operating mode, you don't need to tell it not to delete something; just tell it exactly what you want to delete. So, to do this on a file called input.html: sed '/<ab><\/ab>/d' input.html Or, to edit the file in-place, you can just do: sed -i -e '/<ab><\/ab>/d' input.html Additionally, sed lets you use any character you want as a delimiter; you don't have to use /. So if you'd prefer not to escape your input, you can do: sed '\#<ab></ab>#d' input.html Edit In the comments, you mentioned wanting to delete lines that only contain </ab> and nothing else. To do that, you need to do what's called anchoring the match. The ^ character represents the beginning of the line for anchoring, and $ represents the end of the line. sed '/^<\/ab>$/d' input.html This will only match a line that contains (literally) </ab> and nothing else at all, and delete the line. If you want to match lines that contain whitespace too, but no text other than </ab>: sed '/^[[:blank:]]*<\/ab>[[:blank:]]*$/d' input.html [[:blank:]]* matches "0 or more whitespace characters" and is called a "POSIX bracket expression".
Changing a line in a file on UNIX
I have created a file named "asd.txt" on a UNIX based system. I added four lines by using echo command. Now, I would like to change the first line of this file. I am not allowed to use any text editors, such as vi. I have to do this by using only command line. Can anyone help? Thanks.
Here is how you could do it with sed. sed '1 s/search/replace/' asd.txt If you are feeling up to it and have GNU sed, use the -i switch to do it in place. If you want to replace the entire first line how about doing this? echo "Here is my new first line" && sed '1d' asd.txt For both of these commands you can redirect the output to a new file using the > operator.
#!/bin/bash cat <(echo "Replacement") <(tail -n +2 foo.txt)
How to remove blank lines from a Unix file
I need to remove all the blank lines from an input file and write into an output file. Here is my data as below. 11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321 11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003 11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032 11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001 11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701 11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383 11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746
sed -i '/^$/d' foo This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original: sed '/^$/d' foo > foo.tmp mv foo.tmp foo If you also want to remove lines consisting only of whitespace (not just empty lines) then use: sed -i '/^[[:space:]]*$/d' foo Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too: sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo
awk 'NF' filename awk 'NF > 0' filename sed -i '/^$/d' filename awk '!/^$/' filename awk '/./' filename The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.
Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($): grep -v '^$' infile.txt > outfile.txt If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways: grep -P -v '^\s*$' infile.txt > outfile.txt or, without Perl regular expressions: grep -v '^[[:space:]]*$' infile.txt > outfile.txt
sed -e '/^ *$/d' input > output Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option. If I execute the above command still I have blank lines in my output file. What could be the reason? There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then: sed -e 's/ *$//' -e '/^ *$/d' input > output The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs. Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return: tr -d '\015' < input | sed -e 's/ *$//' -e '/^ *$/d' > output If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against: head -n 2 input | od -c Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.) You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with: echo "Hello World" | sed 's/[[:space:]][[:space:]]*/ /' If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.
grep . file grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.
with awk awk 'NF > 0' filename
To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it: cat file.txt | perl -lane "print if /\S/" Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do. Cheers
You can sed's -i option to edit in-place without using temporary file: sed -i '/^$/d' file
UNIX sed command help
sed -n '$'!p abc.txt | tail +2 > def.txt I have the above mentioned sed command in my code - I am unable to figure out what it does -I am going through sed tutorials to find it out but am not able to - Can some one please help me in figuring out what it does - Thanks
Taking this in stages: sed -n abc.txt "Run abc.txt through sed, but don't print anything out." sed -n '$!p' abc.txt (Note that I've corrected what I think was a misplaced quote mark.) "Run abc.txt through sed; if a line isn't the last line, print it (i.e. print all but the last line)." I guess you know the rest, but note that tail +2 is obsolete syntax-- tail -n 2 would be better. EDIT: To remove the last two lines, try sed 'N;$d' or if that doesn't work, crude but effective: sed '$d' | sed '$d'
As far as the sed command '$'!p is concerned:
the $ matches only the last line of the input file.
the ! negates the sense of the match (so that it matches all but the last line).
the p prints out whatever was matched.
So basically this prints all but the last line of the file.
The -n option stops sed from performing its default action (to print the pattern space) - without that, you'd get one copy of the last line and two copies of all the other lines.
The quotes around $ are to stop the shell from trying to interpret it as a shell variable - I would have quoted the lot myself ('$!p') but that's a style issue, at least on bash. Other shells like csh (which uses ! for command history retrieval) may not be so forgiving.
Replace \n with \r\n in Unix file
I'm trying to do the opposite of this question, replacing Unix line endings with Windows line endings, so that I can use SQL Server bcp over samba to import the file. I have sed installed but not dos2unix. I tried reversing the examples but to no avail. Here's the command I'm using. sed -e 's/\n/\r\n/g' myfile I executed this and then ran od -c myfile, expecting to see \r\n where there used to be \n. But there all still \n. (Or at least they appear to be. The output of od overflows my screen buffer, so I don't get to see the beginning of the file). I haven't been able to figure out what I'm doing wrong. Any suggestions?
When faced with this, I use a simple perl one-liner: perl -pi -e 's/\n/\r\n/' filename because sed behavior varies, and I know this works.
What is the problem with getting dos2unix onto the machine? What is the platform you are working with? Do you have GNU sed or regular non-GNU sed? On Solaris, /usr/bin/sed requires: sed 's/$/^M/' where I entered the '^M' by typing controlV controlM. The '$' matches at the end of the line, and replaces the end of line with the control-M. You can script that, too. Mechanisms expecting sed to expand '\r' or '\\r' to control-M are going to be platform-specific, at best.
You don't need the -e option. $ matches the endline character. This sed command will insert a \r character before the end of line: sed 's/$/\r/' myfile
Just adding a \r (aka ^M, see Jonathan Leffler's answer) in front of \n is not safe because the file might have mixed mode EOL, so then you risk ending up with some lines becomming \r\r\n. The safe thing to do is first remove all '\r' characters, and then insert (a single) \r before \n.
#!/bin/sh
sed 's/^M//g' ${1+"$#"} | sed 's/$/^M/'
Updated to use ^M.
sed 's/\([^^M]\)$/\0^M/' your_file This makes sure you only insert a \r when there is no \r before \n. This worked for me.
Try using: echo " this is output" > input sed 's/$/\r/g' input |od -c
Maybe if you try it this way cat myfile | sed 's/\n/\r\n/g' > myfile.win will work, from my understanding your just making the replacements to the console output, you need to redirect output to a file, in this case myfile.win, then you could just rename it to whatever you want. The whole script would be (running inside a directory full of this kind of files): #!/bin/bash for file in $(find . -type f -name '*') do cat $file | sed 's/\n/\r\n/g' > $file.new mv -f $file.new $file done