Replace \n with \r\n in Unix file - unix

I'm trying to do the opposite of this question, replacing Unix line endings with Windows line endings, so that I can use SQL Server bcp over samba to import the file. I have sed installed but not dos2unix. I tried reversing the examples but to no avail.
Here's the command I'm using.
sed -e 's/\n/\r\n/g' myfile
I executed this and then ran od -c myfile, expecting to see \r\n where there used to be \n. But there all still \n. (Or at least they appear to be. The output of od overflows my screen buffer, so I don't get to see the beginning of the file).
I haven't been able to figure out what I'm doing wrong. Any suggestions?


When faced with this, I use a simple perl one-liner:
perl -pi -e 's/\n/\r\n/' filename
because sed behavior varies, and I know this works.


What is the problem with getting dos2unix onto the machine?
What is the platform you are working with?
Do you have GNU sed or regular non-GNU sed?
On Solaris, /usr/bin/sed requires:
sed 's/$/^M/'
where I entered the '^M' by typing controlV controlM. The '$' matches at the end of the line, and replaces the end of line with the control-M. You can script that, too.
Mechanisms expecting sed to expand '\r' or '\\r' to control-M are going to be platform-specific, at best.


You don't need the -e option.
$ matches the endline character. This sed command will insert a \r character before the end of line:
sed 's/$/\r/' myfile


Just adding a \r (aka ^M, see Jonathan Leffler's answer) in front of \n is not safe because the file might have mixed mode EOL, so then you risk ending up with some lines becomming \r\r\n. The safe thing to do is first remove all '\r' characters, and then insert (a single) \r before \n.
#!/bin/sh
sed 's/^M//g' ${1+"$#"} | sed 's/$/^M/'
Updated to use ^M.


sed 's/\([^^M]\)$/\0^M/' your_file
This makes sure you only insert a \r when there is no \r before \n. This worked for me.


Try using:
echo " this is output" > input
sed 's/$/\r/g' input |od -c


Maybe if you try it this way
cat myfile | sed 's/\n/\r\n/g' > myfile.win
will work, from my understanding your just making the replacements to the console output, you need to redirect output to a file, in this case myfile.win, then you could just rename it to whatever you want. The whole script would be (running inside a directory full of this kind of files):
#!/bin/bash
for file in $(find . -type f -name '*')
do
cat $file | sed 's/\n/\r\n/g' > $file.new
mv -f $file.new $file
done

Related

Conditional statement using awk not working as expected [duplicate]

I have this line inside a file:
ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER
I try to get rid of that ^M (carriage return) character so I used:
sed 's/^M//g'
However this does remove everything after ^M:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER
[root#localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities
What I want to obtain is:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER

Use tr:
tr -d '^M' < inputfile
(Note that the ^M character can be input using Ctrl+VCtrl+M)
EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:
tr -d $'\r' < inputfile

still the same line:
sed -i 's/^M//g' file
when you type the command, for ^M you type Ctrl+VCtrl+M
actually if you have already opened the file in vim, you can just in vim do:
:%s/^M//g
same, ^M you type Ctrl-V Ctrl-M

You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:
sed 's/\r//g' < input.txt > output.txt
Hope that helps.
Note: ^M is actually carriage return character which is represented in code as \r
What dos2unix does is most likely equivalent to:
sed 's/\r\n/\n/g' < input.txt > output.txt
It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.

alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "
Usage:
dos2unix file

If Perl is an option:
perl -i -pe 's/\r\n$/\n/g' file
-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"

In awk:
sub(/\r/,"")
If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.

This is the better way to achieve
tr -d '\015' < inputfile_name > outputfile_name
Later rename the file to original file name.

I agree with #twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:
To run in Mac mode use the command-line option "-c mac" or use the
commands "mac2unix" or "unix2mac"
I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!
I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.

This is clean and simple and it works:
sed -i 's/\r//g' file
where \r of course is the equivalent for ^M.

Simply run the following command:
sed -i -e 's/\r$//' input.file
I verified this as valid in Mac OSX Monterey.

remove any \r :
nawk 'NF+=OFS=_' FS='\r'
gawk 3 ORS= RS='\r'
remove end of line \r :
mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1

How to remove blank lines from a Unix file

I need to remove all the blank lines from an input file and write into an output file. Here is my data as below.
11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321
11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003
11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032
11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001
11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701
11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383
11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746

sed -i '/^$/d' foo
This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original:
sed '/^$/d' foo > foo.tmp
mv foo.tmp foo
If you also want to remove lines consisting only of whitespace (not just empty lines) then use:
sed -i '/^[[:space:]]*$/d' foo
Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too:
sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo

awk 'NF' filename
awk 'NF > 0' filename
sed -i '/^$/d' filename
awk '!/^$/' filename
awk '/./' filename
The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.

Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($):
grep -v '^$' infile.txt > outfile.txt
If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways:
grep -P -v '^\s*$' infile.txt > outfile.txt
or, without Perl regular expressions:
grep -v '^[[:space:]]*$' infile.txt > outfile.txt

sed -e '/^ *$/d' input > output
Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option.
If I execute the above command still I have blank lines in my output file. What could be the reason?
There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then:
sed -e 's/ *$//' -e '/^ *$/d' input > output
The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs.
Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return:
tr -d '\015' < input | sed -e 's/ *$//' -e '/^ *$/d' > output
If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against:
head -n 2 input | od -c
Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.)
You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with:
echo "Hello World" | sed 's/[[:space:]][[:space:]]*/ /'
If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.

grep . file
grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.

with awk
awk 'NF > 0' filename

To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it:
cat file.txt | perl -lane "print if /\S/"
Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do.
Cheers

You can sed's -i option to edit in-place without using temporary file:
sed -i '/^$/d' file

Why sed removes last line?

$ cat file.txt
one
two
three
$ cat file.txt | sed "s/one/1/"
1
two
Where is the word "three"?
UPDATED:
There is no line after the word "three".

As Ivan suggested, your text file is missing the end of line (EOL) marker on the final line. Since that's not present, three is printed out by sed but then immediately over-written by your prompt. You can see it if you force an extra line to be printed.
sed 's/one/1/' file.txt && echo
This is a common problem since people incorrectly think of the EOL as an indication that there's a following line (which is why it's commonly called a "newline") and not as an indication that the current line has ended.

Using comments from other posts:
older versions of sed do not process the last line of a file if no EOL or "new line" is present.
echo can be used to add a new line
Then, to solve the problem you can re-order the commands:
( cat file.txt && echo ) | sed 's/one/1/'

I guess there is no new line character after last line. sed didn't find line separator after last line and ignore it.
Update
I suggest you to rewrite this in perl (if you have it installed):
cat file.txt | perl -pe 's/one/1/'

Instead of cat'ing the file and piping into sed, run sed with the file name as an argument after the substitution string, like so:
sed "s/one/1/" file.txt
When I did it this way, I got the "three" immediately following by the prompt:
1
two
three$

A google search shows that the man page for some versions of sed (not the GNU or BSD versions, which work as you'd expect) indicate that it won't process an incomplete line (one that's not newline-terminated) at the end of a file. The solution is to ensure your files end with a newline, install GNU sed, or use awk or perl instead.

here's an awk solution
awk '{gsub("one","1")}1' file.txt

How do I replace a token with the result of `pwd` in sed?

I'm trying to do something like this:
sed 's/#REPLACE-WITH-PATH/'`pwd`'/'
Unfortunately, I that errors out:
sed: -e expression #1, char 23: unknown option to `s'
Why does this happen?

You need to use a different character instead of /, eg.:
sed 's?#REPLACE-WITH-PATH?'`pwd`'?'
because / appears in the pwd output.

in sed, you can't use / directly, you must use '/'.
#!/bin/bash
dir=$`pwd`/
ls -1 | sed "s/^/${dir//\//\\/}/g"

sed 's:#REPLACE-WITH-PATH:'`pwd`':' config.ini
The problem is one of escaping the output of pwd correctly. Fortunately, as in vim, sed supports using a different delimiter character. In this case, using the colon instead of slash as a delimiter avoids the escaping problem.

instead of fumbling around with quotes like that, you can do it like this
#!/bin/bash
p=`pwd`
# pass the variable p to awk
awk -v p="$p" '$0~p{ gsub("REPLACE-WITH-PATH",p) }1' file >temp
mv temp file
or just bash
p=`pwd`
while read line
do
line=${line/REPLACE-WITH-PATH/$p}
echo $line
done < file > temp
mv temp file

Remove carriage return in Unix

What is the simplest way to remove all the carriage returns \r from a file in Unix?

I'm going to assume you mean carriage returns (CR, "\r", 0x0d) at the ends of lines rather than just blindly within a file (you may have them in the middle of strings for all I know). Using this test file with a CR at the end of the first line only:
$ cat infile
hello
goodbye
$ cat infile | od -c
0000000 h e l l o \r \n g o o d b y e \n
0000017
dos2unix is the way to go if it's installed on your system:
$ cat infile | dos2unix -U | od -c
0000000 h e l l o \n g o o d b y e \n
0000016
If for some reason dos2unix is not available to you, then sed will do it:
$ cat infile | sed 's/\r$//' | od -c
0000000 h e l l o \n g o o d b y e \n
0000016
If for some reason sed is not available to you, then ed will do it, in a complicated way:
$ echo ',s/\r\n/\n/
> w !cat
> Q' | ed infile 2>/dev/null | od -c
0000000 h e l l o \n g o o d b y e \n
0000016
If you don't have any of those tools installed on your box, you've got bigger problems than trying to convert files :-)

tr -d '\r' < infile > outfile
See tr(1)

The simplest way on Linux is, in my humble opinion,
sed -i.bak 's/\r$//g' <filename>
-i will edit the file in place, while the .bak will create a backup of the original file by making a copy of your file and adding the extension .bak at the end. (You can specify what ever you want after the -i, or specify only -i to not create a backup.)
The strong quotes around the substitution operator 's/\r//' are essential. Without them the shell will interpret \r as an escape+r and reduce it to a plain r, and remove all lower case r. That's why the answer given above in 2009 by Rob doesn't work.
And adding the /g modifier ensures that even multiple \r will be removed, and not only the first one.

Old School:
tr -d '\r' < filewithcarriagereturns > filewithoutcarriagereturns

There's a utility called dos2unix that exists on many systems, and can be easily installed on most.

sed -i s/\r// <filename> or somesuch; see man sed or the wealth of information available on the web regarding use of sed.
One thing to point out is the precise meaning of "carriage return" in the above; if you truly mean the single control character "carriage return", then the pattern above is correct. If you meant, more generally, CRLF (carriage return and a line feed, which is how line feeds are implemented under Windows), then you probably want to replace \r\n instead. Bare line feeds (newline) in Linux/Unix are \n.

If you are a Vi user, you may open the file and remove the carriage return with:
:%s/\r//g
or with
:1,$ s/^M//
Note that you should type ^M by pressing ctrl-v and then ctrl-m.

Someone else recommend dos2unix and I strongly recommend it as well. I'm just providing more details.
If installed, jump to the next step. If not already installed, I would recommend installing it via yum like:
yum install dos2unix
Then you can use it like:
dos2unix fileIWantToRemoveWindowsReturnsFrom.txt

Once more a solution... Because there's always one more:
perl -i -pe 's/\r//' filename
It's nice because it's in place and works in every flavor of unix/linux I've worked with.

Removing \r on any UNIX® system:
Most existing solutions in this question are GNU-specific, and wouldn't work on OS X or BSD; the solutions below should work on many more UNIX systems, and in any shell, from tcsh to sh, yet still work even on GNU/Linux, too.
Tested on OS X, OpenBSD and NetBSD in tcsh, and on Debian GNU/Linux in bash.
With sed:
In tcsh on an OS X, the following sed snippet could be used together with printf, as neither sed nor echo handle \r in the special way like the GNU does:
sed `printf 's/\r$//g'` input > output
With tr:
Another option is tr:
tr -d '\r' < input > output
Difference between sed and tr:
It would appear that tr preserves a lack of a trailing newline from the input file, whereas sed on OS X and NetBSD (but not on OpenBSD or GNU/Linux) inserts a trailing newline at the very end of the file even if the input is missing any trailing \r or \n at the very end of the file.
Testing:
Here's some sample testing that could be used to ensure this works on your system, using printf and hexdump -C; alternatively, od -c could also be used if your system is missing hexdump:
% printf 'a\r\nb\r\nc' | hexdump -C
00000000 61 0d 0a 62 0d 0a 63 |a..b..c|
00000007
% printf 'a\r\nb\r\nc' | ( sed `printf 's/\r$//g'` /dev/stdin > /dev/stdout ) | hexdump -C
00000000 61 0a 62 0a 63 0a |a.b.c.|
00000006
% printf 'a\r\nb\r\nc' | ( tr -d '\r' < /dev/stdin > /dev/stdout ) | hexdump -C
00000000 61 0a 62 0a 63 |a.b.c|
00000005
%

If you're using an OS (like OS X) that doesn't have the dos2unix command but does have a Python interpreter (version 2.5+), this command is equivalent to the dos2unix command:
python -c "import sys; import fileinput; sys.stdout.writelines(line.replace('\r', '\n') for line in fileinput.input(mode='rU'))"
This handles both named files on the command line as well as pipes and redirects, just like dos2unix. If you add this line to your ~/.bashrc file (or equivalent profile file for other shells):
alias dos2unix="python -c \"import sys; import fileinput; sys.stdout.writelines(line.replace('\r', '\n') for line in fileinput.input(mode='rU'))\""
... the next time you log in (or run source ~/.bashrc in the current session) you will be able to use the dos2unix name on the command line in the same manner as in the other examples.

you can simply do this :
$ echo $(cat input) > output

Here is the thing,
%0d is the carriage return character. To make it compatabile with Unix. We need to use the below command.
dos2unix fileName.extension fileName.extension

try this to convert dos file into unix file:
fromdos file

For UNIX... I've noticed dos2unix removed Unicode headers form my UTF-8 file. Under git bash (Windows), the following script seems to work nicely. It uses sed. Note it only removes carriage-returns at the ends of lines, and preserves Unicode headers.
#!/bin/bash
inOutFile="$1"
backupFile="${inOutFile}~"
mv --verbose "$inOutFile" "$backupFile"
sed -e 's/\015$//g' <"$backupFile" >"$inOutFile"

If you are running an X environment and have a proper editor (visual studio code), then I would follow the reccomendation:
Visual Studio Code: How to show line endings
Just go to the bottom right corner of your screen, visual studio code will show you both the file encoding and the end of line convention followed by the file, an just with a simple click you can switch that around.
Just use visual code as your replacement for notepad++ on a linux environment and you are set to go.

Using sed
sed $'s/\r//' infile > outfile
Using sed on Git Bash for Windows
sed '' infile > outfile
The first version uses ANSI-C quoting and may require escaping \ if the command runs from a script. The second version exploits the fact that sed reads the input file line by line by removing \r and \n characters. When writing lines to the output file, however, it only appends a \n character. A more general and cross-platform solution can be devised by simply modifying IFS
IFS=$'\r\n' # or IFS+=$'\r' if the lines do not contain whitespace
printf "%s\n" $(cat infile) > outfile
IFS=$' \t\n' # not necessary if IFS+=$'\r' is used
Warning: This solution performs filename expansion (*, ?, [...] and more if extglob is set). Use it only if you are sure that the file does not contain special characters or you want the expansion.
Warning: None of the solutions can handle \ in the input file.

cat input.csv | sed 's/\r/\n/g' > output.csv
worked for me

I've used python for it, here my code;
end1='/home/.../file1.txt'
end2='/home/.../file2.txt'
with open(end1, "rb") as inf:
with open(end2, "w") as fixed:
for line in inf:
line = line.replace("\n", "")
line = line.replace("\r", "")
fixed.write(line)

Though it's a older post, recently I came across with same problem. As I had all the files to rename inside /tmp/blah_dir/ as each file in this directory had "/r" trailing character ( showing "?" at end of file), so doing it script way was only I could think of.
I wanted to save final file with same name (without trailing any character).
With sed, problem was the output filename which I was needed to mention something else ( which I didn't want).
I tried other options as suggested here (not considered dos2unix because of some limitations) but didn't work.
I tried with "awk" finally which worked where I used "\r" as delimiter and taken the first part:
trick is:
echo ${filename}|awk -F"\r" '{print $1}'
Below script snippet I used ( where I had all file had "\r" as trailing character at path /tmp/blah_dir/) to fix my issue:
cd /tmp/blah_dir/
for i in `ls`
do
mv $i $(echo $i | awk -F"\r" '{print $1}')
done
Note: This example is not very exact though close to what I worked (Mentioning here just to give the better idea about what I did)

I made this shell-script to remove the \r character. It works in solaris and red-hat:
#!/bin/ksh
LOCALPATH=/Any_PATH
for File in `ls ${LOCALPATH}`
do
ARCACT=${LOCALPATH}/${File}
od -bc ${ARCACT}|sed -n 'p;n'|sed 's/015/012/g'|awk '{$1=""; print $0}'|sed 's/ /\\/g'|awk '{printf $0;}'>${ARCACT}.TMP
printf "`cat ${ARCACT}.TMP`"|sed '/^$/d'>${ARCACT}
rm ${ARCACT}.TMP
done
exit 0

Resources