Remove last character from a file in unix - unix

Have
08-01-12|07-30-13|08-09-32|12-43-56|
Want
08-01-12|07-30-13|08-09-32|12-43-56
I want to remove just the last |.


quick and dirty
sed 's/.$//' YourFile
a bit secure
sed 's/[|]$//' YourFile
allowing space
sed 's/[|][[:space:]]*$//' YourFile
same for only last char of last line (thansk #amelie for this comment) :
add a $in front so on quick and dirty it gives sed '$ s/.$//' YourFile


Since you tagged with awk, find here two approaches, one for every interpretation of your question:
If you want to remove | if it is the last character:
awk '{sub(/\|$/,"")}1' file
Equivalent to sed s'/|$//' file, only that escaping | because it has a special meaning in regex content ("or").
If you want to remove the last character, no matter what it is:
awk '{sub(/.$/,"")}1' file
Equivalent to sed s'/.$//' file, since . matches any character.
Test
$ cat a
08-01-12|07-30-13|08-09-32|12-43-56|
rrr.
$ awk '{sub(/\|$/,"")}1' a
08-01-12|07-30-13|08-09-32|12-43-56
rrr. # . is kept
$ awk '{sub(/.$/,"")}1' a
08-01-12|07-30-13|08-09-32|12-43-56
rrr # . is also removed

Related

delete text with delimiter in unix

I have a text file in the below format . I need to remove the text between the first and second semicolon (delimiter ), but retain the second semicolon
$cat test.txt
abc;def;ghi;jkl
mno;pqr;stu,xxx
My expected output
abc;ghi;jkl
mno;stu,xxx
I tried using sed 's/^([^;][^;]*);.*$/\1/', but it removes everything after the first semicolon. I also tried with cut -d ';' -f2, this only give the 2nd field as output.

Using cut
cut -d";" -f2 --complement file
-d is for delimeter, i.e ";" in your case
-f is for field, i.e keep the fields listed
--complement is to reverse the selection, i.e remove the fields listed
So:
$ cat test.txt
abc;def;ghi;jkl
mno;pqr;stu;xxx
$ cut -d";" -f2 --complement test.txt
abc;ghi;jkl
mno;stu;xxx

You may use this sed:
sed 's/;[^;]*//' file
abc;ghi;jkl
mno;stu,xxx

You can do it directly by simply removing the 2nd occurrence of the characters in question, e.g.
sed 's/[^;]*;//2' test.txt
Example Use/Output
$ sed 's/[^;]*;//2' test.txt
abc;ghi;jkl
mno;stu,xxx
A thanks to #EdMorton for improvements here as well.
If you did want to use awk, you could simply replace the 2nd field with nothing as well, e.g.
awk -F';' '{sub(/;[^;]*/,"")}1' test.txt
(same output)
With a thanks to #EdMorton for the improvement to the original.
Or as Cyrus suggest with cut, deleting field 2, e.g.
cut -d';' -f-1,3- test.txt
(same output)

Trying to fix OP's attempts here, with sed you could try following code. Simple explanation would be, create 1st back reference which has value till 1st occurrence of ; then from 1st ; to 2nd ; don't keep it in backreference and keep rest of the value in 2nd back reference. Finally while substituting substitute it with 1st and 2nd back reference values.
sed -E 's/^([^;]*);[^;]*;(.*)/\1;\2/' Input_file
OR as per Ed's comment please try following;
sed -E 's/^([^;]*);[^;]*/\1/' Input_file

super lazy awk solution
gawk/mawk/mawk2 'sub(/;[^;]+/,"")'
a more verbose solution but makes it clearer what it's doing
g/mawk 'BEGIN {FS=";+"; OFS=";"} ($2="")||($0=$0)&&($1=$1)'
clean out 2nd field, but since null string is assigned in, it returns 0 (false), thus requiring logical or || to continue.
$0=$0 plus $1=$1 to clean up extra ;, which will also print it.

Remove all text starting from a specific character

I have a file setup like
TEXT1:TEXT2
Basically lines of text separated by a :
I would like all text to the right of the : gone,
so TEXT1:TEXT2 would turn into just TEXT1

Using cut
We tell cut that our field separator is a colon, -d:, and that we want to select the first field, -f1:
$ cut -d: -f1 file
TEXT1
Using sed
We tell sed to remove the first colon on a line and everything after:
$ sed 's/:.*//' file
TEXT1
Using grep
We tell grep to select the first part of each up to but not including the first colon:
$ grep -o '^[^:]*' file
TEXT1

awk -F: '{$0=$1}1' infile
TEXT1
make ":" as your delimiter and then set column1 as your record.

Below script
awk -v FS=":" '{print $1}' file
would also give you the same result.

In AWK, replace everything after the : with nothing:
$ awk 'sub(/:.*/,"",$0)' test
TEXT1

Using sed
sed -i.bkp 's/:.*//' infile.txt
This will also change the file inplace and create a backup file named infile.txt.bkp
Using grep
grep -oP '.*(?=:)' infile.txt

sed: remove digits after word

I have a simple sed question.
I have data like this:
2600,Sale,"Approved 911973",244.72
2601,Sale,"Approved 04735C",490.51
2602,Sale,"Approved 581068",52.82
2603,Sale,"Approved 009275",88.10
How do I make it like this:
2600,Sale,Approved,244.72
2601,Sale,Approved,490.51
2602,Sale,Approved,52.82
2603,Sale,Approved,88.10
Notice the numbers after approved are gone as well as the quotes. I can remove quotes with:
sed 's/,$//gn' file
but I don't know how to remove the spaces and digits.
Thanks!

sed "s/\"Approved[^,]*/Approved/g"
It finds the quoted "Approved" followed by any non-comma character, up until the first comma encountered, and replaces it with Approved (no quotes)
2600,Sale,Approved,244.72
2601,Sale,Approved,490.51
2602,Sale,Approved,52.82
2603,Sale,Approved,88.10

Using extended regex with sed:
sed -r 's/"([^[:space:]]*)[^"]*"/\1/g' file
The above regex targets for any quoted string. If you want to target the string Approved, then:
sed -r 's/"(Approved)[^"]*"/\1/g' file
With basic regex:
sed 's/"\(Approved\)[^"]*"/\1/g' file
To target any quoted string, change Approved to [^[:space:]]*

One way using awk(only if the other columns does not contain multiple words as in your sample):
awk -F"[ ,]" '{gsub("\"","");$1=$1}1' OFS=, file

awk -F'[," ]' '{OFS=","; print $1,$2,$4,$7}' file
Output:
2600,Sale,Approved,244.72
2601,Sale,Approved,490.51
2602,Sale,Approved,52.82
2603,Sale,Approved,88.10
I suppose there is no other whitespace.

How to remove blank lines from a Unix file

I need to remove all the blank lines from an input file and write into an output file. Here is my data as below.
11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321
11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003
11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032
11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001
11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701
11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383
11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746

sed -i '/^$/d' foo
This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original:
sed '/^$/d' foo > foo.tmp
mv foo.tmp foo
If you also want to remove lines consisting only of whitespace (not just empty lines) then use:
sed -i '/^[[:space:]]*$/d' foo
Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too:
sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo

awk 'NF' filename
awk 'NF > 0' filename
sed -i '/^$/d' filename
awk '!/^$/' filename
awk '/./' filename
The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.

Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($):
grep -v '^$' infile.txt > outfile.txt
If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways:
grep -P -v '^\s*$' infile.txt > outfile.txt
or, without Perl regular expressions:
grep -v '^[[:space:]]*$' infile.txt > outfile.txt

sed -e '/^ *$/d' input > output
Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option.
If I execute the above command still I have blank lines in my output file. What could be the reason?
There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then:
sed -e 's/ *$//' -e '/^ *$/d' input > output
The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs.
Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return:
tr -d '\015' < input | sed -e 's/ *$//' -e '/^ *$/d' > output
If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against:
head -n 2 input | od -c
Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.)
You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with:
echo "Hello World" | sed 's/[[:space:]][[:space:]]*/ /'
If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.

grep . file
grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.

with awk
awk 'NF > 0' filename

To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it:
cat file.txt | perl -lane "print if /\S/"
Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do.
Cheers

You can sed's -i option to edit in-place without using temporary file:
sed -i '/^$/d' file

Why sed removes last line?

$ cat file.txt
one
two
three
$ cat file.txt | sed "s/one/1/"
1
two
Where is the word "three"?
UPDATED:
There is no line after the word "three".

As Ivan suggested, your text file is missing the end of line (EOL) marker on the final line. Since that's not present, three is printed out by sed but then immediately over-written by your prompt. You can see it if you force an extra line to be printed.
sed 's/one/1/' file.txt && echo
This is a common problem since people incorrectly think of the EOL as an indication that there's a following line (which is why it's commonly called a "newline") and not as an indication that the current line has ended.

Using comments from other posts:
older versions of sed do not process the last line of a file if no EOL or "new line" is present.
echo can be used to add a new line
Then, to solve the problem you can re-order the commands:
( cat file.txt && echo ) | sed 's/one/1/'

I guess there is no new line character after last line. sed didn't find line separator after last line and ignore it.
Update
I suggest you to rewrite this in perl (if you have it installed):
cat file.txt | perl -pe 's/one/1/'

Instead of cat'ing the file and piping into sed, run sed with the file name as an argument after the substitution string, like so:
sed "s/one/1/" file.txt
When I did it this way, I got the "three" immediately following by the prompt:
1
two
three$

A google search shows that the man page for some versions of sed (not the GNU or BSD versions, which work as you'd expect) indicate that it won't process an incomplete line (one that's not newline-terminated) at the end of a file. The solution is to ensure your files end with a newline, install GNU sed, or use awk or perl instead.

here's an awk solution
awk '{gsub("one","1")}1' file.txt

Resources