find and replace from command line unix - unix

I have a multi line text file where each line has the format
..... Game #29832: ......
I want to append the character '1' to each number on each line (which is different on every line), does anyone know of a way to do this from the command line?
Thanks


sed -i -e 's/Game #[0-9]*/&1/' file
-i is for in-place editing, and & means whatever matched from the pattern. If you don't want to overwrite the file, omit the -i flag.


Using sed:
cat file | sed -e 's/\(Game #[0-9]*\)/\11/'


sed 's/ Game #\([0-9]*\):/ Game #1\1:/' yourfile.txt


GNU awk
awk '{b=gensub(/(Game #[0-9]+)/ ,"\\11","g",$0); print b }' file

Related

Unix command to parse string

I'm trying to figure out a command to parse the following file content:
Operation=GET
Type=HOME
Counters=CacheHit=0,Exception=1,Validated=0
I need to extract Exception=1 into its own line. I'm fiddling with awk, sed and grep but not making much progress. Does anyone have any tips on using any unix command to perform this?
Thanks

Since your file is close to bash syntax, there is a fun little trick you can do to make bash itself parse the file. First, use some program like tr to transform the input into a something bash can parse, and then "source" that, which will create shell variables you can expand later to get the values.
source <(tr , $'\n' < file_name_goes_here)
echo $Exception

Many ways to do this. Here is one assuming the file is called "file.txt". Grab the line you want, replace everything from the start of the line up to Except with just Except, then pull out the first field using comma as the delimiter.
$ grep Exception file.txt | sed 's/.*Except/Except/g' | cut -d, -f 1
Exception=1
If you wanted to use gawk:
$ grep Exception file.txt | sed 's/.*Except/Except/g' | gawk -F, '{print $1}'
Exception=1
or just using grep and sed:
$ grep Exception file.txt | sed 's/.*\(Exception=[0-9]*\).*/\1/g'
Exception=1
or as #sheltter reminded me:
$ egrep -o "Exception=[0-9]+" file.txt
Exception=1

No need to use a mix of commands.
awk -F, 'NR==2 {print RS$1}' RS="Exception" file
Exception=1
Here we split the line by the keyword we look for RS="Exception"
If the line has two record (only when keyword is found), then
print first field, separated using command, with Record selector.
PS This only works if you have one Exception field

Combine two sed commands in one line

I'm looking for an in place command to change all file lines which end with :.:.
From
chr01 1453173 . C T 655.85 PASS . GT:AD:DP:PGT:PID 0/1:25,29:54:.:.
To
chr01 1453173 . C T 655.85 PASS . GT:AD:DP 0/1:25,29:54
In words, I'm basically deleting :PGT:PID and :.:. from any line ending with :.:.

With GNU sed and Solaris sed:
sed '/:\.:\.$/{s///;s/PGT:PID//;}' file
If you want to edit your file with GNU sed "in place" use option -i.

With awk that'd be:
awk 'sub(/:\.:\.$/,""){sub(/:PGT:PID/,"")} 1' file
chr01 1453173 . C T 655.85 PASS . GT:AD:DP 0/1:25,29:54
and for inplace editing with gawk you could add the -i inplace option while with any awk you can just add > tmp && mv tmp file.

You are looking for something like:
sed -i.bak "/^.*:\.:\.$/ {s/:PGT:PID//g; s/:\.:\.//g;}" file
it does inplace replace with file and creates a backup as file.bak
the /^.*:\.:\.$/ restricts the s command to lines ending in :.:. the . need quoting because they are special characters for regexes
the s sommand replaces the strings with the empty string

As a one-liner:
sed -i -e '/:\.:\.$/!n' -e 's///' -e 's/:PGT:PID//g' "$file"
Expanded:
/:\.:\.$/!n # leave lines untouched unless they end in ":.:."
s/// # replace the matched ":.:." with nothing
s/:PGT:PID//g # replace ":PGT:PID" with nothing, everywhere
We use the -i flag to perform in-place edits. We pass each line as a separate -e expression for portability: some sed implementations allow several commands to be concatenated with ;, but that is not required by the POSIX standard.

Changing a line in a file on UNIX

I have created a file named "asd.txt" on a UNIX based system.
I added four lines by using echo command.
Now, I would like to change the first line of this file.
I am not allowed to use any text editors, such as vi.
I have to do this by using only command line. Can anyone help?
Thanks.

Here is how you could do it with sed.
sed '1 s/search/replace/' asd.txt
If you are feeling up to it and have GNU sed, use the -i switch to do it in place.
If you want to replace the entire first line how about doing this?
echo "Here is my new first line" && sed '1d' asd.txt
For both of these commands you can redirect the output to a new file using the > operator.

#!/bin/bash
cat <(echo "Replacement") <(tail -n +2 foo.txt)

How to remove blank lines from a Unix file

I need to remove all the blank lines from an input file and write into an output file. Here is my data as below.
11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321
11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003
11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032
11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001
11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701
11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326
11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383
11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955
11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746

sed -i '/^$/d' foo
This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original:
sed '/^$/d' foo > foo.tmp
mv foo.tmp foo
If you also want to remove lines consisting only of whitespace (not just empty lines) then use:
sed -i '/^[[:space:]]*$/d' foo
Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too:
sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo

awk 'NF' filename
awk 'NF > 0' filename
sed -i '/^$/d' filename
awk '!/^$/' filename
awk '/./' filename
The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.

Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($):
grep -v '^$' infile.txt > outfile.txt
If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways:
grep -P -v '^\s*$' infile.txt > outfile.txt
or, without Perl regular expressions:
grep -v '^[[:space:]]*$' infile.txt > outfile.txt

sed -e '/^ *$/d' input > output
Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option.
If I execute the above command still I have blank lines in my output file. What could be the reason?
There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then:
sed -e 's/ *$//' -e '/^ *$/d' input > output
The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs.
Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return:
tr -d '\015' < input | sed -e 's/ *$//' -e '/^ *$/d' > output
If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against:
head -n 2 input | od -c
Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.)
You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with:
echo "Hello World" | sed 's/[[:space:]][[:space:]]*/ /'
If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.

grep . file
grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.

with awk
awk 'NF > 0' filename

To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it:
cat file.txt | perl -lane "print if /\S/"
Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do.
Cheers

You can sed's -i option to edit in-place without using temporary file:
sed -i '/^$/d' file

Replace \n with \r\n in Unix file

I'm trying to do the opposite of this question, replacing Unix line endings with Windows line endings, so that I can use SQL Server bcp over samba to import the file. I have sed installed but not dos2unix. I tried reversing the examples but to no avail.
Here's the command I'm using.
sed -e 's/\n/\r\n/g' myfile
I executed this and then ran od -c myfile, expecting to see \r\n where there used to be \n. But there all still \n. (Or at least they appear to be. The output of od overflows my screen buffer, so I don't get to see the beginning of the file).
I haven't been able to figure out what I'm doing wrong. Any suggestions?

When faced with this, I use a simple perl one-liner:
perl -pi -e 's/\n/\r\n/' filename
because sed behavior varies, and I know this works.

What is the problem with getting dos2unix onto the machine?
What is the platform you are working with?
Do you have GNU sed or regular non-GNU sed?
On Solaris, /usr/bin/sed requires:
sed 's/$/^M/'
where I entered the '^M' by typing controlV controlM. The '$' matches at the end of the line, and replaces the end of line with the control-M. You can script that, too.
Mechanisms expecting sed to expand '\r' or '\\r' to control-M are going to be platform-specific, at best.

You don't need the -e option.
$ matches the endline character. This sed command will insert a \r character before the end of line:
sed 's/$/\r/' myfile

Just adding a \r (aka ^M, see Jonathan Leffler's answer) in front of \n is not safe because the file might have mixed mode EOL, so then you risk ending up with some lines becomming \r\r\n. The safe thing to do is first remove all '\r' characters, and then insert (a single) \r before \n.
#!/bin/sh
sed 's/^M//g' ${1+"$#"} | sed 's/$/^M/'
Updated to use ^M.

sed 's/\([^^M]\)$/\0^M/' your_file
This makes sure you only insert a \r when there is no \r before \n. This worked for me.

Try using:
echo " this is output" > input
sed 's/$/\r/g' input |od -c

Maybe if you try it this way
cat myfile | sed 's/\n/\r\n/g' > myfile.win
will work, from my understanding your just making the replacements to the console output, you need to redirect output to a file, in this case myfile.win, then you could just rename it to whatever you want. The whole script would be (running inside a directory full of this kind of files):
#!/bin/bash
for file in $(find . -type f -name '*')
do
cat $file | sed 's/\n/\r\n/g' > $file.new
mv -f $file.new $file
done

Resources