I am trying to find out if it is possible to edit a file in a single sed command without manually streaming the edited content into a new file and then renaming the new file to the original file name.
I tried the -i option but my Solaris system said that -i is an illegal option. Is there a different way?
The -i option streams the edited content into a new file and then renames it behind the scenes, anyway.
Example:
sed -i 's/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' filename
while on macOS you need:
sed -i '' 's/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' filename
On a system where sed does not have the ability to edit files in place, I think the better solution would be to use perl:
perl -pi -e 's/foo/bar/g' file.txt
Although this does create a temporary file, it replaces the original because an empty in place suffix/extension has been supplied.
Note that on OS X you might get strange errors like "invalid command code" or other strange errors when running this command. To fix this issue try
sed -i '' -e "s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g" <file>
This is because on the OSX version of sed, the -i option expects an extension argument so your command is actually parsed as the extension argument and the file path is interpreted as the command code. Source: https://stackoverflow.com/a/19457213
The following works fine on my mac
sed -i.bak 's/foo/bar/g' sample
We are replacing foo with bar in sample file. Backup of original file will be saved in sample.bak
For editing inline without backup, use the following command
sed -i'' 's/foo/bar/g' sample
One thing to note, sed cannot write files on its own as the sole purpose of sed is to act as an editor on the "stream" (ie pipelines of stdin, stdout, stderr, and other >&n buffers, sockets and the like). With this in mind you can use another command tee to write the output back to the file. Another option is to create a patch from piping the content into diff.
Tee method
sed '/regex/' <file> | tee <file>
Patch method
sed '/regex/' <file> | diff -p <file> /dev/stdin | patch
UPDATE:
Also, note that patch will get the file to change from line 1 of the diff output:
Patch does not need to know which file to access as this is found in the first line of the output from diff:
$ echo foobar | tee fubar
$ sed 's/oo/u/' fubar | diff -p fubar /dev/stdin
*** fubar 2014-03-15 18:06:09.000000000 -0500
--- /dev/stdin 2014-03-15 18:06:41.000000000 -0500
***************
*** 1 ****
! foobar
--- 1 ----
! fubar
$ sed 's/oo/u/' fubar | diff -p fubar /dev/stdin | patch
patching file fubar
Versions of sed that support the -i option for editing a file in place write to a temporary file and then rename the file.
Alternatively, you can just use ed. For example, to change all occurrences of foo to bar in the file file.txt, you can do:
echo ',s/foo/bar/g; w' | tr \; '\012' | ed -s file.txt
Syntax is similar to sed, but certainly not exactly the same.
Even if you don't have a -i supporting sed, you can easily write a script to do the work for you. Instead of sed -i 's/foo/bar/g' file, you could do inline file sed 's/foo/bar/g'. Such a script is trivial to write. For example:
#!/bin/sh
IN=$1
shift
trap 'rm -f "$tmp"' 0
tmp=$( mktemp )
<"$IN" "$#" >"$tmp" && cat "$tmp" > "$IN" # preserve hard links
should be adequate for most uses.
You could use vi
vi -c '%s/foo/bar/g' my.txt -c 'wq'
sed supports in-place editing. From man sed:
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension supplied)
Example:
Let's say you have a file hello.txtwith the text:
hello world!
If you want to keep a backup of the old file, use:
sed -i.bak 's/hello/bonjour' hello.txt
You will end up with two files: hello.txt with the content:
bonjour world!
and hello.txt.bak with the old content.
If you don't want to keep a copy, just don't pass the extension parameter.
If you are replacing the same amount of characters and after carefully reading “In-place” editing of files...
You can also use the redirection operator <> to open the file to read and write:
sed 's/foo/bar/g' file 1<> file
See it live:
$ cat file
hello
i am here # see "here"
$ sed 's/here/away/' file 1<> file # Run the `sed` command
$ cat file
hello
i am away # this line is changed now
From Bash Reference Manual → 3.6.10 Opening File Descriptors for Reading and Writing:
The redirection operator
[n]<>word
causes the file whose name is the expansion of word to be opened for
both reading and writing on file descriptor n, or on file descriptor 0
if n is not specified. If the file does not exist, it is created.
Like Moneypenny said in Skyfall: "Sometimes the old ways are best."
Kincade said something similar later on.
$ printf ',s/false/true/g\nw\n' | ed {YourFileHere}
Happy editing in place.
Added '\nw\n' to write the file. Apologies for delay answering request.
You didn't specify what shell you are using, but with zsh you could use the =( ) construct to achieve this. Something along the lines of:
cp =(sed ... file; sync) file
=( ) is similar to >( ) but creates a temporary file which is automatically deleted when cp terminates.
mv file.txt file.tmp && sed 's/foo/bar/g' < file.tmp > file.txt
Should preserve all hardlinks, since output is directed back to overwrite the contents of the original file, and avoids any need for a special version of sed.
To resolve this issue on Mac I had to add some unix functions to core-utils following this.
brew install grep
==> Caveats
All commands have been installed with the prefix "g".
If you need to use these commands with their normal names, you
can add a "gnubin" directory to your PATH from your bashrc like:
PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"
Call with gsed instead of sed. The mac default doesn't like how grep -rl displays file names with the ./ preprended.
~/my-dir/configs$ grep -rl Promise . | xargs sed -i 's/Promise/Bluebird/g'
sed: 1: "./test_config.js": invalid command code .
I also had to use xargs -I{} sed -i 's/Promise/Bluebird/g' {} for files with a space in the name.
Very good examples. I had the challenge to edit in place many files and the -i option seems to be the only reasonable solution using it within the find command. Here the script to add "version:" in front of the first line of each file:
find . -name pkg.json -print -exec sed -i '.bak' '1 s/^/version /' {} \;
In case you want to replace stings contain '/',you can use '?'. i.e. replace '/usr/local/bin/python' with '/usr/bin/python3' for all *.py files.
find . -name \*.py -exec sed -i 's?/usr/local/bin/python?/usr/bin/python3?g' {} \;
Related
sed -i s/oldstr/newstr/g
I want to replace all old string with new string in a directory. When I executed this command, It shows all strings are changed to new string on console. But when I entered into file and checked, Old string still remains same. Please help to figure the issue
In case you are in macos you could try:
sed -i '' 's/oldstr/newstr/g' your-file
From the man:
Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved.
So by using -i '' it will not create a backup and apply changes to your file.
to keep a backup this could be used:
sed -i.bak 's/oldstr/newstr/g' your-file
To replace the oldstr in multiple files you could use this:
perl -pi -e 's/oldstr/newstr/g' *
To keep a backup:
perl -i.bak -p -e 's/oldstr/newstr/g' *
#> sed -i \.orig 's/old/new/g' file
That should be the same as:
#> mv file file.orig
#> sed 's/old/new/g' file.orig > file.
I have several txt files under a directory, and I want see first line of every file
So I use
ls *txt | xargs sed -n '1p'
however it only returns one line of the first file
What is wrong?
P.S.: I know I can use head, but what I ask is why sed is not working
Use the argument -t to xargs to see what is going on:
ls *txt | xargs -t sed -n '1p'
You will see that sed is run as:
sed -n '1p' foo.txt bar.txt gar.txt
and as sed only supports one input file, it will print the first line of
the file foo.txt and exit.
xargs is assuming you want to pass the list of input files all together.
To tell it to pass one at a time, you need to use the -L NUM option,
which tells xargs to pass one line at a time from your ls command.
[Note there are other issues you will run into if any file name has a blank in it]
ls *txt | xargs -L 1 -t sed -n '1p'
You will see:
sed -n '1p' foo.txt
sed -n '1p' bar.txt
sed -n '1p' gar.txt
In unix there are many ways to do any task; other ways include:
(if you use /bin/csh or /bin/tcsh):
foreach f (*txt)
echo -n $f:
sed -1p $f
end
If you use /bin/sh or /bin/ksh, then:
for files in *txt;
do
echo -n $files :
sed -n '1p' $files
done
Also consider using the program find; it lets you qualify the types of files you want to look at and can recursively examine sub directories:
`find . -type f -a -name "*txt" -print -a -exec sed -n '1p' {} \;`
First, ls and xargs are not useful here. Please read: "Don't Parse ls". For a more reliable form of the command that works with all kinds of file names, use:
sed -n '1p' *.txt
Second, sed treats its input files all as one stream. So, the above does not do what you want. Use head instead (as you said):
head -n1 *.txt
To suppress the verbose headers that head prints and make the output more like sed 1p, use the -q option:
head -qn1 *.txt
Handling many many files
If you have many many .txt files, where, depending on system configuration, "many" likely means several tens of thousands of such files, then another approach is needed. find is useful:
find . -maxdepth 1 -name '*.txt' -exec head -n1 {} +
This might work for you (GNU sed):
sed -s '1!d' file1 file2 file...
This will print the first line of each file i.e. delete all lines but the first of each file.
Apologies if this has been covered before, I had a search but couldn't find anything. I realise this is probably a simple error, but I'm currently learning UNIX.
I've been trying to do a sed substitution but every time I've tried the command just seems to print out the files and not substitute anything?
The command in question is:
find . -name 'config.xml' -exec sed 's/<name>development<\/name>/<name>releases<\/name>/g' {} \;
After I run this command, if I try to find and grep for <name>development<\/name> it still returns results. Why?
Any help would be appreciated! Thanks.
This is because you're not actually editing the file--just printing the edited version to stdout. Use sed's -i flag:
-i[SUFFIX]
--in-place[=SUFFIX]
This option specifies that files are to be edited in-place. GNU
sed does this by creating a temporary file and sending output to
this file rather than to the standard output.(1).
So:
find . -name 'config.xml' -exec sed -i 's/<name>development<\/name>/<name>releases<\/name>/g' {} \;
You can also use a different character besides / if you don't want to have to escape them:
find . -name 'config.xml' -exec sed -i 's#<name>development</name>#<name>releases</name>#g' {} \;
I have a log file that is written to by a server. I wrote a bash script to send me an email if there is an error in the server. I would now like to remove the lines containing the errors so I don't keep getting emails. I accomplish this by doing the following:
sed -i "/WARNING/d" logs/console.log
After running sed however, no more changes are written to the log. I'm guessing this is because running sed closes any open file descriptors or something. However, when I edit the file and manually remove the warning lines with vi I don't have this problem.
I have also tried redirecting the server output myself with both '>' and '>>' operators and after editing the file with sed the same thing happens (i.e. they are no longer updated).
When sed rewrites the log file the server process probably gets an IO error and doesn't try to reopen the log file. I don't know if this approach can work out. sed definitely doesn't have flags to tweak the way it rewrites files with the -i flag, and I don't know if the server can be tweaked to be more resilient when appending to the log.
So your best option might be a different approach: save the timestamp of the last error look for errors after that timestamp. Something like this:
ts=
file=console.log
while :; do
if test "$ts"; then
if sed -e "1,/$ts/d" $file | grep -q WARNING; then
sed -e "1,/$ts/d" $file | sendmail ...
ts=$(tail -n 1 $file | cut -f1 -d' ')
fi
else
if grep -q WARNING $file; then
sendmail ... < $file
ts=$(tail -n 1 $file | cut -f1 -d' ')
fi
fi
sleep 15
done
This script is just to give you an idea, it can be improved.
I've a folder structure where the starting folder is test. test has two folders in it. test1 and test2, plus a bunch of files. There's a word welcome in these bunch of files as well as the files in test1 and test2. I tried this and it did not work
sed 's/\<welcome\>//g' **/*
What am I doing wrong? How can this be done?
This might work for you (GNU sed):
find test -type f -exec sed -i '/welcome/d' '{}' \;
sed -e 's/welcome//g' test > test2
The file test has several entries including welcome.
This above sed line deletes welcome from test.
You could put it in a loop..
And I see that sputnick has just answered your question to a large extent! :P
Try doing this (GNU sed):
sed -i '/welcome/s/\<welcome\>//g' test/*
The -i switch modify the files, so take care.
If -i switch is not supported on your Unix :
for i in test/*; do
sed '/welcome/s/\<welcome\>//g' "$i" > /tmp/.sed && mv /tmp/.sed "$i"
done
find . -type f|xargs perl -ni -e 'print unless(/<welcome>/)'