How do i find and replace a string on command line in multiple files on unix?
there are many ways .But one of the answers would be:
find . -name '*.html' |xargs perl -pi -e 's/find/replace/g'
Like the Zombie solution (and faster I assume) but with sed (standard on many distros and OSX) instead of Perl :
find . -name '*.py' | xargs sed -i .bak 's/foo/bar/g'
This will replace all foo occurences in your Python files below the current directory with bar and create a backup for each file with the .py.bak extension.
And to remove de .bak files:
find . -name "*.bak" -delete
I always did that with ed scripts or ex scripts.
for i in "$#"; do ex - "$i" << 'eof'; done
%s/old/new/
x
eof
The ex command is just the : line mode from vi.
Using find and sed with name or directories with space use this:
find . -name '*.py' -print0 | xargs -0 sed -i 's/foo/bar/g'
with recent bash shell, and assuming you do not need to traverse directories
for file in *.txt
do
while read -r line
do
echo ${line//find/replace} > temp
done <"file"
mv temp "$file"
done
Related
I have several txt files under a directory, and I want see first line of every file
So I use
ls *txt | xargs sed -n '1p'
however it only returns one line of the first file
What is wrong?
P.S.: I know I can use head, but what I ask is why sed is not working
Use the argument -t to xargs to see what is going on:
ls *txt | xargs -t sed -n '1p'
You will see that sed is run as:
sed -n '1p' foo.txt bar.txt gar.txt
and as sed only supports one input file, it will print the first line of
the file foo.txt and exit.
xargs is assuming you want to pass the list of input files all together.
To tell it to pass one at a time, you need to use the -L NUM option,
which tells xargs to pass one line at a time from your ls command.
[Note there are other issues you will run into if any file name has a blank in it]
ls *txt | xargs -L 1 -t sed -n '1p'
You will see:
sed -n '1p' foo.txt
sed -n '1p' bar.txt
sed -n '1p' gar.txt
In unix there are many ways to do any task; other ways include:
(if you use /bin/csh or /bin/tcsh):
foreach f (*txt)
echo -n $f:
sed -1p $f
end
If you use /bin/sh or /bin/ksh, then:
for files in *txt;
do
echo -n $files :
sed -n '1p' $files
done
Also consider using the program find; it lets you qualify the types of files you want to look at and can recursively examine sub directories:
`find . -type f -a -name "*txt" -print -a -exec sed -n '1p' {} \;`
First, ls and xargs are not useful here. Please read: "Don't Parse ls". For a more reliable form of the command that works with all kinds of file names, use:
sed -n '1p' *.txt
Second, sed treats its input files all as one stream. So, the above does not do what you want. Use head instead (as you said):
head -n1 *.txt
To suppress the verbose headers that head prints and make the output more like sed 1p, use the -q option:
head -qn1 *.txt
Handling many many files
If you have many many .txt files, where, depending on system configuration, "many" likely means several tens of thousands of such files, then another approach is needed. find is useful:
find . -maxdepth 1 -name '*.txt' -exec head -n1 {} +
This might work for you (GNU sed):
sed -s '1!d' file1 file2 file...
This will print the first line of each file i.e. delete all lines but the first of each file.
find . -name '{fileNamePattern}*.bz2' | xargs -n 1 -P 3 bzgrep -H "{patternToSearch}"
I am using the command above to find out a .bz2 file from set of files that have a pattern that I am looking for. It does go through the files because I can see the pattern that I am trying to find being printed on the console but I don't see the file name.
If you look at the bzgrep script (for example this version for OS X) you will see that it pipes the output from bzip2 through grep. That process loses the original filenames. grep never sees them so it cannot print them out (despite your -H flag).
Something like this should do, not exactly what you want but something similar. (You could get the prefix you were expecting by piping the output from bzgrep into sed/awk but that's a bit less simple of a command to write out.)
find . -name '{fileNamePattern}*.bz2' -printf '### %p\n' -exec bzgrep "{patternToSearch}" {} \;
I printed the file name through echo command and xargs.
find . -name "*bz2" | parallel -j 128 echo -n {}\" \" | xargs bzgrep {pattern}
Etan is very close with his answer: grep indeed does not show the filename when only dealing with one file, so you can make grep believe he's looking into multiple files, just by adding the NULL file, so the command becomes:
find . -name '{fileNamePattern}*.bz2' -printf '### %p\n'
-exec bzgrep "{patternToSearch}" {} /dev/null \;
(It's a dirty trick but it's helping me already for more than 15 years :-) )
I've a folder structure where the starting folder is test. test has two folders in it. test1 and test2, plus a bunch of files. There's a word welcome in these bunch of files as well as the files in test1 and test2. I tried this and it did not work
sed 's/\<welcome\>//g' **/*
What am I doing wrong? How can this be done?
This might work for you (GNU sed):
find test -type f -exec sed -i '/welcome/d' '{}' \;
sed -e 's/welcome//g' test > test2
The file test has several entries including welcome.
This above sed line deletes welcome from test.
You could put it in a loop..
And I see that sputnick has just answered your question to a large extent! :P
Try doing this (GNU sed):
sed -i '/welcome/s/\<welcome\>//g' test/*
The -i switch modify the files, so take care.
If -i switch is not supported on your Unix :
for i in test/*; do
sed '/welcome/s/\<welcome\>//g' "$i" > /tmp/.sed && mv /tmp/.sed "$i"
done
find . -type f|xargs perl -ni -e 'print unless(/<welcome>/)'
I ran the following command in a parametrized version of a script:
Script1 as
Nooffiles=`find $1 -mmin $2 -type f -name "$3"|wc -l`
if test $Nooffiles -eq 0
then
exit 1
else
echo "Successful"
find $1 -mmin $2 -type f -name "$3" -exec mv '{}' $4 \;
fi
The script1 works fine. It moves the files from $1 directory to $4. But after it moves the files to the new directory, I have to run another script like this:
Script2 as
for name in `find $1 -type f -name "$2"`
do
filename=`ls $name|xargs -n1 basename`
line=`tail -1 $filename | sed "s/Z/Z|$filename/"`
echo $line >> $3;
echo $filename | xargs -n1 basename;
done
Here, script2 is reading from the directory where the files were moved to by the previous script, script1. They exists there in that directory since the previous moving script worked fine. 'ls' command displays them. But the above script2 says:
File.txt: No such file or directory
Despite ls shows them in the directory, I am getting an error message like this.
Please Help.
Your script really is a mess and please be aware that you should NEVER parse filenames (like the output from ls, or find without -print0 option). See Bash Pitfalls #1.
Apart from that, I think the problem is that in your loop, you truncate the filenames output from find with basename, but then call tail with the base filename as argument, where the file really isn't located in the current folder.
I don't understand what you are doing there, but this is some more correct code that perhaps does next to what you want:
find "$1" -type f -name "$2" -print0 | while read -d '' name
do
filename=`basename "$name"`
tail -1 "$name" | sed "s/Z/Z|$filename/" >> "$3"
echo "$filename"
done
But still, there are pitfalls in this script. It is likely to fail with queer filenames input from find. For example, if your filename contains characters that are special to sed. Or if at some point $filename is --help etc.etc.etc.
When we develop locally, we append ".dev" or ".prod" to files that should be made available only to the development/production server respectively.
What I would like to do is; after deploying the site to the server, recursively find all files with the ".dev" suffix (for example) and remove it (renaming the file). How would I go about doing this, preferably entirely in the shell (without scripts) so I can add it to our deployment script?
Our servers run Ubuntu 10.04.
Try this (not entirely shell-only, requires the find and mv utilities):
find . '(' -name '*.dev' -o -name '*.prod' ')' -type f -execdir sh -c 'mv -- "$0" "${0%.*}"' '{}' ';'
If you have the rename and xargs utilities, you can speed this up a lot:
find . '(' -name '*.dev' -o -name '*.prod' ')' -type f -print0 | xargs -0 rename 's/\.(dev|prod)$//'
Both versions should work with any file name, including file names containing newlines.
It's totally untested, but this should work in the POSIX-like shell of your choice:
remove-suffix () {
local filename
while read filename; do
mv "$filename" "$(printf %s "$filename" | sed "s/\\.$1\$//")"
done
}
find -name '*.dev' | remove-suffix .dev
Note: In the very unusual case that one or more of your filenames contains a newline character, this won't work.
for file in `ls *.dev`; do echo "Old Name $file"; new_name=`echo $file | sed -e 's/dev//'` ; echo "New Name $new_name"; mv $file $new_name; done
In an example of something I used recently this code looks for any file that ends with new.xml changes a date in the filename (filenames were of the form xmlEventLog_2010-03-23T11:16:16_PFM_1_1.xml), removes the _new from the name and renames the filename to the new name :
for file in `ls *new.xml`; do echo "Old Name $file"; new_name=`echo $file | sed -e 's/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/2010-03-23/g' | sed 's/_new//g'` ; echo "New Name $new_name"; mv $file $new_name; done
Is this the type of thing you wanted?
find /fullpath -type f -name "*.dev"|sed 's|\(.*\)\(\.pdf\)|mv & \1.sometag|' | sh