I'm trying to transfer files which have been updated in the 31 days. I'm trying to run
/bin/rsync --remove-source-files --files-from="<(find /tmp/rsync/source -type f -mtime -31 -print0)" /tmp/rsync/source /tmp/rsync/destination
However when trying this, i keep receiving the following error:
failed to open files-from file <(find /tmp/rsync/source -type f -mtime -31 -print0): No such file or directory
The directory exists and is accessible.
This is the output of the find:
$ find /tmp/rsync/source -type f -mtime -31
/tmp/rsync/source/testfile2.txt
/tmp/rsync/source/testfile.txt
/tmp/rsync/source/sourcefile.txt
/bin/rsync --remove-source-files --files-from="<(find /tmp/rsync/source -type f -mtime -31 -print0)" /tmp/rsync/source /tmp/rsync/destination
<( ... ) is Bash Shell Process Substitution - https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html.
So:
You have to be using Bash, i.e. as your command-line shell if running interactively, or as the shell script interpreter, if running in a shell script.
If not using Bash on command-line nor in a shell script or if you are passing that command to something that is executing the rsync program directly and not through a Bash shell, it won't work.
You can't put that syntax inside double quotes as shown. When quoted like that, Bash interprets it as a filename (thus the error).
It's unfortunate this this is coming up first in search results when other questions about this have much better answers.
I've seen people use the --files-from="<(...)" notation several places, but I don't see any reference to it in the rsync manual. Maybe it's a special shell syntax for some people? or distro-added feature? I get the same error message as above when I try to use that notation.
The official way to do it is to either write your list of files into a real file:
find . -mtime -7 > past_week.txt
rsync --files-from=past_week.txt SRC/ DST/
or to pipe the list of files on stdin:
find . -mtime -7 | rsync --files-from=- SRC/ DST/
The single dash - filename here means STDIN, and this is a common convention among unix tools.
If you are worried about files with newlines in the file name, you should use NUL delimiters for the list. (but beware this will also affect --include-from and --exclude-from)
find . -mtime -7 -print0 | rsync -0 --files-from=- SRC/ DST/
You can also list out the files on the command line like rsync -av `find . -mtime -7` DST/ but that doesn't preserve their hierarchy in the tree, and if you have more than a few files that will create a massive command line, and may fail to execute if it exceeds the limit of the operating system.
Tried, this works good for me
find /tmp/rsync/source -type f -mtime -31 | rsync -rcvh /tmp/rsync/source/ /tmp/rsync/destination/ --dry-run
Remove dry-run for actual execution
Since i was forced to use a pre-existing script which was parsing on "{" brackets and where you couldn't run commands before the rsync script i was unable to use the above mentioned solutions.
However i was able to use the following to get it working:
/bin/rsync --recursive --remove-source-files `find /tmp/rsync/source -type f -mtime -31` /tmp/rsync/destination
Related
It's a simple question that I can't seem to figure out. I'm on a Mac with Big Sur with all the latest updates, and I'm going through Terminal to get these commands to run. If there's a better way please let me know.
This is, in basic terms, what I'm trying to do--I want RSYNC to recursively go through a source directory (which in this case would ideally be an entire drive), find any files modified within the last 24 hours, and copy those to another drive, while preserving the folder structure. So if I have:
/Volumes/Drive1/Folder1/File1.file
/Volumes/Drive1/Folder1/File2.file
/Volumes/Drive1/Folder1/File3.file
And File1 has been modified in the last 24 hours, but the other two haven't, I want it to copy that file, so that on the second drive I wind up with:
/Volumes/Drive2/Folder1/File1.file
But without copying File2 and File3.
I've tried a lot of different solutions and strings, but I'm running into problems. The closest I've been able to get is this:
find /Volumes/Drive1/ -type f -mtime -1 -exec cp -a "{}" /Volumes/Drive2/ \;
The problem is that while this one does go through Drive1 and find all the files newer than a day like I want, when it copies them it just dumps them all into the root of Drive2.
This one also seems to come close:
rsync --progress --files-from=<(find /Volumes/Drive1/ -mtime -1 -type f -exec basename {} \;) /Volumes/Drive1/ /Volumes/Drive2/
This one also identifies all the files modified in the last 24 hours, but instead of copying them it gives an error, "link_stat (filename and path) failed: no such file or directory (2)."
I've spent several days trying to figure out what I'm doing wrong but I can't figure it out. Help please!
I think this'll work:
srcDir=/Volumes/Drive1
destDir=/Volumes/Drive2
(cd "$srcDir" && find . -type f -mtime -1 -print0) |
while IFS= read -r -d $'\0' filepath; do
mkdir -p "$(dirname "$destDir/$filepath")"
cp -a "$srcDir/$filepath" "$destDir/$filepath"
done
Explanation:
Using cd "$srcDir"; find . -whatever will generate relative paths (starting with "./") from the source directory to the found files; that means appending the results to $srcDir and $destDir will give the full source and destination paths for each file.
Putting it in parentheses makes it run in a subshell, so the cd won't affect other commands. Coupling cd and find with && means that if cd fails, it won't run find (which would run in the wrong place, generate a list of the wrong file file, and generally cause trouble).
Using -print0 and while IFS= read -r -d $'\0' is a standard weird-filename-safe way of iterating over found files (see BashFAQ #20). Note that if anything in the loop reads from standard input (e.g. cp -i asking for confirmation), it'll steal part of the file list; if this is a worry, use this variant (instead of the pipe) to send the file list over file descriptor #3 instead of standard input:
while IFS= read -r -d $'\0' filepath <&3; do
...
done 3< <(cd "$srcDir" && find . -type f -mtime -1 -print0)
Finally, mkdir -p is used to make sure the destination directory exists, and then cp to copy the file.
I have one directory, with multiple subdirectories. In each subdirectory there is a file on which I want to perform analysis (code already written).
Common for all subdirectories is that they have file with same extension on which analysis should be performed.
Using Unix shell, is there a way to write a commands which will:
for each subdirectory in main directory, use file with certain extension and perform further commands on that file (further commands include creation of some new directories and files)
repeat it for all subdirectories in main directory and files inside them
I will appreciate all suggestions.
Use the find command. find . -type f -name '*.txt' -exec prog \{} \; will execute program prog with the name of every file in the current directory . and below with the extension .txt (i.e. that matches the pattern *.txt). The -type f excludes directories (and pipes and devices). The -exec means execute this command; the \{} will be replaced with the filename; \; means end of command.
This definitely works if your filenames have no spaces, quote marks, or backslashes in them. If they do, it gets a little trickier: find . -type f -name '*.txt' -print0 | xargs -0 -n1 prog, assuming the filename argument goes at the end of the line. The -print0 means output the file with null termination (zero character) and the -0 means input with null termination. xargs takes its input and invokes prog for every null-terminated word. -n1 means only use one argument per invocation; you can omit it if the program accepts multiple filenames as arguments. You can use -i if you need to insert text after the argument.
Note: I am aware that using -exec for various obscure reasons may not be preferable for, say, secure system shell scripts, but for a use case like this it is fine.
Hi I've got lots of folders with the name "#eaDir" all across one of my disks and I'd like to search through, find all of them and delete them and their contents.
I know this is probably a combination of the find and rm command but I can't quite figure them out. Can anyone help?
Try this:
find . -type d -name '#eaDir' -print0 | xargs -rt0 rm -rv
Here's the same thing but using explicit long options for xargs:
find . -type d -name '#eaDir' -print0 | xargs --no-run-if-empty --verbose --null rm -rv
(using long options is always a good idea if you're writing scripts that will need to be maintained/reviewed by other people)
But before anything else:
man find
man xargs
find /path/to/the/disk -type d -name "#eaDir" -delete
Notice that the order here is fundamental: quoting the manpage,
Warnings: Don't forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everything below the starting points you specified.
So, as always, first try your find command with -print, then, when you checked that everything works fine, replace it with -delete. Notice that -delete implies -depth, so, to do meaningful testing with -print, you should explicitly specify it in the expression:
When testing a find command line that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises.
Goto root folder or directory and execute the following command:
find . -path '*/#eaDir/*' -delete -print && find . -path '*/#eaDir' -delete -print
This should work for you.
when I want to execute some shell script in Unix (and let's say that I am in the directory where the script is), I just type:
./someShellScript.sh
and when I want to "source" it (e.g. run it in the current shell, NOT in a new shell), I just type the same command just with the "." (or with the "source" command equivalent) before it:
. ./someShellScript.sh
And now the tricky part. When I want to execute "multiple" shell scripts (let's say all the files with .sh suffix) in the current directory, I type:
find . -type f -name *.sh -exec {} \;
but "what command should I use to "SOURCE" multiple shell scripts in a directory"?
I tried this so far but it DIDN'T work:
find . -type f -name *.sh -exec . {} \;
and it only threw this error:
find: `.': Permission denied
Thanks.
for file in *.sh
do . $file
done
Try the following version of Jonathan's code:
export IFSbak = $IFS;export IFS="\0"
for file in `find . -type f -name '*.sh' -print0`
do source "$file"
end
export IFS=$IFSbak
The problem lies in the way shell's work, and that '.' itself is not a command (neither is source in this). When you run find, the shell will fork itself and then turn into the find process, meaning that any environment variables or other environmental stuff goes into the find process (or, more likely, find forks itself for new processes for each exec).
Also, note that your command (and Jonathan's) will not work if there are spaces in the file names.
You can use find and xargs to do this:
find . -type f -name "*.sh" | xargs -I sh {}
On the UNIX bash shell (specifically Mac OS X Leopard) what would be the simplest way to copy every file having a specific extension from a folder hierarchy (including subdirectories) to the same destination folder (without subfolders)?
Obviously there is the problem of having duplicates in the source hierarchy. I wouldn't mind if they are overwritten.
Example: I need to copy every .txt file in the following hierarchy
/foo/a.txt
/foo/x.jpg
/foo/bar/a.txt
/foo/bar/c.jpg
/foo/bar/b.txt
To a folder named 'dest' and get:
/dest/a.txt
/dest/b.txt
In bash:
find /foo -iname '*.txt' -exec cp \{\} /dest/ \;
find will find all the files under the path /foo matching the wildcard *.txt, case insensitively (That's what -iname means). For each file, find will execute cp {} /dest/, with the found file in place of {}.
The only problem with Magnus' solution is that it forks off a new "cp" process for every file, which is not terribly efficient especially if there is a large number of files.
On Linux (or other systems with GNU coreutils) you can do:
find . -name "*.xml" -print0 | xargs -0 echo cp -t a
(The -0 allows it to work when your filenames have weird characters -- like spaces -- in them.)
Unfortunately I think Macs come with BSD-style tools. Anyone know a "standard" equivalent to the "-t" switch?
The answers above don't allow for name collisions as the asker didn't mind files being over-written.
I do mind files being over-written so came up with a different approach. Replacing each / in the path with - keep the hierarchy in the names, and puts all the files in one flat folder.
We use find to get the list of all files, then awk to create a mv command with the original filename and the modified filename then pass those to bash to be executed.
find ./from -type f | awk '{ str=$0; sub(/\.\//, "", str); gsub(/\//, "-", str); print "mv " $0 " ./to/" str }' | bash
where ./from and ./to are directories to mv from and to.
If you really want to run just one command, why not cons one up and run it? Like so:
$ find /foo -name '*.txt' | xargs echo | sed -e 's/^/cp /' -e 's|$| /dest|' | bash -sx
But that won't matter too much performance-wise unless you do this a lot or have a ton of files. Be careful of name collusions, however. I noticed in testing that GNU cp at least warns of collisions:
cp: will not overwrite just-created `/dest/tubguide.tex' with `./texmf/tex/plain/tugboat/tubguide.tex'
I think the cleanest is:
$ find /foo -name '*.txt' | xargs -i cp {} /dest
Less syntax to remember than the -exec option.
As far as the man page for cp on a FreeBSD box goes, there's no need for a -t switch. cp will assume the last argument on the command line to be the target directory if more than two names are passed.