In my unix shell script, I have a for loop and looping thru the files in a directory, I want to check the modified date of a file is greater than some number of days, say 10 days and do some processing.
Any help in this regards is highly appreciated.
The find command has a built-in test for this:
find <yourdir> -type f -mtime +10 -exec yourproc {} \;
This will run "yourproc matching_filename" on each file in <yourdir> older than 10 days.
The -ctime test works similarly, using the inode change time rather than the
modification time.
find . -mtime +10
This could be integrated into a script in several ways. For one:
isOlder () {
[ "`find $1 -mtime +10`" == "$1" ]
return $?
}
ls | while read fname; do
if isOlder "$fname"; then
echo $fname is older
else
echo $fname is not older
fi
done
I listed the above because it sounded like you mainly wanted a test and were doing other processing. OTOH, if you really just want the older files:
$ find . -mtime +10 | while read fname; do
> echo $fname
> done
Related
I want to delete the graphite's storage whisper's data but there ain't anything in the graphite docs.
One way I did is deleting the the files at /opt/graphite...../whispers/stats... manually.
But this is tedious, so how do I do it?
Currently, deleting files from /opt/graphite/storage/whisper/ is the correct way to clean up whisper data.
As for the tedious side of the process, you could use the find command if there is a certain pattern that your trying to remove.
find /opt/graphite/storage/whisper -name loadavg.wsp -delete
Similar Question on answers.launchpad.net/graphite
I suppose that this is going into Server Fault territory, but I added
the following cron job to delete old metrics of ours that haven't been
written to for over 30 days (e.g. of cloud instances that have been
disposed):
find /mnt/graphite/storage -mtime +30 | grep -E \
"/mnt/graphite/storage/whisper/collectd/app_name/[^/]*" -o \
| uniq | xargs rm -rf
This will delete directories which have valid data.
First:
find whisperDir -mtime +30 -type f | xargs rm
And then delete empty dirs
find . -type d -empty | xargs rmdir
This last step should be repeated, because may be new empty directories will be left.
As people have pointed out, removing the files is the way to go. Expanding on previous answers, I made this script that removes any file that has exceeded its max retention age. Run it as a cronjob fairly regularly.
#!/bin/bash
d=$1
now=$(date +%s)
MINRET=86400
if [ -z "$d" ]; then
echo "Must specify a directory to clean" >&2
exit 1
fi
find $d -name '*.wsp' | while read w; do
age=$((now - $(stat -c '%Y' "$w")))
if [ $age -gt $MINRET ]; then
retention=$(whisper-info.py $w maxRetention)
if [ $age -gt $retention ]; then
echo "Removing $w ($age > $retention)"
rm $w
fi
fi
done
find $d -empty -type d -delete
A couple of bits to be aware of - the whisper-info call is quite heavyweight. To reduce the number of calls to it I've put the MINRET constant in, so that no file will be considered for deletion until it is 1 day old (24*60*60 seconds) - adjust to fit your needs. There are probably other things that can be done to shard the job or generally improve its efficiency, but I haven't had need to as yet.
I know there is this option for unix's find command:
find -version
GNU find version 4.1
-newer file Compares the modification date of the found file with that of
the file given. This matches if someone has modified the found
file more recently than file.
Is there an option that will let me find files that are older than a certain file. I would like to delete all files from a directory for cleanup. So, an alternative where I would find all files older than N days would do the job too.
You can use a ! to negate the -newer operation like this:
find . \! -newer filename
If you want to find files that were last modified more then 7 days ago use:
find . -mtime +7
UPDATE:
To avoid matching on the file you are comparing against use the following:
find . \! -newer filename \! -samefile filename
UPDATE2 (several years later):
The following is more complicated, but does do a strictly older than match. It uses -exec and test -ot to test each file against the comparison file. The second -exec is only executed if the first one (the test) succeeds. Remove the echo to actually remove the files.
find . -type f -exec test '{}' -ot filename \; -a -exec echo rm -f '{}' +
You can just use negation:
find ... \! -newer <reference>
You might also try the -mtime/-atime/-ctime/-Btime family of options. I don't immediately remember how they work, but they might be useful in this situation.
Beware of deleting files from a find operation, especially one running as root; there are a whole bunch of ways an unprivileged, malicious process on the same system can trick it into deleting things you didn't want deleted. I strongly recommend you read the entire "Deleting Files" section of the GNU find manual.
If you only need files that are older than file "foo" and not foo itself, exclude the file by name using negation:
find . ! -newer foo ! -name foo
Please note, that the negation of newer means "older or same timestamp":
As you see in this example, the same file is also returned:
thomas#vm1112:/home/thomas/tmp/ touch test
thomas#vm1112:/home/thomas/tmp/ find ./ ! -newer test
./test
Unfortunately, find doesnt support this
! -newer doesnt mean older. It only means not newer, but it also matches files that have equal modification time. So I rather use
for f in path/files/etc/*; do
[ $f -ot reference_file ] && {
echo "$f is older"
# do something
}
done
find dir \! -newer fencefile -exec \
sh -c '
for f in "$#"; do
[ "$f" -ot fencefile ] && printf "%s\n" "$f"
done
' sh {} + \
;
I am working on a script to create files basically replica of some tables and ftp those files to the remote machine.there is one more requirement to delete the 3days old files on the remote machine before dumping these files.
I need help in writing the ksh for deleting 3 days old files on a remote machine using ftp
Normally, you would use:
find . -mtime +3 -exec rm {} ';'
or something similar (i.e., there may be other limiting clauses like -type f for regular files or -maxdepth 0 to do current directory only, no subdirectories). The -mtime +3 only gets those files whose modification date is 3 days age or more.
Execute man find on your system for full details. Whether Solaris has the same features as GNU find I don't know. It may be more limited (or better).
Update: Please, in the name of whatever gods you worship, please test the command first with echo instead of rm. I take no responsibility for the destruction of your files if you trust the advice of "some random guy on the net who may or may not have your best interests at heart" :-)
And, before anyone jumps in and berates me for not using xargs (or, better yet, find -print0 with xargs -0 where available), I know. But it's not relevant to the specific question at hand. The OP can ask another question if and when the performance of the find -exec is a problem.
If you have a specific file format with the date in it (as you indicate in your comment), you can actually use mdel under ftp. Consider the following script:
# The prefix and suffix of files to delete.
prefix='*_'
suffix='-i.tbl'
# Create FTP script file.
rm -rf temp.ftp
echo "user pax pax_password" >>temp.ftp
echo "cd /my/directory" >>temp.ftp
echo "prompt" >>temp.ftp
# Get current date.
y=$(date +%Y)
m=$(date +%m)
d=$(date +%d)
((lasty = y - 1))
((lastm = m - 1))
# If past Jan 3, delete all of previous year.
if [[ $m -gt 1 || $d -gt 3 ]] ; then
echo "mdel ${prefix}${lasty}????${suffix}" >>temp.ftp
fi
# If past Jan and past the third, delete all of previous month.
if [[ $m -gt 1 && $d -gt 3 ]] ; then
if [[ ${lastm} -lt 10 ]] ; then
echo "mdel ${prefix}${y}0${lastm}??${suffix}" >>temp.ftp
else
echo "mdel ${prefix}${y}${lastm}??${suffix}" >>temp.ftp
fi
fi
# If past the third, delete current month more than three days old.
if [[ $d -gt 3 ]] ; then
((d = d - 3))
if [[ ${m} -lt 10 ]] ; then
m="0${m}"
fi
while [[ ${d} -gt 0 ]] ; do
if [[ ${d} -lt 10 ]] ; then
echo "mdel ${prefix}${y}${m}0${d}${suffix}" >>temp.ftp
else
echo "mdel ${prefix}${y}${m}${d}${suffix}" >>temp.ftp
fi
((d = d - 1))
done
fi
# Finalise script and run it.
echo "bye" >>temp.ftp
ftp -n mymachine.com <temp.ftp
rm -rf temp.ftp
Other than a slight annoyance where you may have up to six days of files left there on month boundaries, this does what you need. You could of course make the code handling the month boundaries a little more intelligent if that's really important.
Just run this script on your box each day and it will clear out files on the target box by using standard ftp tooling. I still think it's easier to run a find on the server box but I'll present this option if that avenue is not available.
I'd rather not do this in PHP so I'm hoping a someone decent at shell scripting can help.
I need a script that runs through directory recursively and finds all files with last modified date is greater than some date. Then, it will tar and zip the file(s) keeping the path information.
as simple as:
find . -mtime -1 | xargs tar --no-recursion -czf myfile.tgz
where find . -mtime -1 will select all the files in (recursively) current directory modified day before. you can use fractions, for example:
find . -mtime -1.5 | xargs tar --no-recursion -czf myfile.tgz
If you have GNU find, then there are a legion of relevant options. The only snag is that the interface to them is less than stellar:
-mmin n (modification time in minutes)
-mtime n (modification time in days)
-newer file (modification time newer than modification time of file)
-daystart (adjust start time from current time to start of day)
Plus alternatives for access time and 'change' or 'create' time.
The hard part is determining the number of minutes since a time.
One option worth considering: use touch to create a file with the required modification time stamp; then use find with -newer.
touch -t 200901031231.43 /tmp/wotsit
find . -newer /tmp/wotsit -print
rm -f /tmp/wotsit
This looks for files newer than 2009-01-03T12:31:43. Clearly, in a script, /tmp/wotsit would be a name with the PID or other value to make it unique; and there'd be a trap to ensure it gets removed even if the user interrupts, and so on and so forth.
You can do this directly with tar and even better:
tar -N '2014-02-01 18:00:00' -jcvf archive.tar.bz2 files
This instructs tar to compress files newer than 1st of January 2014, 18:00:00.
This will work for some number of files. You want to include "-print0" and "xargs -0" in case any of the paths have spaces in them. This example looks for files modified in the last 7 days. To find those modified before the last 7 days, use "+7".
find . -mtime -7 -print0 | xargs -0 tar -cjf /foo/archive.tar.bz2
As this page warns, xargs can cause the tar command to be executed multiple times if there are a lot of arguments, and the "-c" flag could cause problems. In that case, you would want this:
find . -mtime -7 -print0 | xargs -0 tar -rf /foo/archive.tar
You can't update a zipped tar archive with tar, so you would have to bzip2 or gzip it in a second step.
This should show all files modified within the last 7 days.
find . -type f -mtime -7 -print
Pipe that into tar/zip, and you should be good.
I would simply do the following to backup all new files from 7 days ago
tar --newer $(date -d'7 days ago' +"%d-%b") -zcf thisweek.tgz .
note you can also replace '7 days ago' with anything that suits your need
Can be : date -d'yesterday' +"%d-%b"
Or even : date -d'first Sunday last month' +"%d-%b"
well under linux try reading man page of the find command
man find
something like this should
find . -type f -mtime -7 -print -exec cat {} \; | tar cf - | gzip -9
and you have it
You can get a list of files last modified later than x days ago with:
find . -mtime -x
Then you just have to tar and zip files in the resulting list, e.g.:
tar czvf mytarfile.tgz `find . -mtime -30`
for all files modified during last month.
This script will find files having a modification date of two minutes before and after the given date (and you can change the values in the conditions as per your requirement)
PATH_SRC="/home/celvas/Documents/Imp_Task/"
PATH_DST="/home/celvas/Downloads/zeeshan/"
cd $PATH_SRC
TODAY=$(date -d "$(date +%F)" +%s)
TODAY_TIME=$(date -d "$(date +%T)" +%s)
for f in `ls`;
do
# echo "File -> $f"
MOD_DATE=$(stat -c %y "$f")
MOD_DATE=${MOD_DATE% *}
# echo MOD_DATE: $MOD_DATE
MOD_DATE1=$(date -d "$MOD_DATE" +%s)
# echo MOD_DATE: $MOD_DATE
DIFF_IN_DATE=$[ $MOD_DATE1 - $TODAY ]
DIFF_IN_DATE1=$[ $MOD_DATE1 - $TODAY_TIME ]
#echo DIFF: $DIFF_IN_DATE
#echo DIFF1: $DIFF_IN_DATE1
if [[ ($DIFF_IN_DATE -ge -120) && ($DIFF_IN_DATE1 -le 120) && (DIFF_IN_DATE1 -ge -120) ]]
then
echo File lies in Next Hour = $f
echo MOD_DATE: $MOD_DATE
#mv $PATH_SRC/$f $PATH_DST/$f
fi
done
For example you want files having modification date before the given date only, you may change 120 to 0 in $DIFF_IN_DATE parameter discarding the conditions of $DIFF_IN_DATE1 parameter.
Similarly if you want files having modification date 1 hour before and after given date,
just replace 120 by 3600 in if CONDITION.
Let's say I have two directory structures:
/var/www/site1.prod
/var/www/site1.test
I want to use find(1) to see the files that are newer in /var/www/site1.test than their counterparts in /var/www/site1.prod.
Can I do this with find(1), and if so, how?
You also could use rsync -n
rsync -av -n /var/www/site1.test /var/www/site1.prod
should do it.
Using find,
cd /var/www/site1.test
find . -type f -print | while read file ; do
[ "$file" -nt /var/www/site1.prod/"$file" ] && echo "File '$file' changed"
done
This will work with filenames containing blanks, as well as work for a large volume of files.
To distinguish between modified and missing files, as per Eddie's comments,
cd /var/www/site1.test
find . -type f -print | while read file ; do
[ "$file" -nt /var/www/site1.prod/"$file" ] && reason=changed
[ \! -e /var/www/site1.prod/"$file" ] && reason=created
[ -n "$reason" ] && echo "$file $reason"
done
I think you can't do it with find alone, but you can do something like
$ cd /var/www/site1.test
$ files=`find .`
$ for $f in $files; do
if [ $f -nt /var/www/site1.prod/$f ]; then echo "$f changed!"; fi;
done
If you look at the -fprintf option you should be able to create two finds that output into two files that list the files name and modification time. You should then be able to just diff the two files to see what has changed.
I understand that you specifically asked for newer, but I think it's always good to know all your options.
I would use diff -ur dir1 dir2.
Maybe the timestamp changed, maybe it didn't, but that doesn't necessarily tell you if the contents are the same. Diff will tell you if the contents changed. If you really don't want to see the contents use diff -rq to just see the files that changed.