I'd rather not do this in PHP so I'm hoping a someone decent at shell scripting can help.
I need a script that runs through directory recursively and finds all files with last modified date is greater than some date. Then, it will tar and zip the file(s) keeping the path information.
as simple as:
find . -mtime -1 | xargs tar --no-recursion -czf myfile.tgz
where find . -mtime -1 will select all the files in (recursively) current directory modified day before. you can use fractions, for example:
find . -mtime -1.5 | xargs tar --no-recursion -czf myfile.tgz
If you have GNU find, then there are a legion of relevant options. The only snag is that the interface to them is less than stellar:
-mmin n (modification time in minutes)
-mtime n (modification time in days)
-newer file (modification time newer than modification time of file)
-daystart (adjust start time from current time to start of day)
Plus alternatives for access time and 'change' or 'create' time.
The hard part is determining the number of minutes since a time.
One option worth considering: use touch to create a file with the required modification time stamp; then use find with -newer.
touch -t 200901031231.43 /tmp/wotsit
find . -newer /tmp/wotsit -print
rm -f /tmp/wotsit
This looks for files newer than 2009-01-03T12:31:43. Clearly, in a script, /tmp/wotsit would be a name with the PID or other value to make it unique; and there'd be a trap to ensure it gets removed even if the user interrupts, and so on and so forth.
You can do this directly with tar and even better:
tar -N '2014-02-01 18:00:00' -jcvf archive.tar.bz2 files
This instructs tar to compress files newer than 1st of January 2014, 18:00:00.
This will work for some number of files. You want to include "-print0" and "xargs -0" in case any of the paths have spaces in them. This example looks for files modified in the last 7 days. To find those modified before the last 7 days, use "+7".
find . -mtime -7 -print0 | xargs -0 tar -cjf /foo/archive.tar.bz2
As this page warns, xargs can cause the tar command to be executed multiple times if there are a lot of arguments, and the "-c" flag could cause problems. In that case, you would want this:
find . -mtime -7 -print0 | xargs -0 tar -rf /foo/archive.tar
You can't update a zipped tar archive with tar, so you would have to bzip2 or gzip it in a second step.
This should show all files modified within the last 7 days.
find . -type f -mtime -7 -print
Pipe that into tar/zip, and you should be good.
I would simply do the following to backup all new files from 7 days ago
tar --newer $(date -d'7 days ago' +"%d-%b") -zcf thisweek.tgz .
note you can also replace '7 days ago' with anything that suits your need
Can be : date -d'yesterday' +"%d-%b"
Or even : date -d'first Sunday last month' +"%d-%b"
well under linux try reading man page of the find command
man find
something like this should
find . -type f -mtime -7 -print -exec cat {} \; | tar cf - | gzip -9
and you have it
You can get a list of files last modified later than x days ago with:
find . -mtime -x
Then you just have to tar and zip files in the resulting list, e.g.:
tar czvf mytarfile.tgz `find . -mtime -30`
for all files modified during last month.
This script will find files having a modification date of two minutes before and after the given date (and you can change the values in the conditions as per your requirement)
PATH_SRC="/home/celvas/Documents/Imp_Task/"
PATH_DST="/home/celvas/Downloads/zeeshan/"
cd $PATH_SRC
TODAY=$(date -d "$(date +%F)" +%s)
TODAY_TIME=$(date -d "$(date +%T)" +%s)
for f in `ls`;
do
# echo "File -> $f"
MOD_DATE=$(stat -c %y "$f")
MOD_DATE=${MOD_DATE% *}
# echo MOD_DATE: $MOD_DATE
MOD_DATE1=$(date -d "$MOD_DATE" +%s)
# echo MOD_DATE: $MOD_DATE
DIFF_IN_DATE=$[ $MOD_DATE1 - $TODAY ]
DIFF_IN_DATE1=$[ $MOD_DATE1 - $TODAY_TIME ]
#echo DIFF: $DIFF_IN_DATE
#echo DIFF1: $DIFF_IN_DATE1
if [[ ($DIFF_IN_DATE -ge -120) && ($DIFF_IN_DATE1 -le 120) && (DIFF_IN_DATE1 -ge -120) ]]
then
echo File lies in Next Hour = $f
echo MOD_DATE: $MOD_DATE
#mv $PATH_SRC/$f $PATH_DST/$f
fi
done
For example you want files having modification date before the given date only, you may change 120 to 0 in $DIFF_IN_DATE parameter discarding the conditions of $DIFF_IN_DATE1 parameter.
Similarly if you want files having modification date 1 hour before and after given date,
just replace 120 by 3600 in if CONDITION.
Related
I want to delete the graphite's storage whisper's data but there ain't anything in the graphite docs.
One way I did is deleting the the files at /opt/graphite...../whispers/stats... manually.
But this is tedious, so how do I do it?
Currently, deleting files from /opt/graphite/storage/whisper/ is the correct way to clean up whisper data.
As for the tedious side of the process, you could use the find command if there is a certain pattern that your trying to remove.
find /opt/graphite/storage/whisper -name loadavg.wsp -delete
Similar Question on answers.launchpad.net/graphite
I suppose that this is going into Server Fault territory, but I added
the following cron job to delete old metrics of ours that haven't been
written to for over 30 days (e.g. of cloud instances that have been
disposed):
find /mnt/graphite/storage -mtime +30 | grep -E \
"/mnt/graphite/storage/whisper/collectd/app_name/[^/]*" -o \
| uniq | xargs rm -rf
This will delete directories which have valid data.
First:
find whisperDir -mtime +30 -type f | xargs rm
And then delete empty dirs
find . -type d -empty | xargs rmdir
This last step should be repeated, because may be new empty directories will be left.
As people have pointed out, removing the files is the way to go. Expanding on previous answers, I made this script that removes any file that has exceeded its max retention age. Run it as a cronjob fairly regularly.
#!/bin/bash
d=$1
now=$(date +%s)
MINRET=86400
if [ -z "$d" ]; then
echo "Must specify a directory to clean" >&2
exit 1
fi
find $d -name '*.wsp' | while read w; do
age=$((now - $(stat -c '%Y' "$w")))
if [ $age -gt $MINRET ]; then
retention=$(whisper-info.py $w maxRetention)
if [ $age -gt $retention ]; then
echo "Removing $w ($age > $retention)"
rm $w
fi
fi
done
find $d -empty -type d -delete
A couple of bits to be aware of - the whisper-info call is quite heavyweight. To reduce the number of calls to it I've put the MINRET constant in, so that no file will be considered for deletion until it is 1 day old (24*60*60 seconds) - adjust to fit your needs. There are probably other things that can be done to shard the job or generally improve its efficiency, but I haven't had need to as yet.
How can I delete from directory2 and directory3 some files in the same time using the find command?
Some files of directory1 valides the same characteristics of the files in the 2 others directory and directories 1,2,3 figures in the directory0.
I suppose you meant:
(cd directory1 | find -iname 'somename' -print0) |
tee >(cd directory2 && xargs -0 rm -fvi) |
tee >(cd directory3 && xargs -0 rm -fvi)
ad libitum? But I'll make the answer more specific once you do your question
To find (From the man examples)
find / -newerct '1 minute ago' -print
Print out a list of all the files whose inode change
time is more recent than the current time minus one minute
To remove
find / -newerct '1 minute ago' -print | xargs rm
Well,I m trying to list the number of files created today and the count the number of lines in those files.I have to do it in unix.Please suggest how to write script for this.
To find the number of lines:
find / -type f -ctime 0 -mtime 0 -print0 | xargs -0 wc -l
This is almost what you want. There is no file created time in Unix, this is approximation with both file status changed time and file modified time.
If you would like to search only in certain directory, replace / with /path/to/your/dir.
To find the number of files:
find / -type f -ctime 0 -mtime 0 | wc -l
This will find files (-type f) in /path modified in the last 24 hours (-mtime -1 means modified in the last 1 day) and run wc -l to count the number of lines. {} is a placeholder for the file names and + means pass all the file names to a single invocation of wc.
find /path -mtime -1 -type f -exec wc -l {} +
Note that -ctime as suggested in other answers is change time, not creation time. It is the last time a file's owner, group, link count, mode, etc., was changed. Unix does not track the creation time of a file.
find . -maxdepth 1 -daystart -ctime 0 -type f | xargs wc -l
You'll need to change the maxdepth argument value if you need to look deeper.
To count the number of files changed today:
find . -daystart -type f -ctime -1 | wc -l
find finds all the files (-type f) in the current directory (.) created* (-ctime) more recently (-) than one (1) day since the start of this day (-daystart). wc counts the number of lines (-l) in find's output.
To count the lines in those files:
find -daystart -type f -ctime -1 -print0 | wc -l --files0-from=-
The first part is the same, except that find separates the filenames using nulls (-print0). wc counts the lines (-l) in the null-separated files (--files0-from=) on its standard input (-).
* ctime is not actually the creation time, but the time when the file's status was last changed. I don't think the filesystem holds on to the actual creation time.
Determining when a file was created reliably is hard. The mtime is when it was modified, the ctime is when the inode data was changed (change of permissions, for example), the atime is when the file data was last accessed. Usually, the mtime is surrogate for the create time; when a file is created, it records the creation time (as does the ctime and atime), but if the file is subsequently modified, the mtime records the time when the contents of the file was last modified.
find . -mtime -1 -print0 | xargs -0 wc -l
Find all the files under the current directory with a modification time less than 24 hours old and send the names to 'wc -l' - allowing for spaces and other odd characters in the file names.
I am working on a script to create files basically replica of some tables and ftp those files to the remote machine.there is one more requirement to delete the 3days old files on the remote machine before dumping these files.
I need help in writing the ksh for deleting 3 days old files on a remote machine using ftp
Normally, you would use:
find . -mtime +3 -exec rm {} ';'
or something similar (i.e., there may be other limiting clauses like -type f for regular files or -maxdepth 0 to do current directory only, no subdirectories). The -mtime +3 only gets those files whose modification date is 3 days age or more.
Execute man find on your system for full details. Whether Solaris has the same features as GNU find I don't know. It may be more limited (or better).
Update: Please, in the name of whatever gods you worship, please test the command first with echo instead of rm. I take no responsibility for the destruction of your files if you trust the advice of "some random guy on the net who may or may not have your best interests at heart" :-)
And, before anyone jumps in and berates me for not using xargs (or, better yet, find -print0 with xargs -0 where available), I know. But it's not relevant to the specific question at hand. The OP can ask another question if and when the performance of the find -exec is a problem.
If you have a specific file format with the date in it (as you indicate in your comment), you can actually use mdel under ftp. Consider the following script:
# The prefix and suffix of files to delete.
prefix='*_'
suffix='-i.tbl'
# Create FTP script file.
rm -rf temp.ftp
echo "user pax pax_password" >>temp.ftp
echo "cd /my/directory" >>temp.ftp
echo "prompt" >>temp.ftp
# Get current date.
y=$(date +%Y)
m=$(date +%m)
d=$(date +%d)
((lasty = y - 1))
((lastm = m - 1))
# If past Jan 3, delete all of previous year.
if [[ $m -gt 1 || $d -gt 3 ]] ; then
echo "mdel ${prefix}${lasty}????${suffix}" >>temp.ftp
fi
# If past Jan and past the third, delete all of previous month.
if [[ $m -gt 1 && $d -gt 3 ]] ; then
if [[ ${lastm} -lt 10 ]] ; then
echo "mdel ${prefix}${y}0${lastm}??${suffix}" >>temp.ftp
else
echo "mdel ${prefix}${y}${lastm}??${suffix}" >>temp.ftp
fi
fi
# If past the third, delete current month more than three days old.
if [[ $d -gt 3 ]] ; then
((d = d - 3))
if [[ ${m} -lt 10 ]] ; then
m="0${m}"
fi
while [[ ${d} -gt 0 ]] ; do
if [[ ${d} -lt 10 ]] ; then
echo "mdel ${prefix}${y}${m}0${d}${suffix}" >>temp.ftp
else
echo "mdel ${prefix}${y}${m}${d}${suffix}" >>temp.ftp
fi
((d = d - 1))
done
fi
# Finalise script and run it.
echo "bye" >>temp.ftp
ftp -n mymachine.com <temp.ftp
rm -rf temp.ftp
Other than a slight annoyance where you may have up to six days of files left there on month boundaries, this does what you need. You could of course make the code handling the month boundaries a little more intelligent if that's really important.
Just run this script on your box each day and it will clear out files on the target box by using standard ftp tooling. I still think it's easier to run a find on the server box but I'll present this option if that avenue is not available.
In my unix shell script, I have a for loop and looping thru the files in a directory, I want to check the modified date of a file is greater than some number of days, say 10 days and do some processing.
Any help in this regards is highly appreciated.
The find command has a built-in test for this:
find <yourdir> -type f -mtime +10 -exec yourproc {} \;
This will run "yourproc matching_filename" on each file in <yourdir> older than 10 days.
The -ctime test works similarly, using the inode change time rather than the
modification time.
find . -mtime +10
This could be integrated into a script in several ways. For one:
isOlder () {
[ "`find $1 -mtime +10`" == "$1" ]
return $?
}
ls | while read fname; do
if isOlder "$fname"; then
echo $fname is older
else
echo $fname is not older
fi
done
I listed the above because it sounded like you mainly wanted a test and were doing other processing. OTOH, if you really just want the older files:
$ find . -mtime +10 | while read fname; do
> echo $fname
> done