Creating directories from an input file - unix

I have below mentioned directories in an input file and i need them to create in a loop.
data/app_rt_ws/Request/2017_06_27
data/app_rt_ws/Response/2017_06_19
data/app_rt_ws/RTWS
data/app_rt_ws/SDP
data/edge/response/9-20-2016
data/edge/response/9-22-2016
Problem is that i don't need the directories in the yyyy_mm_dd or dd-mm-yyyy format which get created at run time on the server. I need them to be discarded and have the rest of the static path of the directories created.
I am using below mentioned code but can't seem to figure out how to omit the above mentioned part
for i in `cat /tmp/inputfile.txt`
do
echo $i
cd /opt/app/app
awk '/date_year/{print $1}' (need to filter out the entries with date)
mkdir -p $i ( need to create rest of the directories)
done

You may modify your script as followed,
for i in `awk '$0 !~ (/[0-9]{1,2}-[0-9]{1,2}-20[0-9]{2}/ && /20[0-9]{2}_[0-9]{1,2}_[0-9]{1,2}/){print}' inputfile.txt`;
do
echo $i
cd /opt/app/app
mkdir -p $i
done
And the output of the awk command is like this,
$ awk '$0 !~ (/[0-9]{1,2}-[0-9]{1,2}-20[0-9]{2}/ && /20[0-9]{2}_[0-9]{1,2}_[0-9]{1,2}/){print}' inputfile.txt
data/app_rt_ws/RTWS
data/app_rt_ws/SDP

With bash for regexp matching:
$ cat tst.sh
while IFS= read -r path; do
if [[ $path =~ /([0-9]{4}(_[0-9]{2}){2}|([0-9]{1,2}-){2}[0-9]{4})$ ]]; then
path="${path%/*}"
fi
echo mkdir -p "$path"
done < "$1"
$ ./tst.sh file
mkdir -p data/app_rt_ws/Request
mkdir -p data/app_rt_ws/Response
mkdir -p data/app_rt_ws/RTWS
mkdir -p data/app_rt_ws/SDP
mkdir -p data/edge/response
mkdir -p data/edge/response
Remove the echo once you've tested and are happy with the result. Check the regexp - you said you wanted to match dd-mm-yyyy but then your input contained dates as d-mm-yyyy so idk what you really wanted and so I guessed you'd be happy with 1 or 2 digits, hence [0-9]{1,2}.
With other shells use a case statement or similar to match the date at the end of line as a globbing pattern. You do NOT need to call an external tool to do this check.
Using:
for i in `some command`
is always to be avoided because it breaks when the output of some command contains spaces and it's using deprecated backticks instead of modern notation $(some command), and doing:
echo $i
is always to be avoided because you should ALWAYS quote your shell variables (echo "$i") unless you have a specific reason not to so as to avoid accidental word splitting and file name generation.

You could use the bash string substitutions to get rid of the date.
for i in `cat /tmp/inputfile.txt`
do
echo $i
cd /opt/app/app
if [[ $i =~ [0-9] ]]; then
mkdir -p ${i%/*}
else
mkdir -p $i
fi
done
The substitution cuts off everything after the last / so the date is gone.
However if you just want all the ones without numbers then you could do:
for i in `cat /tmp/inputfile.txt`
do
echo $i
cd /opt/app/app
if [[ $i =~ [0-9] ]]; then
:
else
mkdir -p $i
fi
done
Output of the second version:
data/app_rt_ws/RTWS
data/app_rt_ws/SDP
I hope this is what you looked for :)

Related

Issues with iconv command in script

I am trying to create a script which detects if files in a directory have not UTF-8 characters and if they do, grab the file type of that particular file and perform the iconv operation on it.
The code is follows
find <directory> |sed '1d'><directory>/filelist.txt
while read filename
do
file_nm=${filename%%.*}
ext=${filename#*.}
echo $filename
q=`grep -axv '.*' $filename|wc -l`
echo $q
r=`file -i $filename|cut -d '=' -f 2`
echo $r
#file_repair=$file_nm
if [ $q -gt 0 ]; then
iconv -f $r -t utf-8 -c ${file_nm}.${ext} >${file_nm}_repaired.${ext}
mv ${file_nm}_repaired.${ext} ${file_nm}.${ext}
fi
done< <directory>/filelist.txt
While running the code, there are several files that turn into 0 byte files and .bak gets appended to the file name.
ls| grep 'bak' | wc -l
36
Where am I making a mistake?
Thanks for the help.
It's really not clear what some parts of your script are supposed to do.
Probably the error is that you are assuming file -i will output a string which always contains =; but it often doesn't.
find <directory> |
# avoid temporary file
sed '1d' |
# use IFS='' read -r
while IFS='' read -r filename
do
# indent loop body
file_nm=${filename%%.*}
ext=${filename#*.}
# quote variables, print diagnostics to stderr
echo "$filename" >&2
# use grep -q instead of useless wc -l; don't enter condition needlessly; quote variable
if grep -qaxv '.*' "$filename"; then
# indent condition body
# use modern command substitution syntax, quote variable
# check if result contains =
r=$(file -i "$filename")
case $r in
*=*)
# only perform decoding if we can establish encoding
echo "$r" >&2
iconv -f "${r#*=}" -t utf-8 -c "${file_nm}.${ext}" >"${file_nm}_repaired.${ext}"
mv "${file_nm}_repaired.${ext}" "${file_nm}.${ext}" ;;
*)
echo "$r: could not establish encoding" >&2 ;;
esac
fi
done
See also Why is testing “$?” to see if a command succeeded or not, an anti-pattern? (tangential, but probably worth reading) and useless use of wc
The grep regex is kind of mysterious. I'm guessing you want to check if the file contains non-empty lines? grep -qa . "$filename" would do that.

dynamically pass string to Rscript argument with sed

I wrote a script in R that has several arguments. I want to iterate over 20 directories and execute my script on each while passing in a substring from the file path as my -n argument using sed. I ran the following:
find . -name 'xray_data' -exec sh -c 'Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
which results in this error:
ubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
sh: command substitution: line 0: syntax error near unexpected token `('
sh: command substitution: line 0: `sed -e s/.*DeMMO.*[/](.*)_.*[/]xray_data/1/ "./DeMMO1/D1T3rep_Dec2019_Ellison/xray_data"'
When I try to use sed with my pattern on an example file path, it works:
echo "./DeMMO1/D1T1exp_Dec2019_Poorman/xray_data" | sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/'
which produces the correct substring:
D1T1exp_Dec2019
I think there's an issue with trying to use single quotes inside the interpreted string but I don't know how to deal with this. I have tried replacing the single quotes around the sed pattern with double quotes as well as removing the single quotes, both result in this error:
sed: RE error: illegal byte sequence
How should I extract the substring from the file path dynamically in this case?
To loop through the output of find.
while IFS= read -ru "$fd" -d '' files; do
echo "$files" ##: do whatever you want to do with the files here.
done {fd}< <(find . -type f -name 'xray_data' -print0)
No embedded commands in quotes.
It uses a random fd just in case something inside the loop is eating/slurping stdin
Also -print0 delimits the files with null bytes, so it should be safe enough to handle spaces tabs and newlines on the path and file names.
A good start is always put an echo in front of every commands you want to do with the files, so you have an idea what's going to be executed/happen just in case...
This is the solution that ultimately worked for me due to issues with quotes in sed:
for dir in `find . -name 'xray_data'`;
do sampleID="`basename $(dirname $dir) | cut -f1 -d'_'`";
Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f "$dir" -b "$dir/SEM_images" -c "$dir/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "$sampleID";
done

Bash Shell - Print names of all self-referential files

The idea is the write a bash script that prints the names of files in the current directory that contain their own name in their content.
e.g if a file called hello contains the string hello, we print hello, and we do this for all files in the current directory
Here's what I wrote, and I have no idea why it doesn't work.
#!/bin/bash
for file in *
do
if (cat $file | grep $file) 2> /dev/null
then
echo $file
fi
done
Fixed:
#!/bin/bash
for file in *
do
if grep $file $file 2> /dev/null
then
echo $file
fi
done
Apart from quoting issues, potential regex escaping issues, and the useless use of cat and (...), your code should work in principle.
Try this version - if it doesn't work, something else must be going on:
#!/bin/bash
for file in *
do
if grep -qF "$file" "$file" 2> /dev/null
then
echo "$file"
fi
done
-q makes grep not output matching lines (whether a match was found is implied by the exit code).
-F ensures that the search term is treated as a literal (rather than a regex).

Download and insert salt string inside wordpress wp-config.php with Bash

How can I insert the content of the variable $SALT in a specific point (line or string) of a file like wp-contet.php from wordpress using Bash script?
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
I'm not an expert at parsing text files in bash but you should delete the lines that define the things you're downloading from the wordpress salt and then insert the variable at the end... something like:
#!/bin/sh
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
STRING='put your unique phrase here'
printf '%s\n' "g/$STRING/d" a "$SALT" . w | ed -s wp-config.php
OK, now it's fixed... it should look for where the salt is supposed to go and it will replace it with the info retrieved from https://api.wordpress.org/secret-key/1.1/salt/
This version defines new keys if none exist, and also replaces existing keys:
#!/bin/bash
find . -name wp-config.php -print | while read line
do
curl http://api.wordpress.org/secret-key/1.1/salt/ > wp_keys.txt
sed -i.bak -e '/put your unique phrase here/d' -e \
'/AUTH_KEY/d' -e '/SECURE_AUTH_KEY/d' -e '/LOGGED_IN_KEY/d' -e '/NONCE_KEY/d' -e \
'/AUTH_SALT/d' -e '/SECURE_AUTH_SALT/d' -e '/LOGGED_IN_SALT/d' -e '/NONCE_SALT/d' $line
cat wp_keys.txt >> $line
rm wp_keys.txt
done
If you have csplit available, you can split the original wp-config.php file either side of the salt definitions, download new salts, then cat back together. This keeps the PHP define() statements at the same location in wp-config.php instead of than moving them to a different location within the file:
# Download new salts
curl "https://api.wordpress.org/secret-key/1.1/salt/" -o salts
# Split wp-config.php into 3 on the first and last definition statements
csplit wp-config.php '/AUTH_KEY/' '/NONCE_SALT/+1'
# Recombine the first part, the new salts and the last part
cat xx00 salts xx02 > wp-config.php
# Tidy up
rm salts xx00 xx01 xx02
How about using sed?
cat wp-config.php | sed 's/old_string/new_string/g' > wp-config.php
I think I got this one! its a bash script using only commands normally available at the command prompt and it does -everything- (assuming httpd is your web user) except create the databases. here you go.
#!/bin/bash
# wordpress latest auto-install script, by alienation 24 jan 2013. run as root.
# usage: ~/wp-install alien /hsphere/local/home/alien/nettrip.org alien_wpdbname alien_wpdbusername p#sSw0rd
# ( wp-install shell-user folder db-name db-user-name db-user-pw )
# download wordpress to temporary area
cd /tmp
rm -rf tmpwp
mkdir tmpwp
cd tmpwp
wget http://wordpress.org/latest.tar.gz
tar -xvzpf latest.tar.gz
# copy wordpress to where it will live, and go there, removing index placeholder if there is one
mv wordpress/* $2
cd $2
rm index.html
# create config from sample, replacing salt example lines with a real salt from online generator
grep -A 1 -B 50 'since 2.6.0' wp-config-sample.php > wp-config.php
wget -O - https://api.wordpress.org/secret-key/1.1/salt/ >> wp-config.php
grep -A 50 -B 3 'Table prefix' wp-config-sample.php >> wp-config.php
# put the appropriate db info in place of placeholders in our new config file
replace 'database_name_here' $3 -- wp-config.php
replace 'username_here' $4 -- wp-config.php
replace 'password_here' $5 -- wp-config.php
# change file ownership and permissions according to ideal at http://codex.wordpress.org/Hardening_WordPress#File_Permissions
touch .htaccess
chown $1:httpd .htaccess
chown -R $1:httpd *
find . -type d -exec chmod 755 {} \;
find . -type f -exec chmod 644 {} \;
chmod -R 770 wp-content
chmod -R g-w wp-admin wp-includes wp-content/plugins
chmod g+w .htaccess
# thats it!
echo ALL DONE
I built a simple CLI for just that. Try it out. It's called [WP-Salts-Update-CLI][1].
WP-Salts-Update-CLI
WPSUCLI downloads new salts from the WP API and replaces them with the ones in your wp-config.php file for every site on your server.
⚡️ Installation
Open command line terminal (I prefer iTerm2) and run the following command.
bash
sudo wget -qO wpsucli https://git.io/vykgu && sudo chmod +x ./wpsucli && sudo install ./wpsucli /usr/local/bin/wpsucli
This command will perform the following actions:
Use sudo permissions
Use wget to download WPSUCLI and rename it to wpsucli
Make the wpsucli executable
Install wpsucli inside /usr/local/bin/ folder.
🙌 Usage
Just run wpsucli and it will update the salts for every wp-config.php file on your server or PC.
This is the bash script that I came up with that works on my Ubuntu server. I modified the examples from above.
Its a bit of brute force in that it will only replace the 8 keys that currently are required and expects the server to return exactly the same length key every time. The script works well for my use case so I thought I would share it.
CONFIG_FILE=wp-config.php
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
SRC="define('AUTH_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'AUTH_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('SECURE_AUTH_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'SECURE_AUTH_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('LOGGED_IN_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'LOGGED_IN_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('NONCE_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'NONCE_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('AUTH_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'AUTH_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('SECURE_AUTH_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'SECURE_AUTH_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('LOGGED_IN_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'LOGGED_IN_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('NONCE_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'NONCE_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
I tried the accepted solution:
#!/bin/sh
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
STRING='put your unique phrase here'
printf '%s\n' "g/$STRING/d" a "$SALT" . w | ed -s wp-config.php
However it does not work perfectly as for some reason it induces the SALTS to "move down" 1 line in the wp-config.php file each time it is used... it is not ideal if you are going to change SALTS automatically like every week, months with cron for example...
A better solution for me was to create a little function that I call in my script.
This function creates a file with the SALTS (deletes it at the end), deletes every lines containing one of the SALTS then just inserts the SALTS contained in the file in place of the initial SALTS.
This works perfectly.
fct_update_salts() {
# Requires website name as target
curl http://api.wordpress.org/secret-key/1.1/salt/ > ~/SALTS.txt
var_initial_path1=`pwd`
cd ~ #going to home directory
# This scripts eliminates successively all SALT entries, replaces the last one by XXX as a marker, places SALTS.txt, below XXX and deletes XXX
sudo sed -i "/SECURE_AUTH_KEY/d" $1/wp-config.php
sudo sed -i "/LOGGED_IN_KEY/d" $1/wp-config.php
sudo sed -i "/NONCE_KEY/d" $1/wp-config.php
sudo sed -i "/AUTH_SALT/d" $1/wp-config.php
sudo sed -i "/SECURE_AUTH_SALT/d" $1/wp-config.php
sudo sed -i "/LOGGED_IN_SALT/d" $1/wp-config.php
sudo sed -i "/NONCE_SALT/d" $1/wp-config.php
sudo sed -i "/AUTH_KEY/cXXX" $1/wp-config.php
sudo sed -i '/XXX/r SALTS.txt' $1/wp-config.php
sudo sed -i "/XXX/d" $1/wp-config.php
echo "SALTS REPLACED BY:"
echo "====================="
cat ~/SALTS.txt
sudo rm -rf ~/SALTS.txt
cd $var_initial_path1
}
The function is to be called in the script like this:
# Reset SALTS
fct_update_salts $SITE_PATH
Where $SITE_PATH="/var/www/html/YOUR_WEBSITE" or whatever path works for you.
I was challenged with the same issue. Here is the script I wrote to replace the salts and keys from ones downloaded from WordPress. You can use it at any time to replace them if/when needed. I run it as sudo, and the script tests for that. If you use an account that can download to the directory and make updates to the wp-config.php file, then you can delete that part of the script.
#!/bin/sh
# update-WordPress-Salts: Updates WordPress Salts
# written by Wayne Woodward 2017
if [ $# -lt 1 ]; then
echo "Usage: update-WordPress-Salts directory"
exit
fi
if [ "$(whoami)" != "root" ]; then
echo "Please run as root (sudo)"
exit
fi
WPPATH=$1
# Update the salts in the config file
# Download salts from WordPress and save them locally
curl http://api.wordpress.org/secret-key/1.1/salt/ > /var/www/$WPPATH/wp-keys.txt
# Iterate through each "Saltname" and append 1 to it
# For a couple names that may match twice like "AUTH_KEY" adds extra 1s to the end
# But that is OK as when this deletes the lines, it uses the same matching pattern
# (Smarter people may fix this)
for SALTNAME in AUTH_KEY SECURE_AUTH_KEY LOGGED_IN_KEY NONCE_KEY AUTH_SALT SECURE_AUTH_SALT LOGGED_IN_SALT NONCE_SALT
do
sed -i -e "s/$SALTNAME/${SALTNAME}1/g" /var/www/$WPPATH/wp-config.php
done
# Find the line that has the updated AUTH_KEY1 name
# This is so we can insert the file in the same area
line=$(sed -n '/AUTH_KEY1/{=;q}' /var/www/$WPPATH/wp-config.php)
# Insert the file from the WordPress API that we saved into the configuration
sed -i -e "${line}r /var/www/$WPPATH/wp-keys.txt" /var/www/$WPPATH/wp-config.php
# Itererate through the old keys and remove them from the file
for SALTNAME in AUTH_KEY SECURE_AUTH_KEY LOGGED_IN_KEY NONCE_KEY AUTH_SALT SECURE_AUTH_SALT LOGGED_IN_SALT NONCE_SALT
do
sed -i -e "/${SALTNAME}1/d" /var/www/$WPPATH/wp-config.php
done
# Delete the file downloaded from Wordpress
rm /var/www/$WPPATH/wp-keys.txt
Many of the answers rely on the phrase 'put your unique phrase here' being present in the file, so they do not work when you want to change salts after the first time. There are also some that remove the old definitions and append the new ones at the end. While that does work, it's nice to keep the definitions where you would expect them, right after the comment documenting them. My solution addresses those issues.
I made a few attempts with sed, perl and regex, but there are special characters in the salts and the rest of the config file that tend to mess things up. I ended up using grep to search the document for the unique comment structure that opens and closes the salt definition block, which has the following format:
/**##+
<comment documentation>
*/
<salt definitions>
/**##-*/
Note that if that comment structure is removed or altered, this will no longer work. Here's the script:
#!/bin/bash -e
# Set Default Settings:
file='wp-config.php'
# set up temporary files with automatic removal:
trap "rm -f $file_start $file_end $salt" 0 1 2 3 15
file_start=$(mktemp) || exit 1
file_end=$(mktemp) || exit 1
salt=$(mktemp) || exit 1
function find_line {
# returns the first line number in the file which contains the text
# program exits if text is not found
# $1 : text to search for
# $2 : file in which to search
# $3 (optional) : line at which to start the search
line=$(tail -n +${3:-1} $2 | grep -nm 1 $1 | cut -f1 -d:)
[ -z "$line" ] && exit 1
echo $(($line + ${3:-1} - 1))
}
line=$(find_line "/**##+" "$file")
line=$(find_line "\*/" "$file" "$line")
head -n $line $file > $file_start
line=$(find_line "/**##-\*/" "$file" "$line")
tail -n +$line $file > $file_end
curl -Ls https://api.wordpress.org/secret-key/1.1/salt/ > $salt
(cat $file_start $salt; echo; cat $file_end) > $file
exit 0
Strings containing single asterisks, such as "*/" and "/**##-*/" want to expand to directory lists, so that is why those asterisks are escaped.
Here's a pure bash approach. This does not depend on wordpress.org.
I converted the original wp_generate_password() function used by WordPress to generate salt.
#!/bin/bash
set -e
#
# Generates a random password drawn from the defined set of characters.
# Inspired by WordPress function https://developer.wordpress.org/reference/functions/wp_generate_password/
#
# Parameters
# ----------
# $length
# (ing) (Optional) Length of password to generate.
# Default value: 12
# $special_chars
# (bool) (Optional) Whether to include standard special characters.
# Default value: true
# $extra_special_chars
# (bool) (Optional) Whether to include other special characters. Used when generating secret keys and salts.
# Default value: false
#
function wp_generate_password() {
# Args
length="$(test $1 && echo $1 || echo 12 )"
special_chars="$(test $2 && echo $2 || echo 1 )"
extra_special_chars="$(test $3 && echo $3 || echo 0 )"
chars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
[[ $special_chars != 0 ]] && chars="$chars"'!##$%^&*()'
[[ $extra_special_chars != 0 ]] && chars="$chars"'-_ []{}<>~`+=,.;:/?|'
password='';
for i in $(seq 1 $length); do
password="${password}${chars:$(( RANDOM % ${#chars} )):1}"
done
echo "$password"
}
You can then just run SALT="$(wp_generate_password 64 1 1)".
Update
I just published a standalone script to generate WP salt values. You can generate the salt values by running ./wp-generate-salt.sh.
If the wordpress.org API generated SALT values are not necessary for your use case, you can use the pwgen to generate keys on the server and insert those into wp-config.php.
for i in {1..8} ;do unique_key="`pwgen -1 -s 64`";sudo sed -i "0,/put your unique phrase here/s/put your unique phrase here/$unique_key/" /srv/www/wordpress/wp-config.php; done
You may need to fix the ownership of the file after using sudo. You can use a command similar to this for changing the ownership.
chown www-data:www-data /srv/www/wordpress/wp-config.php

Determining age of a file in shell script

G'day,
I need to see if a specific file is more than 58 minutes old from a sh shell script. I'm talking straight vanilla Solaris shell with some POSIX extensions it seems.
I've thought of doing a
touch -t YYYYMMDDHHmm.SS /var/tmp/toto
where the timestamp is 58 minutes ago and then doing a
find ./logs_dir \! -newer /var/tmp/toto -print
We need to postprocess some log files that have been retrieved from various servers using mirror. Waiting for the files to be stable is the way this team decides if the mirror is finished and hence that day's logs are now complete and ready for processing.
Any suggestions gratefully received.
cheers,
I needed something to test age of a specific file, to not re-download too often. So using GNU date and bash:
# if file's modtime hour is less than current hour:
[[ $(date +%k -r GPW/mstall.zip) -lt $(date +%k) ]] && \
wget -S -N \
http://bossa.pl/pub/metastock/mstock/mstall.zip \
Update--this version works much better for me, and is more accurate and understandable:
[[ $(date +%s -r mstall.zip) -lt $(date +%s --date="77 min ago") ]] && echo File is older than 1hr 17min
The BSD variant (tested on a Mac) is:
[[ $(stat -f "%m" mstall.zip) -lt $(date -j -v-77M +%s) ]] && echo File is older than 1hr 17min
You can use different units in the find command, for example:
find . -mtime +0h55m
Will return any files with modified dates older than 55 minutes ago.
This is now an old question, sorry, but for the sake of others searching for a good solution as I was...
The best method I can think of is to use the find(1) command which is the only Un*x command I know of that can directly test file age:
if [ "$(find $file -mmin +58)" != "" ]
then
... regenerate the file ...
fi
The other option is to use the stat(1) command to return the age of the file in seconds and the date command to return the time now in seconds. Combined with the bash shell math operator working out the age of the file becomes quite easy:
age=$(stat -c %Y $file)
now=$(date +"%s")
if (( (now - age) > (58 * 60) ))
then
... regenerate the file ...
fi
You could do the above without the two variables, but they make things clearer, as does use of bash math (which could also be replaced). I've used the find(1) method quite extensively in scripts over the years and recommend it unless you actually need to know age in seconds.
A piece of the puzzle might be using stat. You can pass -r or -s to get a parseable representation of all file metadata.
find . -print -exec stat -r '{}' \;
AFAICR, the 10th column will show the mtime.
Since you're looking to test the time of a specific file you can start by using test and comparing it to your specially created file:
test /path/to/file -nt /var/tmp/toto
or:
touch -t YYYYMMDDHHmm.SS /var/tmp/toto
if [/path/to/file -nt /var/tmp/toto]
...
You can use ls and awk to get what you need as well. Awk has a c-ish printf that will allow you to format the columns any way you want.
I tested this in bash on linux and ksh on solaris.
Fiddle with options to get the best values for your application. Especially "--full-time" in bash and "-E" in ksh.
bash
ls -l foo | awk '{printf "%3s %1s\n", $6, $7}'
2011-04-19 11:37
ls --full-time foo | awk '{printf "%3s %1s\n", $6, $7}'
2011-04-19 11:37:51.211982332
ksh
ls -l bar | awk '{printf "%3s %1s %s\n", $6, $7, $8}'
May 3 11:19
ls -E bar | awk '{printf "%3s %1s %s\n", $6, $7, $8}'
2011-05-03 11:19:23.723044000 -0400

Resources