I am trying to create a script which detects if files in a directory have not UTF-8 characters and if they do, grab the file type of that particular file and perform the iconv operation on it.
The code is follows
find <directory> |sed '1d'><directory>/filelist.txt
while read filename
do
file_nm=${filename%%.*}
ext=${filename#*.}
echo $filename
q=`grep -axv '.*' $filename|wc -l`
echo $q
r=`file -i $filename|cut -d '=' -f 2`
echo $r
#file_repair=$file_nm
if [ $q -gt 0 ]; then
iconv -f $r -t utf-8 -c ${file_nm}.${ext} >${file_nm}_repaired.${ext}
mv ${file_nm}_repaired.${ext} ${file_nm}.${ext}
fi
done< <directory>/filelist.txt
While running the code, there are several files that turn into 0 byte files and .bak gets appended to the file name.
ls| grep 'bak' | wc -l
36
Where am I making a mistake?
Thanks for the help.
It's really not clear what some parts of your script are supposed to do.
Probably the error is that you are assuming file -i will output a string which always contains =; but it often doesn't.
find <directory> |
# avoid temporary file
sed '1d' |
# use IFS='' read -r
while IFS='' read -r filename
do
# indent loop body
file_nm=${filename%%.*}
ext=${filename#*.}
# quote variables, print diagnostics to stderr
echo "$filename" >&2
# use grep -q instead of useless wc -l; don't enter condition needlessly; quote variable
if grep -qaxv '.*' "$filename"; then
# indent condition body
# use modern command substitution syntax, quote variable
# check if result contains =
r=$(file -i "$filename")
case $r in
*=*)
# only perform decoding if we can establish encoding
echo "$r" >&2
iconv -f "${r#*=}" -t utf-8 -c "${file_nm}.${ext}" >"${file_nm}_repaired.${ext}"
mv "${file_nm}_repaired.${ext}" "${file_nm}.${ext}" ;;
*)
echo "$r: could not establish encoding" >&2 ;;
esac
fi
done
See also Why is testing “$?” to see if a command succeeded or not, an anti-pattern? (tangential, but probably worth reading) and useless use of wc
The grep regex is kind of mysterious. I'm guessing you want to check if the file contains non-empty lines? grep -qa . "$filename" would do that.
I wrote a script in R that has several arguments. I want to iterate over 20 directories and execute my script on each while passing in a substring from the file path as my -n argument using sed. I ran the following:
find . -name 'xray_data' -exec sh -c 'Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
which results in this error:
ubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f {} -b "{}/SEM_images" -c "{}/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "`sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/' "{}"`"' sh {} \;
sh: command substitution: line 0: syntax error near unexpected token `('
sh: command substitution: line 0: `sed -e s/.*DeMMO.*[/](.*)_.*[/]xray_data/1/ "./DeMMO1/D1T3rep_Dec2019_Ellison/xray_data"'
When I try to use sed with my pattern on an example file path, it works:
echo "./DeMMO1/D1T1exp_Dec2019_Poorman/xray_data" | sed -e 's/.*DeMMO.*[/]\(.*\)_.*[/]xray_data/\1/'
which produces the correct substring:
D1T1exp_Dec2019
I think there's an issue with trying to use single quotes inside the interpreted string but I don't know how to deal with this. I have tried replacing the single quotes around the sed pattern with double quotes as well as removing the single quotes, both result in this error:
sed: RE error: illegal byte sequence
How should I extract the substring from the file path dynamically in this case?
To loop through the output of find.
while IFS= read -ru "$fd" -d '' files; do
echo "$files" ##: do whatever you want to do with the files here.
done {fd}< <(find . -type f -name 'xray_data' -print0)
No embedded commands in quotes.
It uses a random fd just in case something inside the loop is eating/slurping stdin
Also -print0 delimits the files with null bytes, so it should be safe enough to handle spaces tabs and newlines on the path and file names.
A good start is always put an echo in front of every commands you want to do with the files, so you have an idea what's going to be executed/happen just in case...
This is the solution that ultimately worked for me due to issues with quotes in sed:
for dir in `find . -name 'xray_data'`;
do sampleID="`basename $(dirname $dir) | cut -f1 -d'_'`";
Rscript /Users/Caitlin/Desktop/DeMMO_Pubs/DeMMO_NativeRock/DeMMO_NativeRock/R/scipts/dataStitchR.R -f "$dir" -b "$dir/SEM_images" -c "$dir/../coordinates.txt" -z ".tif" -m ".tif" -a "Unknown|SEM|Os" -d "overview" -y "overview" --overview "overview.*tif" -p FALSE -n "$sampleID";
done
The idea is the write a bash script that prints the names of files in the current directory that contain their own name in their content.
e.g if a file called hello contains the string hello, we print hello, and we do this for all files in the current directory
Here's what I wrote, and I have no idea why it doesn't work.
#!/bin/bash
for file in *
do
if (cat $file | grep $file) 2> /dev/null
then
echo $file
fi
done
Fixed:
#!/bin/bash
for file in *
do
if grep $file $file 2> /dev/null
then
echo $file
fi
done
Apart from quoting issues, potential regex escaping issues, and the useless use of cat and (...), your code should work in principle.
Try this version - if it doesn't work, something else must be going on:
#!/bin/bash
for file in *
do
if grep -qF "$file" "$file" 2> /dev/null
then
echo "$file"
fi
done
-q makes grep not output matching lines (whether a match was found is implied by the exit code).
-F ensures that the search term is treated as a literal (rather than a regex).
My requirement is to attach all the .csv files in a folder and send them in a single mail.
Here is what have tried,
mutt -s "subject" -a *.csv -- abc#gmail.com < subject.txt
The above command is not working (It's not recognizing multiple files) and throwing the error
Error sending message, child exited 67 (User unknown.).
Could not send the message.
Then I tried using multiple -a option as follows,
mutt -s "subject" -a aaa.csv -a bbb.csv -- abc#gmail.com < subject.txt
This works as expected.
But this is not feasible for 100 files for example. I should be able use it with file mask (as like *.csv to take all csv files). Is there is any way we can use like *.csv in single command?
Thanks
Mutt doesn't support such syntax, but it doesn't mean it's impossible. You just have to build the mutt command.
mutt -s "subject" $( printf -- '-a %q ' *.csv ) ...
The command in $( ... ) produces something like this:
-a aaa.csv -a bbb.csv -a ...
Here is the example of sending multiple files using a single command -
mutt -s "Subject" -i "Mail_body text" email_id#abc.com -c email_cc_id#abc.com -a attachment1.pdf -a attachment2.pdf
At the end of the command line use -a for the attachment .
Some linux system have attachment size limit . Mostly it support less size .
I'm getting backslash( \ ) Additionally
Daily_Batch_Status{20131003}.PDF
Daily_System_Monitoring{20131003}.PDF
printf -- '-a %q ' *.PDF
-a Daily_Batch_Status \ {20131003 \ }.PDF -a Daily_System_Monitoring \ {20131003 \ }.PDF
#!/bin/bash
from="me#address.com"
to="target#address.com"
subject="pdfs $(date +%B) $(date +%Y)"
body="You can find the pdfs from $(date +%B) $(date +%Y)"
# here comes the attachments
mutt -s "$subject" $( printf -- ' -a %q' $PWD/*.pdf ) -- $to <<EOF
Dear Mr and Ms,
$(echo $body)
$(cat ~/.signature)
EOF
but it does not work with escape characters in file name like "\[5\]" which can come in MacOs.
I created as a script and collect needed PDFs in a folder and just run the script from that location. So monthly reports are sent... it does not matter how many pdfs (number can vary) but also there should be no white space.
How can I insert the content of the variable $SALT in a specific point (line or string) of a file like wp-contet.php from wordpress using Bash script?
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
I'm not an expert at parsing text files in bash but you should delete the lines that define the things you're downloading from the wordpress salt and then insert the variable at the end... something like:
#!/bin/sh
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
STRING='put your unique phrase here'
printf '%s\n' "g/$STRING/d" a "$SALT" . w | ed -s wp-config.php
OK, now it's fixed... it should look for where the salt is supposed to go and it will replace it with the info retrieved from https://api.wordpress.org/secret-key/1.1/salt/
This version defines new keys if none exist, and also replaces existing keys:
#!/bin/bash
find . -name wp-config.php -print | while read line
do
curl http://api.wordpress.org/secret-key/1.1/salt/ > wp_keys.txt
sed -i.bak -e '/put your unique phrase here/d' -e \
'/AUTH_KEY/d' -e '/SECURE_AUTH_KEY/d' -e '/LOGGED_IN_KEY/d' -e '/NONCE_KEY/d' -e \
'/AUTH_SALT/d' -e '/SECURE_AUTH_SALT/d' -e '/LOGGED_IN_SALT/d' -e '/NONCE_SALT/d' $line
cat wp_keys.txt >> $line
rm wp_keys.txt
done
If you have csplit available, you can split the original wp-config.php file either side of the salt definitions, download new salts, then cat back together. This keeps the PHP define() statements at the same location in wp-config.php instead of than moving them to a different location within the file:
# Download new salts
curl "https://api.wordpress.org/secret-key/1.1/salt/" -o salts
# Split wp-config.php into 3 on the first and last definition statements
csplit wp-config.php '/AUTH_KEY/' '/NONCE_SALT/+1'
# Recombine the first part, the new salts and the last part
cat xx00 salts xx02 > wp-config.php
# Tidy up
rm salts xx00 xx01 xx02
How about using sed?
cat wp-config.php | sed 's/old_string/new_string/g' > wp-config.php
I think I got this one! its a bash script using only commands normally available at the command prompt and it does -everything- (assuming httpd is your web user) except create the databases. here you go.
#!/bin/bash
# wordpress latest auto-install script, by alienation 24 jan 2013. run as root.
# usage: ~/wp-install alien /hsphere/local/home/alien/nettrip.org alien_wpdbname alien_wpdbusername p#sSw0rd
# ( wp-install shell-user folder db-name db-user-name db-user-pw )
# download wordpress to temporary area
cd /tmp
rm -rf tmpwp
mkdir tmpwp
cd tmpwp
wget http://wordpress.org/latest.tar.gz
tar -xvzpf latest.tar.gz
# copy wordpress to where it will live, and go there, removing index placeholder if there is one
mv wordpress/* $2
cd $2
rm index.html
# create config from sample, replacing salt example lines with a real salt from online generator
grep -A 1 -B 50 'since 2.6.0' wp-config-sample.php > wp-config.php
wget -O - https://api.wordpress.org/secret-key/1.1/salt/ >> wp-config.php
grep -A 50 -B 3 'Table prefix' wp-config-sample.php >> wp-config.php
# put the appropriate db info in place of placeholders in our new config file
replace 'database_name_here' $3 -- wp-config.php
replace 'username_here' $4 -- wp-config.php
replace 'password_here' $5 -- wp-config.php
# change file ownership and permissions according to ideal at http://codex.wordpress.org/Hardening_WordPress#File_Permissions
touch .htaccess
chown $1:httpd .htaccess
chown -R $1:httpd *
find . -type d -exec chmod 755 {} \;
find . -type f -exec chmod 644 {} \;
chmod -R 770 wp-content
chmod -R g-w wp-admin wp-includes wp-content/plugins
chmod g+w .htaccess
# thats it!
echo ALL DONE
I built a simple CLI for just that. Try it out. It's called [WP-Salts-Update-CLI][1].
WP-Salts-Update-CLI
WPSUCLI downloads new salts from the WP API and replaces them with the ones in your wp-config.php file for every site on your server.
⚡️ Installation
Open command line terminal (I prefer iTerm2) and run the following command.
bash
sudo wget -qO wpsucli https://git.io/vykgu && sudo chmod +x ./wpsucli && sudo install ./wpsucli /usr/local/bin/wpsucli
This command will perform the following actions:
Use sudo permissions
Use wget to download WPSUCLI and rename it to wpsucli
Make the wpsucli executable
Install wpsucli inside /usr/local/bin/ folder.
🙌 Usage
Just run wpsucli and it will update the salts for every wp-config.php file on your server or PC.
This is the bash script that I came up with that works on my Ubuntu server. I modified the examples from above.
Its a bit of brute force in that it will only replace the 8 keys that currently are required and expects the server to return exactly the same length key every time. The script works well for my use case so I thought I would share it.
CONFIG_FILE=wp-config.php
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
SRC="define('AUTH_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'AUTH_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('SECURE_AUTH_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'SECURE_AUTH_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('LOGGED_IN_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'LOGGED_IN_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('NONCE_KEY'"; DST=$(echo $SALT|cat|grep -o define\(\'NONCE_KEY\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('AUTH_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'AUTH_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('SECURE_AUTH_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'SECURE_AUTH_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('LOGGED_IN_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'LOGGED_IN_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
SRC="define('NONCE_SALT'"; DST=$(echo $SALT|cat|grep -o define\(\'NONCE_SALT\'.\\{70\\}); sed -i "/$SRC/c$DST" $CONFIG_FILE
I tried the accepted solution:
#!/bin/sh
SALT=$(curl -L https://api.wordpress.org/secret-key/1.1/salt/)
STRING='put your unique phrase here'
printf '%s\n' "g/$STRING/d" a "$SALT" . w | ed -s wp-config.php
However it does not work perfectly as for some reason it induces the SALTS to "move down" 1 line in the wp-config.php file each time it is used... it is not ideal if you are going to change SALTS automatically like every week, months with cron for example...
A better solution for me was to create a little function that I call in my script.
This function creates a file with the SALTS (deletes it at the end), deletes every lines containing one of the SALTS then just inserts the SALTS contained in the file in place of the initial SALTS.
This works perfectly.
fct_update_salts() {
# Requires website name as target
curl http://api.wordpress.org/secret-key/1.1/salt/ > ~/SALTS.txt
var_initial_path1=`pwd`
cd ~ #going to home directory
# This scripts eliminates successively all SALT entries, replaces the last one by XXX as a marker, places SALTS.txt, below XXX and deletes XXX
sudo sed -i "/SECURE_AUTH_KEY/d" $1/wp-config.php
sudo sed -i "/LOGGED_IN_KEY/d" $1/wp-config.php
sudo sed -i "/NONCE_KEY/d" $1/wp-config.php
sudo sed -i "/AUTH_SALT/d" $1/wp-config.php
sudo sed -i "/SECURE_AUTH_SALT/d" $1/wp-config.php
sudo sed -i "/LOGGED_IN_SALT/d" $1/wp-config.php
sudo sed -i "/NONCE_SALT/d" $1/wp-config.php
sudo sed -i "/AUTH_KEY/cXXX" $1/wp-config.php
sudo sed -i '/XXX/r SALTS.txt' $1/wp-config.php
sudo sed -i "/XXX/d" $1/wp-config.php
echo "SALTS REPLACED BY:"
echo "====================="
cat ~/SALTS.txt
sudo rm -rf ~/SALTS.txt
cd $var_initial_path1
}
The function is to be called in the script like this:
# Reset SALTS
fct_update_salts $SITE_PATH
Where $SITE_PATH="/var/www/html/YOUR_WEBSITE" or whatever path works for you.
I was challenged with the same issue. Here is the script I wrote to replace the salts and keys from ones downloaded from WordPress. You can use it at any time to replace them if/when needed. I run it as sudo, and the script tests for that. If you use an account that can download to the directory and make updates to the wp-config.php file, then you can delete that part of the script.
#!/bin/sh
# update-WordPress-Salts: Updates WordPress Salts
# written by Wayne Woodward 2017
if [ $# -lt 1 ]; then
echo "Usage: update-WordPress-Salts directory"
exit
fi
if [ "$(whoami)" != "root" ]; then
echo "Please run as root (sudo)"
exit
fi
WPPATH=$1
# Update the salts in the config file
# Download salts from WordPress and save them locally
curl http://api.wordpress.org/secret-key/1.1/salt/ > /var/www/$WPPATH/wp-keys.txt
# Iterate through each "Saltname" and append 1 to it
# For a couple names that may match twice like "AUTH_KEY" adds extra 1s to the end
# But that is OK as when this deletes the lines, it uses the same matching pattern
# (Smarter people may fix this)
for SALTNAME in AUTH_KEY SECURE_AUTH_KEY LOGGED_IN_KEY NONCE_KEY AUTH_SALT SECURE_AUTH_SALT LOGGED_IN_SALT NONCE_SALT
do
sed -i -e "s/$SALTNAME/${SALTNAME}1/g" /var/www/$WPPATH/wp-config.php
done
# Find the line that has the updated AUTH_KEY1 name
# This is so we can insert the file in the same area
line=$(sed -n '/AUTH_KEY1/{=;q}' /var/www/$WPPATH/wp-config.php)
# Insert the file from the WordPress API that we saved into the configuration
sed -i -e "${line}r /var/www/$WPPATH/wp-keys.txt" /var/www/$WPPATH/wp-config.php
# Itererate through the old keys and remove them from the file
for SALTNAME in AUTH_KEY SECURE_AUTH_KEY LOGGED_IN_KEY NONCE_KEY AUTH_SALT SECURE_AUTH_SALT LOGGED_IN_SALT NONCE_SALT
do
sed -i -e "/${SALTNAME}1/d" /var/www/$WPPATH/wp-config.php
done
# Delete the file downloaded from Wordpress
rm /var/www/$WPPATH/wp-keys.txt
Many of the answers rely on the phrase 'put your unique phrase here' being present in the file, so they do not work when you want to change salts after the first time. There are also some that remove the old definitions and append the new ones at the end. While that does work, it's nice to keep the definitions where you would expect them, right after the comment documenting them. My solution addresses those issues.
I made a few attempts with sed, perl and regex, but there are special characters in the salts and the rest of the config file that tend to mess things up. I ended up using grep to search the document for the unique comment structure that opens and closes the salt definition block, which has the following format:
/**##+
<comment documentation>
*/
<salt definitions>
/**##-*/
Note that if that comment structure is removed or altered, this will no longer work. Here's the script:
#!/bin/bash -e
# Set Default Settings:
file='wp-config.php'
# set up temporary files with automatic removal:
trap "rm -f $file_start $file_end $salt" 0 1 2 3 15
file_start=$(mktemp) || exit 1
file_end=$(mktemp) || exit 1
salt=$(mktemp) || exit 1
function find_line {
# returns the first line number in the file which contains the text
# program exits if text is not found
# $1 : text to search for
# $2 : file in which to search
# $3 (optional) : line at which to start the search
line=$(tail -n +${3:-1} $2 | grep -nm 1 $1 | cut -f1 -d:)
[ -z "$line" ] && exit 1
echo $(($line + ${3:-1} - 1))
}
line=$(find_line "/**##+" "$file")
line=$(find_line "\*/" "$file" "$line")
head -n $line $file > $file_start
line=$(find_line "/**##-\*/" "$file" "$line")
tail -n +$line $file > $file_end
curl -Ls https://api.wordpress.org/secret-key/1.1/salt/ > $salt
(cat $file_start $salt; echo; cat $file_end) > $file
exit 0
Strings containing single asterisks, such as "*/" and "/**##-*/" want to expand to directory lists, so that is why those asterisks are escaped.
Here's a pure bash approach. This does not depend on wordpress.org.
I converted the original wp_generate_password() function used by WordPress to generate salt.
#!/bin/bash
set -e
#
# Generates a random password drawn from the defined set of characters.
# Inspired by WordPress function https://developer.wordpress.org/reference/functions/wp_generate_password/
#
# Parameters
# ----------
# $length
# (ing) (Optional) Length of password to generate.
# Default value: 12
# $special_chars
# (bool) (Optional) Whether to include standard special characters.
# Default value: true
# $extra_special_chars
# (bool) (Optional) Whether to include other special characters. Used when generating secret keys and salts.
# Default value: false
#
function wp_generate_password() {
# Args
length="$(test $1 && echo $1 || echo 12 )"
special_chars="$(test $2 && echo $2 || echo 1 )"
extra_special_chars="$(test $3 && echo $3 || echo 0 )"
chars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
[[ $special_chars != 0 ]] && chars="$chars"'!##$%^&*()'
[[ $extra_special_chars != 0 ]] && chars="$chars"'-_ []{}<>~`+=,.;:/?|'
password='';
for i in $(seq 1 $length); do
password="${password}${chars:$(( RANDOM % ${#chars} )):1}"
done
echo "$password"
}
You can then just run SALT="$(wp_generate_password 64 1 1)".
Update
I just published a standalone script to generate WP salt values. You can generate the salt values by running ./wp-generate-salt.sh.
If the wordpress.org API generated SALT values are not necessary for your use case, you can use the pwgen to generate keys on the server and insert those into wp-config.php.
for i in {1..8} ;do unique_key="`pwgen -1 -s 64`";sudo sed -i "0,/put your unique phrase here/s/put your unique phrase here/$unique_key/" /srv/www/wordpress/wp-config.php; done
You may need to fix the ownership of the file after using sudo. You can use a command similar to this for changing the ownership.
chown www-data:www-data /srv/www/wordpress/wp-config.php