I'm trying to write a parallel compress / encrypt backup script for archiving
using GNU parallel, xz and GnuPG. The core part's of script is:
tar --create --format=posix --preserve-permissions --same-owner --directory $BASE/$name --to-stdout . \
| parallel --pipe --recend '' --keep-order --block-size 128M "xz -9 --check=sha256 | gpg --encrypt --recipient $RECIPIENT" \
| pv > $TARGET/$FILENAME
Without GnuPG encryption, it works great (uncompress and untar works),
but after adding parallel encryption, it's fail to decrypt with below error:
[don't know]: invalid packet (ctb=0a)
gpg: WARNING: encrypted message has been manipulated!
gpg: decrypt_message failed: Unexpected error
: Truncated tar archive
tar: Error exit delayed from previous errors.
Because uncompressed size is as same as gnu parallel's block size(around 125M), I assume that it's related GnuPG's support of partial block encryption. How can I solve this problem?
FYI
Another parallel gpg encrption issue about random number generation
https://unix.stackexchange.com/questions/105059/parallel-pausing-and-resuming
Pack
tar --create --format=posix --preserve-permissions --same-owner --directory $BASE/$name --to-stdout . |
parallel --pipe --recend '' --keep-order --block-size 128M "xz -9 --check=sha256 | gpg --encrypt --recipient $RECIPIENT;echo bLoCk EnD" |
pv > $TARGET/$FILENAME
Unpack
cat $TARGET/$FILENAME |
parallel --pipe --recend 'bLoCk EnD\n' -N1 --keep-order --rrs 'gpg --decrypt | xz -d' |
tar tv
-N1 is needed to make sure we pass a single record at a time. GnuPG does not support decrypting multiple merged records.
GnuPG does not support concatenating multiple encryption streams and decrypting them at once. You will have to store multiple files, and decrypt them individually. If I'm not mistaken, your command even mixes up the outputs of all parallel instances of GnuPG, so the result is more or less random garbage.
Anyway: GnuPG also takes care of compression, have a look at the --compression-algo option. If you prefer to use xz, apply --compression-algo none so GnuPG does not try to compress the already-compressed message again. Encryption has massive support by CPU-instructions ourdays, xz -9 might in fact be more time intensive than encryption (although I did not benchmark this).
that's mainly a gpg issue. gpg does not support multithreading and probably
never will. you can search the web about the why.
it even got worse with gpg v2: you cannot even run multiple gpg v2 instances in
parallel because they all lock the gpg-agent which is now doing all the
work........ maybe we should look for an alternative when doing mass encryption.
https://answers.launchpad.net/duplicity/+question/296122
EDIT: No. It is possible to run multiple gpg v2 instances at the same time, without any problem with the gpg-agent.
Related
I am currently writing a batch script to automate signing and encrypting files, on GnuPG version 2.0.30.
Can someone explain why the following will not work on my Windows machine:
echo "passphrase"| gpg2 --batch --yes --always-trust --passphrase-fd 0 -es -r "Public key for encryption" "Path of file to encrypt"
To my knowledge, this should be piping the output of echo (my passphrase) as the input of --passphrase-fd 0 (0 operating on STDIN handle of course). I can't seem to get this to work, and wasn't sure if it was my code, or if it's just an issue running this on Windows.
Whenever I execute this command, I get
"no default secret key: Bad passphrase"
"sign+encrypt failed: Bad passphrase"
As a workaround, I was able to use redirection operater < and passed the filepath to my passphrase, still using the STDIN handle (0):
gpg2 --batch --yes --always-trust --passphrase-fd 0 -es -r "Public key for encryption" "Path of file to encrypt" < "Filepath to passphrase"
I realize it's bad practice to store a passphrase in a local file, and may consider removing the signature altogether, but for my own sake, I want to better understand why this won't work using piping (on Windows).
Thanks!
Unix gurus,
Here is my requirement.
Finding the filelist and using iconv to convert the encoding to UTF-8.
Eg: FILE_LIST=abcfilename1 abcfilename2 abcfilename3 abcfilename4
Note:filename can be anything
Code in Loop:
for f in $FILE_LIST; do
iconv -f ISO-8859-1 -t utf-8 $f_$DATE.txt > $f_$DATE.utf8
mv $f_$DATE.utf8 $f_$DATE.txt
done
This code wait for each file conversion. Taking much time to complete. At a time only single thread/cpu was utilized.
Created Code with multiple background sessions:
for f in $FILE_LIST; do
iconv -f ISO-8859-1 -t utf-8 $f_$DATE.txt > $f_$DATE.utf8 &
done
wait
for f in $FILE_LIST; do
mv $f_$DATE.utf8 $f_$DATE.txt &
done
wait
This creates multiple background session which utilize multiple process. Each process uses single thread/cpu. But if the file size is huge more than 2GB, single thread utilization is not fast enough.
Came across GNU parallel, which utilizes multiple threads/cpus. Not sure how to list or find files based on above scenario and use iconv. My main objective is to utilize maximum resources with less consumption time.
Tried the iconv with 2GB file in size.
Thought of using GNU parallel to utilize multiple CPU and see the performance. Multiple cores were used while running.
GNU parallel:
time find . -name 'filename.txt' | parallel -X iconv -f ISO-8859-1 -t UTF-8 {} \> {}.converted
real 0m14.58s
user 0m23.27s
sys 0m5.38s
Sequential:
time iconv -f ISO-8859-1 -t utf-8 filename.txt > filename.txt.utf8
real 0m6.49s
user 0m5.43s
sys 0m1.07s
I found, sequential timings is much faster than parallel. Am i missing anything in parallel commands?
Kindly suggest, how this scenario can be accomplished?
Thanks
I trying to decrypt AES-128 encrypted m3u8 video files such as this one :
the m3u8 file :
#EXTM3U
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOW-CACHE:NO
#EXT-X-VERSION:2
#EXT-X-FAXS-CM:MII6lAYJKoZIhvcNAQcCoII6hTCCOoECAQExCzAJBgUrDgMCGgUAM... very long key...
#EXT-X-KEY:METHOD=AES-128,URI="faxs://faxs.adobe.com",IV=0X99b74007b6254e4bd1c6e03631cad15b
#EXT-X-TARGETDURATION:8
#EXTINF:8,
video.mp4Frag1Num0.ts
#EXTINF:8,
video.mp4Frag1Num1.ts
...
I've tried with openssl :
openssl aes-128-cbc -d -kfile key.txt -iv 99b74007b6254e4bd1c6e03631cad15b -nosalt -in video_enc.ts -out video_dec.ts
key.txt contains the very long key
-->
bad decrypt
1074529488:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:539:
What am-I doing wrong ?
This might be a bit of a hack, but given a URL to an .m3u8 file, it will download and decrypt the files that make up the stream:
#!/usr/bin/env bash
curl "$1" -s | awk 'BEGIN {c=0} $0 ~ "EXT-X-KEY" {urlpos=index($0,"URI=")+5; ivpos=index($0,"IV="); keyurl=substr($0, urlpos, ivpos-urlpos-2); iv=substr($0, ivpos+5); print "key=`curl -s '\''"keyurl"'\'' | hexdump -C | head -1 | sed \"s/00000000//;s/|.*//;s/ //g\"`"; print "iv="iv} $0 !~ "-KEY" && $0 ~ "http" {printf("curl -s '\''"$0"'\'' | openssl aes-128-cbc -K $key -iv $iv -d >seg%05i.ts\n", c++)}' | bash
This script generates a second script that extracts keys and initialization vectors and uses them to decrypt while downloading. It needs curl, awk, hexdump, sed, and openssl to run. It'll probably choke on an unencrypted stream, or on a stream that uses something other than AES-128 (is any other encryption supported?).
You'll get a bunch of files: seg00000.ts, seg00001.ts, etc. Use tsMuxeR (https://www.videohelp.com/software/tsMuxeR) to merge these into a single file (simple concatenation didn't work for me...it's what I tried first):
(echo "MUXOPT --no-pcr-on-video-pid --new-audio-pes --vbr --vbv-len=500"; (echo -n "V_MPEG4/ISO/AVC, "; for i in seg*.ts; do echo -n "\"$i\"+"; done; echo ", fps=30, insertSEI, contSPS, track=258") | sed "s/+,/,/"; (echo -n "A_AAC, "; for i in seg*.ts; do echo -n "\"$i\"+"; done; echo ", track=257") | sed "s/+,/,/") >video.meta
tsMuxeR video.meta video.ts
(Track IDs and framerate may need adjustment...get the values to use by passing one of the downloaded files to tsMuxeR.)
Then use ffmpeg to remux to something a bit more widely understood:
ffmpeg -i video.ts -vcodec copy -acodec copy video.m4v
In order to decrypt encrypted video stream you need encryption key.
This key is not part of the stream. It should be obtained separately.
EXT-X-FAXS-CM header contains DRM meta-data and not the key.
This is excert from Adobe Media Server developer guide:
The Adobe Access Server protected variant playlist also needs to include the #EXT-X-FAXS-CM tag. The value of #EXT-X-FAXS-CM tag in variant playlist is the relative URI referring to the DRM metadata of one of the individual streams.At the client, the #EXT-X-FAXS-CM tag in variant playlist will be used to create the DRM session. The same DRM session will be used for all encrypted M3U8 files inside the variant playlist.
Full guide can be found here:
http://help.adobe.com/en_US/adobemediaserver/devguide/WS5262178513756206-4b6aabd1378392bb59-7fe8.html
There is also mention that faxs://faxs.adobe.com URI is for local key serving.
So key obtained locally from a device.
While some of the bash scripts in the existing answers get you part (or even all) of the way, depending which site you're trying to download from, you might hit other obstacles (different auth method, custom license server mount, etc.)
I've found streamlink to be the most robust solution for this, which also lets you stream directly (rather than download), if that's what you're after, and it has all the site-specific work already done for you for a long list of sites (see plugins section, but keep in mind it's under active development and the latest release was in June, so for some of the newer ones you'll have to git clone and install from source).
In many cases, VLC will happily convert an .m3u8 video to an unencrypted .ts or .mp4. In the VLC graphical interface, go to Media > Convert/Save.
Even through this file includes AES encrypted data, openssl don't know the m3u8 format. However FFmpeg might be able to handle it.
i am using mysql-5.5 and rhel5 and my intention is to use mysqldump to take the encrypted backup and compressed backup
as i am using mysqldump as below
mysqldump -u root -p db_name | gzip >file_name.sql.gz
it will give compressed backup but not encrypted one
How about this:
mysqldump -u root -p db_name | gpg --encrypt -r 'user_id' | gzip >file_name.sql.gz
of course you need the public key of the user that you want to encrypt for.
e.g.
gpg --import keyfile
Instead of using GPG which is frankly, kind of overkill unless you really like GPG, you can use OpenSSL which is likely built-in and has no real dependency structure for making easily portable and decryptable backups. This way you can readily decrypt the backup on just about any Linux system (and many other platforms) without any keyring, just knowing the passphrase.
Read more at this link about how do so.
Backup one database, change what is inside [..]
mysqldump -u root --single-transaction [DataBaseName] | gzip | openssl enc -pbkdf2 -k [MyPassword] > database.sql.zip.enc
Backup all databases separately:
date=`date "+%Y%m%d"`
for DB in $(mysql -u root -e 'show databases' -s --skip-column-names); do
mysqldump -u root --single-transaction $DB | gzip | openssl enc -pbkdf2 -k [MyPassword] > db-$DB-$date.sql.gz.enc;
done
Also note that using -p via command line is really bad practise as the password can be read out via ps aux.
I suggest using openssl as pgp is getting to slow on big files.
The best solution I have found so far which I am regularly using at work now is mysqldump-secure.
It offers openssl encryption and compression as well as other more features and even ships with a nagios monitoring plugin.
I use the following Bash script that uses Dropbox to sync the backups directly to our own company server (followed by automatic backups of that data). Replace the script variables with your own. Then I just add that to my crontab to run it every 12 hours.
FILENAME=dbname.$(date +%Y-%m-%d-%H-%M)
SQLFILE=/root/Desktop/$FILENAME.sql
ZIPFILE=/root/Desktop/$FILENAME.zip
GPGFILE=/root/Dropbox/SQL-Backups/$FILENAME.gpg
mysqldump --user=dbuser --password=password --port=3306 --default-character-set=utf8 --single-transaction=TRUE --databases "dbname" --result-file="$SQLFILE"
zip -9 $ZIPFILE $SQLFILE
gpg --output "$GPGFILE" --encrypt --recipient "recipient#company.com" "$ZIPFILE"
unlink $ZIPFILE
unlink $SQLFILE
This uses GnuPG to encrypt the resulting zipped SQL dump. Remember to never import the private key to the web server. The web server's GPG setup only needs the public key.
You can use the GPG software available for most platforms to create your key and publish the public key to a key server.
I am trying to add my GPG public key as a part of our appliance installation process. The purpose of it to encrypt any important files like logs before admin pulling them into his local using admin portal and then decrypt them using private key.
The plan is to export public key into a file and make appliance installation process to import it using gpg --import command. But I realized, the key is needed to be trusted/signed before do any encryption.
How to make this key is trusted without any human intervention at the time of installation?
Btw, our appliance os is ubuntu vm and we use kickstart to automate.
Advance thanks for all help.
Your question is really "How do I encrypt to a key without gpg balking at the fact that the key is untrusted?"
One answer is you could sign the key.
gpg --edit-key YOUR_RECIPIENT
sign
yes
save
The other is you could tell gpg to go ahead and trust.
gpg --encrypt --recipient YOUR_RECIPIENT --trust-model always YOUR_FILE
Coincidentally I have a similar situation to the OP - I'm trying to use public/private keys to sign and encrypt firmware for different embedded devices. Since no answer yet shows how to add trust to a key you already have imported, here is my answer.
After creating and testing the keys on a test machine, I exported them as ascii:
$ gpg --export -a <hex_key_id> > public_key.asc
$ gpg --export-secret-keys -a <hex_key_id> > private_key.asc
Then secure-copied and imported them to the build server:
$ gpg --import public_key.asc
$ gpg --import private_key.asc
Important: add trust
Now edit the key to add ultimate trust:
$ gpg --edit-key <user#here.com>
At the gpg> prompt, type trust, then type 5 for ultimate trust, then y to confirm, then quit.
Now test it with a test file:
$ gpg --sign --encrypt --yes --batch --status-fd 1 --recipient "recipient" --output testfile.gpg testfile.txt
which reports
...
[GNUPG:] END_ENCRYPTION
without adding trust, I get various errors (not limited to the following):
gpg: There is no assurance this key belongs to the named user
gpg: testfile.bin: sign+encrypt failed: Unusable public key
There's an easier way to tell GPG to trust all of its keys by using the --trust-model option:
gpg -a --encrypt -r <recipient key name> --trust-model always
From the man page:
--trust-model pgp|classic|direct|always|auto
Set what trust model GnuPG should follow. The models are:
always Skip key validation and assume that used
keys are always fully trusted. You generally
won't use this unless you are using some
external validation scheme. This option also
suppresses the "[uncertain]" tag printed
with signature checks when there is no evidence
that the user ID is bound to the key. Note that
this trust model still does not allow the use
of expired, revoked, or disabled keys.
Add trusted-key 0x0123456789ABCDEF to your ~/.gnupg/gpg.conf replacing the keyid. This is equivalent to ultimately trusting this key which means that certifications done by it will be accepted as valid. Just marking this key as valid without trusting it is harder and either requires a signature or switching the trust-model to direct. If you are sure to only import valid keys you can simply mark all keys as valid by adding trust-model always. In the latter case ensure that you disable automatic key retrieval (not enabled by default).
This worked for me:
Trying to encrypt a file responds with this:
gpg -e --yes -r <uid> <filename>
It is NOT certain that the key belongs to the person named
in the user ID. If you *really* know what you are doing,
you may answer the next question with yes.
Use this key anyway? (y/N)
That causes my shell script to fail.
So I:
$gpg --edit-key <uid>
gpg> trust
Please decide how far you trust this user to correctly verify other
users' keys (by looking at passports, checking fingerprints from
different sources, etc.)
1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
5 = I trust ultimately
m = back to the main menu
Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y
Please note that the shown key validity is not necessarily correct
unless you restart the program.
gpg> quit
Now the encrypt works properly.
Based on #tersmitten's article and a bit of trial and error, I ended up with the following command line to trust all keys in a given keyring without user interaction. I use it for keys used with both StackEschange Blackbox and hiera-eyaml-gpg:
# The "-E" makes this work with both GNU sed and OS X sed
gpg --list-keys --fingerprint --with-colons |
sed -E -n -e 's/^fpr:::::::::([0-9A-F]+):$/\1:6:/p' |
gpg --import-ownertrust
Personally, I prefer a solution which stores the results in the trustdb file itself rather than depends on user environment outside the shared Git repo.
Here's a trick I've figured out for automation of GnuPG key management, hint heredoc + --command-fd 0 is like magic. Below is an abridged version of one of the scripts that's been written to aid in automation with GnuPG.
#!/usr/bin/env bash
## First argument should be a file path or key id
Var_gnupg_import_key="${1}"
## Second argument should be an integer
Var_gnupg_import_key_trust="${2:-1}"
## Point to preferred default key server
Var_gnupg_key_server="${3:-hkp://keys.gnupg.net}"
Func_import_gnupg_key_edit_trust(){
_gnupg_import_key="${1:-${Var_gnupg_import_key}}"
gpg --no-tty --command-fd 0 --edit-key ${_gnupg_import_key} <<EOF
trust
${Var_gnupg_import_key_trust}
quit
EOF
}
Func_import_gnupg_key(){
_gnupg_import_key="${1:-${Var_gnupg_import_key}}"
if [ -f "${_gnupg_import_key}" ]; then
echo "# ${0##*/} reports: importing key file [${_gnupg_import_key}]"
gpg --no-tty --command-fd 0 --import ${_gnupg_import_key} <<EOF
trust
${Var_gnupg_import_key_trust}
quit
EOF
else
_grep_string='not found on keyserver'
gpg --dry-run --batch --search-keys ${_gnupg_import_key} --keyserver ${Var_gnupg_key_server} | grep -qE "${_grep_string}"
_exit_status=$?
if [ "${_exit_status}" != "0" ]; then
_key_fingerprint="$(gpg --no-tty --batch --dry-run --search-keys ${_gnupg_import_key} | awk '/key /{print $5}' | tail -n1)"
_key_fingerprint="${_key_fingerprint//,/}"
if [ "${#_key_fingerprint}" != "0" ]; then
echo "# ${0##*/} reports: importing key [${_key_fingerprint}] from keyserver [${Var_gnupg_key_server}]"
gpg --keyserver ${Var_gnupg_key_server} --recv-keys ${_key_fingerprint}
Func_import_gnupg_key_edit_trust "${_gnupg_import_key}"
else
echo "# ${0##*/} reports: error no public key [${_gnupg_import_key}] as file or on key server [${Var_gnupg_key_server}]"
fi
else
echo "# ${0##*/} reports: error no public key [${_gnupg_import_key}] as file or on key server [${Var_gnupg_key_server}]"
fi
fi
}
if [ "${#Var_gnupg_import_key}" != "0" ]; then
Func_import_gnupg_key "${Var_gnupg_import_key}"
else
echo "# ${0##*/} needs a key to import."
exit 1
fi
Run with script_name.sh 'path/to/key' '1' or script_name.sh 'key-id' '1' to import a key and assign a trust value of 1 or edit all values with script_name.sh 'path/to/key' '1' 'hkp://preferred.key.server'
Encryption should now be without complaint but even if it does the following --always-trust option should allow encryption even with complaint.
gpg --no-tty --batch --always-trust -e some_file -r some_recipient -o some_file.gpg
If you wish to see this in action, then check the Travis-CI build logs and how the helper script GnuPG_Gen_Key.sh is used for both generating and importing keys in the same operation... version two of this helper script will be much cleaner and modifiable but it's a good starting point.
This oneliner updates the trustdb with the ownertrust values from STDIN -- by extracting the fingerprint to the format required by --import-ownertrust flag.
This flag, as detailed on gpg man page, should be used In case of a severely damaged trustdb and/or if you have a recent backup of the ownertrust values, you may re-create the trustdb.
gpg --list-keys --fingerprint \
| grep ^pub -A 1 \
| tail -1 \
| tr -d ' ' \
| awk 'BEGIN { FS = "\n" } ; { print $1":6:" }' \
| gpg --import-ownertrust
One way to trust imported gpg keys:
gpg --import <user-id.keyfile>
fpr=`gpg --with-colons --fingerprint <user-id> |awk -F: '$1 == "fpr" {print$10; exit}'`
gpg --export-ownertrust && echo $fpr:6: |gpg --import-ownertrust
here, I assume that you import a key with the <user-id> from <user-id.keyfile>. The second line only extracts fingerprint, you can drop it if you know the fingerprint beforehand.
I think, I figured way to do this.
I used 'gpg --import-ownertrust' to export my trust db into a text file then removed all of my keys from it except public key I needed to push. And then imported my public key and edited owner-trust file on to server. This seems like working.
Now I am having trouble implementing these steps in Kickstart file:-(
With powershell, here is how to trust john.doe#foo.bar (adapted from #tersmitten blog post):
(gpg --fingerprint john.doe#foo.bar | out-string) -match 'fingerprint = (.+)'
$fingerprint = $Matches[1] -replace '\s'
"${fingerprint}:6:" | gpg --import-ownertrust
Note: using cinst gpg4win-vanilla
There is a way to autotrust key using --edit-key, but without getting into interactive shell (so can be automated in script). Below is a sample for windows:
(echo trust &echo 5 &echo y &echo quit) | gpg --command-fd 0 --edit-key your#email.com
Unix based:
echo -e "5\ny\n" | gpg --homedir . --command-fd 0 --expert --edit-key user#exaple.com trust;
For more info read this post. It details if you are creating more than one key.
I used following script for import key:
#!/bin/bash
function usage() {
cat >&2 <<EOF
Usage: $0 path_of_private_key
Example: gpg_import.sh ~/.ssh/my_gpg_private.key
Import gpg key with trust.
EOF
exit 1
}
[[ $# -lt 1 ]] && usage
KEY_PATH=$1
KEY_ID=$(gpg --list-packets ${KEY_PATH}/${GPG_PRIVATE_KEY} | awk '/keyid:/{ print $2 }' | head -1)
gpg --import ${KEY_PATH}/${GPG_PRIVATE_KEY}
(echo trust &echo 5 &echo y &echo quit) | gpg --command-fd 0 --edit-key $KEY_ID
I am using windows with gpgwin4.0.2 installed.
Open the Kleopatra (the GUI) -> Certificates -> Right Click -> Certify. Once the has been certify, this message will not show any.
Try this :
(echo trust &echo 5 &echo y &echo quit &echo save) | gpg --homedir 'gpgDirectory/' --batch --command-fd 0 --edit-key 'youKey'
--homedir : not required