It's quite helpful to have the autogenerated checksums, ie, md5sum, sha1, sha256 and sha512 when querying a file, ie:-
https://server/artifactory/files/file.txt.md5
https://server/artifactory/files/artifactory/files/file.txt.sha1
https://server/artifactory/files/artifactory/files/file.txt.sha256
etc.
My problem is that an downstream system requires the output one would get with the cmd, ie
$ md5sum file.txt
3f768759030461237774e050cafea1e3 file.txt
or if piped
3f768759030461237774e050cafea1e3 -
Artifactory only sends the hash itself so the downstream app does not know how to parse it:-
3f768759030461237774e050cafea1e3
Is there config option, query string or plugin that I can apply so the hash outputs like md5sum, sha1sum, sha256sum and sha512sum, would?
Related
I'm trying to process a remote m3u8 playlist containing (a possibly encrypted) HLS stream. The output I'm looking for is a mp4 container with MPEG-4 inside.
The playlist is a result of an ended live stream and may contain EXT-X-DISCONTINUITY tags. As I understand, there's no "built-in" way to process it and indeed, there are plenty of warnings like "Non-monotonous DTS in output stream" and the resulting file always has some playback issues.
There are a couple of options to "glue" it. On an unencrypted stream, I found concat demuxer to produce the result with the least playback problems. The command is:
LIST=chunks.list; ffmpeg -loglevel 'debug' -f concat -safe 0 -protocol_whitelist "file,http,https,tcp,tls,crypto" -i $LIST -c copy -movflags frag_keyframe -y output_concat.mp4
where chunks.list is something like:
file 'https://www.example.org/chunk1.ts'
file 'https://www.example.org/chunk2.ts'
file 'https://www.example.org/chunk3.ts'
Now, I'm trying to use concat demuxer to process encrypted chunks. I've tried passing -key and -iv options in different places and changing chunks.list to be like file 'crypto+https...' but it won't pick up encryption key:
Opening an input file: chunks.list.
[concat # 0x7f9fb6800c00] Opening 'chunks.list' for reading
[NULL # 0x7f9fb6007e00] Opening 'crypto+https://www.example.org/chunk1.ts' for reading
[crypto # 0x7f9fb5700a00] decryption key not set
[concat # 0x7f9fb6800c00] Impossible to open 'crypto+https://www.example.org/chunk1.ts'
[AVIOContext # 0x7f9fb5700780] Statistics: 5094 bytes read, 0 seeks
chunks.list: Invalid argument
Docs mention encryption options for crypto protocol, so it looks it's just a matter of passing these in a proper way.
When, instead of using concat demuxer, I try to process and decrypt only one chunk like:
ffmpeg -i crypto+https://www.example.org/chunk1.ts -key <my_hex_key> -iv <my_iv> chunk1.ts it works fine. There's nothing wrong with the key itself, I'm able to decrypt it with other tools (openssl etc).
Is it possible for concat demuxer to handle decryption? If so, where should I pass key and iv options?
I have a large (8GB) packet capture (.pcap) that has generated a number of Snort alerts.
I suspect that some data may have been exfiltrated while encoded. Is there an easy way to determine if anything has gone out as Base64 encoded without having to sift through Wireshark to find the problem, perhaps using the terminal?
My worry is that an actor could conduct lateral movement within the system and extract files to a 3rd party system which wouldn't be recognized by Snort alerts.
I've tried parsing the large .pcap into 200MB files to allow closer examination (my VM has memory limitations).
This inspired me to write pdml2flow-base64strings a plugin for pdml2flow. Using the power of pdml2flow the plugin searches in all fields known by wireshark/tshark for valid base64 encoded data. If it finds base64 strings it then decodes and prints the raw data for you. You can use the --minlength switch, if you want to limit the search only for data of a certain size. It also allows you to narrow down your search to only ascii (--ascii) or utf-8 (--utf8).
For example if you want to extract all utf-8 strings encoded in any field known by wireshark/tshark you could use pdml2flow with the plugin:
tshark -r dump.cap -Tpdml | pdml2flow +base64strings --utf8
or if you don't want flow aggregation use pdml2frame
tshark -r dump.cap -Tpdml | pdml2frame +base64strings --utf8
I hope this helps. Any input is welcome, thank you.
Disclosure: I am the author of pdml2flow and pdml2flow-base64strings
Not an easy task but you could try using tsharkon the command line.
Generated base64 payload:
echo "base64 encoded payload" | base64
YmFzZTY0IGVuY29kZWQgcGF5bG9hZAo=
Setup a listening socket:
netcat -l -p 8090
POST to it
curl -X POST -d 'YmFzZTY0IGVuY29kZWQgcGF5bG9hZAo=' http://localhost:8090
^C
Parsing on command line after capturing with wireshark:
tshark -r ~/tmp/base64.pcapng -T fields -e http.file_data | grep -E '^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?$'
Result (plus empty lines for non matches, not shown for simplicity's sake):
YmFzZTY0IGVuY29kZWQgcGF5bG9hZAo=
There is one file I want to encrypt with GnuPG by
gpg2 --homedir=~/.gnupg --always-trust=true --recipient="BlahBlah" --encrypt=/path/to/file --output=/path/to/output_file
However this command seems to hang forever and never return. Interestingly, after I interrupt process, there is indeed /path/to/output_file created , however the bytes written there is much bigger than raw payload (for example my /path/to/file is only of 5 bytes but it turns out there are nearly 200 bytes written to /path/to/output_file).
There must be something wrong, but I really couldn't figure out what is it.
I have in advance imported the key for BlahBlah by gpg --import key.asc. It happens both for GnuPG 1 and GnuPG 2.
You're applying --encrypt in a wrong way. --encrypt does not expect any parameters, the file(s) to be worked on are passed as very last arguments. Additionally, following the documentation you should pass --output /path/to/output_file instead of --output=/path/to/output_file. Finally, GnuPG distinguishes between options and commands, and options should precede commands.
What you observe is that GnuPG starts writing header information, but then waits for input from STDIN (until interrupted).
The GnuPG command line you're looking for is
gpg2 --homedir=~/.gnupg --always-trust=true --recipient="BlahBlah" --output /path/to/output_file --encrypt /path/to/file
One last hint: the combination of --always-trust=true and resolving a recipient by user ID is a very bad idea, as any other key with the same user ID in the local keyring might be used. Pass the full key's fingerprint instead, which specifically selects a distinct key (using short key IDs is not secure, either).
I have a job that runs periodically and signs/encrypts a file like so:
$ gpg --homedir /path/to/.gnupg -r key1#mydomain.com -r key2#mydomain.com --local-user sig1#mydomain.com --batch --passphrase-file /path/to/gpg-password --sign -ea myfile
The command encrypts the file for two recipients: key1#mydomain.com and key2#mydomain.com. It signs the file with sig1#mydomain.com. It runs in batch mode as there is no human interactivity - this is an automated process. It gets the passphrase for the signature from /path/to/gpg-password.
What I would like to do is now sign the file with two signatures at the same time. Like so:
$ gpg --homedir /path/to/.gnupg -r key1#mydomain.com -r key2#mydomain.com --local-user sig1#mydomain.com --local-user sig2#mydomain.com --batch --passphrase-file /path/to/gpg-password --sign -ea myfile
gpg: skipped "sig2#mydomain.com": bad passphrase
It works fine in interactive mode (eg. without --batch), I just have to supply the two passphrases via the command line. However, in batch mode it fails as it tries to get the signatures from the file. The signature is only valid for one of the signing keys.
From the man page:
--passphrase-file file
Read the passphrase from file file. Only the first line will be read from file file. This can only be used if only one passphrase is supplied.
How do I tell it what the password is for each key?
You have different options.
Completely remove the passwords, since they're stored somewhere anyway.
Use the same password (as you already discovered).
Use the gpg-agent and preset the passphrase. I'm unsure whether this is GnuPG 2-only (usually installed as gpg2, maybe to be installed from a gnupg2 package). Presetting the passphrase is as easy as running gpg-preset-passphrase --preset [fingerprint]. You will have to run this command for each of the keys individually, and make sure to cache the passphrase for a given time (at least the processing time of adding all the passphrases, and then signing the file you want to sign).
For the sake of completeness, but impractical: sign the file individually for each key, then take apart the OpenPGP packets and recombine them adding all the signatures one after the other. Signing with multiple keys just creates multiple signature packets.
For anyone else in the same situation as me, I ended up working around this apparent deficiency of gpg by editing one of the signing keys to have the same password as the other (the password stored in the gpg-password file). This doesn't compromise security in this instance since the password is stored in a text file anyway - the real security is the password of the user that this commands runs from and the fact that the secret keys are kept secret). You can change the password on a key by doing gpg --edit-key <key_id>, then passwd. Don't forget to save after.
I've been trying to set up encrypted folder following the instructions in https://wiki.archlinux.org/index.php/ECryptfs#Without_ecryptfs-utils. In that case a mount passphrase is manually created and wrapped using a chosen password ("Arch" in that example). This is working as expected.
There is yet another possibility how to manually set up things. Namely using simple mount e.g.:
mount -t ecryptfs ~/.Private/ ~/Private -o key=passphrase,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=y
In this case I'm prompted for a [password] and after that folder is mounted.
My question is: where is the mount passphrase hidden in the second example? And how is my entered [password] related to it and to the FEKEK in this case.
Could anybody please explain?
Thank you in advance.
I think the entered passphrase becomes the "file encryption key, encryption key, or FEKEK" itself, and is temporarily stored in the kernel keyring (root's keyring if you need to mount with sudo) while the folder is mounted. You can see the keyring before & after mounting & unmounting with sudo keyctl show.
It does not store the passphrase/FEKEK anywhere on disk, unlike the method "Without ecryptfs-utils" in the ArchWiki you liked to, and the one in the ecryptfs-utils below.
FYI, using the ecryptfs-utils tools it will create a random (16 byte) passphrase to use, if you haven't specified your own passphrase, and it apparently is the file encryption key (FEKEK) / FNEK.
Looking in a couple of the ecryptfs-utils tools, (several are shell scripts) inside ecryptfs-migrate-home it collects some data and calls ecryptfs-setup-private, which has this function to generate a more secure 16 byte random passphrase/FEKEK/FNEK (that then gets "wrapped"/encrypted with your login passphrase and stored on disk):
random_passphrase () {
bytes=$1
# Pull $1 of random data from /dev/urandom,
# and convert to a string of hex digits
od -x -N $bytes --width=$bytes /dev/urandom | head -n 1 | sed "s/^0000000//" | sed "s/\s*//g"
}