openssl smime: aes vs rsa - which one encrypts? - encryption

Question

When I want to encrypt large files, I have found the following command:
openssl smime -encrypt -binary -aes-256-cbc \
-in large_file.img -out large_file.img.dat \
-outform DER \
public-rsa2048.pem
The private/public keys have been created like this:
openssl req -x509 -nodes -newkey rsa:2048 \
-keyout private-rsa2048.pem -out public-rsa2048.pem
So I am using aes-256-cbs as well as an rsa-key.
Which one is now responsible for the encrpytion?
Which one does impact the performance and how?
If I increase/lower the public key (rsa512 vs rsa4096), does this impact the performance?
If I choose a different aes algorithm, does this impact the performance as well?
Could someone bring some light into this matter please?
Are there any other methods to encrypt large files?

Answer 1

Using the openssl smime command to encrypt data encrypts it in such a way that it can be received and decrypted by a recipient using email. This means that it uses certificates representing various entities (sender, recipient) and containing keys uniquely associated with those entities to both sign, encrypt, decrypt, and verify signatures on the contained data.
As Artjom pointed out in their comment, a hybrid cryptosystem is used. This means a combination of symmetric and asymmetric encryption algorithms are used, as each has benefits and drawbacks. Symmetric encryption (such as AES) is extremely fast in comparison to asymmetric encryption (such as RSA). However, AES/CBC provides only confidentiality, not integrity, authentication, or non-repudiation. Asymmetric encryption can provide integrity, authentication, and non-repudiation. Asymmetric encryption also has fairly low limits for the amount of data it can encrypt (for example, an RSA-4096 bit key can encrypt at most 446 bytes at once). By combining both methods, we can exercise each for their strengths and mitigate the weaknesses of the other.
In this case, let our message be M. An AES/CBC key Kaes of length 256 bits is generated and used to encrypt the raw data such that the cipher text C is C = E(Kaes, M). The recipient's public key Krpub is then used to encrypt Kaes as C' = E(Krpub, Kaes). The cost of encrypting this small amount of data is relatively low, even though we are using an expensive operation.
Note prior to encrypting the ephemeral session key, it is likely signed using the sender's private key Kspriv unless digital signing is disabled. I am not 100% certain with S/MIME, but this is how PGP works, as when sending to multiple recipients, it would be much more expensive to encrypt the session key with n recipient's public keys and then sign each encrypted key with the sender's private key O(2n) vs. O(n+1) (not really Big-O notation, but effective for communicating the point). The signature of the session key is the same regardless of recipient because it depends only on the sender's private key.
Now all that is left is to concatenate the encrypted session key C' and the cipher text C and transmit them to the recipient. In a real S/MIME message, the identifying information about the sender's public key is transmitted as well in order to facilitate signature verification. The recipient parses the complete encrypted message, decrypts the session key using the recipient's private key Krpriv and then uses the session key Kaes to decrypt the cipher text C.
However, it seems like all of this may be overkill for your situation. If you do not need to integrate with an email communication system and are simply trying to encrypt large files for storage, openssl also offers a simple enc command. You can use it as follows:
Password-based encryption:
$ openssl enc -aes-256-cbc -in large_plain_file.dmg -out large_encrypted_file.enc -k thisIsABadPassword
-k allows you to enter a password on the command line from which the key will be derived. OpenSSL uses a deterministic key derivation function (KDF) (effectively key, iv = MD5(password||salt)). You can run this command with -p on the end to get the key, salt, and IV printed to the console. The IV is deterministically derived from the password and salt. A KDF like PBKDF2 is recommended but the library function in OpenSSL does not expose it to the command-line tool for some reason.
Warning you can run this and specify -nosalt to skip the salt generation, but this is not recommended as it makes an already extremely weak KDF even weaker.
Keyed encryption:
$ openssl enc -aes-256-cbc -in large_plain_file.dmg -out large_encrypted_file.enc K 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -IV 0123456789ABCDEFFEDCBA9876543210
Running with the actual key and IV provided. The key is 32 bytes (256 bits) and the IV is 16 bytes (the size of one AES block).
To decrypt the data, run the command with the -d flag:
$openssl enc -aes-256-cbc -d -in large_encrypted_file.dmg -out large_decrypted_file.enc
This is going to be simpler for you (no certificates/key pairs needed) and faster (no RSA encryption).
Something to keep in mind for all of this is that the password/key will be kept in terminal history if you provide it in the command invocation. Running without -k or -K will prompt the password to be entered in a secure prompt. Or use -pass to read from an environment variable, file, or file descriptor.
Update to address original questions explicitly:
The AES key is responsible for encrypting the data. The RSA key is used to encrypt the AES key.
Both impact the performance. The AES key is encrypting much more data, but is much faster than RSA encryption.
Yes, changing the size of the RSA key impacts performance, (higher key sizes are slower), but for a large file this is unlikely to be the limiting factor on performance.
Yes, changing the AES mode of operation may have a substantial impact on performance. AES/CBC, AES/OFB, and AES/CFB are similar in that they are serial operations (depending on the result of one block operation to proceed to the next), but AES/CTR operates as a stream cipher and can be parallelized to provide both encryption and decryption.
See answer above.