I just start learning Bouncy Castle for AES encryption/decryption. I am using AES/CBC/PKCS7PADDING with 256-bit key.
BC can encrypt and decrypt text successfully, however after decryption I notice that there are always a few padding of null (0x00), which therefore fails my hash comparison. For example, suppose original input string is “1234567890”, the decrypted byte array is always:
{0x49,0x50,0x51,0x52,0x53,0x54,0x55,0x56,0x57,0x48,0x00,0x00,0x00,0x00,0x00,0x00}
Why the padding is not 0x06,0x06,0x06,0x06,0x06,0x06? And is there any way to deterministically tell the padding length (could be 0) after encryption so that I can get exactly the same string before encryption?
When you specify PKCS7, BC will add the padding to the data before encrypting, and remove it again when decrypting. PKCS7 with AES would always add at least 1 byte of padding, and will add enough data to make the input a multiple of the AES block size. When decrypting the padding is verified to be correct, and in the case of PKCS7 also serve as an indicator of how much of the last block of decrypted data is padding, and how much is real data.
If you try decrypting the encrypted and padded data without specifying PKCS7 in the decrypt step, the padding would still be in the decrypted data.
Edit:
To illustrate my point .. here is some Java code that encrypts "1234567890" with AES/CBC/PKCS7, and then decrypts it again both with and without the PKCS7 padding:
public class BCTest {
public static void doTest() throws Exception {
Security.addProvider(new BouncyCastleProvider());
byte[] clearData = "1234567890".getBytes();
SecretKey secretKey = new SecretKeySpec("0123456789ABCDEF".getBytes(), "AES");
AlgorithmParameterSpec IVspec = new IvParameterSpec("0123456789ABCDEF".getBytes());
// encrypt with PKCS7 padding
Cipher encrypterWithPad = Cipher.getInstance("AES/CBC/PKCS7PADDING", "BC");
encrypterWithPad.init(Cipher.ENCRYPT_MODE, secretKey, IVspec);
byte[] encryptedData = encrypterWithPad.doFinal(clearData);
System.out.println("Encryped data (" + encryptedData.length + " bytes): \t" + toHexString(encryptedData));
// decrypt with PKCS7 pad
Cipher decrypterWithPad = Cipher.getInstance("AES/CBC/PKCS7PADDING", "BC");
decrypterWithPad.init(Cipher.DECRYPT_MODE, secretKey, IVspec);
byte[] buffer1 = new byte[encryptedData.length];
int decryptLen1 = decrypterWithPad.doFinal(encryptedData, 0, encryptedData.length, buffer1);
System.out.println("Decrypted with Pad (" + decryptLen1 + " bytes): \t" + toHexString(buffer1));
// decrypt without PKCS7 pad
Cipher decrypterWithoutPad = Cipher.getInstance("AES/CBC/NOPADDING", "BC");
decrypterWithoutPad.init(Cipher.DECRYPT_MODE, secretKey, IVspec);
byte[] buffer2 = new byte[encryptedData.length];
int decryptLen2 = decrypterWithoutPad.doFinal(encryptedData, 0, encryptedData.length, buffer2);
System.out.println("Decrypted without Pad (" + decryptLen2 + " bytes):\t" + toHexString(buffer2));
}
private static String toHexString(byte[] bytes) {
return javax.xml.bind.DatatypeConverter.printHexBinary(bytes);
}
public static void main(String[] args) throws Exception {
BCTest.doTest();
}
}
Output:
Encryped data (16 bytes): 602CAE14358D0AC5C96E2D46D17E58E3
Decrypted with Pad (10 bytes): 31323334353637383930000000000000
Decrypted without Pad (16 bytes): 31323334353637383930060606060606
When decrypting with the padding option, the output have been striped of the padding - and the cipher indicates 10 bytes of decrypted data - the rest of the buffer is 0 filled. Decrypting without the padding option, results in the padding now being part of the decrypted data.
Edit2:
Now seeing the original code, confirms my hunch. The methode GetOutputSize don't return the output size of the decrypted string, but only the maximum needed space in an output buffer. The methode have the following documentation in the BC code:
/**
* return the size of the output buffer required for an update plus a
* doFinal with an input of len bytes.
*
* #param len the length of the input.
* #return the space required to accommodate a call to update and doFinal
* with len bytes of input.
*/
DoFinal returns the actual length of the decrypted data put in the buffer.
So in
byte[] plainTextBuffer = new byte[cipher.GetOutputSize(data.Length - IV_LENGTH)];
int length = cipher.DoFinal(data, iv.Length, data.Length - iv.Length, plainTextBuffer, 0);
The plainTextBuffer would be slightly larger than the actual decrypted data - the actual length of data would be in length.
i am using c# from bouncycastle. looks to me this might be a bug from bouncycastle, or at least bouncycastle c# implementation does not follow pkcs7 spec exactly.
my solution is to chop off the trailing bytes that are not included in the return length of DoFinal. still not very sure why there are padding of 0x00, which as said should not exist at all.
below is the code. i used AES/CBC/PKCS7PADDING for both encryption and decryption.
encryption --->
ICipherParameters keyParams = ParameterUtilities.CreateKeyParameter("AES", keyByte);
ICipherParameters aesIVKeyParam = new ParametersWithIV(keyParams, StringToByteArray(IV_STRING));
byte[] iv = ((ParametersWithIV) aesIVKeyParam).GetIV();
IBufferedCipher cipher = CipherUtilities.GetCipher("AES/CBC/PKCS7PADDING");
cipher.Init(true, aesIVKeyParam);
byte[] cipherText = new byte[iv.Length + cipher.GetOutputSize(data.Length)];
Array.Copy(iv, 0, cipherText, 0, iv.Length);
int length = cipher.DoFinal(data, 0, data.Length, cipherText, iv.Length);
decryption --->
ICipherParameters keyParams = ParameterUtilities.CreateKeyParameter("AES", keyByte);
byte[] iv = new byte[IV_LENGTH];
Array.Copy(data, 0, iv, 0, IV_LENGTH);
ICipherParameters aesIVKeyParam = new ParametersWithIV(keyParams, iv);
IBufferedCipher cipher = CipherUtilities.GetCipher("AES/CBC/PKCS7PADDING");
cipher.Init(false, aesIVKeyParam);
byte[] plainTextBuffer = new byte[cipher.GetOutputSize(data.Length - IV_LENGTH)];
int length = cipher.DoFinal(data, iv.Length, data.Length - iv.Length, plainTextBuffer, 0);
Related
In the (PK)ZIP specification at https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT, specifically in the Strong Encryption Specification (SES) section, there is a line on deriving a key from a password:
MasterSessionKey = DeriveKey(SHA1(Password))
What's DeriveKey?
(In WinZip's AES documentation at https://www.winzip.com/en/support/aes-encryption/, they use PBKDF2 with 1000 iterations. I don't see any similar explanation in APPNOTE)
PKWARE implemented a strong encryption in version 5, but did not provide the algorithm of encoding/decoding (Method For Strongly Encrypted .ZIP Files - Patent US 2020/0250329 A1). In this algorithm AES encryption was implemented as part of it. You can define this by strong encryption (bit 6) = yes in General Purpose Flag.
After that WinZip could not use this algo, so it invented another one. You can define this by strong encryption (bit 6) = no in General Purpose Flag and AesExtraFieldRecord with signature 0x990.
As you can see there're two ways to encrypt a zip file. All open source software use the second one. The first one is available only by PKWARE SecureZIP
You can find example of this alogirthm in (7zip) Strong.cpp:35. In java it should look like this:
public static byte[] getMasterKey(String password) {
byte[] data = password.getBytes(StandardCharsets.UTF_8);
byte[] sha1 = DigestUtils.sha1(data);
return DeriveKey(sha1);
}
private static byte[] DeriveKey(byte[] digest) {
byte[] buf = new byte[kDigestSize * 2]; // kDigestSize = 20
DeriveKey2(digest, (byte)0x36, buf, 0);
DeriveKey2(digest, (byte)0x5C, buf, kDigestSize);
return Arrays.copyOfRange(buf, 0, 32);
}
private static void DeriveKey2(byte[] digest, byte c, byte[] dest, int offs) {
byte[] buf = new byte[64];
Arrays.fill(buf, c);
for (int i = 0; i < kDigestSize; i++)
buf[i] ^= digest[i];
byte[] sha1 = DigestUtils.sha1(buf);
System.arraycopy(sha1, 0, dest, offs, sha1.length);
}
Demo:
String password = "JohnDoe";
byte[] masterKey = getMasterKey(password);
The next paragraph 'defines' it
7.2.5.3 The function names and parameter requirements will depend on
the choice of the cryptographic toolkit selected. Almost any
toolkit supporting the reference implementations for each
algorithm can be used. The RSA BSAFE(r), OpenSSL, and Microsoft
CryptoAPI libraries are all known to work well.
I guess it's up to you to decide which of the encryption algorithms you want to use and go from there
I want to decrypt fmp4 segment.
This segment was encrypt with HLS Apple Tools (https://developer.apple.com/documentation/http_live_streaming/about_apple_s_http_live_streaming_tools)
METHOD is AES-128
IV is 1d48fc5dee84b5a3e9a428f055e03c2e
I have a key and IV (you can got the key, and segment in google drive https://drive.google.com/drive/folders/1xF-C9EXFvT8qjI--sBB6QMPn8cNW7L-D?usp=sharing)
To decrypt I use Poco library.
This is my code:
Poco::Crypto::Cipher::ByteVec readKey(const std::string& uri) {
Poco::Crypto::Cipher::ByteVec key;
auto stream = Stream::makeStream(uri);
if (stream->open(uri, {})) {
key.resize(KEY_SIZE);
stream->read((char*)&key[0], KEY_SIZE);
}
return key;
}
std::vector<uint8_t> _key = readKey("./unit-tests/resources/cipher-stream/file.key");
std::string ivSrc = "1d48fc5dee84b5a3e9a428f055e03c2e";
Poco::Crypto::Cipher::ByteVec iv {ivSrc.begin(), ivSrc.end()};
Poco::Crypto::CipherKey key("aes-128-cbc", _key, iv);
Poco::Crypto::Cipher::Ptr cipher = Poco::Crypto::CipherFactory::defaultFactory().createCipher(key);
Poco::FileInputStream src("./unit-tests/resources/cipher-stream/fileSequence1.m4s");
Poco::FileOutputStream dst("./unit-tests/resources/cipher-stream/fileSequence1_dec.m4s");
Poco::Crypto::CryptoOutputStream decryptor(dst, cipher->createDecryptor());
Poco::StreamCopier::copyStream(src, decryptor);
// decryptor.close();
src.close();
dst.close();
Problem description:
After decryption I got distorted data. You can see this at the beginning of the file. Please see picture below. On the right side of the image file is distorted.
The correct data you can see on the left side.
You're using the wrong IV; that will lead to the first block (16 bytes) being corrupted. Your IV hex value is 1d48fc5dee84b5a3e9a428f055e03c2e, but you're interpreting that as ASCII. It's using the first 16 bytes of your string and ignoring the rest.
I haven't used Poco in a long time and don't remember if there's a hex parser handy, but that's what you need. Or write the IV directly in hex rather than as an ASCII string.
I'm working on a project to decrypt an AES-128 encrypted string in ColdFusion that is passed as a URL parameter.
The vendor takes a pass phrase and converts it to a valid AES-128 key "using an algorithm equivalent to Microsoft's CryptDeriveKey using the SHA-1 hash function." I need to replicate this generatedKey in ColdFusion so I can use the value in my decrypt() call.
When using CryptDeriveKey you pass the encryption type, the Hash type, the block length and a 0 iv array and it returns the Hash. Source: Generating a Key from a Password
// generate an RC2 key
byte[] iv = new byte[] { 0, 0, 0, 0, 0, 0, 0, 0 };
byte[] key = cdk.CryptDeriveKey(“RC2”, “SHA1”, 128, iv);
In the vendor's test tool, the pass phrase "test1234" results in a Hash of:
A6455C7A24BC5E869B0DDF647238F5DA
I found the genAESKeyFromPW() UDF, which seems to be the closest, but requires a salt which CryptDeriveKey does not use. I have also tried the code below. However, it is not working as the Hash() is not creating a valid AES-128 key:
<cfset generatedKey = Hash('test1234', 'SHA-1')>
<cfset decrypted=decrypt(encryptedString, generatedKey, 'AES/CBC/PKCS7Padding', 'Base64', '0')>
What steps do I need to replicate CryptDeriveKey function?
Update:
The vendor provided this C# example of the decryption:
public static byte[] AesDecryptBytes(byte[] cipherText, byte[] key)
{
byte[] IV = new byte[16];
AesManaged aes = new AesManaged();
aes.Mode = CipherMode.CBC;
aes.Padding = PaddingMode.PKCS7;
ICryptoTransform decryptor = aes.CreateDecryptor(key, IV);
byte[] plainBytes;
using (MemoryStream memoryStream = new MemoryStream())
{
using (CryptoStream cryptoStream = new CryptoStream(memoryStream, decryptor, CryptoStreamMode.Write))
{
cryptoStream.Write(cipherText, 0, cipherText.Length);
}
plainBytes = memoryStream.ToArray();
}
return plainBytes;
}
From what I have read in this documentation, it sounds like the function essentially hashes the password binary then does an XOR with two arrays. I am not 100%, but I do not think the method they described is the same as PBKDF1 or PBKDF2.
Let n be the required derived key length, in bytes. The derived key is
the first n bytes of the hash value after the hash computation has
been completed by CryptDeriveKey. If the hash is not a member of the
SHA-2 family and the required key is for either 3DES or AES, the key
is derived as follows:
Form a 64-byte buffer by repeating the constant 0x36 64 times. Let k be the length of the hash value that is represented by the input
parameter hBaseData. Set the first k bytes of the buffer to the result
of an XOR operation of the first k bytes of the buffer with the hash
value that is represented by the input parameter hBaseData.
Form a 64-byte buffer by repeating the constant 0x5C 64 times. Set the first k bytes of the buffer to the result of an XOR operation of
the first k bytes of the buffer with the hash value that is
represented by the input parameter hBaseData.
Hash the result of step 1 by using the same hash algorithm as that used to compute the hash value that is represented by the hBaseData
parameter.
Hash the result of step 2 by using the same hash algorithm as that used to compute the hash value that is represented by the hBaseData
parameter.
Concatenate the result of step 3 with the result of step 4.
Use the first n bytes of the result of step 5 as the derived key.
CF Key Generation
Start by hashing the password and converting it into binary with binaryDecode:
hBaseData = binaryDecode(hash("test1234", "SHA1"), "hex");
Build and populate the two buffers with the specified constants:
// 0x36 (i.e. 54 decimal)
buff1 = listToArray(repeatString("54,", 64));
// 0x5C (i.e. 92 decimal)
buff2 = listToArray(repeatString("92,", 64));
Then do a bitwise XOR, storing the results in the buffer:
for (k = 1; k <= arrayLen(hBaseData); k++) {
buff1[k] = BitXOR( buff1[k], hBaseData[k]);
buff2[k] = BitXOR( buff2[k], hBaseData[k]);
}
Next hash() both buffers and concatenate the results:
hash1 = hash( javacast("byte[]", buff1), "SHA1");
hash2 = hash( javacast("byte[]", buff2), "SHA1");
combined = hash1 & hash2;
Finally, extract the first n bytes (16 == 128 bits / 8) as the new key. Since CF's hash() function returns hexadecimal (always two characters per byte) string functions can be used here.
keySize = 128 / 8;
newKey = left(combined, keySize *2);
Result: A6455C7A24BC5E869B0DDF647238F5DA
Decrypting in CF
Before you can decrypt, a few important notes:
CF's encrypt/decrypt functions expect keys to be encoded as base64. The generated key above is in hexadecimal format. So it must be converted first:
"PKCS7Padding" is not valid for CF/Java. Instead use PKCS5Padding.
"CBC" mode always requires an IV. The IV is a binary array whose length is the same as the algorithm's block size (AES block size = 16 bytes). It must be "...the same value [used to encrypt] to successfully decrypt the data." Based on the descriptions in your API, your IV should be all zeros. (This is not good in practice, but is okay for a test case).
For more details, see Strong encryption in ColdFusion
Example:
encrypted = "1lqcm0Jiy4Rs29tz2jpuoQ==";
newKeyHex = "A6455C7A24BC5E869B0DDF647238F5DA";
keyBase64 = binaryEncode(binaryDecode(newKeyHex, "hex"), "base64");
iv = javacast("byte[]", [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]);
decrypted = decrypt(encrypted, keyBase64, "AES/CBC/PKCS5Padding", "Base64", iv);
writeOutput("<br>"& decrypted);
Result: recordID=000001
Background
I've been struggling with decrypting an apparently well-formed cipher text for about a day. Assume we've got the following hex-encoded cipher text which contains exactly 160 characters thereby having 80 bytes.
QString c = "1BFAC407AF0D440A2D6176C0B5D125AA96088490299AC18C74623C0EF1BB1372E554FC4150A8066220E943697BE2491D8AE13AA036B298425AC510A8A917D59EBB69708B9040AB3A84C63043EAD4AB07";
QString k = CryptoUtils::hexEncode("abc");
QString p = CryptoUtils::decrypt(c, k);
qDebug() << p;
Provided we're using AES 256, AFAIK, the key must be of length 32 bytes and cipher text of a length of multiple of 16 bytes, which all these consditions are met regarding my snippet code.
Please note that I'm using SHA256 feeding with a pass phrase to generate a 32 bytes key. So, this ensures that all keys are of length 32 bytes.
Full source codes of those function can be found on my repo on GitHub (at branch Part1).
My Question
When I want to run this code, my app crashes. Here's the exception:
terminate called after throwing an instance of 'CryptoPP::InvalidCiphertext'
what(): StreamTransformationFilter: invalid PKCS #7 block padding found
The program has unexpectedly finished.
I searched around about this problem and figured out it could be because of the trailing \0 once you encrypted the plain text. However, I couldn't just solve the problem. Please help me out, it's just driving me crazy.
Full source codes of those function can be found on my repo on GitHub
I'd make these changes at minimum:
QString CryptoUtils::encrypt(QString text, QString keyhex)
{
...
// name the variable, kill the memory leak
SHA256 sha256;
StringSource ss1(decodedKey, size, true, new HashFilter(sha256, new ArraySink(key, AES::MAX_KEYLENGTH)));
...
// name the variable
StringSource ss2(plain, true, new StreamTransformationFilter(Encryptor, new HexEncoder(new StringSink(encrypted))));
// verify embedded NULLs don't affect results
QString qs = QString::fromStdString(encrypted);
assert(qs.length() == encrypted.length());
}
And:
QString CryptoUtils::decrypt(QString text, QString keyhex)
{
// bad karma here...
string encrypted = text.toStdString();
assert(encrypted.length() == text.length());
...
// name the variable, kill the memory leak
SHA256 sha256;
StringSource ss1(decodedKey, size, true, new HashFilter(sha256, new ArraySink(key, AES::MAX_KEYLENGTH)));
...
// name the variable,
StringSource ss2(encrypted, true, new HexDecoder(new StreamTransformationFilter(Decryptor, new StringSink(plain))));
// verify embedded NULLs don't affect results
QString qs = QString::fromStdString(plain);
assert(qs.length() == plain.length());
}
The hexEncode function seems to misbehave:
QString CryptoUtils::hexEncode(QString text)
{
byte *bytearray = (byte *) text.toLatin1().data();
int length = text.toLatin1().length();
return hexEncode(bytearray, length);
}
Should be replaced with:
QString CryptoUtils::hexEncode(QString text)
{
byte *bytearray = (byte *) text.toStdString().data();
int length = text.length();
return hexEncode(bytearray, length);
}
So, the question is how to correctly make a binary data of RSA *rsa key?
I do
RSA *rsa = RSA_generate_key(2048, 65537, 0, 0);
uchar *pp = (uchar *)malloc(2560);
int bytes = i2d_RSAPublicKey(rsa, &pp);
write(client, pp, bytes);
and reciever does d2i_RSAPublicKey and the program crashes. what i do wrong?
Don't malloc pp yourself, instead set it NULL. i2d_RSAPublicKey will do the malloc for you and set pp to point to the newly malloced public key.