I want to decrypt fmp4 segment.
This segment was encrypt with HLS Apple Tools (https://developer.apple.com/documentation/http_live_streaming/about_apple_s_http_live_streaming_tools)
METHOD is AES-128
IV is 1d48fc5dee84b5a3e9a428f055e03c2e
I have a key and IV (you can got the key, and segment in google drive https://drive.google.com/drive/folders/1xF-C9EXFvT8qjI--sBB6QMPn8cNW7L-D?usp=sharing)
To decrypt I use Poco library.
This is my code:
Poco::Crypto::Cipher::ByteVec readKey(const std::string& uri) {
Poco::Crypto::Cipher::ByteVec key;
auto stream = Stream::makeStream(uri);
if (stream->open(uri, {})) {
key.resize(KEY_SIZE);
stream->read((char*)&key[0], KEY_SIZE);
}
return key;
}
std::vector<uint8_t> _key = readKey("./unit-tests/resources/cipher-stream/file.key");
std::string ivSrc = "1d48fc5dee84b5a3e9a428f055e03c2e";
Poco::Crypto::Cipher::ByteVec iv {ivSrc.begin(), ivSrc.end()};
Poco::Crypto::CipherKey key("aes-128-cbc", _key, iv);
Poco::Crypto::Cipher::Ptr cipher = Poco::Crypto::CipherFactory::defaultFactory().createCipher(key);
Poco::FileInputStream src("./unit-tests/resources/cipher-stream/fileSequence1.m4s");
Poco::FileOutputStream dst("./unit-tests/resources/cipher-stream/fileSequence1_dec.m4s");
Poco::Crypto::CryptoOutputStream decryptor(dst, cipher->createDecryptor());
Poco::StreamCopier::copyStream(src, decryptor);
// decryptor.close();
src.close();
dst.close();
Problem description:
After decryption I got distorted data. You can see this at the beginning of the file. Please see picture below. On the right side of the image file is distorted.
The correct data you can see on the left side.
You're using the wrong IV; that will lead to the first block (16 bytes) being corrupted. Your IV hex value is 1d48fc5dee84b5a3e9a428f055e03c2e, but you're interpreting that as ASCII. It's using the first 16 bytes of your string and ignoring the rest.
I haven't used Poco in a long time and don't remember if there's a hex parser handy, but that's what you need. Or write the IV directly in hex rather than as an ASCII string.
I'm trying to store binary data in a QR code. Apparently QR codes do support storing raw binary data (or ISO-8859-1 / Latin1). Here is what I want to encode (hex):
d1 50 01 00 00 00 f6 5f 05 2d 8f 0b 40 e2 01
I've tried the following encoders:
qr.js
Google Charts
qrcode.js
Decoding with zxing.org produces various incorrect results. The two javascript ones produce this (it's wrong; the first text character should be Ñ.
Whereas Google Charts produces this...
What is going on? Are any of these correct? What's really weird is that if I encode this sequence (with the JS ones at least) then it works fine - I would have thought the issue was non-ASCII characters but Ñ (0xd1) is non-ASCII.
d1 50 01 00 00 00 01 02 03 04 05 06 40 e2 01
Does anyone know what is going on?
Update
It occurred to me to try scanning them with a ZBar-based scanner app I found. It scans both JS versions ok (at least they start with ÑP). The Google Charts one is just wrong. So it seems like the issue is with ZXing (which is surprisingly shit - I wouldn't recommend it to anyone).
Update 2
ZBar can't handle null bytes. :-(
"What is going on? Are any of these correct?"
Except for the google chart (which is just empty), your QR codes are correct.
You can see the binary data from zxing is what you would expect:
4: Byte mode indicator
0f: length of 15 byte
d15001...: your 15 bytes of data
ec11 is just padding
The problem comes from the decoding. Because most decoders will try to interpret it as text. But since it's binary data, you should not try to handle it as text. Even if you think you can convert it from text to binary, as you saw this may cause issues with values which are not valid text.
So the solution is to use a decoder that will output you the binary data, and not text data.
Now about interpreting the QR code binary data as text, you said the first character should be 'Ñ' which is true if interpreted it as "ISO-8859-1",
which according to the QR code standard, is what should be done when there is no ECI mode defined.
But in practice, most smartphone QR code reader will interpret it as UTF-8 in this case (or at least try to auto-detect the encoding).
Even though this is not the standard, this had become common practice:
binary mode with no ECI, UTF-8 encoded text.
Maybe the reason behind it is that no one wants to waste these precious bytes adding an ECI mode specifying UTF-8. And actually, not all decoders support ECI.
There are two issues that you have to overcome to store binary data in QR codes.
ISO-8859-1 does not allow bytes in ranges of 00-1F and 7F-9F. If you
need to encode these bytes anyway, quote or encode them, i.e. use
quoted-printable or Base-64 encoding to avoid these ranges.
Since you are trying to store binary data in QR codes, you have to
rely only on your own scanner that will handle this binary data. You
don’t have to display text from your QR codes by other software,
like web application at zxing.org, because most QR decoders,
including that of zxing.org use heuristics to detect the character
set used. These heuristics may detect a character set other than
ISO-8859-1 and thus fail to properly display your binary data. Some
scanners use heuristics to detect a character set even if the
character set is explicitly given by ECI. This is why providing ECI
may not help much – scanners still use heuristics even with ECI.
So, using US-ASCII printable characters only (e.g., binary data encoded in Base64 before passing it to a QR Code generator) is the safest choice for QR code against the heuristics. This will also overcome another complication: that ISO-8859-1 was not the default encoding in earlier QR code standard published in 2000 (ISO/IEC 18004:2000). That standard did specify 8-bit Latin/Kana character set in accordance with JIS X 0201 (JIS8 also known as ISO-2022-JP) as default encoding for 8-bit mode, while the updated standard published in 2005 did change the default to ISO-8859-1.
As an alternative to Base-64, you can encode each byte with two hexadecimal characters (0-9, A-F), so, in the QR code your data will be encoded in the alphanumeric mode, not in 8-bit mode. This will disable all heuristics for sure and should not produce larger QR Code than with Base-64, because each character in the alphanumeric mode takes only 6 bits in the QR code stream.
Update:
I recently went back and published the referenced code as a project on GitHub for anyone who wants to use it.
https://github.com/yurelle/Base45Encoder
This is a bit necro, but I just hit this problem, and figured out a solution.
The problem with reading QR Codes with ZXING is that it assumes all QR Payloads are Strings. If you're willing to generate the QR Code in java with ZXING, I developed a solution which enables storing a binary payload in ZXING QR Codes with a storage efficiently loss of only -8%; better than the 33% inflation from Base64.
It exploits an internal compression optimization of the ZXING library based around pure Alphanum Strings. If you want a full explanation, with math and Unit Tests, check out my other answer.
But the short answer is this:
Solution
I implemented it as a self-contained static utility class, so all you have to do is call:
//Encode
final byte[] myBinaryData = ...;
final String encodedStr = BinaryToBase45Encoder.encodeToBase45QrPayload(myBinaryData);
//Decode
final byte[] decodedBytes = BinaryToBase45Encoder.decodeBase45QrPayload(encodedStr);
Alternatively, you can also do it via InputStreams:
//Encode
final InputStream in_1 = ... ;
final String encodedStr = BinaryToBase45Encoder.encodeToBase45QrPayload(in_1);
//Decode
final InputStream in_2 = ... ;
final byte[] decodedBytes = BinaryToBase45Encoder.decodeBase45QrPayload(in_2);
Here's the implementation
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;
/**
* For some reason none of the Java QR Code libraries support binary payloads. At least, none that
* I could find anyway. The commonly suggested workaround for this is to use Base64 encoding.
* However, this results in a 33% payload size inflation. If your payload is already near the size
* limit of QR codes, this is a lot.
*
* This class implements an encoder which takes advantage of a built-in compression optimization
* of the ZXING QR Code library, to enable the storage of Binary data into a QR Code, with a
* storage efficiency loss of only -8%.
*
* The built-in optimization is this: ZXING will automatically detect if your String payload is
* purely AlphaNumeric (by their own definition), and if so, it will automatically compress 2
* AlphaNumeric characters into 11 bits.
*
*
* ----------------------
*
*
* The included ALPHANUMERIC_TABLE is the conversion table used by the ZXING library as a reverse
* index for determining if a given input data should be classified as alphanumeric.
*
* See:
*
* com.google.zxing.qrcode.encoder.Encoder.chooseMode(String content, String encoding)
*
* which scans through the input string one character at a time and passes them to:
*
* getAlphanumericCode(int code)
*
* in the same class, which uses that character as a numeric index into the the
* ALPHANUMERIC_TABLE.
*
* If you examine the values, you'll notice that it ignores / disqualifies certain values, and
* effectively converts the input into base 45 (0 -> 44; -1 is interpreted by the calling code
* to mean a failure). This is confirmed in the function:
*
* appendAlphanumericBytes(CharSequence content, BitArray bits)
*
* where they pack 2 of these base 45 digits into 11 bits. This presents us with an opportunity.
* If we can take our data, and convert it into a compatible base 45 alphanumeric representation,
* then the QR Encoder will automatically pack that data into sub-byte chunks.
*
* 2 digits in base 45 is 2,025 possible values. 11 bits has a maximum storage capacity of 2,048
* possible states. This is only a loss of 1.1% in storage efficiency behind raw binary.
*
* 45 ^ 2 = 2,025
* 2 ^ 11 = 2,048
* 2,048 - 2,025 = 23
* 23 / 2,048 = 0.01123046875 = 1.123%
*
* However, this is the ideal / theoretical efficiency. This implementation processes data in
* chunks, using a Long as a computational buffer. However, since Java Long's are singed, we
* can only use the lower 7 bytes. The conversion code requires continuously positive values;
* using the highest 8th byte would contaminate the sign bit and randomly produce negative
* values.
*
*
* Real-World Test:
*
* Using a 7 byte Long to encode a 2KB buffer of random bytes, we get the following results.
*
* Raw Binary Size: 2,048
* Encoded String Size: 3,218
* QR Code Alphanum Size: 2,213 (after the QR Code compresses 2 base45 digits to 11 bits)
*
* This is a real-world storage efficiency loss of only 8%.
*
* 2,213 - 2,048 = 165
* 165 / 2,048 = 0.08056640625 = 8.0566%
*/
public class BinaryToBase45Encoder {
public final static int[] ALPHANUMERIC_TABLE;
/*
* You could probably just copy & paste the array literal from the ZXING source code; it's only
* an array definition. But I was unsure of the licensing issues with posting it on the internet,
* so I did it this way.
*/
static {
final Field SOURCE_ALPHANUMERIC_TABLE;
int[] tmp;
//Copy lookup table from ZXING Encoder class
try {
SOURCE_ALPHANUMERIC_TABLE = com.google.zxing.qrcode.encoder.Encoder.class.getDeclaredField("ALPHANUMERIC_TABLE");
SOURCE_ALPHANUMERIC_TABLE.setAccessible(true);
tmp = (int[]) SOURCE_ALPHANUMERIC_TABLE.get(null);
} catch (NoSuchFieldException e) {
e.printStackTrace();//Shouldn't happen
tmp = null;
} catch (IllegalAccessException e) {
e.printStackTrace();//Shouldn't happen
tmp = null;
}
//Store
ALPHANUMERIC_TABLE = tmp;
}
public static final int NUM_DISTINCT_ALPHANUM_VALUES = 45;
public static final char[] alphaNumReverseIndex = new char[NUM_DISTINCT_ALPHANUM_VALUES];
static {
//Build AlphaNum Index
final int len = ALPHANUMERIC_TABLE.length;
for (int x = 0; x < len; x++) {
// The base45 result which the alphanum lookup table produces.
// i.e. the base45 digit value which String characters are
// converted into.
//
// We use this value to build a reverse lookup table to find
// the String character we have to send to the encoder, to
// make it produce the given base45 digit value.
final int base45DigitValue = ALPHANUMERIC_TABLE[x];
//Ignore the -1 records
if (base45DigitValue > -1) {
//The index into the lookup table which produces the given base45 digit value.
//
//i.e. to produce a base45 digit with the numeric value in base45DigitValue, we need
//to send the Encoder a String character with the numeric value in x.
alphaNumReverseIndex[base45DigitValue] = (char) x;
}
}
}
/*
* The storage capacity of one digit in the number system; i.e. the maximum
* possible number of distinct values which can be stored in 1 logical digit
*/
public static final int QR_PAYLOAD_NUMERIC_BASE = NUM_DISTINCT_ALPHANUM_VALUES;
/*
* We can't use all 8 bytes, because the Long is signed, and the conversion math
* requires consistently positive values. If we populated all 8 bytes, then the
* last byte has the potential to contaminate the sign bit, and break the
* conversion math. So, we only use the lower 7 bytes, and avoid this problem.
*/
public static final int LONG_USABLE_BYTES = Long.BYTES - 1;
//The following mapping was determined by brute-forcing -1 Long (all bits 1), and compressing to base45 until it hit zero.
public static final int[] BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION = new int[] {0,2,3,5,6,8,9,11,12};
public static final int NUM_BASE45_DIGITS_PER_LONG = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION[LONG_USABLE_BYTES];
public static final Map<Integer, Integer> BASE45_TO_BINARY_DIGIT_COUNT_CONVERSION = new HashMap<>();
static {
//Build Reverse Lookup
int len = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION.length;
for (int x=0; x<len; x++) {
int numB45Digits = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION[x];
BASE45_TO_BINARY_DIGIT_COUNT_CONVERSION.put(numB45Digits, x);
}
}
public static String encodeToBase45QrPayload(final byte[] inputData) throws IOException {
return encodeToBase45QrPayload(new ByteArrayInputStream(inputData));
}
public static String encodeToBase45QrPayload(final InputStream in) throws IOException {
//Init conversion state vars
final StringBuilder strOut = new StringBuilder();
int data;
long buf = 0;
// Process all input data in chunks of size LONG.BYTES, this allows for economies of scale
// so we can process more digits of arbitrary size before we hit the wall of the binary
// chunk size in a power of 2, and have to transmit a sub-optimal chunk of the "crumbs"
// left over; i.e. the slack space between where the multiples of QR_PAYLOAD_NUMERIC_BASE
// and the powers of 2 don't quite line up.
while(in.available() > 0) {
//Fill buffer
int numBytesStored = 0;
while (numBytesStored < LONG_USABLE_BYTES && in.available() > 0) {
//Read next byte
data = in.read();
//Push byte into buffer
buf = (buf << 8) | data; //8 bits per byte
//Increment
numBytesStored++;
}
//Write out in lower base
final StringBuilder outputChunkBuffer = new StringBuilder();
final int numBase45Digits = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION[numBytesStored];
int numB45DigitsProcessed = 0;
while(numB45DigitsProcessed < numBase45Digits) {
//Chunk out a digit
final byte digit = (byte) (buf % QR_PAYLOAD_NUMERIC_BASE);
//Drop digit data from buffer
buf = buf / QR_PAYLOAD_NUMERIC_BASE;
//Write Digit
outputChunkBuffer.append(alphaNumReverseIndex[(int) digit]);
//Track output digits
numB45DigitsProcessed++;
}
/*
* The way this code works, the processing output results in a First-In-Last-Out digit
* reversal. So, we need to buffer the chunk output, and feed it to the OutputStream
* backwards to correct this.
*
* We could probably get away with writing the bytes out in inverted order, and then
* flipping them back on the decode side, but just to be safe, I'm always keeping
* them in the proper order.
*/
strOut.append(outputChunkBuffer.reverse().toString());
}
//Return
return strOut.toString();
}
public static byte[] decodeBase45QrPayload(final String inputStr) throws IOException {
//Prep for InputStream
final byte[] buf = inputStr.getBytes();//Use the default encoding (the same encoding that the 'char' primitive uses)
return decodeBase45QrPayload(new ByteArrayInputStream(buf));
}
public static byte[] decodeBase45QrPayload(final InputStream in) throws IOException {
//Init conversion state vars
final ByteArrayOutputStream out = new ByteArrayOutputStream();
int data;
long buf = 0;
int x=0;
// Process all input data in chunks of size LONG.BYTES, this allows for economies of scale
// so we can process more digits of arbitrary size before we hit the wall of the binary
// chunk size in a power of 2, and have to transmit a sub-optimal chunk of the "crumbs"
// left over; i.e. the slack space between where the multiples of QR_PAYLOAD_NUMERIC_BASE
// and the powers of 2 don't quite line up.
while(in.available() > 0) {
//Convert & Fill Buffer
int numB45Digits = 0;
while (numB45Digits < NUM_BASE45_DIGITS_PER_LONG && in.available() > 0) {
//Read in next char
char c = (char) in.read();
//Translate back through lookup table
int digit = ALPHANUMERIC_TABLE[(int) c];
//Shift buffer up one digit to make room
buf *= QR_PAYLOAD_NUMERIC_BASE;
//Append next digit
buf += digit;
//Increment
numB45Digits++;
}
//Write out in higher base
final LinkedList<Byte> outputChunkBuffer = new LinkedList<>();
final int numBytes = BASE45_TO_BINARY_DIGIT_COUNT_CONVERSION.get(numB45Digits);
int numBytesProcessed = 0;
while(numBytesProcessed < numBytes) {
//Chunk out 1 byte
final byte chunk = (byte) buf;
//Shift buffer to next byte
buf = buf >> 8; //8 bits per byte
//Write byte to output
//
//Again, we need to invert the order of the bytes, so as we chunk them off, push
//them onto a FILO stack; inverting their order.
outputChunkBuffer.push(chunk);
//Increment
numBytesProcessed++;
}
//Write chunk buffer to output stream (in reverse order)
while (outputChunkBuffer.size() > 0) {
out.write(outputChunkBuffer.pop());
}
}
//Return
out.flush();
out.close();
return out.toByteArray();
}
}
Just at a glance, the qr formats are different. I'd compare the qr formats to see if it's a problem of error correction or encoding or something else.
It turned out that ZXing is just crap, and ZBar does some weird stuff with the data (converting it to UTF-8 for example). I managed to get it to output the raw data including null bytes though. Here is a patch for the best Android ZBar library I found, that has now been merged.
I used System.Convert.ToBase64String to convert the supplied sample byte array into a Base64-encoded string, then I used ZXing to create a QRCode image.
Next I called ZXing to read the string back from the generated QRCode, and then called System.Convert.FromBase64String to convert the string back into a byte array.
I confirm that the data completed the round trip successfully.
The informational RFC 9285 - The Base45 Data Encoding document describing the optimal scheme for storing binary data within the constraints of QR Alphanumeric Mode was recently published by the IETF.
(one positive side-effect of ongoing standardization work surrounding Health Certificate QR-codes)
Background
I've been struggling with decrypting an apparently well-formed cipher text for about a day. Assume we've got the following hex-encoded cipher text which contains exactly 160 characters thereby having 80 bytes.
QString c = "1BFAC407AF0D440A2D6176C0B5D125AA96088490299AC18C74623C0EF1BB1372E554FC4150A8066220E943697BE2491D8AE13AA036B298425AC510A8A917D59EBB69708B9040AB3A84C63043EAD4AB07";
QString k = CryptoUtils::hexEncode("abc");
QString p = CryptoUtils::decrypt(c, k);
qDebug() << p;
Provided we're using AES 256, AFAIK, the key must be of length 32 bytes and cipher text of a length of multiple of 16 bytes, which all these consditions are met regarding my snippet code.
Please note that I'm using SHA256 feeding with a pass phrase to generate a 32 bytes key. So, this ensures that all keys are of length 32 bytes.
Full source codes of those function can be found on my repo on GitHub (at branch Part1).
My Question
When I want to run this code, my app crashes. Here's the exception:
terminate called after throwing an instance of 'CryptoPP::InvalidCiphertext'
what(): StreamTransformationFilter: invalid PKCS #7 block padding found
The program has unexpectedly finished.
I searched around about this problem and figured out it could be because of the trailing \0 once you encrypted the plain text. However, I couldn't just solve the problem. Please help me out, it's just driving me crazy.
Full source codes of those function can be found on my repo on GitHub
I'd make these changes at minimum:
QString CryptoUtils::encrypt(QString text, QString keyhex)
{
...
// name the variable, kill the memory leak
SHA256 sha256;
StringSource ss1(decodedKey, size, true, new HashFilter(sha256, new ArraySink(key, AES::MAX_KEYLENGTH)));
...
// name the variable
StringSource ss2(plain, true, new StreamTransformationFilter(Encryptor, new HexEncoder(new StringSink(encrypted))));
// verify embedded NULLs don't affect results
QString qs = QString::fromStdString(encrypted);
assert(qs.length() == encrypted.length());
}
And:
QString CryptoUtils::decrypt(QString text, QString keyhex)
{
// bad karma here...
string encrypted = text.toStdString();
assert(encrypted.length() == text.length());
...
// name the variable, kill the memory leak
SHA256 sha256;
StringSource ss1(decodedKey, size, true, new HashFilter(sha256, new ArraySink(key, AES::MAX_KEYLENGTH)));
...
// name the variable,
StringSource ss2(encrypted, true, new HexDecoder(new StreamTransformationFilter(Decryptor, new StringSink(plain))));
// verify embedded NULLs don't affect results
QString qs = QString::fromStdString(plain);
assert(qs.length() == plain.length());
}
The hexEncode function seems to misbehave:
QString CryptoUtils::hexEncode(QString text)
{
byte *bytearray = (byte *) text.toLatin1().data();
int length = text.toLatin1().length();
return hexEncode(bytearray, length);
}
Should be replaced with:
QString CryptoUtils::hexEncode(QString text)
{
byte *bytearray = (byte *) text.toStdString().data();
int length = text.length();
return hexEncode(bytearray, length);
}
For public key encryption and diffie-hellman in libsodium, I typically make private keys simply by generating 32 random bytes with randombytes_buf and then derive the public key (when needed) using crypto_scalarmult_base.
Is there any benefit to using crypto_box_keypair to generate a keypair (other than syntax)? Or does this function basically do exactly that?
This is exactly what the crypto_box_keypair() function does.
The benefits of this function are clarity, and guarantee that the secret key is properly generated.
https://download.libsodium.org/doc/public-key_cryptography/public-key_signatures.html
for example:
unsigned char pk[crypto_sign_PUBLICKEYBYTES]; //Variable declarations
unsigned char sk[crypto_sign_SECRETKEYBYTES]; Variable declarations
crypto_sign_keypair(pk, sk);
NSData *privateKeyData = [NSData dataWithBytes:sk length:crypto_box_SECRETKEYBYTES];
NSData *publicKeyData = [NSData dataWithBytes:pk length:crypto_box_PUBLICKEYBYTES];
NSLog(#"%#",privateKeyData); // target publick key data and secret key data
NSLog(#"%#",publicKeyData);
//Other
NSLog(#"%s\n\n=====\n\n\n%s",pk,sk); //(nullable const void *)bytes
Byte *byte = (Byte *)[publicKeyData bytes];
NSLog(#"%s",byte);
I want to store a password (no hash) on my disk. it's nothing sensitive but i just don't want it in plaintext on my disk.
what i tried till now is:
converting the string in binary and XOR it with the binary of a key.
bool ok = true;
QByteArray qbaPW("mypass");
long long intPW = qbaPW.toHex().toLongLong( &ok, 16 );
QString binPW = QString::number( intPW, 2);
but the thing is, that it only works with short passwords. if they are too long intPW gets too big for longlong. any ideas how can avoid that thing?
cheers
A QByteArray is like a char array[len] in C. You can access individual members and do whatever you please with them. For example:
QByteArray const key("mykey");
QByteArray password("password");
for (int ik = 0, ip = 0;
ip < password.length();
++ ip, ik = (ik+1 < key.length() ? ik+1 : 0)) {
password[ip] = password[ip] ^ key[ik];
}
Since this just XORs with the key, you repeat this procedure to decrypt the password. A good key will be generated randomly and will be longer than the longest password you envisage (say 64 characters).
Do note that this method is only reasonably safe if the users are explicitly informed not to reuse any other password in your application - otherwise you're essentially leaking passwords that are supposed to be secure.