Can someone please clear up a bit of MDC and data encryption for me? in rfc 4880, it says:
The plaintext of the data to be
encrypted is passed through the SHA-1
hash function, and the result of the
hash is appended to the plaintext in a
Modification Detection Code packet.
The input to the hash function
includes the prefix data described
above; it includes all of the
plaintext, and then also includes two
octets of values 0xD3, 0x14. These
represent the encoding of a
Modification Detection Code packet tag
and length field of 20 octets.
at first, it seems like the mdc (without its header data) is just: sha1([data]) -> hash_value
then the second sentence up to the semicolon makes it seem like sha1(OpenPGP_CFB_extra_data + [data]) -> hash_value
the stuff after the semicolon makes it seem like I am supposed to do sha1([data] + "\xd3\x14") -> hash_value. (this doesnt make sense at all, but it seems to be what is written)
what is going on?
after getting the correct MDC, what is done with it? is it its own packet, or something like this (according to my understanding) done?:
tag18_header + encrypt(plaintext + "\xd3\x14" + 20 byte hash)
After reading RFC 4880 and parts of the GnuPG source code (g10/cipher.c seems to be the place where this is handled), I interpret it is like this:
0xd3 is the MDC packet tag.
0x14 is the MDC packet length (20 bytes).
The MDC hash is computed like this:
MCD_hash = SHA-1(OpenPGP_CFB_extra_data + [plaintext] + "\xd3\x14")
Then this is appended to the plaintext message and encrypted:
encrypt(OpenPGP_CFB_extra_data + [plaintext] + "\xd3\x14" + MDC_hash)
When decrypted, this hash is verified by computing SHA-1 of everything but the last 20 bytes and comparing the result to the last 20 bytes, as RFC 4880 writes (page 50):
During decryption, the plaintext data should be hashed with SHA-1, including the prefix data as well as the packet tag and length field of the Modification Detection Code packet. The body of the MDC packet, upon decryption, is compared with the result of the SHA-1 hash.
Related
We are developing an application that has to work with data that is enycrpted by LoraWan (https://www.lora-alliance.org)
We have already found the documentation of how they encrypt their data, and have been reading through it for the past few days (https://www.lora-alliance.org/sites/default/files/2018-04/lorawantm_specification_-v1.1.pdf) but currently still can't solve our problem.
We have to use AES 128-bit ECB decryption with zero-padding to decrypt the messages, but the problem is it's not working because the encrypted messages we are receiving are not long enough for AES 128 so the algorithm returns a "Data is not a complete block" exception on the last line.
An example key we receive is like this: D6740C0B8417FF1295D878B130784BC5 (not a real key). It is 32 characters long, so 32 bytes, but if treat it as hexadecimal, then it becomes 16 bytes long, which is what is needed for AES 128-bit. This is the code we use to convert the Hex from String:
public static string HextoString(string InputText)
{byte[] hex= Enumerable.Range(0, InputText.Length)
.Where(x => x % 2 == 0)
.Select(x => Convert.ToByte(InputText.Substring(x, 2), 16))
.ToArray();
return System.Text.Encoding.ASCII.GetString(hex);}
(A small thing to note for the above code is that we are not sure what Encoding to use, as we could not find it in the Lora documentation and they have not told us, but depending on this small setting we could be messing up our decryption (though we have tried all possible combinations, ascii, utf8, utf7, etc))
An example message we receive is: d3 73 4c which we are assuming is also in hexadecimal. This is only 6 bytes, and 3 bytes if we convert it from hexa to normal, compared to the 16 bytes we'd need minimum to match the key length.
This is the code for Aes 128 decrypt we are using:
private static string Aes128Decrypt(string cipherText, string key){
string decrypted = null;
var cipherPlainTextBytes = HexStringToByteArray(cipherText);
//var cipherPlainTextBytes = ForcedZeroPadding(HexStringToByteArray(cipherText));
var keyBytes = HexStringToByteArray(key);
using (var aes = new AesCryptoServiceProvider())
{
aes.KeySize = 128;
aes.Key = keyBytes;
aes.Mode = CipherMode.ECB;
aes.Padding = PaddingMode.Zeros;
ICryptoTransform decryptor = aes.CreateDecryptor(aes.Key, aes.IV);
using (MemoryStream ms = new MemoryStream(cipherPlainTextBytes, 0, cipherPlainTextBytes.Length))
{
using (CryptoStream cs = new CryptoStream(ms, decryptor, CryptoStreamMode.Read))
{
using (StreamReader sr = new StreamReader(cs))
{
decrypted = sr.ReadToEnd();
}
}
}
}
return decrypted;}
So obviously this is going to return "Data is an incomplete block" at sr.ReadToEnd().
As you can see from the example, in that one commented out line, we have also tried to "Pad" the text to the correct size with a full zero byte array of correct length (16 - cipherText), in which case the algorithm runs fine, but it returns complete gibberish and not the original text.
We already have tried all of the modes of operation and messed around with padding modes as well. They are not providing us with anything but a cipherText, and a key for that text. No Initialization vector either, so we are assuming we are supposed to be generating that every time (but for ECB it isn't even needed iirc)
What's more is, they are able to encrypt-decrypt their messages just fine. What is most puzzling about this is that I have been googling this for days now and I cannot find a SINGLE example on google where the CIPHERTEXT is shorter than the key during decryption.
Obviously I have found examples where the message they are Encrypting is shorter than what is needed, but that is what padding is for on the ENCRYPTION side (right?). So that when you then receive the padded message, you can tell the algorithm what padding mode was used to make it correct length, so then it can seperate the padding from the message. But in all of those cases the recieved message during decryption is of correct length.
So the question is - what are we doing wrong? is there some way to decrypt with ciphertexts that are shorter than the key? Or are they messing up somewhere by producing ciphers that are too short?
Thanks for any help.
In AES-ECB, the only valid ciphertext shorter than 16-byte is empty. That 16-byte limit is the block (not key) size of AES, which happens to match the key size for AES-128.
Therefore, the question's
An example message we receive is: d3 73 4c
does not show an ECB encrypted message (since a comment tells that's from a JSON, that can't be bytes that happen to show as hex). And that's way too short to be a FRMPayload (per this comment) for a Join-Accept, since the spec says of the later:
1625 The message is either 16 or 32 bytes long.
Could it be that whatever that JSON message contains is not a full FRMPayload, but a fragment of a packet, encoded as hexadecimal pair with space separator? As long as it is not figured out how to build a FRMPayload, there's not point in deciphering it.
Update: If that mystery message is always 3 bytes, and if it is always the same for a given key (or available a single time per key), then per Maarten Bodewes's comment it might be a Key Check Value. The KCV is often the first 3 bytes of the encryption of the all-zero value with the key per the raw block cipher (equivalently: per ECB). Herbert Hanewinkel's javascript AES can work fully offline (which is necessary to not expose the key), and be used to manually validate an hypothesis. It tells that for the 16-byte key given in the question, a KCV would be cd15e1 (or c076fc per the variant in the next section).
Also: it is used CreateDecryptor to craft a gizmo in charge of the ECB decryption. That's likely incorrect in the context of decryption of a LoraWan payload, which requires ECB encryption for decryption of some fields:
1626 Note: AES decrypt operation in ECB mode is used to encrypt the join-accept message so that the end-device can use an AES encrypt operation to decrypt the message. This way an end device only has to implement AES encrypt but not AES decrypt.
In the context of decryption of a LoraWan packets, you want to communicate with the AES engine using byte arrays, not strings. Strings have an encoding, when LoraWan ciphertext and corresponding plaintext does not. Others seems to have managed to coerce the nice .NET do-it-all crypto API to get a low-level job done.
In the HextoString code, I vaguely get that the intention and perhaps outcome is that hex becomes the originally hex input as a byte array (fully rid of hexadecimal and other encoding sin; in which case the variable hex should be renamed to something on the tune of pure_bytes). But then I'm at loss about System.Text.Encoding.ASCII.GetString(hex). I'd be surprised if it just created a byte string from a byte array, or turned the key back to hexadecimal for later feeding to HexStringToByteArray in Aes128Decrypt. Plus this makes me fear that any byte in [0x80..0xFF] might turn to 0x3F, which is not nice for key, ciphertext, and corresponding LoraWan payload. These have no character encoding when de-hexified.
My conclusion is that if HexStringToByteArray does what its name suggests, and given the current interface of Aes128Decrypt, HextoString should simply remove whitespace (or is unneeded if HexStringToByteArray removes whitespace on the fly). But my recommendation is to change the interface to use byte arrays, not strings (see previous section).
As an aside: creating an ICryptoTransform object from its key is supposed to be performed once for multiple uses of the object.
Ok, the original task is to track users among 2 "friendly" web-sites who are able to share users cookies (lets say, I have example.com and my friend has mysite.com and also he has a domain simple.example.com so he can set cookies on .example.com).
To track users activity we want to set unique cookie, this cookie should be unique and 32 bytes long (ascii). Quite simple from this point of view and can be implemented as such:
md5(microtime)
and that's it, but now we have new constraints:
we should be able to tell who exactly has set the cookie: exmaple.com engine or mysite.com engine
32 bytes longs is a must, still
we should be able to encrypt timestamp (when cookies was issued)
first and last character of the resulting cookie value should be different so we can do A/B testing basing on the cookie (so we could always say if last character of the cookie is "> K", show this users "feature A")
Given that the resulting string should always be 32 or less characters long and data should be encrypted and decrypted (not by users, of course) and the string should be unique for the users, it makes the task quite complex.
My thought and questions:
we should use symmetric key encryption (solves constraints 1 and 3), but it this case how do we ensure that resulting string is no longer than 32 chars (constraint 2)?
is there other solution on the problem given that amount of data we need to encrypt is: timestamp and microseconds (14 bytes), site-issuer flag (1 byte) = 15 bytes total
My first take was to pack data into binary string and than base64-encode it. The result would be 8-chars long base64-encoded string:
def encode():
base64( pack('Lv', timestamp, microseconds) )
Add site-issuer flag and chars at the beginning and the end:
def getCookie():
rand('a'...'Z') + encode() + issuerFlagChar() + rand('a'...'Z')
So, the result is 11 chars long and we meet constraint 2 easily.
But the problem is: this algorithm is not secure for sure, I'm not sure if the resulting string for millions of websites users is unique.
I wonder if I could use DES or AES for this purpose but I'm not sure that the resulting string will always meet constraint 2 (resulting string should be no longer than 32 ascii chars).
Is there symmetric key algorithms that ensure something like "if you encrypt N bytes with M-bytes key you will have resulting data length of Math.Ceil(N*2+1/M) bytes"? So the resulting length would be predictable?
Setting aside the fact that you should indeed consult a security consultant, the actual question you pose can easily be answered:
Is there symmetric key algorithms that ensure something like "if you encrypt N bytes with M-bytes key you will have resulting data length of Math.Ceil(N*2+1/M) bytes"? So the resulting length would be predictable?
Yes there are. And they are called Block Ciphers.
By definition, every block cipher has the property that the length of the ciphertext is equal to the length of the plain text. In practice most block ciphers (inclusing DES and AES) cheat a bit because they require the plaintext to be padded to the length of the block before they start encrypting.
In other words, given a plaintext of N bytes and a block size of B, the ciphertext will have a length of B*(Math.ceil(N/B)) bytes.
Note how I am talking about the block size, which is different from the key size. The key size is actually irrelevant in this case.
For example, AES uses a block size of 128 bits, or 16 bytes. This means that if your plain text is between 17 and 32 bytes long, AES will guarantee that your ciphertext is 32 bytes long. This is independent from the key size you choose, which can be one of 128, 192 or 256 bits (16, 24 or 32 bytes).
First of all, you need to know whether you want to encrypt or sign the data.
Encrypting will prevent users from seeing the data, but they are still able to modify it in some ways depending on the encryption type. For example, decrypting a modified ciphertext will simply give corrupted data, it won't fail.
Signing, on the other hand, will prevent users from modifying the data, that is, your code will be able to detect the data has been modified. A simple algorithm for this is HMAC.
I'll assume you want both. My solution below does both.
Your cookie must be 32 bytes long, which is 256 bits. We are going to use 128 bits for encrypted data and 128 bits for the HMAC.
For the data, I will encode the timestamp as a 64bit integer (more than enough even if you want to store it to microsecond precision). The site that issued the cookie can be stored as 1 bit if you have two sites, but I'll store it in a 32bit integer because we have plenty of space. Same for a tag you can use for a/b testing.
All the data is exactly 128 bits, 16 bytes. This is the exact size of an AES block. So, we will encrypt it with AES!
The other 16 bytes will be a MAC of the ciphertext (Encrypt then MAC). I used HMAC-SHA256, which has 256bits of output. We only have room for 128bits, so I have truncated it. In theory this makes it less secure, but in practice 128bit is big enough to make a brute-force attempt impossible.
Decrypting the cookie is the reverse process: calculate the HMAC of the given ciphertext and check it matches the given MAC. If so, proceeed to decrypt the ciphertext and unpack the data.
Here's the code:
from struct import pack, unpack
from Crypto.Cipher import AES
import hashlib
import hmac
AES_KEY = hashlib.sha256(b"secret key 1 asdfasdf").digest()
HMAC_KEY = hashlib.sha256(b"secret key 2 asdfasdf").digest()
# timestamp: 64bit unix timestamp
# site: 32bit integer, which site issued the cookie
# tag: 32bit integer, tag used for a/b testing.
def encrypt_cookie(timestamp, site, tag):
# Pack the data
data = pack('QII', timestamp, site, tag)
# Encrypt it
aes = AES.new(AES_KEY, AES.MODE_ECB, 'This is an IV456')
ciphertext = aes.encrypt(data)
# Do HMAC of the ciphertext
sig = hmac.new(HMAC_KEY, ciphertext, hashlib.sha256).digest()
sig = sig[:16] # Truncate to only first 16 bytes.
return ciphertext + sig
def decrypt_cookie(cookie):
# Do HMAC of the ciphertext
sig = hmac.new(HMAC_KEY, cookie[:16], hashlib.sha256).digest()
sig = sig[:16] # Truncate to only first 16 bytes.
# Check the HMAC is ok
if sig != cookie[16:]:
raise Exception("Cookie has been tampered with")
# Decrypt it
aes = AES.new(AES_KEY, AES.MODE_ECB, 'This is an IV456')
data = aes.decrypt(cookie[:16])
# unPack the data
timestamp, site, tag = unpack('QII', data)
return timestamp, site, tag
cookie = encrypt_cookie(1, 2, 3)
print(len(cookie)) # prints: 32
print(decrypt_cookie(cookie)) # prints: 1, 2, 3
# Change a single byte in the cookie, the last one
cookie = cookie[:31] + b'0'
print(decrypt_cookie(cookie)) # raises the exception
I'm curious to know why the cookie must be 32bytes though. Seems a weird requirement, and if you didn't have it, you'd be able to use many libraries that are designed to solve exactly this problem, such as Django signing if you're using Django.
The documents just don't seem to provide an answer..
Microsoft tried to explain the subject clearly, but it is still ambiguous. At least in our case.
We have an encrypted MP4 stream. It contains "SampleEncryptionBox"es or "PIFF" boxes, which contain 8-byte = 64-bit Initialization Vectors for encrypted blocks. BUT: The actual "counter block" for decrypting the "AES-128 Counter Mode"-encrypted video data is 128-bit. I don't know where exactly to put the IV in it!!
PIFF document says 16-byte IV is the entire counter block (obviously) for AES-CTR mode. Also, 8-byte IV is put at the beginning of the counter block, for AES-ECB mode (page 17). But for 8-byte IV in AES-CTR mode, it says nothing!
This RFC document says that the 128-bit should comprise 4-byte Nonce + 8-byte IV + 4-byte counter. And the Nonce value should be taken from the extra 4 bytes supplied for the main 128-bit AES key. I can only obtain the 128-bit key by the Protection Header, where should I get the 4-byte Nonce??
Any bit of extra knowledge will be highly appreciated.
Try NIST SP 800-38A instead, section B.2. Note that this document is the first one that is referenced by the Microsoft document:
A second approach to satisfying the uniqueness property across messages is to assign to each message a unique string of b/2 bits (rounding up, if b is odd), in other words, a message nonce, and to incorporate the message nonce into every counter block for the message. The leading b/2 bits (rounding up, if b is odd) of each counter block would be the message nonce, and the standard incrementing function would be applied to the remaining m bits to provide an index to the counter blocks for the message. Thus, if N is the message nonce for a given message, then the jth counter block is given by Tj = N | [j]m, for j = 1...n. The number of blocks, n, in any m message must satisfy n < 2 . A procedure should be established to ensure the uniqueness of the message nonces.
Note that you would need 2 ^ 64 blocks of data to get to the next "message nonce" or IV. This is just a sample method of generating a counter; unfortunately NIST does the bad habbit of not specifying any default ones. So it depends on the protocol as well, but above is most common.
Ok, I found the explanation.. It is written clearly in "ISO/IEC JTC 1/SC 29 N" document.
If the IV_size field is 8, then its value is copied to bytes 0 to 7 of the InitializationVector and bytes 8 to 15 of the InitializationVector are set to zero. The IV_size field shall not be 0 when the IsEncrypted flag is 0x1.
AES-ECB Mode has nothing to do with it.
I am using 128-bit Rijndael with ECB cipher mode encryption to convert my string. It formats some kind of string with symbols == or = at the end.
In my code I need some preliminary decision was this string encrypted. Can I suggest that if it contains = symbol at the end it is encrypted or possible cases when I will not get = symbol in the end in encrypted string?
First of all, if you are using ECB you are not really "encrypting" since it is a broken cipher mode (see: http://bobnalice.wordpress.com/2009/01/28/friends-don%E2%80%99t-let-friends-use-ecb-mode-encryption/ )
Secondly, the = signs you are seeing are the Base64 padding characters. They are only tagentially related to encryption since Base64 is used for any kind of binary data, not just encrypted date.
Thirdly, you can't even rely on the = sign always being present for Base64 data... it is only added for certain lengths of data (i.e. you could have encrypted, Base64 data that has no = sign)
As #JoelFan points out, those '=' characters are Base64 padding which aren't part of the encryption at all, and aren't always there either if the data comes out to the right number of characters naturally.
You need to add something "out of band". I would replace the potentially-encrypted string with a class which has a string member for the data, and an "encrypted" flag. Once you're there, throw in a getData() method that checks if the instance's data is encrypted, returns the unencrypted value of the data if so (and may cache the plaintext in a private member for later, if it'll be accessed a lot), or just returns the plain text if it's not encrypted.
As the title says, I am trying to decrypt DUKPT encrypted track data coming from a DUKPT enabled scanner.
I have the ANSI Standard (X9.24) for DUKPT and have successfully implemented the ability to generate the IPEK from the KSN and BDK. Furthermore, I have successfully implemented the ability to generate the Left and Right MAC Request and Response Keys by XORing the PIN Encryption Keys. Lastly, I am able to generate the EPB.
From here, I don't understand how to generate the MAC Request and Response from the L/R Keys that I have generated.
Lastly, once I get to that step, what comes next? When do I actually have the key that decrypts the track data sent by a DUKPT enabled device?
I am aware of the Thales Simulator and jPOS. My code is currently referencing the Thales Simulator to do all of its work. But, the file decryption process just isn't returning the expected data.
If anybody can offer some insight into decrypting track data, it would be much appreciated.
http://thalessim.codeplex.com/
http://jpos.org/
I spent too much time studying the horrible X9.24 spec and finally got both the encryption and decryption working with my vendor’s examples and marketing promptly decided to switch vendors. Since it is a standard, you would think that anybody’s implementation would be the same. I wish. Anyway, there are variations on how things are implemented. You have to study the fine print to make sure you are working things the same as your other side.
But that is not your question.
First if you need to decrypt a data track from a credit card, you are probably interested in producing a key that will decrypt the data based upon the original super secret Base Derivation Key. That has nothing to do with the MAC generation and is only mentioned in passing in that dreadful spec. You need to generate the IPEK for that key serial number and device ID and repeatedly apply the “Non-reversible Key Generation Process” from the spec if bits are set in the counter part of the full key serial number from the HSM.
That part of my code looks like this: (Sorry for the long listing in a posting.)
/*
* Bit "zero" set (this is a 21 bit register)(ANSI counts from the left)
* This will be used to test each bit of the encryption counter
* to decide when to find another key.
*/
testBit=0x00100000;
/*
* We have to "encrypt" the IPEK repeatedly to find the current key
* (See Section A.3). Each time we encrypt (generate a new key),
* we need to use the all prior bits to the left of the current bit.
* The Spec says we will have a maximum of ten bits set at any time
* so we should not have to generate more than ten keys to find the
* current encryption key.
*/
cumBits=0;
/*
* For each of the 21 possible key bits,
* if it is set, we need to OR that bit into the cumulative bit
* variable and set that as the KSN count and "encrypt" again.
* The encryption we are using the goofy ANSI Key Generation
* subroutine from page 50.
*/
for(int ii=0; ii<21; ii++)
{
if( (keyNumber&testBit) != 0)
{
char ksr[10];
char eightByte[8]={0};
cumBits |= testBit;
ksn.count=cumBits; /* all bits processed to date */
memcpy(ksr, &ksn,10); /* copy bit structure to char array*/
memcpy(crypt,&ksr[2],8); /* copy bytes 2 through 9 */
/*
* Generate the new Key overwriting the old.
* This will apply the "Non-reversible Key Generation Process"
* to the lower 64 bits of the KSN.
*/
keyGen(&key, &crypt, &key);
}
testBit>>=1;
}
Where
keyNumber is the current counter from the ksn
ksn is an 80 bit structure that contains the 80 bit Key Serial Number from the HSM
crypt is a 64 bit block of data I have it of type DES_cblock since I am using openSSL.
key is a 128 bit double DES_cblock structure.
The keyGen routine is almost verbatim from the “Non-reversible Key Generation Process” local subroutine on page 50 of the spec.
At the end of this, the key variable will contain the key that can be used for the decryption, almost. The dudes that wrote the spec added some “variant” behavior to the key to keep us on our toes. If the key is to be used for decrypting a data stream such as a credit card track, you will need to XOR bytes 5 and 13 with 0xFF and Triple DES encrypt the key with itself (ECB mode). My code looks like:
DOUBLE_KEY keyCopy;
char *p;
p=(char*)&key;
p[ 5]^=0xff;
p[13]^=0xff;
keyCopy=key;
des3(&keyCopy, (DES_cblock *)&key.left, &key.left);
des3(&keyCopy, (DES_cblock *)&key.right, &key.right);
If you are using this to decrypt a PIN block, you will need to XOR bytes 7 and 15 with 0xFF. (I am not 100% sure this should not be applied for the stream mode as well but my vendor is leaving it out.)
If it is a PIN block, it will be encrypted with 3-DES in ECB mode. If it is a data stream, it will be encrypted in CBC mode with a zero initialization vector.
(Did I mention I don’t much care for the spec?) It is interesting to note that the encryption side could be used in a non-hardware, tamper resistant security module if the server side (above) remembers and rejects keys that have been used previously. The technology is pretty neat. The ANSI spec leaves something to be desired but the technology is all right.
Good luck.
/Bob Bryan
For data encryption, the variant is 0000000000FF0000.0000000000FF0000 so you need to XOR bytes 5 and 13 instead of 7 and 15. In addition, you need an additional 3DES self-encryption step of each key parts (left and right).
Here is the relevant code in jPOS
https://github.com/jpos/jPOS/blob/master/jpos/src/main/java/org/jpos/security/jceadapter/JCESecurityModule.java#L1843-1856