I am struggling for an entire week now, failing at meeting the encryption requirements of HERE traffic api, precisely TPEG API.
Steps to do:
<?xml version="1.0" encoding="UTF-8" ?>
<get-messages>
<locations>
<loc lat="52.55121" lon="13.16565"/>
</locations>
</get-messages>
This xml body must be enrypted, HERE API documents it as follows:
Encrypt and compress all traffic information requests:
Compress the XML body using gzip.
Calculate the length in bytes of the gzip file.
Prepend the length of the gzipped data to the compressed body as a little endian 32 bit integer.
Pad the combination of gziped content and length with zeros to make it evenly divisible by 16 bytes.
Using AES 128, encrypt the resultant padded combination of content and length as follows:
a) Create a random integer 16 bytes long.
b) AES encrypt the result of step 4, in mode CBC using the integer generated in step 5.a as the initialization vector and the key from the InitSession response. Do not apply additional padding.
Send the resulting block of AES encrypted data as an HTTP POST request, prepended by the integer generated in step 5.a as content type application/octet-stream to the URL in the initsession response
So many things are just unclear here, what is the desired result of the gziped xml? Base64? binary?
What is the type of the 32 little endian int, binary?
The key has a length of 32 characters. Since AES128 only fits 16 bytes long keys, I assume the key must be interpreted as hex values. Do all values need to be defined as hex values?
What is the type of the IV? Hex? Text? Binary?
What is the type of the encrypted result? Hex? Binary? Text? Base64?
The http header must contain content_type appilcation/octet-stream.
Related
I implemented AES 128 bit encryption using initialization vector and padding, as seen in the code below. I happen to be using ColdFusion, but I don’t think that matters. The encrypted result shows some repeating patterns, which I would not have expected, but then again I don't know the characteristics of correct output for this. Am I doing initialization vector and padding correctly?
<!---
To encrypt this, for example:
"String1"
Prefix the string with an Initialization Vector of 16 random characters,
plus enough padding ("000000001") to make the entire string a multiple of 16 characters (32 characters, here)
"HoMoz4yT0+WAU7CX000000001String1"
Now encrypt the string to this (64 characters):
"Bn0k3q9aGJt91nWNA0xun6va8t8+OiJVmCqv0RzUzPWFyT4jUMzZ56pG5uFt6bGG"
--->
<cfoutput>
<cfset EncryptKey="LpEecqQe3OderPakcZeMcw==">
<cfloop index="StringToEncrypt" list="String1,String2,String3,String3">
<!--- Make random Initialization Vector (IV) of length 16
(create it from GenerateSecretKey(), but GenerateSecretKey is NOT the key that we encrypt/decrypt with) --->
<cfset IV=left(GenerateSecretKey("AES",128),16)>
<!--- Pad the string so its length is a multiple of 16 --->
<cfset padlength=16 - (len(StringToEncrypt) mod 16)>
<cfset padding=repeatstring("0",padlength-1) & "1">
<cfset NewStringToEncrypt=IV & padding & StringToEncrypt>
<cfset EncryptedString=encrypt(NewStringToEncrypt,EncryptKey,"AES","Base64")>
<pre>Original string: #StringToEncrypt#
StringToEncrypt: #NewStringToEncrypt#
EncryptedString: #EncryptedString#</pre>
</cfloop>
</cfoutput>
Below is sample output:
Original string: String1
StringToEncrypt: QLkApY6XKka7mQge000000001String1
EncryptedString: BOAVeSKidQyyHrEa15x9Uava8t8+OiJVmCqv0RzUzPWFyT4jUMzZ56pG5uFt6bGG
Original string: String2
StringToEncrypt: DboCmHHuVrU05oTV000000001String2
EncryptedString: 4Yk14F0ffz9+djbvSiwA1/X3FHhS5Vhta7Q8iocBPhmFyT4jUMzZ56pG5uFt6bGG
Original string: String3
StringToEncrypt: 8om5VbbWQgvRWK7Q000000001String3
EncryptedString: 01AF+pmF9sDsUHcIXSVfom8Egv8Oiyb2yy12hiVcJjqFyT4jUMzZ56pG5uFt6bGG
Original string: String3
StringToEncrypt: T4qJodVe6aEv0p1E000000001String3
EncryptedString: aAjCbSBRZ+cd7ZwpFPZUxW8Egv8Oiyb2yy12hiVcJjqFyT4jUMzZ56pG5uFt6bGG
Each EncryptedString ends with the same 21 characters:
FyT4jUMzZ56pG5uFt6bGG
When the original string is the same ("String3" in the 3rd and 4th example), the EncryptedString ends with the same 42 characters:
8Egv8Oiyb2yy12hiVcJjqFyT4jUMzZ56pG5uFt6bGG
Update: Per the accepted answer, I should not do my own padding or initialization vector. Coldfusion's encrypt/decrypt functions can handle that automatically, and the encrypted values will have no repeating patterns. For example:
EncryptedString=encrypt(StringToEncrypt, EncryptKey, 'AES/CBC/PKCS5Padding', 'Base64')
DecryptedString=decrypt(EncryptedString, EncryptKey, 'AES/CBC/PKCS5Padding', 'Base64')
Do not do your own padding, but let the encrypt function do it. It's appended to the plaintext, not prepended. The usual padding is called PKCS5 padding and adds $t$ times the byte $t \in {1,2,3,\ldots,16}$ to make up full blocks.
Also, the iv is an argument to the encrypt function and not prepended before the plaintext. It's actually (when you use a chain-mode like CBC, OFB, CFB) prepended to the ciphertext.
So you could generate the IV as you do (though there are probably better functions to do it) and then use the plaintext as is and encrypt:
EncryptedString=encrypt(StringToEncrypt, EncryptKey, 'AES/CBC/PKCS5Padding', 'Base64', binaryDecode(IV, "base64"))
(based on this documentation)
added 2
The iv should be binary according to these docs, so the base64 from generateKey has to be converted. Thx to comments by #Agax and his "cftry" examples...
added 1 It turns out that omitting the IV will cause the function
to generate its own IV, and in a seemingly non-repeating way.
In any case we can use
EncryptedString=encrypt(StringToEncrypt, EncryptKey, 'AES/CBC/PKCS5Padding', 'Base64')
instead. It's essential to use a chaining mode like CBC to get the propagation effect from a random IV that masks identical plaintexts and which the default 'AES' leaves visible (this does ECB mode, presumably).
The IV is actually not prepended to the ciphertext, when we give it as an argument, so you have to remember it somehow to be able to decrypt the first block of plain text. Quite non-standard. When you let encrypt generate it itself, it does prepend it. So you have to use the same signature version of decrypt.
A possible reason for this is that the mode of AES you are using doesnt use the previous block with the current block before encrypting the current block.
For example, CBC mode performs an XOR with the previous block cipher text and the current block plaintext before encrypting the current block. This means that even with the same string ("string3") and same key, a different IV will yield a completely different result, and there will never be string repetition.
Other modes of AES that use XORs include: CBC, PCBC, OFB, CFB. This link describes the different modes of AES in more detail (under the "common modes" section).
May a URL contain raw binary data in a GET-request?
Is it possible to create a URL, www.example.com/**binary-data**, where www.example.com/ are ordinary ASCII characters, and **binary-data** are arbitrary raw byte-values, e.g., 0x10.
I don't won't to encode the binary data, but just create a string, e.g., char* in C, that contains both the ASCII characters and the binary data.
Or is POST-request the only way to send raw binary data as part of the body?
No, but could percent-escape the non-URI characters.
No. A URL transmitted in an HTTP GET request is percent-encoded, UTF-8 encoded Unicode text (resulting in an "ASCII" string).
Again, it's Unicode text.
Unicode encodings do not produce arbitrary binary data. There is no equivalent text for some arbitrary binary data.
Moving away from "raw", the server and request can, of course, agree on the use of a scheme such as Base64 to turn arbitrary binary data into Unicode text. By that point, though, you might as well use an HTTP request with a body, and although as far as HTTP is concerned the bodies are raw binary data, HTTP headers can indicate a standard format. Such requests include POST and PUT.
There are also practical limits to the length of the URL.
Thank you for your help.
I have a problem with sjcl library or cryptojs aes library.
Everytime i encrypt my file with aes the file size is * 2.
For example: When i upload a 4mb file then there is a 8mb file on my server.
I split the file on block which bytesize can be divided by 16.
Please help me
This is the way i encrypt the file:
var l_EncryptetData = CryptoJS.AES.encrypt(p_ChunkData, p_FileKey).toString();
And this is a example array with block sizes from the 4mb file:
0: 1572864
1: 1572864
2: 867005
length: 3
And the FileKey is a SHA-256 Hash
You are converting the cipher output (a CipherParams object) to a character string. By default, CryptoJS does this by representing the result in hexadecimal, where each byte is encoded as two hex digits. You can override this behavior by providing a custom format strategy. Or better yet, don't convert the cipher text to a character string; keep it as binary data.
Please warn users of your application that it is not secure.
My company is working on a project that will put card readers in the field. The readers use DUKPT TripleDES encryption, so we will need to develop software that will decrypt the card data on our servers.
I have just started to scratch the surface on this one, but I find myself stuck on a seemingly simple problem... In trying to generate the IPEK (the first step to recreating the symmetric key).
The IPEK's a 16 byte hex value created by concatenating two triple DES encrypted 8 byte hex strings.
I have tried ECB and CBC (zeros for IV) modes with and without padding, but the result of each individual encoding is always 16 bytes or more (2 or more blocks) when I need a result that's the same size as the input. In fact, throughout this process, the cyphertexts should be the same size as the plaintexts being encoded.
<cfset x = encrypt("FFFF9876543210E0",binaryEncode(binaryDecode("0123456789ABCDEFFEDCBA98765432100123456789ABCDEF", "hex"), "base64") ,"DESEDE/CBC/PKCS5Padding","hex",BinaryDecode("0000000000000000","hex"))>
Result: 3C65DEC44CC216A686B2481BECE788D197F730A72D4A8CDD
If you use the NoPadding flag, the result is:
3C65DEC44CC216A686B2481BECE788D1
I have also tried encoding the plaintext hex message as base64 (as the key is). In the example above that returns a result of:
DE5BCC68EB1B2E14CEC35EB22AF04EFC.
If you do the same, except using the NoPadding flag, it errors with "Input length not multiple of 8 bytes."
I am new to cryptography, so hopefully I'm making some kind of very basic error here. Why are the ciphertexts generated by these block cipher algorithms not the same lengths as the plaintext messages?
For a little more background, as a "work through it" exercise, I have been trying to replicate the work laid out here:
https://www.parthenonsoftware.com/blog/how-to-decrypt-magnetic-stripe-scanner-data-with-dukpt/
I'm not sure if it is related and it may not be the answer you are looking for, but I spent some time testing bug ID 3842326. When using different attributes CF is handling seed and salt differently under the hood. For example if you pass in a variable as the string to encrypt rather than a constant (hard coded string in the function call) the resultant string changes every time. That probably indicates different method signatures - in your example with one flag vs another flag you are seeing something similar.
Adobe's response is, given that the resulting string can be unecrypted in either case this is not really a bug - more of a behavior to note. Can your resultant string be unencrypted?
The problem is encrypt() expects the input to be a UTF-8 string. So you are actually encrypting the literal characters F-F-F-F-9.... rather than the value of that string when decoded as hexadecimal.
Instead, you need to decode the hex string into binary, then use the encryptBinary() function. (Note, I did not see an iv mentioned in the link, so my guess is they are using ECB mode, not CBC.) Since the function also returns binary, use binaryEncode to convert the result to a more friendly hex string.
Edit: Switching to ECB + "NoPadding" yields the desired result:
ksnInHex = "FFFF9876543210E0";
bdkInHex = "0123456789ABCDEFFEDCBA98765432100123456789ABCDEF";
ksnBytes = binaryDecode(ksnInHex, "hex");
bdkBase64 = binaryEncode(binaryDecode(bdkInHex, "hex"), "base64");
bytes = encryptBinary(ksnBytes, bdkBase64, "DESEDE/ECB/NoPadding");
leftRegister = binaryEncode(bytes, "hex");
... which produces:
6AC292FAA1315B4D
In order to do this we want to start with our original 16 byte BDK
... and XOR it with the following mask ....
Unfortunately, most of the CF math functions are limited to 32 bit integers. So you probably cannot do that next step using native CF functions alone. One option is to use java's BigInteger class. Create a large integer from the hex strings and use the xor() method to apply the mask. Finally, use the toString(radix) method to return the result as a hex string:
bdkText ="0123456789ABCDEFFEDCBA9876543210";
maskText = "C0C0C0C000000000C0C0C0C000000000";
// use radix=16 to create integers from the hex strings
bdk = createObject("java", "java.math.BigInteger").init(bdkText, 16);
mask = createObject("java", "java.math.BigInteger").init(maskText, 16);
// apply the mask and convert the result to hex (upper case)
newKeyHex = ucase( bdk.xor(mask).toString(16) );
WriteOutput("<br>newKey="& newKeyHex);
writeOutput("<br>expected=C1E385A789ABCDEF3E1C7A5876543210");
That should be enough to get you back on track. Given some of CF's limitations here, java would be a better fit IMO. If you are comfortable with it, you could write a small java class and invoke that from CF instead.
Who can tell me the encoding type of these data? It doesn't seem to be base64.
/zZ/u00GIaP9HW010G000G01003/sm1302WS7YCU6IWZ8ICjAoWmF6H1F3StF7jONKbaaO2Pbe+
0Z8gWjER3eAhQhOgCoF/Bskxr////cy7////w/+Rz//Z/sm130IijBJmrF7P1GNRufOob+FZu+F
Zu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZu+FZ/m00H200X07430
I800X410n41/yG07m000GK10G410G400000000000420mG51WS82GeB/yG0jH000W430m840mK5
10G0005z0G8300GH1H8XCK464r5X1o9n53A1aQ488qAnmHLIqV0aCs9oWWaA5XSO6Heb9YSeAIe
qDJOtE3awGqH5HaT8IKfJL5LMLrXPMcDaPMPdQ6bgStHrTdTuUNg3X8M6XuY9YfAJb9MMbvYPcg
AZfAMcfwYfghApjBMsjxYvkiB3nCN6nyZ9ojBJrDNMrzZPsk7Yu+JbvkVewUhnylFqzVRt+Fdw/
yG07m400m410G410G410G00000000420mG51WS82GeB/yG0jH400W4210G310S510G00G9t0042
0n441I4n1X91KGTXSHCYCe4854AHeR712ICpKl0LOdBH2XOaDE4byHSO6Hec9oWfAZKsDpWvEaD
4HKP7I4bAKrHLLbTOMLfZP6LcPsXfQdDqTNPtU7bwWeE4XOQ7Y8cAafEKbPQNc9cQegEafQQdgA
cgihEqjRQtkBcwmiF4nSR7oCdAqjFKrTRNsDdQukFavURdwEdgylFqzVRt+Fdw/ze030C1008H0
n40Fm2vQMjkzf0pGHVSKab1ad5I/PBPkbl5Z/S7D5dyrb1dfvQ/ZnIotCAIB6x7B38LLB5lm7Qc
G9zajcwMyMFzmSr18sd82pnn1586H5a4aP7EFIfblRUMRoVCsj/TO5IVph7kBAUH1TnkMJPl18s
bU/JFuvxydwWpLYZi9s8YItR7OAkV/m1LFIsjPLLrjujX08FbWPhdxUEIvlnvESbzsuZE1dgSd+
jT7DF77jyni1ZXG0IMFq52JUmCRzajcwMyMFy0S7D7sIsRfRnO/m1mSqXlRSo17VPaP0TIkVpxK
iy+C95jQHssgfFV6Sds0fyhMuYbBFPBUHsyTh4+M2imK31pzAjImsKKPbaWYMDUfyiS/fM8xgcg
ix7v1EIJrutLSdqoUxccdSX2I0e7Ex0ndhmE/Vy0ngQIjOQBqCTZSblAXYPKE2H6C4/N5IVPBPk
bl5Z/071pMHRwQ3V97vmEq1sKZ3O/0d+VlMpBSHHZzv8gZhZkVmzAWFGRzajcwMyMFzmSqVPBPk
bl5Z/S7DBzfYPrGahk+xkKZT+TJVV/0Dt+T0Qd6qKKKYZgxFvhA3FJor/7Yi+rbaRKBif5vpbzf
F2XL1nr/BZlYj2p+QoWpqyjVnugl5ljRYLNHtXcSoAw8JrwWu/3wqo1VEZakaIxjbZaF+hSuODW
yOFzAfHtKgj1O+KZgGkL2bICW7a+eF9EEsQkp1hwvX2HbOeM3cHq8px3FwrFBPmN4WaTCaSWXYC
dru/3ds50p1byrBpVRAoC+S8WzEm0wZZkFK7a6j4oksgkVACiYe0g361m2VcFrFrpLo6pWZVUY3
e02UJZ6C0+d+UcAYn91TFAoj97C536DUX0nqzEl+UkbDsk9wYpJ8P45vRg49mid3BdwaSVvxKv5
bCiapnAt92yu9K7W3sxzidZfKTtklWi4KR1IGnaT201xPxrU+//0Blyw9Q90SvgCPMyas/SPYm9
mESPFFy08VJ7NdRUQIApCio1olB2FE2Czi+t+SK2pWPjs7FkP6EUdlx3yXKWXZC8Y0Fb3W0a/m0
wKfNIGRb3Hcy/xHCxPTt+PSzktuSdyglI97l+q6CiLN0mCa/GVv/AajxQA5I8a037B7+yVyAYbb
dIut6DfBSZ0239pKeQrUX3VJ6TOeoZHHkGJ98E1MZz/m3tVvrHkNUyZycE1nd1tEk0Ajn9jYHCv
LL0p/UflOgMoHo5555G0KKKK0555501HHHG0KKKK0555501HHHG0KKKK0555507/za
(Line endings inserted for readability)
It appears to be base64. If you add a single equal ('=') for padding to the end your decoder should be happy (see https://en.wikipedia.org/wiki/Base64#Padding).
Decoded it's 1568 bytes which mod 16 is zero. This histogram of byte value occurance is flat. So I'd guess something encrypted with a 128 bit block cipher like AES.
It does look like base64 to me. Most variants of base64 include the following characters:
A-Z a-z 0-9 + / (and = for padding)
However, if it were proper base64 it would end with a single = as padding, as the 2091 characters don't exactly fit a number of bytes.
Your data doesn't seem to decode to anything readable, so it might be binary data, or encrypted (or both). Only with thorough knowledge of cryptography systems, and a lot of hints and luck, might some expert be able to figure out the encryption used (if any), but that's beyond the scope of this site.
Without more information as to the source of the data, we can only guess.