Decoding a larger DES data fails with Arduino - arduino

Question

I am trying to add an NODE_MCU_V3 to a small BMS system I have running with Raspberry's and a Java library.
The device makes a call to the server, and gets a JSON response. This response is DES encrypted and then Base64 encoded. The sending of the information works fine with encoding, and receiving a response works on smaller responses. But if the JSON is more than 208 characters the decryption fails. I am not that familiar with Arduino and C but would have a guess it has to do with the size.
This is the relevant code. The result is a String taken from the server response.
The result of the Base64 decoder is what is expected, the problem is with the DES decipher. I added examples to the bottom in HEX.
char encoded[result.length()];
result.toCharArray(encoded, result.length());
// Convert back.
int decodedLength = Base64.decodedLength(encoded, sizeof(encoded));
char decodedChar[decodedLength];
Base64.decode(decodedChar, encoded, sizeof(encoded));
Serial.print("Decoded: "); printArray((byte*) decodedChar, sizeof(decodedChar));
byte jsonByte[decodedLength];
for (int i = 0; i < (des.get_size() / 8); i++) {
byte intermitInput[8];
byte intermitResult[8];
for (int j = 0; j < 8; j++) {
intermitInput[j] = (byte) decodedChar[(i * 8) + j];
}
des.decrypt(intermitResult, intermitInput, key);
for (int j = 0; j < 8; j++) {
jsonByte[(i * 8) + j] = intermitResult[j];
}
}
Serial.print("Decrypted: "); printArray(jsonByte, sizeof(jsonByte));
char json[sizeof(jsonByte) + 1];
for (int i = 0; i < sizeof(jsonByte); i++) {
json[i] = (char)jsonByte[i];
}
json[sizeof(jsonByte)] = '\0';
Serial.print("Decripted result:\t");
Serial.println(json);
StaticJsonDocument<256> doc;
DeserializationError error = deserializeJson(doc, json);
// Test if parsing succeeds.
if (error) {
Serial.print(F("deserializeJson() failed."));
return;
}
const char* resultStatus = doc["result"];
if (strstr(resultStatus, "SUCCESS") < 0) {
Serial.print("Problem with the response: "); Serial.println(resultStatus);
return;
}
Serial.print(" Value: "); Serial.println(resultStatus);
Server:
Json - Datalength: 359 - {"result":"SUCCESS","message":"(none)", "actions":[{"action":"initiation","data":[{"type":"IO_CONTROLLER","inputPins":"","outputPinsManual":"","relayDefaultState":"","inputResistance":"","inputPinsManual":"","outputPins":"4"}], "hostname":"AR-259bfc91250", "macAddress":"null"},{"action":"campAlarm","value":"false"},{"action":"warningBeep","value":"false"}]}
Clear JSON HEX: 7b22726573756c74223a2253554343455353222c226d657373616765223a22286e6f6e6529222c2022616374696f6e73223a5b7b22616374696f6e223a22696e6974696174696f6e222c2264617461223a5b7b2274797065223a22494f5f434f4e54524f4c4c4552222c22696e70757450696e73223a22222c226f757470757450696e734d616e75616c223a22222c2272656c617944656661756c745374617465223a22222c22696e707574526573697374616e6365223a22222c22696e70757450696e734d616e75616c223a22222c226f757470757450696e73223a2234227d5d2c2022686f73746e616d65223a2241522d3235396266633931323530222c20226d616341646472657373223a226e756c6c227d2c7b22616374696f6e223a2263616d70416c61726d222c2276616c7565223a2266616c7365227d2c7b22616374696f6e223a227761726e696e6742656570222c2276616c7565223a2266616c7365227d5d7d
Encrypted before Base64: 1e64fd8c074b2eda044f2ed820dab06949e5a2c5602918b13779e906c2733c1719965a456e2127fef0c910cbbbfcd137c535c9423cc14e7e8497de8fd340441f0a48634ccaa6d64e5d23bd2bc4f06d3998adc310f77631d17478f8c85168663e9ecba5551d1bac0c06b26e778a96e4292de903683e4fadeb5c94050f7b30aa1de5a11408a366bef1584115b530ea4b9efffcc0abef06ed6b0baca92b67ef54587eaa0e71976261eacf7942ec2a566d962246ebf7aeacda6d4a6b5246ca2281cf1e6db1af905a5260c74276ba7bcbba00e4ce82abb260f606afee1277f7cbf42ab3cac3cc09c5fb9676c5feb30c47dcd437d64a7886376435eeee517e14673b45ead6ddf30b238e46c53f6bbbedbab1e74ca1fbb5dbac628a12ae007471017d969757d37a2ffa5f992c5ecc9c9fad3f500638b7a135695eba21b54947a3492fdc98b3a2570fbea4bae0d50d426152ee8012779de0f5ba9e96ae15cef583353a3116499bea9e766488
Base64 send: HmT9jAdLLtoETy7YINqwaUnlosVgKRixN3npBsJzPBcZllpFbiEn/vDJEMu7/NE3xTXJQjzBTn6El96P00BEHwpIY0zKptZOXSO9K8TwbTmYrcMQ93Yx0XR4+MhRaGY+nsulVR0brAwGsm53ipbkKS3pA2g+T63rXJQFD3swqh3loRQIo2a+8VhBFbUw6kue//zAq+8G7WsLrKkrZ+9UWH6qDnGXYmHqz3lC7CpWbZYiRuv3rqzabUprUkbKIoHPHm2xr5BaUmDHQna6e8u6AOTOgquyYPYGr+4Sd/fL9CqzysPMCcX7lnbF/rMMR9zUN9ZKeIY3ZDXu7lF+FGc7RerW3fMLI45GxT9ru+26sedMofu126xiihKuAHRxAX2Wl1fTei/6X5ksXsycn60/UAY4t6E1aV66IbVJR6NJL9yYs6JXD76kuuDVDUJhUu6AEned4PW6npauFc71gzU6MRZJm+qedmSI
Arduino:
Base64 coming in: HmT9jAdLLtoETy7YINqwaUnlosVgKRixN3npBsJzPBcZllpFbiEn/vDJEMu7/NE3xTXJQjzBTn6El96P00BEHwpIY0zKptZOXSO9K8TwbTmYrcMQ93Yx0XR4+MhRaGY+nsulVR0brAwGsm53ipbkKS3pA2g+T63rXJQFD3swqh3loRQIo2a+8VhBFbUw6kue//zAq+8G7WsLrKkrZ+9UWH6qDnGXYmHqz3lC7CpWbZYiRuv3rqzabUprUkbKIoHPHm2xr5BaUmDHQna6e8u6AOTOgquyYPYGr+4Sd/fL9CqzysPMCcX7lnbF/rMMR9zUN9ZKeIY3ZDXu7lF+FGc7RerW3fMLI45GxT9ru+26sedMofu126xiihKuAHRxAX2Wl1fTei/6X5ksXsycn60/UAY4t6E1aV66IbVJR6NJL9yYs6JXD76kuuDVDUJhUu6AEned4PW6npauFc71gzU6MRZJm+qedmSI
Base64 Decoded: 1E 64 FD 8C 07 4B 2E DA 04 4F 2E D8 20 DA B0 69 49 E5 A2 C5 60 29 18 B1 37 79 E9 06 C2 73 3C 17 19 96 5A 45 6E 21 27 FE F0 C9 10 CB BB FC D1 37 C5 35 C9 42 3C C1 4E 7E 84 97 DE 8F D3 40 44 1F 0A 48 63 4C CA A6 D6 4E 5D 23 BD 2B C4 F0 6D 39 98 AD C3 10 F7 76 31 D1 74 78 F8 C8 51 68 66 3E 9E CB A5 55 1D 1B AC 0C 06 B2 6E 77 8A 96 E4 29 2D E9 03 68 3E 4F AD EB 5C 94 05 0F 7B 30 AA 1D E5 A1 14 08 A3 66 BE F1 58 41 15 B5 30 EA 4B 9E FF FC C0 AB EF 06 ED 6B 0B AC A9 2B 67 EF 54 58 7E AA 0E 71 97 62 61 EA CF 79 42 EC 2A 56 6D 96 22 46 EB F7 AE AC DA 6D 4A 6B 52 46 CA 22 81 CF 1E 6D B1 AF 90 5A 52 60 C7 42 76 BA 7B CB BA 00 E4 CE 82 AB B2 60 F6 06 AF EE 12 77 F7 CB F4 2A B3 CA C3 CC 09 C5 FB 96 76 C5 FE B3 0C 47 DC D4 37 D6 4A 78 86 37 64 35 EE EE 51 7E 14 67 3B 45 EA D6 DD F3 0B 23 8E 46 C5 3F 6B BB ED BA B1 E7 4C A1 FB B5 DB AC 62 8A 12 AE 00 74 71 01 7D 96 97 57 D3 7A 2F FA 5F 99 2C 5E CC 9C 9F AD 3F 50 06 38 B7 A1 35 69 5E BA 21 B5 49 47 A3 49 2F DC 98 B3 A2 57 0F BE A4 BA E0 D5 0D 42 61 52 EE 80 12 77 9D E0 F5 BA 9E 96 AE 15 CE F5 83 35 3A 31 16 49 9B EA 9E 76 64 7F
Decrypted HEX: 7B 22 72 65 73 75 6C 74 22 3A 22 53 55 43 43 45 53 53 22 2C 22 6D 65 73 73 61 67 65 22 3A 22 28 6E 6F 6E 65 29 22 2C 20 22 61 63 74 69 6F 6E 73 22 3A 5B 7B 22 61 63 74 69 6F 6E 22 3A 22 69 6E 69 74 69 61 74 69 6F 6E 22 2C 22 64 61 74 61 22 3A 5B 7B 22 74 79 70 65 22 3A 22 49 4F 5F 43 4F 4E 54 52 4F 4C 4C 45 52 22 2C 22 69 6E 70 75 74 50 69 6E 73 22 3A 22 22 2C 22 6F 75 74 70 75 74 50 69 6E 73 4D 61 6E 75 61 6C 22 3A 22 22 2C 22 72 65 6C 61 79 44 65 66 61 75 6C 74 53 74 61 74 65 22 3A 22 22 2C 22 69 6E 70 75 74 52 65 73 69 73 74 61 6E 63 65 22 3A 22 22 2C 22 69 6E 70 75 74 50 69 6E 73 4D 61 6E 75 61 6C 22 3A 22 22 2C 42 89 FE 3F 00 00 00 00 0D 0A 00 25 35 78 20 40 42 89 FE 3F D0 00 00 00 20 00 00 00 C7 08 10 40 09 00 00 00 00 00 00 00 F0 A7 C6 4B 0F 00 00 00 54 58 20 40 10 E9 FE 3F 40 89 FE 3F 60 58 20 40 54 58 20 40 10 E9 FE 3F 40 89 FE 3F 21 5B 20 40 C8 FB FF 3F 60 FA FF 3F 10 E9 FE 3F 18 5C 20 40 54 58 20 40 10 E9 FE 3F ED 88 FE 3F 5E 15 20 40 68 01 00 00 10 E9 FE 3F ED 88 FE 3F 21 5B 20 40 10 E9 FE 3F 68 01 00 00 00 00 00 00 B0 FD FF 3F 10 E9 FE 3F 68 01 00 00
Decrypted JSON String: {"result":"SUCCESS","message":"(none)", "actions":[{"action":"initiation","data":[{"type":"IO_CONTROLLER","inputPins":"","outputPinsManual":"","relayDefaultState":"","inputResistance":"","inputPinsManual":"",B⸮⸮?

Answer 1

If you suspect you are running out of memory, You can try to reclaim memory buffers by dual-purposing them. For example, reuse the input buffer that contained the base64 data fto store the decoded json, this will save you at least 208 byes of RAM. It may be enough for your case. All you'd need to do to test this theory is this:
byte* jsonByte = (byte*)encoded; // save decodedLength bytes of RAM
For futher savings, you can also try to decode the base64 in-place, using a single buffer for both input and output. If your base64 decode can't do it (it likely can), then write your own that can, it's definitely feasible for very little pain. This can save you another 208 bytes, which is quite a lot. For testing this, change the declarion of decodedChar to:
char* decodedChar = encoded;
Base64.decodedLength() will always return (result.length() * 5) / 8, which is always smaller than the number of bytes coming in.
Also, there are more potential savings in the decryption. For one, the decryption can be done in_place.
for (int i = 0; i < (des.get_size() / 8); i++) {
byte intermitInput[8];
byte intermitResult[8]; // not needed
for (int j = 0; j < 8; j++) {
intermitInput[j] = (byte) decodedChar[(i * 8) + j];
}
des.decrypt(intermitResult, intermitInput, key);
for (int j = 0; j < 8; j++) {
jsonByte[(i * 8) + j] = intermitResult[j];
}
}
// could be done in-place with no loss of functionality,
// and a noticeable performance gain. Copying data takes
// time, too.
for (int i = 0; i < des.get_size(); i += 8) {
byte intermitInput[8];
for (int j = 0; j < 8; j++) {
intermitInput[j] = (byte) decodedChar[i + j];
}
des.decrypt((byte*)decodedChar + i, intermitInput, key);
}
// result is in array decodedCher. array jsonBytes is not needed anymore.
And here:
// you use 208 bytes of RAM only to add a null terminating character.
// when you coud have simply allocated 1 more byte in jsonByte[], avoided
// this entire loop, and made your function run faster at the same time.
char json[sizeof(jsonByte) + 1];
for (int i = 0; i < sizeof(jsonByte); i++) {
json[i] = (char)jsonByte[i];
}
json[sizeof(jsonByte)] = '\0';
My estimate is that you can reclaim at least 600 bytes of RAM in this function, which would alow you to process larger jsons.
With the details given in your question, it is impossible to give you an estimate of how large a json you'd be able to process.
Also, since your arduino bugs in printArray(), I'd check in there for any less than useful large intermediate arrays. In a print function, any array larger than 16 bytes would already be way too big for the task at hand. Ideally there should be no intermediate array at all in a print function.
To recap: You can and should reorganize your code to use only one array for input, all intermediate results and the final json. RAM resources are very limited on small microprocessors and Arduinos. With use, you will notice that embedded algorithms and libraries, like the DES encryption, are especially built so resource cost is as small as possible.
Another area to explore: Is the json result parsed by the Arduino? If not, the entire sequence could very likely be done 8 bytes at a time, read in a block of 10 bytes, and send the resulting 8 bytes to the Pi as you go, giving you the priceless capability to process jsons of unlimited size, at least on the arduino side of things.