Squid proxy discarding request made with netcat, but not curl - squid

I need to make a REST call within a RHEL kickstart %pre script, and thus I am limited to using netcat (since the wget packaged in the RHEL %pre environment can't configure the HTTP method). I'd of course love to use curl (since it has the lovely -X option) but alas it's not available in the %pre environment.
That said, here's a relevant curl command and, importantly, the exact stream of bytes it sends to the server:
$ curl -X POST http://pkrizak-globalpxe.anonycom.com/univac/api/record/pkrizak-sles10.anonycom.com/_install_log --data-binary '[ ]' --trace /tmp/foo.log
$ cat /tmp/foo.log (truncated)
== Info: About to connect() to pkrizak-globalpxe.anonycom.com port 80 (#0)
== Info: Trying 10.46.174.117... == Info: connected
== Info: Connected to pkrizak-globalpxe.anonycom.com (10.46.174.117) port 80 (#0)
=> Send header, 312 bytes (0x138)
0000: 50 4f 53 54 20 2f 75 6e 69 76 61 63 2f 61 70 69 POST /univac/api
0010: 2f 72 65 63 6f 72 64 2f 70 6b 72 69 7a 61 6b 2d /record/pkrizak-
0020: 73 6c 65 73 31 30 2e 61 63 6f 63 79 63 6f 6d 2e sles10.anonycom.
0030: 63 6f 6d 2f 5f 69 6e 73 74 61 6c 6c 5f 6c 6f 67 com/_install_log
0040: 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 65 72 2d HTTP/1.1..User-
0050: 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e 31 39 Agent: curl/7.19
0060: 2e 37 20 28 78 38 36 5f 36 34 2d 72 65 64 68 61 .7 (x86_64-redha
0070: 74 2d 6c 69 6e 75 78 2d 67 6e 75 29 20 6c 69 62 t-linux-gnu) lib
0080: 63 75 72 6c 2f 37 2e 31 39 2e 37 20 4e 53 53 2f curl/7.19.7 NSS/
0090: 33 2e 31 34 2e 33 2e 30 20 7a 6c 69 62 2f 31 2e 3.14.3.0 zlib/1.
00a0: 32 2e 33 20 6c 69 62 69 64 6e 2f 31 2e 31 38 20 2.3 libidn/1.18
00b0: 6c 69 62 73 73 68 32 2f 31 2e 34 2e 32 0d 0a 48 libssh2/1.4.2..H
00c0: 6f 73 74 3a 20 70 6b 72 69 7a 61 6b 2d 67 6c 6f ost: pkrizak-glo
00d0: 62 61 6c 70 78 65 2e 61 63 6f 63 79 63 6f 6d 2e balpxe.anonycom.
00e0: 63 6f 6d 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a com..Accept: */*
00f0: 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 ..Content-Length
0100: 3a 20 33 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 : 3..Content-Typ
0110: 65 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 e: application/x
0120: 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 -www-form-urlenc
0130: 6f 64 65 64 0d 0a 0d 0a oded....
=> Send data, 3 bytes (0x3)
0000: 5b 20 5d [ ]
== Info: HTTP 1.0, assume close after body
<= Recv header, 24 bytes (0x18)
I've truncated the rest, but suffice to say that the transaction continues without problems.
Now, sending the exact same stream of bytes using netcat:
$ ./mypost.sh | hexdump -C
00000000 50 4f 53 54 20 2f 75 6e 69 76 61 63 2f 61 70 69 |POST /univac/api|
00000010 2f 72 65 63 6f 72 64 2f 70 6b 72 69 7a 61 6b 2d |/record/pkrizak-|
00000020 73 6c 65 73 31 30 2e 61 63 6f 63 79 63 6f 6d 2e |sles10.anonycom.|
00000030 63 6f 6d 2f 5f 69 6e 73 74 61 6c 6c 5f 6c 6f 67 |com/_install_log|
00000040 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 65 72 2d | HTTP/1.1..User-|
00000050 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e 31 39 |Agent: curl/7.19|
00000060 2e 37 20 28 78 38 36 5f 36 34 2d 72 65 64 68 61 |.7 (x86_64-redha|
00000070 74 2d 6c 69 6e 75 78 2d 67 6e 75 29 20 6c 69 62 |t-linux-gnu) lib|
00000080 63 75 72 6c 2f 37 2e 31 39 2e 37 20 4e 53 53 2f |curl/7.19.7 NSS/|
00000090 33 2e 31 34 2e 33 2e 30 20 7a 6c 69 62 2f 31 2e |3.14.3.0 zlib/1.|
000000a0 32 2e 33 20 6c 69 62 69 64 6e 2f 31 2e 31 38 20 |2.3 libidn/1.18 |
000000b0 6c 69 62 73 73 68 32 2f 31 2e 34 2e 32 0d 0a 48 |libssh2/1.4.2..H|
000000c0 6f 73 74 3a 20 70 6b 72 69 7a 61 6b 2d 67 6c 6f |ost: pkrizak-glo|
000000d0 62 61 6c 70 78 65 2e 61 63 6f 63 79 63 6f 6d 2e |balpxe.anonycom.|
000000e0 63 6f 6d 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a |com..Accept: */*|
000000f0 0d 0a 43 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 |..Content-length|
00000100 3a 20 33 0d 0a 43 6f 6e 74 65 6e 74 2d 74 79 70 |: 3..Content-typ|
00000110 65 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 |e: application/x|
00000120 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 |-www-form-urlenc|
00000130 6f 64 65 64 0d 0a 0d 0a 5b 20 5d |oded....[ ]|
0000013b
...doesn't work:
$ ./mypost.sh | nc pkrizak-globalpxe.anonycom.com 80
$ # no response
The tricky bit here is that there is a squid (reverse) proxy listening on port 80 of the host I'm connecting to. So I'm not actually talking to Apache or Nginx or even my custom Perl application -- I'm trying to talk to Squid (which when approached with curl actually forwards the request properly to my application). And apparently something is different about how curl talks to Squid than how nc is talking to squid, despite the content of the request being identical.
I've tried changing the request to use HTTP/1.0 and even tried leaving the HTTP/ part off, but that does not help.
I'm really baffled at this point -- what is going on under the hood that I'm missing? Why is netcat behaving differently?


Turns out this is an oddity in the way that netcat behaves when piped data on STDIN.
After tracing the packets with wireshark, I found that netcat, after sending the data piped to STDIN, immediately sends a FIN,ACK packet to the server. The squid server, naturally, aborts processing of the request upon receipt of the FIN,ACK packet and closes the connection.
This behavior can be avoided by using the -i option to netcat, which specifies an interval time between transactions. Using -i 1, for example, waits one second before sending the FIN,ACK after sending its data. This is long enough for the squid proxy to return with an answer.
Another solution is to have the script generating the input to STDIN pause after writing out the HTTP POST information. A simple sleep 1 at the end of the script appears to be sufficient to get the proxy to complete the request.

Related

How can I validate an HTTP response?

I'm creating a low level HTTP server and I want to validate that my responses are correct.
Some of them are fine, but others fail. I have saved some responses as bytes and viewing them in HxD, and they look correct, but they obviously aren't in some way.
Here's the start of one HTTP response formatted as hex:
48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D 0A 43 4F 4E 54 45 4E
54 2D 54 59 50 45 3A 20 69 6D 61 67 65 2F 70 6E 67 0D 0A 4C 41 53 54
2D 4D 4F 44 49 46 49 45 44 3A 20 53 75 6E 2C 20 30 31 20 4D 61 79 20
32 30 32 32 20 32 32 3A 32 34 3A 32 34 20 47 4D 54 0D 0A 41 43 43 45
50 54 2D 52 41 4E 47 45 53 3A 20 62 79 74 65 73 0D 0A 45 54 41 47 3A
20 22 37 35 37 35 35 33 36 61 61 35 64 64 38 31 3A 30 22 0D 0A 53 45
52 56 45 52 3A 20 4D 69 63 72 6F 73 6F 66 74 2D 49 49 53 2F 31 30 2E
30 0D 0A 58 2D 50 4F 57 45 52 45 44 2D 42 59 3A 20 41 53 50 2E 4E 45
54 0D 0A 44 41 54 45 3A 20 57 65 64 2C 20 30 34 20 4D 61 79 20 32 30
32 32 20 30 39 3A 30 31 3A 34 38 20 47 4D 54 0D 0A 43 4F 4E 54 45 4E
54 2D 4C 45 4E 47 54 48 3A 20 39 38 37 35 37 0D 0A 43 6F 6E 6E 65 63
74 69 6F 6E 3A 20 63 6C 6F 73 65 0D 0A 0D 0A 0D 0A 89 50 4E 47
How can I validate that the response I'm creating is valid according to RFC 2616?
I'm working in .NET but I don't mind if it's an online tool or a different platform.
Edit:
I've solved my issue with this particular example (3 crlfs instead of 2 between head and body) but I'd still like to know how I can validate these requests automatically.

I ended up achieving this with this nuget package and the following code:
if (System.Diagnostics.Debugger.IsAttached)
{
using (var handler = new HttpParserDelegate())
using (var parser = new HttpCombinedParser(handler))
{
parser.Execute(arr);
}
if (!handler.HttpRequestResponse.IsEndOfMessage)
{
System.Diagnostics.Debugger.Break();
throw new InvalidOperationException($"An HTTP response was serialized but it is not a valid response");
}
}

Decoding base64 string in R gives unexpected binary output

I have this string:
x <- "W3siY29kZSI6IkIxMTgyIiwiZGVzY3JpcHRpb24iOiJUaXJlIHByZXNzdXJlIG1vbml0b3IgbW9kdWxlIiwiZmF1bHRJbmZvcm1hdGlvbnMiOlt7ImRlc2NyaXB0aW9uIjoiLS0tIn0seyJkZXNjcmlwdGlvbiI6IlRSVUUiLCJkZXNjcmlwdGlvbl9lbmciOiJUUlVFIiwiZGVzY3JpcHRpb25fZnJlIjoiVFJVRSIsImRlc2NyaXB0aW9uX2dlciI6IlRSVUUiLCJkZXNjcmlwdGlvbl9zcGEiOiJUUlVFIiwiZGVzY3JpcHRpb25faXRhIjoiVFJVRSIsImRlc2NyaXB0aW9uX3R1ciI6IlRSVUUifV0sImZyZWV6ZUZyYW1lcyI6W10sImVuaGFuY2VkRHRjSW5mb3MiOltdLCJzdGF0dXMiOiJBQ1RJVkUiLCJzeXN0ZW0iOiJNVUxUSUZVTkNUSU9OIiwic3lzdGVtTWV0YSI6eyJzeXN0ZW1OYW1lIjpudWxsLCJzdWJTeXN0ZW1OYW1lIjpudWxsfSwiZGVzY3JpcHRpb25fZW5nIjoiVGlyZSBwcmVzc3VyZSBtb25pdG9yIG1vZHVsZSIsImRlc2NyaXB0aW9uX2ZyZSI6Ik1vZHVsZSBkZSBzdXJ2ZWlsbGFuY2UgZGUgbGEgcHJlc3Npb24gZGVzIHBuZXVzIiwiZGVzY3JpcHRpb25fZ2VyIjoiUmVpZmVuZHJ1Y2vDvGJlcndhY2h1bmciLCJkZXNjcmlwdGlvbl9zcGEiOiJNw7NkdWxvIGRlIGNvbnRyb2wgZGUgbGEgcHJlc2nDs24gZGUgbG9zIG5ldW3DoXRpY29zIiwiZGVzY3JpcHRpb25faXRhIjoiTW9kdWxvIG1vbml0b3IgZGkgcHJlc3Npb25lIHBuZXVtYXRpY28iLCJkZXNjcmlwdGlvbl90dXIiOiJMYXN0aWsgYmFzxLFuY8SxIGl6bGVtZSBtb2TDvGzDvCJ9LHsiY29kZSI6IkIxMjREIiwiZGVzY3JpcHRpb24iOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJmYXVsdEluZm9ybWF0aW9ucyI6W3siZGVzY3JpcHRpb24iOiItLS0ifSx7ImRlc2NyaXB0aW9uIjoiVFJVRSIsImRlc2NyaXB0aW9uX2VuZyI6IlRSVUUiLCJkZXNjcmlwdGlvbl9mcmUiOiJUUlVFIiwiZGVzY3JpcHRpb25fZ2VyIjoiVFJVRSIsImRlc2NyaXB0aW9uX3NwYSI6IlRSVUUiLCJkZXNjcmlwdGlvbl9pdGEiOiJUUlVFIiwiZGVzY3JpcHRpb25fdHVyIjoiVFJVRSJ9XSwiZnJlZXplRnJhbWVzIjpbXSwiZW5oYW5jZWREdGNJbmZvcyI6W10sInN0YXR1cyI6IkFDVElWRSIsInN5c3RlbSI6Ik1VTFRJRlVOQ1RJT04iLCJzeXN0ZW1NZXRhIjp7InN5c3RlbU5hbWUiOm51bGwsInN1YlN5c3RlbU5hbWUiOm51bGx9LCJkZXNjcmlwdGlvbl9lbmciOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJkZXNjcmlwdGlvbl9mcmUiOiJDYXB0ZXVyLCBwcmVzc2lvbiBkZXMgcG5ldXMiLCJkZXNjcmlwdGlvbl9nZXIiOiJTZW5zb3IgUmVpZmVuZHJ1Y2siLCJkZXNjcmlwdGlvbl9zcGEiOiJTZW5zb3IgZGUgcHJlc2nDs24gZGVsIG5ldW3DoXRpY28iLCJkZXNjcmlwdGlvbl9pdGEiOiJTZW5zb3JlLCBwcmVzc2lvbmUgcG5ldW1hdGljaSIsImRlc2NyaXB0aW9uX3R1ciI6IlNlbnPDtnIsIGxhc3RpayBoYXZhIGJhc8SxbmPEsSJ9XQ=="
I know it has been encoded using base64. Doing a quick search on internet, I found that in R there are some packages to encode/decode base64.
Package jsonlite with the function base64dec()
Package base64decode with the function base64decode().
In both cases, when running...
base64dec(x)
base64decode(x)
I get this result, which is a binary output:
[1] 5b 7b 22 63 6f 64 65 22 3a 22 42 31 31 38 32 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 69 72 65 20 70 72 65 73 73 75 72 65 20 6d 6f 6e 69 74 6f 72
[54] 20 6d 6f 64 75 6c 65 22 2c 22 66 61 75 6c 74 49 6e 66 6f 72 6d 61 74 69 6f 6e 73 22 3a 5b 7b 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 2d 2d 2d 22 7d 2c 7b
[107] 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 65 6e 67 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69
[160] 70 74 69 6f 6e 5f 66 72 65 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 67 65 72 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69
[213] 6f 6e 5f 73 70 61 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 69 74 61 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f
[266] 74 75 72 22 3a 22 54 52 55 45 22 7d 5d 2c 22 66 72 65 65 7a 65 46 72 61 6d 65 73 22 3a 5b 5d 2c 22 65 6e 68 61 6e 63 65 64 44 74 63 49 6e 66 6f 73 22 3a 5b 5d
[319] 2c 22 73 74 61 74 75 73 22 3a 22 41 43 54 49 56 45 22 2c 22 73 79 73 74 65 6d 22 3a 22 4d 55 4c 54 49 46 55 4e 43 54 49 4f 4e 22 2c 22 73 79 73 74 65 6d 4d 65
[372] 74 61 22 3a 7b 22 73 79 73 74 65 6d 4e 61 6d 65 22 3a 6e 75 6c 6c 2c 22 73 75 62 53 79 73 74 65 6d 4e 61 6d 65 22 3a 6e 75 6c 6c 7d 2c 22 64 65 73 63 72 69 70
[425] 74 69 6f 6e 5f 65 6e 67 22 3a 22 54 69 72 65 20 70 72 65 73 73 75 72 65 20 6d 6f 6e 69 74 6f 72 20 6d 6f 64 75 6c 65 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e
[478] 5f 66 72 65 22 3a 22 4d 6f 64 75 6c 65 20 64 65 20 73 75 72 76 65 69 6c 6c 61 6e 63 65 20 64 65 20 6c 61 20 70 72 65 73 73 69 6f 6e 20 64 65 73 20 70 6e 65 75
[531] 73 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 67 65 72 22 3a 22 52 65 69 66 65 6e 64 72 75 63 6b c3 bc 62 65 72 77 61 63 68 75 6e 67 22 2c 22 64 65 73 63 72
[584] 69 70 74 69 6f 6e 5f 73 70 61 22 3a 22 4d c3 b3 64 75 6c 6f 20 64 65 20 63 6f 6e 74 72 6f 6c 20 64 65 20 6c 61 20 70 72 65 73 69 c3 b3 6e 20 64 65 20 6c 6f 73
[637] 20 6e 65 75 6d c3 a1 74 69 63 6f 73 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 69 74 61 22 3a 22 4d 6f 64 75 6c 6f 20 6d 6f 6e 69 74 6f 72 20 64 69 20 70 72
[690] 65 73 73 69 6f 6e 65 20 70 6e 65 75 6d 61 74 69 63 6f 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 74 75 72 22 3a 22 4c 61 73 74 69 6b 20 62 61 73 c4 b1 6e 63
[743] c4 b1 20 69 7a 6c 65 6d 65 20 6d 6f 64 c3 bc 6c c3 bc 22 7d 2c 7b 22 63 6f 64 65 22 3a 22 42 31 32 34 44 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 53
[796] 65 6e 73 6f 72 2c 20 74 79 72 65 20 70 72 65 73 73 75 72 65 22 2c 22 66 61 75 6c 74 49 6e 66 6f 72 6d 61 74 69 6f 6e 73 22 3a 5b 7b 22 64 65 73 63 72 69 70 74
[849] 69 6f 6e 22 3a 22 2d 2d 2d 22 7d 2c 7b 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 65 6e 67 22 3a 22
[902] 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 66 72 65 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 67 65 72 22 3a 22 54 52 55
[955] 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 73 70 61 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 69 74 61 22 3a
However, when I copy that string into this web, I get this result:
[{"code":"B1182","description":"Tire pressure monitor module","faultInformations":[{"description":"---"},{"description":"TRUE","description_eng":"TRUE","description_fre":"TRUE","description_ger":"TRUE","description_spa":"TRUE","description_ita":"TRUE","description_tur":"TRUE"}],"freezeFrames":[],"enhancedDtcInfos":[],"status":"ACTIVE","system":"MULTIFUNCTION","systemMeta":{"systemName":null,"subSystemName":null},"description_eng":"Tire pressure monitor module","description_fre":"Module de surveillance de la pression des pneus","description_ger":"Reifendrucküberwachung","description_spa":"Módulo de control de la presión de los neumáticos","description_ita":"Modulo monitor di pressione pneumatico","description_tur":"Lastik basıncı izleme modülü"},{"code":"B124D","description":"Sensor, tyre pressure","faultInformations":[{"description":"---"},{"description":"TRUE","description_eng":"TRUE","description_fre":"TRUE","description_ger":"TRUE","description_spa":"TRUE","description_ita":"TRUE","description_tur":"TRUE"}],"freezeFrames":[],"enhancedDtcInfos":[],"status":"ACTIVE","system":"MULTIFUNCTION","systemMeta":{"systemName":null,"subSystemName":null},"description_eng":"Sensor, tyre pressure","description_fre":"Capteur, pression des pneus","description_ger":"Sensor Reifendruck","description_spa":"Sensor de presión del neumático","description_ita":"Sensore, pressione pneumatici","description_tur":"Sensör, lastik hava basıncı"}]
This is actually the expected output. What's wrong with the code?

Looks like the output is raw encoded. So you'll need to use rawToChar.
library(base64)
rawToChar(base64_decode(x))
[1] "[{\"code\":\"B1182\",\"description\":\"Tire pressure monitor module\",\"faultInformations\":[{\"description\":\"---\"},{\"description\":\"TRUE\",\"description_eng\":\"TRUE\",\"description_fre\":\"TRUE\",\"description_ger\":\"TRUE\",\"description_spa\":\"TRUE\",\"description_ita\":\"TRUE\",\"description_tur\":\"TRUE\"}],\"freezeFrames\":[],\"enhancedDtcInfos\":[],\"status\":\"ACTIVE\",\"system\":\"MULTIFUNCTION\",\"systemMeta\":{\"systemName\":null,\"subSystemName\":null},\"description_eng\":\"Tire pressure monitor module\",\"description_fre\":\"Module de surveillance de la pression des pneus\",\"description_ger\":\"Reifendrucküberwachung\",\"description_spa\":\"Módulo de control de la presión de los neumáticos\",\"description_ita\":\"Modulo monitor di pressione pneumatico\",\"description_tur\":\"Lastik basıncı izleme modülü\"},{\"code\":\"B124D\",\"description\":\"Sensor, tyre pressure\",\"faultInformations\":[{\"description\":\"---\"},{\"description\":\"TRUE\",\"description_eng\":\"TRUE\",\"description_fre\":\"TRUE\",\"description_ger\":\"TRUE\",\"description_spa\":\"TRUE\",\"description_ita\":\"TRUE\",\"description_tur\":\"TRUE\"}],\"freezeFrames\":[],\"enhancedDtcInfos\":[],\"status\":\"ACTIVE\",\"system\":\"MULTIFUNCTION\",\"systemMeta\":{\"systemName\":null,\"subSystemName\":null},\"description_eng\":\"Sensor, tyre pressure\",\"description_fre\":\"Capteur, pression des pneus\",\"description_ger\":\"Sensor Reifendruck\",\"description_spa\":\"Sensor de presión del neumático\",\"description_ita\":\"Sensore, pressione pneumatici\",\"description_tur\":\"Sensör, lastik hava basıncı\"}]"
Edit
To better understand why this is necessary we can check the class of the output of base64_decode().
> class(base64_decode(x))
#[1] "raw"
The ?raw helpfully tells us:
The raw type is intended to hold raw bytes. It is possible to extract subsequences of bytes, and to replace elements (but only by elements of a raw vector). ... A raw vector is printed with each byte separately represented as a pair of hex digits. If you want to see a character representation (with escape sequences for non-printing characters) use rawToChar.
So we just need to convert the raw bytes to character.
Data
x <- "W3siY29kZSI6IkIxMTgyIiwiZGVzY3JpcHRpb24iOiJUaXJlIHByZXNzdXJlIG1vbml0b3IgbW9kdWxlIiwiZmF1bHRJbmZvcm1hdGlvbnMiOlt7ImRlc2NyaXB0aW9uIjoiLS0tIn0seyJkZXNjcmlwdGlvbiI6IlRSVUUiLCJkZXNjcmlwdGlvbl9lbmciOiJUUlVFIiwiZGVzY3JpcHRpb25fZnJlIjoiVFJVRSIsImRlc2NyaXB0aW9uX2dlciI6IlRSVUUiLCJkZXNjcmlwdGlvbl9zcGEiOiJUUlVFIiwiZGVzY3JpcHRpb25faXRhIjoiVFJVRSIsImRlc2NyaXB0aW9uX3R1ciI6IlRSVUUifV0sImZyZWV6ZUZyYW1lcyI6W10sImVuaGFuY2VkRHRjSW5mb3MiOltdLCJzdGF0dXMiOiJBQ1RJVkUiLCJzeXN0ZW0iOiJNVUxUSUZVTkNUSU9OIiwic3lzdGVtTWV0YSI6eyJzeXN0ZW1OYW1lIjpudWxsLCJzdWJTeXN0ZW1OYW1lIjpudWxsfSwiZGVzY3JpcHRpb25fZW5nIjoiVGlyZSBwcmVzc3VyZSBtb25pdG9yIG1vZHVsZSIsImRlc2NyaXB0aW9uX2ZyZSI6Ik1vZHVsZSBkZSBzdXJ2ZWlsbGFuY2UgZGUgbGEgcHJlc3Npb24gZGVzIHBuZXVzIiwiZGVzY3JpcHRpb25fZ2VyIjoiUmVpZmVuZHJ1Y2vDvGJlcndhY2h1bmciLCJkZXNjcmlwdGlvbl9zcGEiOiJNw7NkdWxvIGRlIGNvbnRyb2wgZGUgbGEgcHJlc2nDs24gZGUgbG9zIG5ldW3DoXRpY29zIiwiZGVzY3JpcHRpb25faXRhIjoiTW9kdWxvIG1vbml0b3IgZGkgcHJlc3Npb25lIHBuZXVtYXRpY28iLCJkZXNjcmlwdGlvbl90dXIiOiJMYXN0aWsgYmFzxLFuY8SxIGl6bGVtZSBtb2TDvGzDvCJ9LHsiY29kZSI6IkIxMjREIiwiZGVzY3JpcHRpb24iOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJmYXVsdEluZm9ybWF0aW9ucyI6W3siZGVzY3JpcHRpb24iOiItLS0ifSx7ImRlc2NyaXB0aW9uIjoiVFJVRSIsImRlc2NyaXB0aW9uX2VuZyI6IlRSVUUiLCJkZXNjcmlwdGlvbl9mcmUiOiJUUlVFIiwiZGVzY3JpcHRpb25fZ2VyIjoiVFJVRSIsImRlc2NyaXB0aW9uX3NwYSI6IlRSVUUiLCJkZXNjcmlwdGlvbl9pdGEiOiJUUlVFIiwiZGVzY3JpcHRpb25fdHVyIjoiVFJVRSJ9XSwiZnJlZXplRnJhbWVzIjpbXSwiZW5oYW5jZWREdGNJbmZvcyI6W10sInN0YXR1cyI6IkFDVElWRSIsInN5c3RlbSI6Ik1VTFRJRlVOQ1RJT04iLCJzeXN0ZW1NZXRhIjp7InN5c3RlbU5hbWUiOm51bGwsInN1YlN5c3RlbU5hbWUiOm51bGx9LCJkZXNjcmlwdGlvbl9lbmciOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJkZXNjcmlwdGlvbl9mcmUiOiJDYXB0ZXVyLCBwcmVzc2lvbiBkZXMgcG5ldXMiLCJkZXNjcmlwdGlvbl9nZXIiOiJTZW5zb3IgUmVpZmVuZHJ1Y2siLCJkZXNjcmlwdGlvbl9zcGEiOiJTZW5zb3IgZGUgcHJlc2nDs24gZGVsIG5ldW3DoXRpY28iLCJkZXNjcmlwdGlvbl9pdGEiOiJTZW5zb3JlLCBwcmVzc2lvbmUgcG5ldW1hdGljaSIsImRlc2NyaXB0aW9uX3R1ciI6IlNlbnPDtnIsIGxhc3RpayBoYXZhIGJhc8SxbmPEsSJ9XQ=="

Byte truncation casting from void* on Arduino

I'm hoping that someone can help me diagnose this odd behavior that I'm experiencing. I'm programming for the Arduino Due and using the Adafruit FONA modem. The code in question submits an array of bytes to be sent over the modem. sendData is taking an unsigned char[] which I cast to void*. data_len is the number of bytes written to the buffer rather than the size of the buffer itself.
The board I'm working with doesn't use printf/sprintf to print to stdout. Instead this is done using Serial.print and Serial.println. Serial here is an Arduino class, not the standard C++ class. sprintf_ByteArray I wrote to examine the contents of a byte buffer, with the results reflected as hex bytes.
As I mentioned my ultimate goal here is to write data to the modem. This is done in the fonaSerial->println and fonaSerial->write calls. The first issues a modem command with the size of the data to be written. The second writes the data.
But I'm experiencing something screwy. The result of printing data and String message is displayed below. Inexplicably to me, message only contains the last 119 bytes of data. At first, this didn't surprise me as data includes null bytes and I assumed that those were somehow to blame. But, message actually includes null bytes! And, I would have expected message to represent the first N bytes of data, up until the first null byte.
Lastly: I end by casting data to a uint8_t* and passing to fonaSerial->write. Initially I had been passing a String but after I had trouble casting I decided to try this alternative tack. I then send this data to a server I'm running. The server only gets a payload of 119 bytes too.
For the record, earlier I successfully sent a smaller payload. I have no idea what I'm doing now that is causing the problem.
bool sendData(void *data, size_t data_len) {
char tmp[1024] = {0};
Serial.print("data_len=");Serial.println(data_len);
sprintf_ByteArray(tmp, (unsigned char*) data, data_len);
Serial.print("Payload (void*)=");Serial.println(tmp);
String message = (const char*) data;
Serial.print("Payload (String)=");Serial.println(message);
fonaSerial->print(F("AT+CIPSEND="));
fonaSerial->println(data_len);
fona.readline(3000);
fonaSerial->write((uint8_t*) data, data_len);
fonaSerial->flush();
fona.readline(3000); // wait up to 3 seconds to send the data
Serial.print (F("\t<--- ")); Serial.println(fona.replybuffer);
return (strcmp(fona.replybuffer, "SEND OK") == 0);
}
Payload (void*)= 8a a5 62 69 64 00 69 63 6f 6e 64 69 74 69 6f 6e 01 67 62 61 74 74 65 72 79 18 1d 65 73 74 61 74 65 02 69 74 69 6d 65 73 74 61 6d 70 19 0f 95 a5 62 69 64 01 69 63 6f 6e 64 69 74 69 6f 6e 01 67 62 61 74 74 65 72 79 18 42 65 73 74 61 74 65 62 69 74 69 6d 65 73 74 61 6d 70 19 0f 97 a5 62 69 64 02 69 63 6f 6e 64 69 74 69 6f 6e 00 67 62 61 74 74 65 72 79 18 29 65 73 74 61 74 65 06 69 74 69 6d 65 73 74 61 6d 70 19 0f 98 a5 62 69 64 03 69 63 6f 6e 64 69 74 69 6f 6e 01 67 62 61 74 74 65 72 79 18 26 65 73 74 61 74 65 01 69 74 69 6d 65 73 74 61 6d 70 19 0f 99 a5 62 69 64 04 69 63 6f 6e 64 69 74 69 6f 6e 01 67 62 61 74 74 65 72 79 18 2f 65 73 74 61 74 65 62 69 74 69 6d 65 73 74 61 6d 70 19 0f 9a a5 62 69 64 05 69 63 6f 6e 64 69 74 69 6f 6e 01 67 62 61 74 74 65 72 79 18 1d 65 73 74 61 74 65 06 69 74 69 6d 65 73 74 61 6d 70 19 0f 9c a5 62 69 64 18 42 69 63 6f 6e 64 69 74 69 6f 6e 18 75 67 62 61 74 74 65 72 79 18 69 65 73 74 61 74 65 18 6c 69 74 69 6d 65 73 74 61 6d 70 1a 67 6e 69 64 a5 62 69 64 18 20 69 63 6f 6e 64 69 74 69 6f 6e 18 6c 67 62 61 74 74 65 [72 79 18 6f 65 73 74 61 74 65 18 74 69 74 69 6d 65 73 74 61 6d 70 1a 00 6e 39 20 a5 62 69 64 62 69 63 6f 6e 64 69 74 69 6f 6e 00 67 62 61 74 74 65 72 79 18 40 65 73 74 61 74 65 74 69 74 69 6d 65 73 74 61 6d 70 19 0f a0 a5 62 69 64 74 69 63 6f 6e 64 69 74 69 6f 6e 00 67 62 61 74 74 65 72 79 18 43 65 73 74 61 74 65 74 69 74 69 6d 65 73 74 61 6d 70 19 0f a1]
length 471
Payload (String)= 72 79 18 6f 65 73 74 61 74 65 18 74 69 74 69 6d 65 73 74 61 6d 70 1a 00 6e 39 20 a5 62 69 64 62 69 63 6f 6e 64 69 74 69 6f 6e 00 67 62 61 74 74 65 72 79 18 40 65 73 74 61 74 65 74 69 74 69 6d 65 73 74 61 6d 70 19 0f a0 a5 62 69 64 74 69 63 6f 6e 64 69 74 69 6f 6e 00 67 62 61 74 74 65 72 79 18 43 65 73 74 61 74 65 74 69 74 69 6d 65 73 74 61 6d 70 19 0f a1
length 119

try this 'to_string' converter:
String to_string(unsigned char *data,unsigned int len){
String memtmp="";
unsigned char tmpdata;
for(unsigned int i=0;i<len;i++){
tmpdata=*(data+i);
memtmp+=String(tmpdata,HEX);
memtmp+=" ";
}
return memtmp;
}
I hope this can help. Good luck!

Extra bytes and EVP_DecryptFinal_ex Error on OpenSSL with ApplePay

I am trying to decrypt the PaymentToken returned by ApplePay with OpenSSL. The decryption works but I am seeing an extra 16 bytes appended to the plaintext as well as an error during EVP_DecryptFinal_ex. My first thought was I needed to turn off padding but that does not seem to be the issue. Any help is appreciated!
Apple's docs say:
Use the symmetric key to decrypt the value of the data key using AES–256 (id-aes256-GCM 2.16.840.1.101.3.4.1.46), with an initialization vector of sixteen null bytes and no associated authentication data.
The java sample provided by Apple uses BouncyCastle and initializes the Cipher with "AES/GCM/NoPadding"
My code looks like this:
static const unsigned char gcm_iv[] = {
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00
};
/* Create the context */
EVP_CIPHER_CTX *decrypt_ctx;
decrypt_ctx = EVP_CIPHER_CTX_new();
/* Select cipher */
EVP_DecryptInit_ex(decrypt_ctx, EVP_aes_256_gcm(), NULL, NULL, NULL);
/* Set IV length */
EVP_CIPHER_CTX_ctrl(decrypt_ctx, EVP_CTRL_GCM_SET_IVLEN, sizeof(gcm_iv), NULL);
/* Specify key and IV */
EVP_DecryptInit_ex(decrypt_ctx, NULL, NULL, (const unsigned char *)gcm_key, gcm_iv);
/* Turn off padding */
EVP_CIPHER_CTX_set_padding(decrypt_ctx, 0);
unsigned char outbuf[4096];
int outlen = sizeof(outbuf);
int rv;
int plaintext_len;
/* Decrypt plaintext */
rv = EVP_DecryptUpdate(decrypt_ctx, outbuf, &outlen, (const unsigned char *)data, data_length);
if(!rv)
{
printf("\nOpenSSL Error after DecryptUpdate: %s\n", ERR_error_string(ERR_get_error(), NULL));
err = 1;
}
/* Output decrypted block */
printf("\nPlaintext after DecryptUpdate (%d bytes):\n\n", outlen);
/* Remember plaintext length */
plaintext_len = outlen;
BIO_dump_fp(stdout, (const char *)outbuf, plaintext_len);
rv = EVP_DecryptFinal_ex(decrypt_ctx, outbuf + outlen, &outlen);
if(!rv)
{
printf("\nOpenSSL Error after DecryptFinal_ex: %s\n", ERR_error_string(ERR_get_error(), NULL));
err = 1;
}
/* Output decrypted block */
printf("\nPlaintext after DecryptFinal(%d bytes):\n\n", outlen);
BIO_dump_fp(stdout, (const char *)outbuf, plaintext_len);
And the output looks like this (plaintext altered for security):
Plaintext after DecryptUpdate (319 bytes):
0000 - 7b 22 61 70 70 6c 69 63-61 74 69 6f 6e 50 72 69 {"applicationPri
0010 - 6d 61 72 79 41 63 63 6f-75 6e 74 4e 75 6d 62 65 maryAccountNumbe
0020 - 72 22 3a 22 34 30 38 38-xx xx xx xx xx xx xx xx r":"4088xxxxxxxx
0030 - 30 30 32 32 22 2c 22 61-70 70 6c 69 63 61 74 69 0022","applicati
0040 - 6f 6e 45 78 70 69 72 61-74 69 6f 6e 44 61 74 65 onExpirationDate
0050 - 22 3a 22 32 30 30 32 32-38 22 2c 22 63 75 72 72 ":"200228","curr
0060 - 65 6e 63 79 43 6f 64 65-22 3a 22 38 34 30 22 2c encyCode":"840",
0070 - 22 74 72 61 6e 73 61 63-74 69 6f 6e 41 6d 6f 75 "transactionAmou
0080 - 6e 74 22 3a 33 30 30 2c-22 64 65 76 69 63 65 4d nt":300,"deviceM
0090 - 61 6e 75 66 61 63 74 75-72 65 72 49 64 65 6e 74 anufacturerIdent
00a0 - 69 66 69 65 72 22 3a 22-xx xx xx xx xx xx xx xx ifier":"xxxxxxxx
00b0 - xx xx xx xx 22 2c 22 70-61 79 6d 65 6e 74 44 61 xxxx","paymentDa
00c0 - 74 61 54 79 70 65 22 3a-22 33 44 53 65 63 75 72 taType":"3DSecur
00d0 - 65 22 2c 22 70 61 79 6d-65 6e 74 44 61 74 61 22 e","paymentData"
00e0 - 3a 7b 22 6f 6e 6c 69 6e-65 50 61 79 6d 65 6e 74 :{"onlinePayment
00f0 - 43 72 79 70 74 6f 67 72-61 6d 22 3a 22 xx xx xx Cryptogram":"xxx
0100 - xx xx xx xx xx xx xx xx-xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0110 - xx xx xx xx xx xx xx xx-xx 22 2c 22 65 63 69 49 xxxxxxxxx","eciI
0120 - 6e 64 69 63 61 74 6f 72-22 3a 22 35 22 7d 7d 0b ndicator":"5"}}.
0130 - ce 56 54 48 49 1c 73 1a-b3 a4 89 e8 b2 11 f7 .VTHI.s........
OpenSSL Error after DecryptFinal_ex: error:00000000:lib(0):func(0):reason(0)
Plaintext after DecryptFinal(0 bytes):
0000 - 7b 22 61 70 70 6c 69 63-61 74 69 6f 6e 50 72 69 {"applicationPri
0010 - 6d 61 72 79 41 63 63 6f-75 6e 74 4e 75 6d 62 65 maryAccountNumbe
0020 - 72 22 3a 22 34 30 38 38-xx xx xx xx xx xx xx xx r":"4088xxxxxxxx
0030 - 30 30 32 32 22 2c 22 61-70 70 6c 69 63 61 74 69 0022","applicati
0040 - 6f 6e 45 78 70 69 72 61-74 69 6f 6e 44 61 74 65 onExpirationDate
0050 - 22 3a 22 32 30 30 32 32-38 22 2c 22 63 75 72 72 ":"200228","curr
0060 - 65 6e 63 79 43 6f 64 65-22 3a 22 38 34 30 22 2c encyCode":"840",
0070 - 22 74 72 61 6e 73 61 63-74 69 6f 6e 41 6d 6f 75 "transactionAmou
0080 - 6e 74 22 3a 33 30 30 2c-22 64 65 76 69 63 65 4d nt":300,"deviceM
0090 - 61 6e 75 66 61 63 74 75-72 65 72 49 64 65 6e 74 anufacturerIdent
00a0 - 69 66 69 65 72 22 3a 22-xx xx xx xx xx xx xx xx ifier":"xxxxxxxx
00b0 - xx xx xx xx 22 2c 22 70-61 79 6d 65 6e 74 44 61 xxxx","paymentDa
00c0 - 74 61 54 79 70 65 22 3a-22 33 44 53 65 63 75 72 taType":"3DSecur
00d0 - 65 22 2c 22 70 61 79 6d-65 6e 74 44 61 74 61 22 e","paymentData"
00e0 - 3a 7b 22 6f 6e 6c 69 6e-65 50 61 79 6d 65 6e 74 :{"onlinePayment
00f0 - 43 72 79 70 74 6f 67 72-61 6d 22 3a 22 xx xx xx Cryptogram":"xxx
0100 - xx xx xx xx xx xx xx xx-xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0110 - xx xx xx xx xx xx xx xx-xx 22 2c 22 65 63 69 49 xxxxxxxxx","eciI
0120 - 6e 64 69 63 61 74 6f 72-22 3a 22 35 22 7d 7d 0b ndicator":"5"}}.
0130 - ce 56 54 48 49 1c 73 1a-b3 a4 89 e8 b2 11 f7 .VTHI.s........

HTTP 400 on IIS 6.0

I'm trying to create HTTP request to XML service and I'm getting 400 errors from IIS server.
This request WORKS:
T 192.168.0.10:52584 -> 193.189.144.141:80 [AP]
50 4f 53 54 20 2f 73 63 72 69 70 74 73 2f 58 4d POST /scripts/XM
4c 5f 49 6e 74 65 72 66 61 63 65 2e 64 6c 6c 20 L_Interface.dll
48 54 54 50 2f 31 2e 30 0d 0a HTTP/1.0..
###
T 192.168.0.10:52584 -> 193.189.144.141:80 [AP]
48 6f 73 74 3a 20 77 77 77 31 2e 67 6e 74 2e 6c Host: www1.gnt.l
74 0d 0a 43 6f 6e 74 65 6e 74 2d 74 79 70 65 3a t..Content-type:
20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 application/x-w
77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 ww-form-urlencod
65 64 0d 0a 43 6f 6e 74 65 6e 74 2d 6c 65 6e 67 ed..Content-leng
74 68 3a 20 37 33 0d 0a 43 6f 6e 6e 65 63 74 69 th: 73..Connecti
6f 6e 3a 20 63 6c 6f 73 65 0d 0a 0d 0a 4d 66 63 on: close....Mfc
49 53 41 50 49 43 6f 6d 6d 61 6e 64 3d 44 65 66 ISAPICommand=Def
61 75 6c 74 26 55 53 45 52 4e 41 4d 45 3d 73 65 ault&USERNAME=se
63 72 65 74 26 50 41 53 53 57 4f 52 44 3d 73 65 cret&PASSWORD=se
63 72 65 74 26 43 48 45 43 4b 3d 73 65 63 72 65 cret&CHECK=secre
74 26 58 4d 4c 3d 0d 0a 0d 0a t&XML=....
Although this one DOESN'T WORK:
T 192.168.0.10:52592 -> 193.189.144.141:80 [AP]
50 4f 53 54 20 2f 73 63 72 69 70 74 73 2f 58 4d POST /scripts/XM
4c 5f 49 6e 74 65 72 66 61 63 65 2e 64 6c 6c 20 L_Interface.dll
48 54 54 50 2f 31 2e 30 0d 0a 43 6f 6e 74 65 6e HTTP/1.0..Conten
74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 t-Type: applicat
69 6f 6e 2f 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 ion/x-www-form-u
72 6c 65 6e 63 6f 64 65 64 0d 0a 43 6f 6e 74 65 rlencoded..Conte
6e 74 2d 4c 65 6e 67 74 68 3a 20 37 32 0d 0a 48 nt-Length: 72..H
6f 73 74 3a 20 77 77 77 31 2e 67 6e 74 2e 6c 74 ost: www1.gnt.lt
0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 50 79 ..User-Agent: Py
74 68 6f 6e 2d 75 72 6c 6c 69 62 2f 31 2e 31 37 thon-urllib/1.17
0d 0a 0d 0a 58 4d 4c 3d 26 55 53 45 52 4e 41 4d ....XML=&USERNAM
45 3d 73 65 63 72 65 74 26 50 41 53 53 57 4f 52 E=secret&PASSWOR
44 3d 73 65 63 72 65 74 26 4d 66 63 49 53 41 50 D=secret&MfcISAP
49 43 6f 6d 6d 61 6e 64 3d 44 65 66 61 75 6c 74 ICommand=Default
26 43 48 45 43 4b 3d 63 68 65 63 6b &CHECK=check
Exact error message:
HTTP/1.1 400 Bad Request.
Server: Microsoft-IIS/6.0.
X-Powered-By: ASP.NET.
Date: Wed, 13 Jul 2011 20:53:07 GMT.
Connection: close.
.
Any ideas where's the issue?

Resolved by changing order of arguments.
Made MfcISAP=Default the first argument

Resources