I'm creating a low level HTTP server and I want to validate that my responses are correct.
Some of them are fine, but others fail. I have saved some responses as bytes and viewing them in HxD, and they look correct, but they obviously aren't in some way.
Here's the start of one HTTP response formatted as hex:
48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D 0A 43 4F 4E 54 45 4E
54 2D 54 59 50 45 3A 20 69 6D 61 67 65 2F 70 6E 67 0D 0A 4C 41 53 54
2D 4D 4F 44 49 46 49 45 44 3A 20 53 75 6E 2C 20 30 31 20 4D 61 79 20
32 30 32 32 20 32 32 3A 32 34 3A 32 34 20 47 4D 54 0D 0A 41 43 43 45
50 54 2D 52 41 4E 47 45 53 3A 20 62 79 74 65 73 0D 0A 45 54 41 47 3A
20 22 37 35 37 35 35 33 36 61 61 35 64 64 38 31 3A 30 22 0D 0A 53 45
52 56 45 52 3A 20 4D 69 63 72 6F 73 6F 66 74 2D 49 49 53 2F 31 30 2E
30 0D 0A 58 2D 50 4F 57 45 52 45 44 2D 42 59 3A 20 41 53 50 2E 4E 45
54 0D 0A 44 41 54 45 3A 20 57 65 64 2C 20 30 34 20 4D 61 79 20 32 30
32 32 20 30 39 3A 30 31 3A 34 38 20 47 4D 54 0D 0A 43 4F 4E 54 45 4E
54 2D 4C 45 4E 47 54 48 3A 20 39 38 37 35 37 0D 0A 43 6F 6E 6E 65 63
74 69 6F 6E 3A 20 63 6C 6F 73 65 0D 0A 0D 0A 0D 0A 89 50 4E 47
How can I validate that the response I'm creating is valid according to RFC 2616?
I'm working in .NET but I don't mind if it's an online tool or a different platform.
Edit:
I've solved my issue with this particular example (3 crlfs instead of 2 between head and body) but I'd still like to know how I can validate these requests automatically.
I ended up achieving this with this nuget package and the following code:
if (System.Diagnostics.Debugger.IsAttached)
{
using (var handler = new HttpParserDelegate())
using (var parser = new HttpCombinedParser(handler))
{
parser.Execute(arr);
}
if (!handler.HttpRequestResponse.IsEndOfMessage)
{
System.Diagnostics.Debugger.Break();
throw new InvalidOperationException($"An HTTP response was serialized but it is not a valid response");
}
}
I have this string:
x <- "W3siY29kZSI6IkIxMTgyIiwiZGVzY3JpcHRpb24iOiJUaXJlIHByZXNzdXJlIG1vbml0b3IgbW9kdWxlIiwiZmF1bHRJbmZvcm1hdGlvbnMiOlt7ImRlc2NyaXB0aW9uIjoiLS0tIn0seyJkZXNjcmlwdGlvbiI6IlRSVUUiLCJkZXNjcmlwdGlvbl9lbmciOiJUUlVFIiwiZGVzY3JpcHRpb25fZnJlIjoiVFJVRSIsImRlc2NyaXB0aW9uX2dlciI6IlRSVUUiLCJkZXNjcmlwdGlvbl9zcGEiOiJUUlVFIiwiZGVzY3JpcHRpb25faXRhIjoiVFJVRSIsImRlc2NyaXB0aW9uX3R1ciI6IlRSVUUifV0sImZyZWV6ZUZyYW1lcyI6W10sImVuaGFuY2VkRHRjSW5mb3MiOltdLCJzdGF0dXMiOiJBQ1RJVkUiLCJzeXN0ZW0iOiJNVUxUSUZVTkNUSU9OIiwic3lzdGVtTWV0YSI6eyJzeXN0ZW1OYW1lIjpudWxsLCJzdWJTeXN0ZW1OYW1lIjpudWxsfSwiZGVzY3JpcHRpb25fZW5nIjoiVGlyZSBwcmVzc3VyZSBtb25pdG9yIG1vZHVsZSIsImRlc2NyaXB0aW9uX2ZyZSI6Ik1vZHVsZSBkZSBzdXJ2ZWlsbGFuY2UgZGUgbGEgcHJlc3Npb24gZGVzIHBuZXVzIiwiZGVzY3JpcHRpb25fZ2VyIjoiUmVpZmVuZHJ1Y2vDvGJlcndhY2h1bmciLCJkZXNjcmlwdGlvbl9zcGEiOiJNw7NkdWxvIGRlIGNvbnRyb2wgZGUgbGEgcHJlc2nDs24gZGUgbG9zIG5ldW3DoXRpY29zIiwiZGVzY3JpcHRpb25faXRhIjoiTW9kdWxvIG1vbml0b3IgZGkgcHJlc3Npb25lIHBuZXVtYXRpY28iLCJkZXNjcmlwdGlvbl90dXIiOiJMYXN0aWsgYmFzxLFuY8SxIGl6bGVtZSBtb2TDvGzDvCJ9LHsiY29kZSI6IkIxMjREIiwiZGVzY3JpcHRpb24iOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJmYXVsdEluZm9ybWF0aW9ucyI6W3siZGVzY3JpcHRpb24iOiItLS0ifSx7ImRlc2NyaXB0aW9uIjoiVFJVRSIsImRlc2NyaXB0aW9uX2VuZyI6IlRSVUUiLCJkZXNjcmlwdGlvbl9mcmUiOiJUUlVFIiwiZGVzY3JpcHRpb25fZ2VyIjoiVFJVRSIsImRlc2NyaXB0aW9uX3NwYSI6IlRSVUUiLCJkZXNjcmlwdGlvbl9pdGEiOiJUUlVFIiwiZGVzY3JpcHRpb25fdHVyIjoiVFJVRSJ9XSwiZnJlZXplRnJhbWVzIjpbXSwiZW5oYW5jZWREdGNJbmZvcyI6W10sInN0YXR1cyI6IkFDVElWRSIsInN5c3RlbSI6Ik1VTFRJRlVOQ1RJT04iLCJzeXN0ZW1NZXRhIjp7InN5c3RlbU5hbWUiOm51bGwsInN1YlN5c3RlbU5hbWUiOm51bGx9LCJkZXNjcmlwdGlvbl9lbmciOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJkZXNjcmlwdGlvbl9mcmUiOiJDYXB0ZXVyLCBwcmVzc2lvbiBkZXMgcG5ldXMiLCJkZXNjcmlwdGlvbl9nZXIiOiJTZW5zb3IgUmVpZmVuZHJ1Y2siLCJkZXNjcmlwdGlvbl9zcGEiOiJTZW5zb3IgZGUgcHJlc2nDs24gZGVsIG5ldW3DoXRpY28iLCJkZXNjcmlwdGlvbl9pdGEiOiJTZW5zb3JlLCBwcmVzc2lvbmUgcG5ldW1hdGljaSIsImRlc2NyaXB0aW9uX3R1ciI6IlNlbnPDtnIsIGxhc3RpayBoYXZhIGJhc8SxbmPEsSJ9XQ=="
I know it has been encoded using base64. Doing a quick search on internet, I found that in R there are some packages to encode/decode base64.
Package jsonlite with the function base64dec()
Package base64decode with the function base64decode().
In both cases, when running...
base64dec(x)
base64decode(x)
I get this result, which is a binary output:
[1] 5b 7b 22 63 6f 64 65 22 3a 22 42 31 31 38 32 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 69 72 65 20 70 72 65 73 73 75 72 65 20 6d 6f 6e 69 74 6f 72
[54] 20 6d 6f 64 75 6c 65 22 2c 22 66 61 75 6c 74 49 6e 66 6f 72 6d 61 74 69 6f 6e 73 22 3a 5b 7b 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 2d 2d 2d 22 7d 2c 7b
[107] 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 65 6e 67 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69
[160] 70 74 69 6f 6e 5f 66 72 65 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 67 65 72 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69
[213] 6f 6e 5f 73 70 61 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 69 74 61 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f
[266] 74 75 72 22 3a 22 54 52 55 45 22 7d 5d 2c 22 66 72 65 65 7a 65 46 72 61 6d 65 73 22 3a 5b 5d 2c 22 65 6e 68 61 6e 63 65 64 44 74 63 49 6e 66 6f 73 22 3a 5b 5d
[319] 2c 22 73 74 61 74 75 73 22 3a 22 41 43 54 49 56 45 22 2c 22 73 79 73 74 65 6d 22 3a 22 4d 55 4c 54 49 46 55 4e 43 54 49 4f 4e 22 2c 22 73 79 73 74 65 6d 4d 65
[372] 74 61 22 3a 7b 22 73 79 73 74 65 6d 4e 61 6d 65 22 3a 6e 75 6c 6c 2c 22 73 75 62 53 79 73 74 65 6d 4e 61 6d 65 22 3a 6e 75 6c 6c 7d 2c 22 64 65 73 63 72 69 70
[425] 74 69 6f 6e 5f 65 6e 67 22 3a 22 54 69 72 65 20 70 72 65 73 73 75 72 65 20 6d 6f 6e 69 74 6f 72 20 6d 6f 64 75 6c 65 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e
[478] 5f 66 72 65 22 3a 22 4d 6f 64 75 6c 65 20 64 65 20 73 75 72 76 65 69 6c 6c 61 6e 63 65 20 64 65 20 6c 61 20 70 72 65 73 73 69 6f 6e 20 64 65 73 20 70 6e 65 75
[531] 73 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 67 65 72 22 3a 22 52 65 69 66 65 6e 64 72 75 63 6b c3 bc 62 65 72 77 61 63 68 75 6e 67 22 2c 22 64 65 73 63 72
[584] 69 70 74 69 6f 6e 5f 73 70 61 22 3a 22 4d c3 b3 64 75 6c 6f 20 64 65 20 63 6f 6e 74 72 6f 6c 20 64 65 20 6c 61 20 70 72 65 73 69 c3 b3 6e 20 64 65 20 6c 6f 73
[637] 20 6e 65 75 6d c3 a1 74 69 63 6f 73 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 69 74 61 22 3a 22 4d 6f 64 75 6c 6f 20 6d 6f 6e 69 74 6f 72 20 64 69 20 70 72
[690] 65 73 73 69 6f 6e 65 20 70 6e 65 75 6d 61 74 69 63 6f 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 74 75 72 22 3a 22 4c 61 73 74 69 6b 20 62 61 73 c4 b1 6e 63
[743] c4 b1 20 69 7a 6c 65 6d 65 20 6d 6f 64 c3 bc 6c c3 bc 22 7d 2c 7b 22 63 6f 64 65 22 3a 22 42 31 32 34 44 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 53
[796] 65 6e 73 6f 72 2c 20 74 79 72 65 20 70 72 65 73 73 75 72 65 22 2c 22 66 61 75 6c 74 49 6e 66 6f 72 6d 61 74 69 6f 6e 73 22 3a 5b 7b 22 64 65 73 63 72 69 70 74
[849] 69 6f 6e 22 3a 22 2d 2d 2d 22 7d 2c 7b 22 64 65 73 63 72 69 70 74 69 6f 6e 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 65 6e 67 22 3a 22
[902] 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 66 72 65 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 67 65 72 22 3a 22 54 52 55
[955] 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 73 70 61 22 3a 22 54 52 55 45 22 2c 22 64 65 73 63 72 69 70 74 69 6f 6e 5f 69 74 61 22 3a
However, when I copy that string into this web, I get this result:
[{"code":"B1182","description":"Tire pressure monitor module","faultInformations":[{"description":"---"},{"description":"TRUE","description_eng":"TRUE","description_fre":"TRUE","description_ger":"TRUE","description_spa":"TRUE","description_ita":"TRUE","description_tur":"TRUE"}],"freezeFrames":[],"enhancedDtcInfos":[],"status":"ACTIVE","system":"MULTIFUNCTION","systemMeta":{"systemName":null,"subSystemName":null},"description_eng":"Tire pressure monitor module","description_fre":"Module de surveillance de la pression des pneus","description_ger":"Reifendrucküberwachung","description_spa":"Módulo de control de la presión de los neumáticos","description_ita":"Modulo monitor di pressione pneumatico","description_tur":"Lastik basıncı izleme modülü"},{"code":"B124D","description":"Sensor, tyre pressure","faultInformations":[{"description":"---"},{"description":"TRUE","description_eng":"TRUE","description_fre":"TRUE","description_ger":"TRUE","description_spa":"TRUE","description_ita":"TRUE","description_tur":"TRUE"}],"freezeFrames":[],"enhancedDtcInfos":[],"status":"ACTIVE","system":"MULTIFUNCTION","systemMeta":{"systemName":null,"subSystemName":null},"description_eng":"Sensor, tyre pressure","description_fre":"Capteur, pression des pneus","description_ger":"Sensor Reifendruck","description_spa":"Sensor de presión del neumático","description_ita":"Sensore, pressione pneumatici","description_tur":"Sensör, lastik hava basıncı"}]
This is actually the expected output. What's wrong with the code?
Looks like the output is raw encoded. So you'll need to use rawToChar.
library(base64)
rawToChar(base64_decode(x))
[1] "[{\"code\":\"B1182\",\"description\":\"Tire pressure monitor module\",\"faultInformations\":[{\"description\":\"---\"},{\"description\":\"TRUE\",\"description_eng\":\"TRUE\",\"description_fre\":\"TRUE\",\"description_ger\":\"TRUE\",\"description_spa\":\"TRUE\",\"description_ita\":\"TRUE\",\"description_tur\":\"TRUE\"}],\"freezeFrames\":[],\"enhancedDtcInfos\":[],\"status\":\"ACTIVE\",\"system\":\"MULTIFUNCTION\",\"systemMeta\":{\"systemName\":null,\"subSystemName\":null},\"description_eng\":\"Tire pressure monitor module\",\"description_fre\":\"Module de surveillance de la pression des pneus\",\"description_ger\":\"Reifendrucküberwachung\",\"description_spa\":\"Módulo de control de la presión de los neumáticos\",\"description_ita\":\"Modulo monitor di pressione pneumatico\",\"description_tur\":\"Lastik basıncı izleme modülü\"},{\"code\":\"B124D\",\"description\":\"Sensor, tyre pressure\",\"faultInformations\":[{\"description\":\"---\"},{\"description\":\"TRUE\",\"description_eng\":\"TRUE\",\"description_fre\":\"TRUE\",\"description_ger\":\"TRUE\",\"description_spa\":\"TRUE\",\"description_ita\":\"TRUE\",\"description_tur\":\"TRUE\"}],\"freezeFrames\":[],\"enhancedDtcInfos\":[],\"status\":\"ACTIVE\",\"system\":\"MULTIFUNCTION\",\"systemMeta\":{\"systemName\":null,\"subSystemName\":null},\"description_eng\":\"Sensor, tyre pressure\",\"description_fre\":\"Capteur, pression des pneus\",\"description_ger\":\"Sensor Reifendruck\",\"description_spa\":\"Sensor de presión del neumático\",\"description_ita\":\"Sensore, pressione pneumatici\",\"description_tur\":\"Sensör, lastik hava basıncı\"}]"
Edit
To better understand why this is necessary we can check the class of the output of base64_decode().
> class(base64_decode(x))
#[1] "raw"
The ?raw helpfully tells us:
The raw type is intended to hold raw bytes. It is possible to extract subsequences of bytes, and to replace elements (but only by elements of a raw vector). ... A raw vector is printed with each byte separately represented as a pair of hex digits. If you want to see a character representation (with escape sequences for non-printing characters) use rawToChar.
So we just need to convert the raw bytes to character.
Data
x <- "W3siY29kZSI6IkIxMTgyIiwiZGVzY3JpcHRpb24iOiJUaXJlIHByZXNzdXJlIG1vbml0b3IgbW9kdWxlIiwiZmF1bHRJbmZvcm1hdGlvbnMiOlt7ImRlc2NyaXB0aW9uIjoiLS0tIn0seyJkZXNjcmlwdGlvbiI6IlRSVUUiLCJkZXNjcmlwdGlvbl9lbmciOiJUUlVFIiwiZGVzY3JpcHRpb25fZnJlIjoiVFJVRSIsImRlc2NyaXB0aW9uX2dlciI6IlRSVUUiLCJkZXNjcmlwdGlvbl9zcGEiOiJUUlVFIiwiZGVzY3JpcHRpb25faXRhIjoiVFJVRSIsImRlc2NyaXB0aW9uX3R1ciI6IlRSVUUifV0sImZyZWV6ZUZyYW1lcyI6W10sImVuaGFuY2VkRHRjSW5mb3MiOltdLCJzdGF0dXMiOiJBQ1RJVkUiLCJzeXN0ZW0iOiJNVUxUSUZVTkNUSU9OIiwic3lzdGVtTWV0YSI6eyJzeXN0ZW1OYW1lIjpudWxsLCJzdWJTeXN0ZW1OYW1lIjpudWxsfSwiZGVzY3JpcHRpb25fZW5nIjoiVGlyZSBwcmVzc3VyZSBtb25pdG9yIG1vZHVsZSIsImRlc2NyaXB0aW9uX2ZyZSI6Ik1vZHVsZSBkZSBzdXJ2ZWlsbGFuY2UgZGUgbGEgcHJlc3Npb24gZGVzIHBuZXVzIiwiZGVzY3JpcHRpb25fZ2VyIjoiUmVpZmVuZHJ1Y2vDvGJlcndhY2h1bmciLCJkZXNjcmlwdGlvbl9zcGEiOiJNw7NkdWxvIGRlIGNvbnRyb2wgZGUgbGEgcHJlc2nDs24gZGUgbG9zIG5ldW3DoXRpY29zIiwiZGVzY3JpcHRpb25faXRhIjoiTW9kdWxvIG1vbml0b3IgZGkgcHJlc3Npb25lIHBuZXVtYXRpY28iLCJkZXNjcmlwdGlvbl90dXIiOiJMYXN0aWsgYmFzxLFuY8SxIGl6bGVtZSBtb2TDvGzDvCJ9LHsiY29kZSI6IkIxMjREIiwiZGVzY3JpcHRpb24iOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJmYXVsdEluZm9ybWF0aW9ucyI6W3siZGVzY3JpcHRpb24iOiItLS0ifSx7ImRlc2NyaXB0aW9uIjoiVFJVRSIsImRlc2NyaXB0aW9uX2VuZyI6IlRSVUUiLCJkZXNjcmlwdGlvbl9mcmUiOiJUUlVFIiwiZGVzY3JpcHRpb25fZ2VyIjoiVFJVRSIsImRlc2NyaXB0aW9uX3NwYSI6IlRSVUUiLCJkZXNjcmlwdGlvbl9pdGEiOiJUUlVFIiwiZGVzY3JpcHRpb25fdHVyIjoiVFJVRSJ9XSwiZnJlZXplRnJhbWVzIjpbXSwiZW5oYW5jZWREdGNJbmZvcyI6W10sInN0YXR1cyI6IkFDVElWRSIsInN5c3RlbSI6Ik1VTFRJRlVOQ1RJT04iLCJzeXN0ZW1NZXRhIjp7InN5c3RlbU5hbWUiOm51bGwsInN1YlN5c3RlbU5hbWUiOm51bGx9LCJkZXNjcmlwdGlvbl9lbmciOiJTZW5zb3IsIHR5cmUgcHJlc3N1cmUiLCJkZXNjcmlwdGlvbl9mcmUiOiJDYXB0ZXVyLCBwcmVzc2lvbiBkZXMgcG5ldXMiLCJkZXNjcmlwdGlvbl9nZXIiOiJTZW5zb3IgUmVpZmVuZHJ1Y2siLCJkZXNjcmlwdGlvbl9zcGEiOiJTZW5zb3IgZGUgcHJlc2nDs24gZGVsIG5ldW3DoXRpY28iLCJkZXNjcmlwdGlvbl9pdGEiOiJTZW5zb3JlLCBwcmVzc2lvbmUgcG5ldW1hdGljaSIsImRlc2NyaXB0aW9uX3R1ciI6IlNlbnPDtnIsIGxhc3RpayBoYXZhIGJhc8SxbmPEsSJ9XQ=="
I need to make a REST call within a RHEL kickstart %pre script, and thus I am limited to using netcat (since the wget packaged in the RHEL %pre environment can't configure the HTTP method). I'd of course love to use curl (since it has the lovely -X option) but alas it's not available in the %pre environment.
That said, here's a relevant curl command and, importantly, the exact stream of bytes it sends to the server:
$ curl -X POST http://pkrizak-globalpxe.anonycom.com/univac/api/record/pkrizak-sles10.anonycom.com/_install_log --data-binary '[ ]' --trace /tmp/foo.log
$ cat /tmp/foo.log (truncated)
== Info: About to connect() to pkrizak-globalpxe.anonycom.com port 80 (#0)
== Info: Trying 10.46.174.117... == Info: connected
== Info: Connected to pkrizak-globalpxe.anonycom.com (10.46.174.117) port 80 (#0)
=> Send header, 312 bytes (0x138)
0000: 50 4f 53 54 20 2f 75 6e 69 76 61 63 2f 61 70 69 POST /univac/api
0010: 2f 72 65 63 6f 72 64 2f 70 6b 72 69 7a 61 6b 2d /record/pkrizak-
0020: 73 6c 65 73 31 30 2e 61 63 6f 63 79 63 6f 6d 2e sles10.anonycom.
0030: 63 6f 6d 2f 5f 69 6e 73 74 61 6c 6c 5f 6c 6f 67 com/_install_log
0040: 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 65 72 2d HTTP/1.1..User-
0050: 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e 31 39 Agent: curl/7.19
0060: 2e 37 20 28 78 38 36 5f 36 34 2d 72 65 64 68 61 .7 (x86_64-redha
0070: 74 2d 6c 69 6e 75 78 2d 67 6e 75 29 20 6c 69 62 t-linux-gnu) lib
0080: 63 75 72 6c 2f 37 2e 31 39 2e 37 20 4e 53 53 2f curl/7.19.7 NSS/
0090: 33 2e 31 34 2e 33 2e 30 20 7a 6c 69 62 2f 31 2e 3.14.3.0 zlib/1.
00a0: 32 2e 33 20 6c 69 62 69 64 6e 2f 31 2e 31 38 20 2.3 libidn/1.18
00b0: 6c 69 62 73 73 68 32 2f 31 2e 34 2e 32 0d 0a 48 libssh2/1.4.2..H
00c0: 6f 73 74 3a 20 70 6b 72 69 7a 61 6b 2d 67 6c 6f ost: pkrizak-glo
00d0: 62 61 6c 70 78 65 2e 61 63 6f 63 79 63 6f 6d 2e balpxe.anonycom.
00e0: 63 6f 6d 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a com..Accept: */*
00f0: 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 ..Content-Length
0100: 3a 20 33 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 : 3..Content-Typ
0110: 65 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 e: application/x
0120: 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 -www-form-urlenc
0130: 6f 64 65 64 0d 0a 0d 0a oded....
=> Send data, 3 bytes (0x3)
0000: 5b 20 5d [ ]
== Info: HTTP 1.0, assume close after body
<= Recv header, 24 bytes (0x18)
I've truncated the rest, but suffice to say that the transaction continues without problems.
Now, sending the exact same stream of bytes using netcat:
$ ./mypost.sh | hexdump -C
00000000 50 4f 53 54 20 2f 75 6e 69 76 61 63 2f 61 70 69 |POST /univac/api|
00000010 2f 72 65 63 6f 72 64 2f 70 6b 72 69 7a 61 6b 2d |/record/pkrizak-|
00000020 73 6c 65 73 31 30 2e 61 63 6f 63 79 63 6f 6d 2e |sles10.anonycom.|
00000030 63 6f 6d 2f 5f 69 6e 73 74 61 6c 6c 5f 6c 6f 67 |com/_install_log|
00000040 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 65 72 2d | HTTP/1.1..User-|
00000050 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e 31 39 |Agent: curl/7.19|
00000060 2e 37 20 28 78 38 36 5f 36 34 2d 72 65 64 68 61 |.7 (x86_64-redha|
00000070 74 2d 6c 69 6e 75 78 2d 67 6e 75 29 20 6c 69 62 |t-linux-gnu) lib|
00000080 63 75 72 6c 2f 37 2e 31 39 2e 37 20 4e 53 53 2f |curl/7.19.7 NSS/|
00000090 33 2e 31 34 2e 33 2e 30 20 7a 6c 69 62 2f 31 2e |3.14.3.0 zlib/1.|
000000a0 32 2e 33 20 6c 69 62 69 64 6e 2f 31 2e 31 38 20 |2.3 libidn/1.18 |
000000b0 6c 69 62 73 73 68 32 2f 31 2e 34 2e 32 0d 0a 48 |libssh2/1.4.2..H|
000000c0 6f 73 74 3a 20 70 6b 72 69 7a 61 6b 2d 67 6c 6f |ost: pkrizak-glo|
000000d0 62 61 6c 70 78 65 2e 61 63 6f 63 79 63 6f 6d 2e |balpxe.anonycom.|
000000e0 63 6f 6d 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a |com..Accept: */*|
000000f0 0d 0a 43 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 |..Content-length|
00000100 3a 20 33 0d 0a 43 6f 6e 74 65 6e 74 2d 74 79 70 |: 3..Content-typ|
00000110 65 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 |e: application/x|
00000120 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 |-www-form-urlenc|
00000130 6f 64 65 64 0d 0a 0d 0a 5b 20 5d |oded....[ ]|
0000013b
...doesn't work:
$ ./mypost.sh | nc pkrizak-globalpxe.anonycom.com 80
$ # no response
The tricky bit here is that there is a squid (reverse) proxy listening on port 80 of the host I'm connecting to. So I'm not actually talking to Apache or Nginx or even my custom Perl application -- I'm trying to talk to Squid (which when approached with curl actually forwards the request properly to my application). And apparently something is different about how curl talks to Squid than how nc is talking to squid, despite the content of the request being identical.
I've tried changing the request to use HTTP/1.0 and even tried leaving the HTTP/ part off, but that does not help.
I'm really baffled at this point -- what is going on under the hood that I'm missing? Why is netcat behaving differently?
Turns out this is an oddity in the way that netcat behaves when piped data on STDIN.
After tracing the packets with wireshark, I found that netcat, after sending the data piped to STDIN, immediately sends a FIN,ACK packet to the server. The squid server, naturally, aborts processing of the request upon receipt of the FIN,ACK packet and closes the connection.
This behavior can be avoided by using the -i option to netcat, which specifies an interval time between transactions. Using -i 1, for example, waits one second before sending the FIN,ACK after sending its data. This is long enough for the squid proxy to return with an answer.
Another solution is to have the script generating the input to STDIN pause after writing out the HTTP POST information. A simple sleep 1 at the end of the script appears to be sufficient to get the proxy to complete the request.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 years ago.
Improve this question
I'm analyzing a TCPdump file with Wireshark. Within the connection traces, I saw "brvread" port 1054 as both source and destination ports. When I searched it in the Internet, the only thing I found out was that it may be an indication of an attack or vulnerability. But apart from that I could not find any other information about what brvread is. Does anybody have any idea about it?
The package looks like this (Data from Angelo Neuschitzer):
# Time Source Target Protoco Length Info
1 0.000000000 192.168.2.107 239.255.255.250 UDP 1310 Source port: brvread Destination port: us-cli
Content:
NOTIFY * HTTP/1.1
x-type: localDvr
x-filter: 5107dcd0-aed6-4f2a-aa93-b5fea9caffec
x-lastUserActivity: 12/23/2013 10:03:29 AM
x-location: http://192.168.2.107:8080/dvrfs/info.xml
x-device: 3244238e-0e41-4f90-ae8a-35b8c84a11a2
x-debug: http://192.168.2.107:8080
<node count='961525'>
<activities>
<p15n stamp='08CF44F5A880AA10ECE09BE967E7'/>
<schedver dver='3' ver='600' pendcap='False' />
<x/>
<recreq src='udp://239.35.20.43:10000:79b02293-93df-46f1-976d-c651c578fed7?r=3537009&p=1&ssrc0=514095545&r0=3537009&ch=11&profile=multicastICC&forceDetune=true&age=-1&skip=0' st='0x0' et='0xFFFFFFFFFFFFFFFF' postpad='0' rate='3537009' pri='1'/>
<recordver ver='1' verid='0' size='137438953472' free='136834973696' />
<x/>
<tune src='udp://239.35.20.43:10000:79b02293-93df-46f1-976d-c651c578fed7' pipe='FULLSCREEN' ct='0xd6628ecdc4833e0d' pil='0x0' rate='0x35f871' stopped='false'/>
<tune src='udp://239.35.20.43:10000:79b02293-93df-46f1-976d-c651c578fed7' rate='0x35f871' pil='0x0'/>
<record url='http://192.168.2.107:8080/dvrfs/v17' src='udp://239.35.20.43:10000:79b02293-93df-46f1-976d-c651c578fed7' pri='1' st='0xd66288ea884a134e' et='0xd6628eced30f4a81' stopped='false'/>
</activities>
</node>
(Indentation manually, there are no newlines etc. in package)
Following is a dump from wireshark:
0000 01 00 5e 7f ff fa 00 23 a3 97 87 d1 08 00 45 60 ..^....#......E`
0010 05 10 12 4d 00 00 01 11 ef 22 c0 a8 02 6b ef ff ...M....."...k..
0020 ff fa 04 1e 1f 92 04 fc 98 bd 02 57 9c 74 3e 4c ...........W.t>L
0030 ad cc 43 83 bb 3c a2 de 24 9c 64 00 21 00 10 00 ..C..<..$.d.!...
0040 31 04 c3 d6 62 8e d5 8a 16 3f dd 7d 1a a7 28 ac 1...b....?.}..(.
0050 4c 21 c7 d1 24 5f a6 55 a6 5b e5 4e 4f 54 49 46 L!..$_.U.[.NOTIF
0060 59 20 2a 20 48 54 54 50 2f 31 2e 31 0d 0a 78 2d Y * HTTP/1.1..x-
0070 74 79 70 65 3a 20 6c 6f 63 61 6c 44 76 72 0d 0a type: localDvr..
0080 78 2d 66 69 6c 74 65 72 3a 20 35 31 30 37 64 63 x-filter: 5107dc
0090 64 30 2d 61 65 64 36 2d 34 66 32 61 2d 61 61 39 d0-aed6-4f2a-aa9
00a0 33 2d 62 35 66 65 61 39 63 61 66 66 65 63 0d 0a 3-b5fea9caffec..
00b0 78 2d 6c 61 73 74 55 73 65 72 41 63 74 69 76 69 x-lastUserActivi
00c0 74 79 3a 20 31 32 2f 32 33 2f 32 30 31 33 20 31 ty: 12/23/2013 1
00d0 30 3a 30 33 3a 32 39 20 41 4d 0d 0a 78 2d 6c 6f 0:03:29 AM..x-lo
00e0 63 61 74 69 6f 6e 3a 20 68 74 74 70 3a 2f 2f 31 cation: http://1
00f0 39 32 2e 31 36 38 2e 32 2e 31 30 37 3a 38 30 38 92.168.2.107:808
0100 30 2f 64 76 72 66 73 2f 69 6e 66 6f 2e 78 6d 6c 0/dvrfs/info.xml
0110 0d 0a 78 2d 64 65 76 69 63 65 3a 20 33 32 34 34 ..x-device: 3244
0120 32 33 38 65 2d 30 65 34 31 2d 34 66 39 30 2d 61 238e-0e41-4f90-a
0130 65 38 61 2d 33 35 62 38 63 38 34 61 31 31 61 32 e8a-35b8c84a11a2
0140 0d 0a 78 2d 64 65 62 75 67 3a 20 68 74 74 70 3a ..x-debug: http:
0150 2f 2f 31 39 32 2e 31 36 38 2e 32 2e 31 30 37 3a //192.168.2.107:
0160 38 30 38 30 0d 0a 0d 0a 3c 6e 6f 64 65 20 63 6f 8080....<node co
0170 75 6e 74 3d 27 39 36 31 35 32 35 27 3e 3c 61 63 unt='961525'><ac
0180 74 69 76 69 74 69 65 73 3e 3c 70 31 35 6e 20 73 tivities><p15n s
0190 74 61 6d 70 3d 27 30 38 43 46 34 34 46 35 41 38 tamp='08CF44F5A8
01a0 38 30 41 41 31 30 45 43 45 30 39 42 45 39 36 37 80AA10ECE09BE967
01b0 45 37 27 2f 3e 3c 73 63 68 65 64 76 65 72 20 64 E7'/><schedver d
01c0 76 65 72 3d 27 33 27 20 76 65 72 3d 27 36 30 30 ver='3' ver='600
01d0 27 20 70 65 6e 64 63 61 70 3d 27 46 61 6c 73 65 ' pendcap='False
01e0 27 20 2f 3e 3c 78 2f 3e 3c 72 65 63 72 65 71 20 ' /><x/><recreq
01f0 73 72 63 3d 27 75 64 70 3a 2f 2f 32 33 39 2e 33 src='udp://239.3
0200 35 2e 32 30 2e 34 33 3a 31 30 30 30 30 3a 37 39 5.20.43:10000:79
0210 62 30 32 32 39 33 2d 39 33 64 66 2d 34 36 66 31 b02293-93df-46f1
0220 2d 39 37 36 64 2d 63 36 35 31 63 35 37 38 66 65 -976d-c651c578fe
0230 64 37 3f 72 3d 33 35 33 37 30 30 39 26 61 6d 70 d7?r=3537009&
0240 3b 70 3d 31 26 61 6d 70 3b 73 73 72 63 30 3d 35 ;p=1&ssrc0=5
0250 31 34 30 39 35 35 34 35 26 61 6d 70 3b 72 30 3d 14095545&r0=
0260 33 35 33 37 30 30 39 26 61 6d 70 3b 63 68 3d 31 3537009&ch=1
0270 31 26 61 6d 70 3b 70 72 6f 66 69 6c 65 3d 6d 75 1&profile=mu
0280 6c 74 69 63 61 73 74 49 43 43 26 61 6d 70 3b 66 lticastICC&f
0290 6f 72 63 65 44 65 74 75 6e 65 3d 74 72 75 65 26 orceDetune=true&
02a0 61 6d 70 3b 61 67 65 3d 2d 31 26 61 6d 70 3b 73 amp;age=-1&s
02b0 6b 69 70 3d 30 27 20 73 74 3d 27 30 78 30 27 20 kip=0' st='0x0'
02c0 65 74 3d 27 30 78 46 46 46 46 46 46 46 46 46 46 et='0xFFFFFFFFFF
02d0 46 46 46 46 46 46 27 20 70 6f 73 74 70 61 64 3d FFFFFF' postpad=
02e0 27 30 27 20 72 61 74 65 3d 27 33 35 33 37 30 30 '0' rate='353700
02f0 39 27 20 70 72 69 3d 27 31 27 2f 3e 3c 72 65 63 9' pri='1'/><rec
0300 6f 72 64 76 65 72 20 76 65 72 3d 27 31 27 20 76 ordver ver='1' v
0310 65 72 69 64 3d 27 30 27 20 73 69 7a 65 3d 27 31 erid='0' size='1
0320 33 37 34 33 38 39 35 33 34 37 32 27 20 66 72 65 37438953472' fre
0330 65 3d 27 31 33 36 38 33 34 39 37 33 36 39 36 27 e='136834973696'
0340 20 2f 3e 3c 78 2f 3e 3c 74 75 6e 65 20 73 72 63 /><x/><tune src
0350 3d 27 75 64 70 3a 2f 2f 32 33 39 2e 33 35 2e 32 ='udp://239.35.2
0360 30 2e 34 33 3a 31 30 30 30 30 3a 37 39 62 30 32 0.43:10000:79b02
0370 32 39 33 2d 39 33 64 66 2d 34 36 66 31 2d 39 37 293-93df-46f1-97
0380 36 64 2d 63 36 35 31 63 35 37 38 66 65 64 37 27 6d-c651c578fed7'
0390 20 70 69 70 65 3d 27 46 55 4c 4c 53 43 52 45 45 pipe='FULLSCREE
03a0 4e 27 20 63 74 3d 27 30 78 64 36 36 32 38 65 63 N' ct='0xd6628ec
03b0 64 63 34 38 33 33 65 30 64 27 20 70 69 6c 3d 27 dc4833e0d' pil='
03c0 30 78 30 27 20 72 61 74 65 3d 27 30 78 33 35 66 0x0' rate='0x35f
03d0 38 37 31 27 20 73 74 6f 70 70 65 64 3d 27 66 61 871' stopped='fa
03e0 6c 73 65 27 2f 3e 3c 74 75 6e 65 20 73 72 63 3d lse'/><tune src=
03f0 27 75 64 70 3a 2f 2f 32 33 39 2e 33 35 2e 32 30 'udp://239.35.20
0400 2e 34 33 3a 31 30 30 30 30 3a 37 39 62 30 32 32 .43:10000:79b022
0410 39 33 2d 39 33 64 66 2d 34 36 66 31 2d 39 37 36 93-93df-46f1-976
0420 64 2d 63 36 35 31 63 35 37 38 66 65 64 37 27 20 d-c651c578fed7'
0430 72 61 74 65 3d 27 30 78 33 35 66 38 37 31 27 20 rate='0x35f871'
0440 70 69 6c 3d 27 30 78 30 27 2f 3e 3c 72 65 63 6f pil='0x0'/><reco
0450 72 64 20 75 72 6c 3d 27 68 74 74 70 3a 2f 2f 31 rd url='http://1
0460 39 32 2e 31 36 38 2e 32 2e 31 30 37 3a 38 30 38 92.168.2.107:808
0470 30 2f 64 76 72 66 73 2f 76 31 37 27 20 73 72 63 0/dvrfs/v17' src
0480 3d 27 75 64 70 3a 2f 2f 32 33 39 2e 33 35 2e 32 ='udp://239.35.2
0490 30 2e 34 33 3a 31 30 30 30 30 3a 37 39 62 30 32 0.43:10000:79b02
04a0 32 39 33 2d 39 33 64 66 2d 34 36 66 31 2d 39 37 293-93df-46f1-97
04b0 36 64 2d 63 36 35 31 63 35 37 38 66 65 64 37 27 6d-c651c578fed7'
04c0 20 70 72 69 3d 27 31 27 20 73 74 3d 27 30 78 64 pri='1' st='0xd
04d0 36 36 32 38 38 65 61 38 38 34 61 31 33 34 65 27 66288ea884a134e'
04e0 20 65 74 3d 27 30 78 64 36 36 32 38 65 63 65 64 et='0xd6628eced
04f0 33 30 66 34 61 38 31 27 20 73 74 6f 70 70 65 64 30f4a81' stopped
0500 3d 27 66 61 6c 73 65 27 2f 3e 3c 2f 61 63 74 69 ='false'/></acti
0510 76 69 74 69 65 73 3e 3c 2f 6e 6f 64 65 3e vities></node>
Ok, it is much easier to proceed having all those details available.
What you actually see is Simple Service Discovery Protocol (SSDP) message. (Wikipedia)
brvread is an old name, came from IANA and used by Wireshark.
This port is also associated with AckCmd trojan. Interesting fact:
The interesting feature about this backdoor is that it only uses ACK
packets. This means that a standard connection is not established;
rather, data will be transmitted directly using ACK packets. This
makes it possible for the Trojan to bypass some firewalls.
So I think the actual explanation is quite trivial: some other service sits on that port and talks SSDP.
I'm trying to create HTTP request to XML service and I'm getting 400 errors from IIS server.
This request WORKS:
T 192.168.0.10:52584 -> 193.189.144.141:80 [AP]
50 4f 53 54 20 2f 73 63 72 69 70 74 73 2f 58 4d POST /scripts/XM
4c 5f 49 6e 74 65 72 66 61 63 65 2e 64 6c 6c 20 L_Interface.dll
48 54 54 50 2f 31 2e 30 0d 0a HTTP/1.0..
###
T 192.168.0.10:52584 -> 193.189.144.141:80 [AP]
48 6f 73 74 3a 20 77 77 77 31 2e 67 6e 74 2e 6c Host: www1.gnt.l
74 0d 0a 43 6f 6e 74 65 6e 74 2d 74 79 70 65 3a t..Content-type:
20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 application/x-w
77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 ww-form-urlencod
65 64 0d 0a 43 6f 6e 74 65 6e 74 2d 6c 65 6e 67 ed..Content-leng
74 68 3a 20 37 33 0d 0a 43 6f 6e 6e 65 63 74 69 th: 73..Connecti
6f 6e 3a 20 63 6c 6f 73 65 0d 0a 0d 0a 4d 66 63 on: close....Mfc
49 53 41 50 49 43 6f 6d 6d 61 6e 64 3d 44 65 66 ISAPICommand=Def
61 75 6c 74 26 55 53 45 52 4e 41 4d 45 3d 73 65 ault&USERNAME=se
63 72 65 74 26 50 41 53 53 57 4f 52 44 3d 73 65 cret&PASSWORD=se
63 72 65 74 26 43 48 45 43 4b 3d 73 65 63 72 65 cret&CHECK=secre
74 26 58 4d 4c 3d 0d 0a 0d 0a t&XML=....
Although this one DOESN'T WORK:
T 192.168.0.10:52592 -> 193.189.144.141:80 [AP]
50 4f 53 54 20 2f 73 63 72 69 70 74 73 2f 58 4d POST /scripts/XM
4c 5f 49 6e 74 65 72 66 61 63 65 2e 64 6c 6c 20 L_Interface.dll
48 54 54 50 2f 31 2e 30 0d 0a 43 6f 6e 74 65 6e HTTP/1.0..Conten
74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 t-Type: applicat
69 6f 6e 2f 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 ion/x-www-form-u
72 6c 65 6e 63 6f 64 65 64 0d 0a 43 6f 6e 74 65 rlencoded..Conte
6e 74 2d 4c 65 6e 67 74 68 3a 20 37 32 0d 0a 48 nt-Length: 72..H
6f 73 74 3a 20 77 77 77 31 2e 67 6e 74 2e 6c 74 ost: www1.gnt.lt
0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 50 79 ..User-Agent: Py
74 68 6f 6e 2d 75 72 6c 6c 69 62 2f 31 2e 31 37 thon-urllib/1.17
0d 0a 0d 0a 58 4d 4c 3d 26 55 53 45 52 4e 41 4d ....XML=&USERNAM
45 3d 73 65 63 72 65 74 26 50 41 53 53 57 4f 52 E=secret&PASSWOR
44 3d 73 65 63 72 65 74 26 4d 66 63 49 53 41 50 D=secret&MfcISAP
49 43 6f 6d 6d 61 6e 64 3d 44 65 66 61 75 6c 74 ICommand=Default
26 43 48 45 43 4b 3d 63 68 65 63 6b &CHECK=check
Exact error message:
HTTP/1.1 400 Bad Request.
Server: Microsoft-IIS/6.0.
X-Powered-By: ASP.NET.
Date: Wed, 13 Jul 2011 20:53:07 GMT.
Connection: close.
.
Any ideas where's the issue?
Resolved by changing order of arguments.
Made MfcISAP=Default the first argument