Decoding http2 frame header/data in Go - http

I am trying to read from a tcp connection which contains HTTP/2 data. Below is the code for reading HEADERS frame -
framer := http2.NewFramer(conn, conn)
frame, _ := framer.ReadFrame()
fmt.Printf("fh type: %s\n", frame.Header().Type)
fmt.Printf("fh type: %d\n", frame.Header().Type)
fmt.Printf("fh flag: %d\n", frame.Header().Flags)
fmt.Printf("fh length: %d\n", frame.Header().Length)
fmt.Printf("fh streamid: %d\n", frame.Header().StreamID)
headersframe := (frame1.(*http2.HeadersFrame))
fmt.Printf("stream ended? %v\n", headersframe.StreamEnded())
fmt.Printf("block fragment: %x\n", headersframe.HeaderBlockFragment())
I send request using curl as -
curl -v https://127.0.0.1:8000/ -k --http2
This is the output I get (after reading connection preface and SETTINGS), if I read from the conn using above code -
fh type: HEADERS
fh type: 1
fh flag: 5
fh length: 30
fh streamid: 1
stream ended? true
block fragment: 828487418a089d5c0b8170dc6c4d8b7a8825b650c3abb6f2e053032a2f2a
I understand the ouput, except the block fragment part and how to decode it into ascii string? I want to know the protocol/method/url path information.

The "header block fragment" is encoded using HPACK.
Go has an implementation to encode and decode HPACK, so you don't have to write your own.
You can find here an example of using both the encoder and decoder Go API.

I figured it out using Go hpack library (https://godoc.org/golang.org/x/net/http2/hpack) -
decoder := hpack.NewDecoder(2048, nil)
hf, _ := decoder.DecodeFull(headersframe.HeaderBlockFragment())
for _, h := range hf {
fmt.Printf("%s\n", h.Name + ":" + h.Value)
}
This prints -
:method:GET
:path:/
:scheme:https
:authority:127.0.0.1:5252
user-agent:curl/7.58.0
accept:*/*

Related

Lua - encrypt / decrypt commands with aes cbc to pair with a Panasonic TV

I’m trying to rework a script I found online to control a Panasonic TV, which requires a secure/encrypted pairing to occur so I can control it remotely. (The full code here -> https://forum.logicmachine.net/showthread.php?tid=232&pid=16580#pid16580)
Because it seems to be built on LuaJIT and has some other proprietary Lua elements; I’m trying to find alternatives that will allow it to work with the 5.1 Lua install on a Vera Home Automation controller (a relatively closed system).
Also, and perhaps most important for me is that I’d love to make as much of the converted code have minimal requirements to call external modules. I should add I’ve only recently started learning Lua, but one way I like to learn is to convert/repurpose code I find online..
So far i’ve managed to find alternatives for a number of the modules being used, e.g
encdec.base64dec -> Lua Base64 Encode
lmcore.hextostr -> https://github.com/tst2005/binascii/blob/master/binascii.lua
storage.set -> Alternative found in Vera Home Controllers
storage.get -> Alternative found in Vera Home Controllers
bit.ban -> Bitware module in Vera Home Controllers
bit.bxor -> Bitware module in Vera Home Controllers
Where I’m stuck is with the following..
aes:new
aes.cipher
user.aes
encdec.hmacsha256
Here’s an extract of the code where the above are used.
function encrypt_soap_payload(data, key, hmac_key, iv)
payload = '000000000000'
n = #data
payload = payload .. string.char(bit.band(bit.rshift(n, 24), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 16), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 8), 0xFF))
payload = payload .. string.char(bit.band(n, 0xFF))
payload = payload .. data
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 1)
ciphertext = aes_cbc:encrypt(payload)
sig = encdec.hmacsha256(ciphertext, hmac_key, true)
encrypted_payload = encdec.base64enc(ciphertext .. sig)
return encrypted_payload
end
function decrypt_soap_payload(data, key, hmac_key, iv)
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 0)
decrypted = aes_cbc:decrypt(encdec.base64dec(data))
decrypted = string.gsub(string.sub(lmcore.strtohex(decrypted), 33), '%x%x', function(value) return string.char(tonumber(value, 16)) end)
return decrypted
end
I can get the the point where I can create the parameters for the payload encrypt request (example below), it’s the encryption/decryption I can do..
data="1234"
key="\\S„ßÍ}/Ìa5!"
hmac_key="¹jz¹2¸F\r}òcžÎ„ 臧.ª˜¹=¤µæŸ"
iv=" {¬£áæ‚2žâ3ÐÞË€ú "
I’ve found an aes.lua module online, but that requires loads of others modules most notably ffi.lua. Ideally I’d like to avoid using that. I also came across this aes128.lua but i’m not sure how that handles all the other parameters e.g cbc etc. Finally there’s this aes256ecb.lua script, could that be converted to aes 128 cbc and then used in the above?
Is anyone aware (or maybe has) a Lua script that can handle the aes cbc requirements above ?
Many thanks !
In the end I found out that I could do aes.cbc by calling openssl from the command line, e.g.
local payload = "ENTER HERE"
Local key = "ENTER HERE"
local iv = "ENTER HERE"
local buildsslcommand = "openssl enc -aes-128-cbc -nosalt -e -a -A "..payload.." -K "..key.." -iv "..iv
-- print("Command to send = " ..buildsslcommand)
local file = assert(io.popen(buildsslcommand, 'r'))
local output = file:read('*all')
file:close()
-- print(string.len(output)) --> just count what's returned.
-- print(output) -- > Prints the output of the command.
FYI - It looks like I could do encdec.hmacsha256 via openSSL as well, but I’ve not been able to do that :-( ..

Translate a working curl request into HTTP

I'm using curl to perform a POST request, but I can't assume my target platform to have curl available, so I'm trying to rewrite my curl request in HTTP (which is guaranteed to be available). My knowledge of both curl and HTTP is very limited, so I'm hoping someone can point out what I'm doing wrong.
My curl request (command line):
curl.exe POST https://xxxxxx.ingest.sentry.io/api/xxxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxx -F upload_file_minidump=#"C:\path\Minidump.dmp" -F upload_file_log=#"C:\path\program.log"
A relevant part of curl's output is shown below. This is after connecting to the server and sending it the POST request. The server now lets the client know the first file can be sent, and curl responds first by sending the file's own header, and then the data (clamped here)
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 175 bytes (0xaf)
0000: --------------------------f2a4a742c08bf427
002c: Content-Disposition: form-data; name="upload_file_minidump"; fil
006c: ename="UE4Minidump.dmp"
0085: Content-Type: application/octet-stream
00ad:
=> Send data, 16384 bytes (0x4000)
0000: MDMP..a..... .......m/S`.........................;..............
0040: 8Z......T...=...........`.......8...........T................[..
0080: .........\...........]..........= ..............................
00c0: ....................................aJ.......`......Lw..........
0100: ............T........?..i/S`........ ... ... ............ ......
0140: ............G.M.T. .S.t.a.n.d.a.r.d. .T.i.m.e...................
0180: ................................G.M.T. .D.a.y.l.i.g.h.t. .T.i.m.
01c0: e...................................................1.9.0.4.1...
..etc..
By reading the verbose output of curl, I've created a HTTP request looking like this (c++ code using unreal engine 4 libraries):
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> httpRequest = FHttpModule::Get().CreateRequest();
httpRequest->SetURL(TEXT("https://xxxxxx.ingest.sentry.io/api/xxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxx"));
httpRequest->SetVerb(TEXT("POST"));
const FString boundary(TEXT("------------------------f2a4a742c08bf427"));
httpRequest->SetHeader(TEXT("Content-Type"), TEXT("multipart/form-data; boundary=") + boundary);
const FString fileName(FPaths::Combine(path, crashToReport.folderName, TEXT("UE4Minidump.dmp")));
ensure(FPaths::FileExists(fileName));
const FString prefixBoundary(TEXT("\r\n--") + boundary + TEXT("\r\n"));
const FString fileHeader(TEXT("Content-Disposition: form-data; name=\"upload_file_minidump\"; filename=\"UE4Minidump.dmp\"\r\nContent-Type: application/octet-stream\r\n\r\n"));
FString fileContents;
FFileHelper::LoadFileToString(fileContents, *fileName);
const FString suffixBoundary(TEXT("\r\n--") + boundary + TEXT("--\r\n"));
const FString content(prefixBoundary + fileHeader + fileContents + suffixBoundary);
httpRequest->SetContentAsString(content);
This works to a degree, the server now accepts this, and will receive the file - however the file ends up being unreadable server-side, leading me to think I'm not sending it in the right format.
What kind of data is expected in a multipart/form-data request?
A thing I notice is that the curl request sends the file's header separately (the first chunk of 175 bytes). I would love some information on how to achieve that!
I finally figured it out. I'm not sure how to report what I was doing wrong, but I think it had to do with what happens under the hood in:
httpRequest->SetContentAsString(..)
, which reliably caused the backend to fail to interpret the binary file I was trying to send. I ended up reading the binary file .. as a binary file:
TArray<uint8> dumpFileData;
FFileHelper::LoadFileToArray(dumpFileData, *FPaths::Combine(path,crashToReport.folderName, TEXT("UE4Minidump.dmp")));
Then send it via the POST request, similarly as before, but adding the complete form data part as binary data:
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> httpRequest = FHttpModule::Get().CreateRequest();
httpRequest->SetURL(TEXT("https://xxxxx.ingest.sentry.io/api/xxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxxxxx"));
httpRequest->SetVerb(TEXT("POST"));
const FString boundary(TEXT("------------------------bb33b671b1212234"));
httpRequest->SetHeader(TEXT("Content-Type"), TEXT("multipart/form-data; boundary=") + boundary);
httpRequest->SetHeader(TEXT("Accept"), TEXT("*/*"));
httpRequest->SetHeader(TEXT("Expect"), TEXT("100-continue"));
{
const FString prefixBoundary(TEXT("--") + boundary + TEXT("\r\n"));
const FString fileHeader(TEXT("Content-Disposition: form-data; name=\"upload_file_minidump\"; filename=\"UE4Minidump.dmp\"\nContent-Type: application/octet-stream\r\n\r\n"));
const FString suffixBoundary(TEXT("\r\n--") + boundary + TEXT("--\r\n"));
TArray<uint8> CombinedContent;
CombinedContent.Append(FStringToUint8(prefixBoundary + fileHeader));
CombinedContent.Append(dumpFileData);
CombinedContent.Append(FStringToUint8(suffixBoundary));
httpRequest->SetContent(CombinedContent);
}
httpRequest->ProcessRequest();
For completeness, FStringToUint8 is defined as follows:
// Convert FString to UTF8 and put it in a TArray
TArray<uint8> FStringToUint8(const FString& InString)
{
TArray<uint8> OutBytes;
// Handle empty strings
if (InString.Len() > 0)
{
FTCHARToUTF8 Converted(*InString); // Convert to UTF8
OutBytes.Append(reinterpret_cast<const uint8*>(Converted.Get()), Converted.Length());
}
return OutBytes;
}

NGINX LUA Content-Length +1 Byte Lost

I am having an interesting bug and method issue
Lua mentions that the js_content variable has a length of 80 bytes.
But when I don't use the "Content-Length" header, firefox mentions that 81 bytes of data are transferred.
I don't know where the +1 byte excess comes from
I will be glad if you can help, an application I wrote with VBNet gives an error when I noticed that the "Content-Length" header is 80 bytes while parsing json data from my remote server, but it works fine when I add +1.
local ref_array = {1, 2, 3}
local sArray = {}
sArray["1"] = "One"
sArray["2"] = "Two"
sArray["3"] = "Tree"
local ctable = {}
for index, data in ipairs(ref_array) do
if sArray[tostring(data)] ~= nil then
local cinfo = {}
cinfo["X"] = tostring(data)
cinfo["Y"] = sArray[tostring(data)]
cinfo["Z"] = 0
table.insert(ctable, cinfo)
end
end
local js_content = cjson.encode(ctable)
ngx.header['Content-Type'] = 'application/json'
ngx.header['Content-Length'] = #js_content -- 80 byte
ngx.say(js_content)
ngx.exit(200)
I guess the problem is the Line Feed Character character at the end.
ngx.say always adds linefeed
ngx.print is just output
problem solved
Linefeed Character

Accessing JSON data after Indy POST

I am struggling to know how to access the response to an Indy POST request. I post the data either as JSON or paramstring. My code when using JSON is as follows.
params := TStringList.Create;
try
params.Text :=
'{'
+ format ('"client_secret":"%s",', [FilesFrm.ClientSecret])
+ format ('"client_id":"%s",', [FilesFrm.ClientId])
+ '"grant_type":"authorization_code",'
+ '"redirect_uri":"http://localhost:8080",'
+ format ('"code":"%s"', [fCode])
+ '}';
idLogFile1.Active := true;
// Make sure it uses HTTP 1.1, not 1.0
IdHTTP1.HTTPOptions := IdHTTP1.HTTPOptions + [hoKeepOrigProtocol];
IdHTTP1.Request.ContentType := 'application/json';
IdHttp1.Request.Accept := 'application/vnd.hmrc.1.0+json';
try
result := IdHTTP1.Post (
'https://test-api.service.hmrc.gov.uk/oauth/token',
params);
except
on E: Exception do
memo1.lines.add (E.ClassName + ': ' + E.message);
end;
memo1.Lines.add (result);
memo1.Lines.add (idHTTP1.ResponseText);
finally
params.free;
end;
The result of printing out result and RepsonseText is just
EIdHTTPProtocolException: HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
However, because I have a TidLogFile component attached to the TidHTTP, I can see what actually arrives, which is the following.
Recv 2/1/2019 7:56:07 AM: HTTP/1.1 400 Bad Request<EOL>
Content-Type: application/json<EOL>
Content-Length: 72<EOL>
Cache-Control: no-cache,no-store,etc, etc...
; Secure; HTTPOnly<EOL><EOL>
{"error":"invalid_request","error_description":"grant_type is required"}
Aside from the fact that grant_type appears to be in the original request data, I would like to be able to access the JSON response at the end, since "grant_type_is_required" is much more helpful than "Bad request", but I cannot find where it is.
I have subsequently found Response.ContentLength, which contains the value 72, and Response.ContentStream, which should in theory contain the 72 bytes of data, but produces access violations when I try to extract the data.
len := idHTTP1.Response.ContentLength;
memo1.Lines.Add(format ('Length = %d', [len]));
if assigned (idHTTP1.Response.ContentStream) then
begin
//idHTTP1.Response.ContentStream.Position := 0;
result := ReadStringFromStream (idHTTP1.Response.ContentStream, len);
end;
memo1.lines.add (result);
In addition to mjn42's answer, which is technically correct, TIdHTTP also has optional hoNoProtocolErrorException and hoWantProtocolErrorContent flags in its HTTPOptions property, which you can enable to avoid the EIdHTTPProtocolException being raised and to populate your result variable with error data, respectively:
params := TStringStream.Create(
'{'
+ format ('"client_secret":"%s",', [FilesFrm.ClientSecret])
+ format ('"client_id":"%s",', [FilesFrm.ClientId])
+ '"grant_type":"authorization_code",'
+ '"redirect_uri":"http://localhost:8080",'
+ format ('"code":"%s"', [fCode])
+ '}',
TEncoding.UTF8);
try
IdLogFile1.Active := true;
// Make sure it uses HTTP 1.1, not 1.0,
// and disable EIdHTTPProtocolException on errors
IdHTTP1.ProtocolVersion := pv1_1;
IdHTTP1.HTTPOptions := IdHTTP1.HTTPOptions + [hoKeepOrigProtocol, hoNoProtocolErrorException, hoWantProtocolErrorContent];
IdHTTP1.Request.ContentType := 'application/json';
IdHTTP1.Request.Accept := 'application/vnd.hmrc.1.0+json';
try
result := IdHTTP1.Post('https://test-api.service.hmrc.gov.uk/oauth/token', params);
except
on E: Exception do begin
Memo1.Lines.Add(E.ClassName + ': ' + E.message);
raise;
end;
end;
Memo1.Lines.Add(result);
finally
params.Free;
end;
Here is an example which shows how the HTTP body can be accessed.
The body can be accessed if you catch the EIdHTTPProtocolException exception.
on E: EIdHTTPProtocolException do
begin
WriteLn(E.Message);
WriteLn(E.ErrorMessage);
end;
Full example code:
program JSONPostExample;
{$APPTYPE CONSOLE}
uses
IdHTTP, IdGlobal, SysUtils, Classes;
var
HTTP: TIdHTTP;
RequestBody: TStream;
ResponseBody: string;
begin
HTTP := TIdHTTP.Create;
try
try
RequestBody := TStringStream.Create('{"日本語":42}',
TEncoding.UTF8);
try
HTTP.Request.Accept := 'application/json';
HTTP.Request.ContentType := 'application/json';
ResponseBody := HTTP.Post('https://httpbin.org/post',
RequestBody);
WriteLn(ResponseBody);
WriteLn(HTTP.ResponseText);
finally
RequestBody.Free;
end;
except
on E: EIdHTTPProtocolException do
begin
WriteLn(E.Message);
WriteLn(E.ErrorMessage);
end;
on E: Exception do
begin
WriteLn(E.Message);
end;
end;
finally
HTTP.Free;
end;
ReadLn;
ReportMemoryLeaksOnShutdown := True;
end.
Note that you must not use a TStringList for the POST body. That version of TIdHTTP.Post() formats the data according to the application/x-www-form-urlencoded media type, which is not appropriate for JSON and will corrupt it.

What would cause curl POST with --data and #filename to only send 46% of the data?

I run:
curl -i -X POST -H "Content-Type: application/octet-stream" --data #race.mov "http://127.0.0.1:8080/api/v1/put_file"
where race.mov is 4134329 bytes. But only 1931871 are received by the program listening on port 8080.
so I try another file that's 2482905 bytes. But only 1150635 are received. Isn't that weird?
irb(main):002:0> 1931871.0 / 4134329.0
=> 0.4672755845023461
irb(main):003:0> 1150635.0 / 2482905.0
=> 0.4634228856923644
What's going on to make curl only send 46% of the binary data?
Update this same go program:
dat, err := ioutil.ReadFile("/Users/aa/Movies/race.mov")
fmt.Println(err)
endpoint := fmt.Sprintf("http://127.0.0.1:8080/api/v1/put_file")
fmt.Println(endpoint)
buffer_reader := bytes.NewReader(dat)
resp, err := http.Post(endpoint, "application/octet-stream", buffer_reader)
fmt.Println(resp, err)
makes the correct length happen so it MUST be curl

Resources