How to parse chunked HTTP content with Lua on nodemcu? - http

Question

I have script which coomunicates between nodemcu and my server. It works good on my localhost and is parsing response retrieved from my server when I send GET request. Problem is when I upload it all on my website where transfer encoding is chunked. I am not able to retrieve content, although request is legitimate and correct. Code is written in Lua and I am trying to work on my NodeMCU device.
conn=net.createConnection(net.TCP, 0)
conn:on("connection",function(conn, payload)
conn:send("GET /mypath/node.php?id=1&update"..
" HTTP/1.1\r\n"..
"Host: www.mydomain.com\r\n"..
"Accept: */*\r\n"..
"User-Agent: Mozilla/4.0 (compatible; esp8266 Lua;)"..
"\r\n\r\n")
end)
conn:on("receive", function(conn, payload)
if string.find(payload, "UPDATE")~=nil then
node.restart()
end
conn:close()
conn = nil
end)
conn:connect(80,"www.mydomain.com")
end
Just to repeat that this GET request works and is tested manualy and on localhost. Only problem is with chunked content, I don't know how to parse it.
Update: I managed to remove chunked encoding by changing HTTP/1.1 to HTTP/1.0, but still I have problem
using this code
conn:on("receive", function(conn, payload)
print(payload)
I get this response
HTTP/1.1 200 OK
Date: Tue, 09 Jan 2018 02:34:25 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=9m226vr20r4baa634bagk8k2k3; path=/
Connection: close
Content-Type: text/html; charset=utf-8
Update 2.
I have just created one file http.php with text included "php". I have uploaded it to localhost and to my domain. Once I tried to access my localhost from nodemcu, and then to domain. Results were different
This is the request
conn:send("GET /"..s.path.."/http.php"..
" HTTP/1.0\r\n"..
"Host: "..s.domain.."\r\n"..
"Accept: */*\r\n"..
"User-Agent: Mozilla/4.0 (compatible; esp8266 Lua;)"..
"\r\n\r\n")
end)
s.domain and s.path correcponds to different paths and domains on localhost and my domain
Result on domain
HTTP/1.1 200 OK
Date: Tue, 09 Jan 2018 03:09:28 GMT
Server: Apache
Connection: close
Content-Type: text/html; charset=UTF-8
result on localhost
TTP/1.1 200 OK
Date: Tue, 09 Jan 2018 03:08:48 GMT
Server: Apache/2.4.27 (Win64) PHP/7.0.23
X-Powered-By: PHP/7.0.23
Content-Length: 3
Connection: close
Content-Type: text/html; charset=UTF-8
php
As you can see, localhost is showing content "php", and domain is showing only header. When I type some file which does not exists domain is showing me html code.

Answer 1

I'm using the following code to put the chunks together. I'm wondering anyways, why your response from the server is missing the Content-Length header.
conn:on("receive", function(client, payload)
-- Inspired by https://github.com/marcoskirsch/nodemcu-httpserver/blob/master/httpserver.lua
-- Collect data packets until the size of HTTP body meets the Content-Length stated in header
if payload:find("Content%-Length:") or bBodyMissing then
if fullPayload then fullPayload = fullPayload .. payload else fullPayload = payload end
if (tonumber(string.match(fullPayload, "%d+", fullPayload:find("Content%-Length:")+16)) > #fullPayload:sub(fullPayload:find("\r\n\r\n", 1, true)+4, #fullPayload)) then
bBodyMissing = true
return
else
payload = fullPayload
fullPayload, bBodyMissing = nil
end
end
if (bBodyMissing == nil) then
local _, headerEnd = payload:find("\r\n\r\n")
local body = payload:sub(headerEnd + 1)
print (body)
end
end)