I am sending HTTP requests using socket.http to a Shelly 2.5 relay IOT device that contains a Mongoose/6.18 server running on an ESP8266, but am having problems.
HTTP requests to this device work as expected when initiated from postman or browser: the device's server responds as expected.
However if I send requests using Lua 5.3 socket.http module, the server does NOT respond. I've used tcpdump to see what's going on and from what I see, unlike browsers or postman which send the HTTP request in one TCP message containing both the HTTP startline and headers, Lua seems to send two separate TCP messages, first for the startline only, then a second for the headers. Mongoose responds with an ack when the startline is sent, then Lua sends the headers and Mongoose does not respond. Lua then tries 5 times to resend the headers before giving up.
I don't normally have problems using socket.http when sending HTTP requests to other servers.
I'm not sure if the 2-part HTTP message is what is causing the issue with this particular device, or if I'm perhaps misinterpreting the tcpdump output:
18:24:23.823886 IP 192.168.1.140.41894 > 192.168.1.136.http: Flags [P.], seq 1:151, ack 1, win 65392, length 150: HTTP: GET /settings/actions?index=0&name=out_on_url&enabled=true&urls[]=http://192.168.1.107:6667/b6cb7ffc-7919-41e6-ab80-6bed8b117753/relay/0/on HTTP/1.1
E.....#.#.6............Pj.n.....P..p....GET /settings/actions?index=0&name=out_on_url&enabled=true&urls[]=http://192.168.1.107:6667/b6cb7ffc-7919-41e6-ab80-6bed8b117753/relay/0/on HTTP/1.1
18:24:23.898872 IP 192.168.1.136.http > 192.168.1.140.41894: Flags [.], ack 151, win 2770, length 0
E..(.6#...v5.........P......j.o:P.
.......5...
18:24:23.898946 IP 192.168.1.140.41894 > 192.168.1.136.http: Flags [P.], seq 151:447, ack 1, win 65392, length 296: HTTP
E..P..#.#.5{...........Pj.o:....P..p....Cache-Control: max-age=0
Host: 192.168.1.136
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Connection: keep-alive
User-Agent: LuaSocket 3.0-rc1
TE: trailers
Accept-Language: en-US,en;q=0.9
18:24:24.125746 IP 192.168.1.140.41894 > 192.168.1.136.http: Flags [P.], seq 151:447, ack 1, win 65392, length 296: HTTP
E..P..#.#.5z...........Pj.o:....P..p....Cache-Control: max-age=0
Host: 192.168.1.136
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Connection: keep-alive
User-Agent: LuaSocket 3.0-rc1
TE: trailers
Accept-Language: en-US,en;q=0.9
18:24:24.575743 IP 192.168.1.140.41894 > 192.168.1.136.http: Flags [P.], seq 151:447, ack 1, win 65392, length 296: HTTP
E..P..#.#.5y...........Pj.o:....P..p....Cache-Control: max-age=0
Host: 192.168.1.136
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Connection: keep-alive
User-Agent: LuaSocket 3.0-rc1
TE: trailers
Accept-Language: en-US,en;q=0.9
18:24:25.495791 IP 192.168.1.140.41894 > 192.168.1.136.http: Flags [P.], seq 151:447, ack 1, win 65392, length 296: HTTP
E..P..#.#.5x...........Pj.o:....P..p....Cache-Control: max-age=0
Host: 192.168.1.136
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Connection: keep-alive
User-Agent: LuaSocket 3.0-rc1
TE: trailers
Accept-Language: en-US,en;q=0.9
18:24:27.328180 ARP, Request who-has 192.168.1.136 tell 192.168.1.1, length 46
........`.'.................>.[p.....!.....
>.
18:24:27.335732 IP 192.168.1.140.41894 > 192.168.1.136.http: Flags [P.], seq 151:447, ack 1, win 65392, length 296: HTTP
E..P..#.#.5w...........Pj.o:....P..p....Cache-Control: max-age=0
Host: 192.168.1.136
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Connection: keep-alive
User-Agent: LuaSocket 3.0-rc1
TE: trailers
Accept-Language: en-US,en;q=0.9
Just to follow up on this - it turns out the device I was sending the message to was spontaneously restarting as soon as it read the first part of the message - thus its failure to receive the rest and respond. Obviously a bug in their firmware, and unfortunately Shelly wasn't too interested in fixing it after several messages and data dumps sent to their tech support.
Related
Thats from wikipedia:
For version 1.1 of the HTTP protocol, the chunked transfer mechanism is considered to be always and anyways acceptable, even if not listed in the TE (transfer encoding) request header field
Thats what I get from clients (Mozilla, Opera):
GET http://www.google.com/ HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Apparently there is neither Transfer-Encoding field there, nor I see any chunks (I've checked with HEX editor, no additional symbols).
I open connection as follows (Python)
socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Is it lower level handling joins chunks into message? Is so, how can I know where the HTTP message ends so that I can stop reading the request and start handling it?
You should read the specification.
But simply, in this case, since it's a GET, and there's not content, there's not going to be a Content-Length header. So, you stop reading when you get the empty line with just a CR/LF.
Otherwise, you read past that blank line, and read Content-Length bytes.
I am sending the following HTTP request:
POST /input/8dZ8bgapvjfYzmwWno6W.txt HTTP/1.1
Host: data.sparkfun.com
Phant-Private-Key: pz5ga4pkydHgpEb8v608
Connection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: 7
temp=44
In my code, I am sending it using UART tx requests to the xbee module, for which the translates to:
POST /input/8dZ8bgapvjfYzmwWno6Wr.txt HTTP/1.1\r\n
Host: data.sparkfun.com\r\n
Phant-Private-Key: pz5ga4pkydHgpEb8v608\r\n
Connection: close\r\n
Content-Type: application/x-www-form-urlencoded\r\n
Content-Length: 7\r\n
\r\n
temp=44
This is to communicate to the phant dataserver at data.sparkfun.com, and it responds with the following data:
HTTP/1.0 400 Bad request
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>
I found the answer:
The packet is correct.
While configuring the Xbee Wifi module using XCTU, I had to give the correct port numbers of the server and client Xbee, which were wrong.
Server was 80, client was any port I think.
My client is sending:
POST /xxx/yyy HTTP/1.1
Host: localhost:9009
User-Agent: gSOAP/2.8
Content-Type: text/xml; charset=utf-8
Content-Length: 2442
Connection: keep-alive
SOAPAction: ""
But the server replies:
HTTP/1.1 200 OK
Content-Type: text/xml;charset=UTF-8
Content-Length: 11182
Server: Jetty(8.1.14.v20131031)
Isn't the server supposed to return "Connection: keep-alive" too?
I see that afterwards the client closes the connection although it is configure to keep the connection open.
I assumed it is because the server didn't provide the keep-alive in the reply (Is that the RFC?).
In my case the reason gSoap closed the connection wasn't related to the HTTP headers returned from server, but to the fact you need to specify the keep-alive flags on both directions by calling:
soap_set_imode(this, SOAP_IO_KEEPALIVE);
soap_set_omode(this, SOAP_IO_KEEPALIVE);
From what I've read in HTTP 1.1 persistent connections are the default so if the server didn't return "Connection: close" the connection can be used for next request too.
I am loading a MP3 stream from IceCast 2.3.2-kh29 server in the Android app with MediaPlayer class.
Playing works well, but sometimes stops happen. If see the server responses in IcyStreamMeta class for ID3 tags, there is 404 error for this case.
Also it happens in Windows 7: Firefox and other browsers.
Here are normal headers (some data ***ed):
http://***:14534/***.mp3
GET /***.mp3 HTTP/1.1
Host: ***:14534
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Server: nginx/1.4.1
Date: Tue, 23 Jul 2013 21:22:00 GMT
Content-Type: audio/mpeg
Transfer-Encoding: chunked
Connection: keep-alive
icy-br: 192
ice-audio-info: bitrate=192;samplerate=44100;channels=2
icy-description: MP3 192 Kbps
icy-genre: ***
icy-name: ***
icy-pub: 1
icy-url: ***
Cache-Control: no-cache
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
So, the stream sometimes plays only about a minute or less, sometimes seconds and stops. What's the possible reason of 404 error? In other devices there were tests with stable work. Internet speed is well. Can router cause such things? Also, maybe some special HTTP request headers are needed for IceCast (and if they're not present, it gives 404)? Or it's an internal server error for some cases?
So, from WireShark:
2973 53.630385000 SERVER'S IP 192.168.100.6 TCP 1466 14534 > 59847 [ACK] Seq=1284017 Ack=1 Win=63 Len=1412
2976 53.636352000 SERVER'S IP 192.168.100.6 TCP 1157 14534 > 59847 [PSH, ACK] Seq=1285429 Ack=1 Win=63 Len=1103
2978 53.671606000 SERVER'S IP 192.168.100.6 TCP 60 14534 > 59847 [PSH, ACK] Seq=1286532 Ack=1 Win=63 Len=5
2980 53.678606000 SERVER'S IP 192.168.100.6 TCP 60 14534 > 59847 [FIN, ACK] Seq=1286537 Ack=2 Win=63 Len=0
The issue is your chunked encoding. You're proxying your stream through Nginx, and Nginx is "fixing" the output to be compatible with HTTP/1.0. Don't do that.
You can try turning off chunked encoding in your Nginx config:
chunked_transfer_encoding off
I'd like to pass a raw HTTP request like:
GET /foo/bar HTTP/1.1
Host: example.org
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Accept: */*
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: http://example.org/test
Cookie: foo=bar; lorem=ipsum;
to a HTTP client.
I tried cat raw.http | curl but without success.
Any suggestion?
Thx.
Raw data in, raw data out:
nc example.org 80 < raw.http
If you need to pipe the data through some program:
cat raw.http | someprogram | nc example.org 80
Manual page
The question is tagged curl so I thought it was about time there was a curl answer
cat raw.http | curl "telnet://TARGETHOST:80"
For normal use just need to set the TARGETHOST to be the same as "host" header value.
For my purposes(not normal) I was hitting a TARGETHOST that was an ip address with a server that was listening for host headers of specific hosts.
Note that neither of these solutions would work if your need httpS instead of http. In this case you can send it this way:
$ cat raw.http | openssl s_client -connect server:443