When looking at client->server interaction for fetching images, I see the following HTTP GET request from client where the packet contains 2 HTTP GET requests and I am not sure how the server would respond to such requests?
Will the server ignore the second GET request?
Will the server send the response one by one to each GET request?
This doesn't seem to be HTTP pipelining. Please advise if it is.
Transmission Control Protocol, Src Port: 59649 (59649), Dst Port: 8080 (8080), Seq: 1, Ack: 1, Len: 648
Source Port: 59649
Destination Port: 8080
[Stream index: 86]
[TCP Segment Len: 648]
Sequence number: 1 (relative sequence number)
[Next sequence number: 649 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
Header Length: 32 bytes
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: *******AP***]
Window size value: 683
[Calculated window size: 43712]
[Window size scaling factor: 64]
Checksum: [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
Timestamps: TSval 6345, TSecr 6344
Kind: Time Stamp Option (8)
Length: 10
Timestamp value: 6345
Timestamp echo reply: 6344
[SEQ/ACK analysis]
[iRTT: 0.000099000 seconds]
[Bytes in flight: 648]
Hypertext Transfer Protocol
GET HTTP/1.1\r\n
[Expert Info (Chat/Sequence): GET HTTP/1.1\r\n]
[GET HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]enter code here
Request Method: GET
Request URI:
Request Version: HTTP/1.1
Host: \r\n
sent: \r\n
User-Agent: \r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-GB,*\r\n
Connection: keep-alive\r\n
\r\n
[Full request URI: ]
[HTTP request 2/2]
[Prev request in frame: 1254]
[Response in frame: 1272]
Hypertext Transfer Protocol
GET \r\n
[Expert Info (Chat/Sequence): GET HTTP/1.1\r\n]
[GET HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: GET
Request URI:
Request Version: HTTP/1.1
Host: \r\n
sent: \r\n
User-Agent: \r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-GB,*\r\n
Connection: keep-alive\r\n
\r\n
[Full request URI: ]
[HTTP request 2/2]
[Prev request in frame: 1254]
[Response in frame: 1272]
Are there any online tool that I can use to test such requests?
It is perfectly acceptable for multiple HTTP requests to be in a single TCP packet, if they fit.
What you are seeing is indeed HTTP pipelining, which is covered in RFC 2616 Section 8 and RFC 7230 Section 6.3.2 of the HTTP 1.1 spec. The client is sending a new GET request without first waiting for a response to a previous GET request. That is the very definition of pipelining:
HTTP requests and responses can be pipelined on a connection. Pipelining allows a client to make multiple requests without waiting for each response, allowing a single TCP connection to be used much more efficiently, with much lower elapsed time.
TCP is just optimizing things by using a single TCP packet for both HTTP requests. The client likely has send coalescing (aka the "Nagle algorithm") enabled (which most socket libraries do by default) to reduce network traffic.
In order for the server to respond to pipelined requests, a persistent connection MUST be used, which is another requirement of pipelining, and is clearly visible in your example (the Connection: keep-alive request header).
TCP is a byte stream, the lower level TCP framing does not matter to the higher level protocol layers. A properly written HTTP receiver will be able to separate the individual HTTP messages regardless of the TCP framing used, and process them individually as needed. The HTTP 1.1 spec requires all requests to be responded to in the same order that they were received (HTTP 2.0 changes that, but that is a much more involved process to handle - multiplexing - which I won't get into).
So, to answer your questions:
Will the server ignore the second GET request? - NO
Will the server send the response one by one to each GET request? - YES
This doesn't seem to be HTTP pipelining. Please advise if it is. - IT IS, but not for the reason you are thinking.
Related
I found that a get message header looks like:
:method: GET
:scheme: https
:authority: server.net
:path: /config
accept: */*
accept-encoding: gzip,deflate
What a connect message header should look like?
This example is from the RFC of http2:
GET /resource HTTP/1.1 HEADERS
Host: example.org ==> + END_STREAM
Accept: image/jpeg + END_HEADERS
:method = GET
:scheme = https
:path = /resource
host = example.org
accept = image/jpeg
I want to know the equivalent of the connect header in http2.
In Http1 is:
CONNECT example.org:443 HTTP/1.1
Host: example.org:443
The format of the CONNECT method in HTTP/2 is specified in section 8.3.
With the formatting you used above looks like:
:method: CONNECT
:authority: proxy.net:8080
As specified, :scheme and :path must be omitted.
The HTTP/2 CONNECT method can also be used for bootstrapping other protocols (see for example WebSocket over HTTP/2), so that, additionally, the :protocol pseudo-header may also be present.
Remember however that this is only a textual representation of HTTP/2; the bytes that actually travel over the network are different since you must encode them using HPACK.
Unless you are actually writing an HTTP/2 implementation, it is better that you use existing libraries (available in virtually any programming language) to send HTTP/2 requests (of any kind): the libraries will take care of converting your CONNECT request into the proper bytes to send over the network.
I'm writing a HTTP server in Go that will receive requests from clients with a Expect: 100-continue header. However, it appears that the net/http server doesn't send a HTTP/1.1 100 Continue unless the client also sends a Transfer-Encoding: Chunked header, which some clients (for example, ffmpeg with an icecast:// destination) do not.
Here's a minimal server, that writes into a bytes.Buffer (I've reproduced the same behaviour with a more complicated server that, for example, uses io.Copy() to write into a file):
func main() {
http.HandleFunc("/", func(writer http.ResponseWriter, r *http.Request) {
log.Printf("Expect header: %v\n", r.Header.Get("Expect"))
log.Printf("Transfer-Encoding header: %v\n", r.Header.Get("Transfer-Encoding"))
buf := new(bytes.Buffer)
defer func() {
log.Printf("Buffer size: %d\n", buf.Len())
}()
defer r.Body.Close()
log.Println("Writing.")
io.Copy(buf, r.Body)
})
log.Fatal(http.ListenAndServe(":3948", nil))
}
And here's a transcript of two HTTP conversations (via telnet), where the server sends a 100 in one but not in the other:
PUT /telnetlol HTTP/1.1
Host: localhost
Expect: 100-continue
HTTP/1.1 200 OK
Date: Thu, 18 Mar 2021 10:59:09 GMT
Content-Length: 0
PUT /telnetlol HTTP/1.1
Host: localhost
Expect: 100-continue
Transfer-Encoding: chunked
HTTP/1.1 100 Continue
test
HTTP/1.1 200 OK
Date: Thu, 18 Mar 2021 10:59:35 GMT
Content-Length: 0
Connection: close
Is this a bug in Go, or am I misunderstanding the HTTP spec? The spec reads:
Upon receiving a request which includes an Expect request-header field with the "100-continue" expectation, an origin server MUST respond with 100 (Continue) status and continue to read from the input stream, or respond with a final status code. The origin server MUST NOT wait for the request body before sending the 100 (Continue) response.
Edit: Sending a non-zero Content-Length header in the initial request also makes the server reply with a 100 Continue. (Although, if I understand the spec correctly, it should still reply with a Continue irregardless.)
The net/http server correctly handles the request:
PUT /telnetlol HTTP/1.1
Host: localhost
Expect: 100-continue
with this response:
HTTP/1.1 200 OK
Date: Thu, 18 Mar 2021 10:59:09 GMT
Content-Length: 0
The request does not have a message body per RFC 7230 3.3:
The presence of a message body in a request is signaled by a
Content-Length or Transfer-Encoding header field.
A server may omit sending the a 100 response when there is no message body per RFC 7231 5.1.1:
A server MAY omit sending a 100 (Continue) response if it has
already received some or all of the message body for the
corresponding request, or if the framing indicates that there is
no message body.
In addition, the client request is bad per RFC 7231 5.1.1:
A client MUST NOT generate a 100-continue expectation in a request that does not include a message body.
I am trying to create a MediaServer UPNP program in order to stream video from my phones camera to my PC.
I used Intel device spy to send an M-SEARCH request and used Wireshark to capture the network packets.
Here is the M-SEARCH packet
(Src: 192.168.1.28, Dst: 239.255.255.250; Src Port: 50852, Dst Port: 1900, time 2.09)
M-SEARCH * HTTP/1.1
ST: upnp:rootdevice
MAN: "ssdp:discover"
MX: 5
HOST: 239.255.255.250:1900
Here is the UDP reply
(Src: 192.168.1.23, Dst: 192.168.1.28; Src Port: 53359, Dst Port: 50852)
HTTP/1.1 200 OK
CACHE-CONTROL: max-age=1810
DATE: Wed, 1 Feb 2017 02:07:36 GMT
EXT:
LOCATION: http://192.168.1.23:49156/details.xml
SERVER: Linux/2.x.x, UPnP/1.0, pvConnect UPnP SDK/1.0, TwonkyMedia UPnP SDK/1.1
ST: upnp:rootdevice
USN: uuid:3d64febc-ae6a-4584-853a-85368ca80800::upnp:rootdevice
Content-Length: 0
I do not get a following HTTP GET request to 192.168.1.23. I compared it to other UPNP device responses that worked and could see no difference.
I tried different source ports but with no sucess. Any ideas?
#simonc, Thank you. I did have a \r\n at the end of my message, but I added another one (to the NOTIFY message as well) and now I can see my device.
I'm making a clojure web app that streams data to clients using chunked HTTP responses. This works great when I run it locally using foreman, but doesn't work properly when I deploy it to Heroku.
A minimal example exhibiting this behaviour can be found on my github here. The frontend (in resources/index.html) performs an AJAX GET request and prints the response chunks as they arrive. The server uses http-kit to send a new chunk to connected clients every second. By design, the HTTP request never completes.
When the same code is deployed to Heroku, the HTTP connection is closed by the server immediately after the first chunk is sent. It seems to be Heroku's routing mesh which is causing this disconnection to occur.
This can also be seen by performing the GET request using curl:
$ curl -v http://arcane-headland-2284.herokuapp.com/stream
* About to connect() to arcane-headland-2284.herokuapp.com port 80 (#0)
* Trying 54.243.166.168...
* Adding handle: conn: 0x6c3be0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x6c3be0) send_pipe: 1, recv_pipe: 0
* Connected to arcane-headland-2284.herokuapp.com (54.243.166.168) port 80 (#0)
> GET /stream HTTP/1.1
> User-Agent: curl/7.31.0
> Host: arcane-headland-2284.herokuapp.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Date: Sat, 17 Aug 2013 16:57:24 GMT
* Server http-kit is not blacklisted
< Server: http-kit
< transfer-encoding: chunked
< Connection: keep-alive
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
The time is currently Sat Aug 17 16:57:24 UTC 2013 <-- this is the first chunk
Can anybody suggest why this is happening? HTTP streaming is supposed to be supported in Heroku's Cedar stack. The fact the code runs correctly using foreman suggests it is something in Heroku's routing mesh causing it to break.
Live demo of the failing project: http://arcane-headland-2284.herokuapp.com/
This was due to a bug in http-kit which will be fixed shortly.
https://devcenter.heroku.com/articles/request-timeout may be relevant: "long-polling" requests like yours have to send data every 55 seconds or be terminated.
I'm writing some code to integrate an in-house app into a DVR to retrieve a video file. This is all reverse engineered as there isn't any official documentation, and I'm having trouble understanding the following sequence of events (captured by playing with the DVR's Android app).
936 72.985204 192.168.0.1 192.168.0.200 HTTP 468 POST /cgi-bin/supervisor/NetworkBk.cgi HTTP/1.1 (application/x-www-form-urlencoded)
937 72.985368 192.168.0.200 192.168.0.1 TCP 54 mit-ml-dev > 41859 [ACK] Seq=1 Ack=415 Win=65535 Len=0
938 73.933676 192.168.0.200 192.168.0.1 HTTP 275 HTTP/1.0 200 OK (video/mpeg4)
939 73.933983 192.168.0.1 192.168.0.200 TCP 54 41859 > mit-ml-dev [ACK] Seq=415 Ack=222 Win=15544 Len=0
940 74.004433 192.168.0.200 192.168.0.1 TCP 74 [TCP segment of a reassembled PDU]
941 74.004887 192.168.0.1 192.168.0.200 TCP 54 41859 > mit-ml-dev [ACK] Seq=415 Ack=242 Win=15544 Len=0
942 74.024669 192.168.0.200 192.168.0.1 HTTP 1346 Continuation or non-HTTP traffic
The HTTP POST requests the video file, which then results in an HTTP OK. I get confused as to what happens next. Isn't the request complete when the HTTP 200 is received? Why then is it continuing to receive TCP data and then getting a HTTP Continuation or non-HTTP traffic? The subsequent TCP packets contain the video file I'm intending to download. When I manually craft a HTTP POST I get the HTTP OK response and then I'm stumped.
This is the code I use to simulate the HTTP POST.
import requests
dc = {"action":"download", "start_time":"2013 7 1 13 59 00", "end_time":"2013 7 14 3 0", "num":"255", "ch":"5"}
r = requests.post("http://192.168.0.200/cgi-bin/supervisor/NetworkBk.cgi", data=dc, auth=(username, password))
This code gets the HTTP 200 OK response, how do I get to the Continuation or non-HTTP traffic? I'm new to this and so am unsure if I've provided enough details. I can provide the HTTP headers if that will help.
Addendum
This is the RAW response of the HTTP OK reply. As far as I can tell, there is nothing there about expecting extra content.
HTTP/1.0 200 OK
Date: Mon, 01 Jul 2013 15:01:34 GMT
nServer: Linux/2.x UPnP/1.0 Avtech/1.0
Expires: 0
Pragma: no-cache
Cache-Control: no-cache
Connection: close
Content-Type: video/mpeg4
Content-Length: 5
0
OK
Why then is it continuing to receive TCP data and then getting a HTTP
Continuation or non-HTTP traffic?
By default, when you make a request, the body of the response is downloaded immediately.
So in this case once the successful POST request is made the DVR will immediately start sending the video data over TCP - most probably as a H.264 Byte Stream. That would account for the non-HTTP traffic you are seeing.
This code gets the HTTP 200 OK response, how do I get to the
Continuation or non-HTTP traffic?
You can override this default behaviour and defer downloading the response body until you access the Response.content attribute with the stream parameter: You could then use something like r.iter_content to iterate over the response data in chunks and then write them to a file. e.g.
import requests
url = "http://192.168.0.200/cgi-bin/supervisor/NetworkBk.cgi"
dc = {"action":"download", "start_time":"2013 7 1 13 59 00", "end_time":"2013 7 14 3 0", "num":"255", "ch":"5"}
r = requests.post(url, data=dc, auth=(username, password), stream=True)
if r.status_code == 200:
with open(path, 'wb') as f:
for chunk in r.iter_content():
f.write(chunk)