I'm writing some code to integrate an in-house app into a DVR to retrieve a video file. This is all reverse engineered as there isn't any official documentation, and I'm having trouble understanding the following sequence of events (captured by playing with the DVR's Android app).
936 72.985204 192.168.0.1 192.168.0.200 HTTP 468 POST /cgi-bin/supervisor/NetworkBk.cgi HTTP/1.1 (application/x-www-form-urlencoded)
937 72.985368 192.168.0.200 192.168.0.1 TCP 54 mit-ml-dev > 41859 [ACK] Seq=1 Ack=415 Win=65535 Len=0
938 73.933676 192.168.0.200 192.168.0.1 HTTP 275 HTTP/1.0 200 OK (video/mpeg4)
939 73.933983 192.168.0.1 192.168.0.200 TCP 54 41859 > mit-ml-dev [ACK] Seq=415 Ack=222 Win=15544 Len=0
940 74.004433 192.168.0.200 192.168.0.1 TCP 74 [TCP segment of a reassembled PDU]
941 74.004887 192.168.0.1 192.168.0.200 TCP 54 41859 > mit-ml-dev [ACK] Seq=415 Ack=242 Win=15544 Len=0
942 74.024669 192.168.0.200 192.168.0.1 HTTP 1346 Continuation or non-HTTP traffic
The HTTP POST requests the video file, which then results in an HTTP OK. I get confused as to what happens next. Isn't the request complete when the HTTP 200 is received? Why then is it continuing to receive TCP data and then getting a HTTP Continuation or non-HTTP traffic? The subsequent TCP packets contain the video file I'm intending to download. When I manually craft a HTTP POST I get the HTTP OK response and then I'm stumped.
This is the code I use to simulate the HTTP POST.
import requests
dc = {"action":"download", "start_time":"2013 7 1 13 59 00", "end_time":"2013 7 14 3 0", "num":"255", "ch":"5"}
r = requests.post("http://192.168.0.200/cgi-bin/supervisor/NetworkBk.cgi", data=dc, auth=(username, password))
This code gets the HTTP 200 OK response, how do I get to the Continuation or non-HTTP traffic? I'm new to this and so am unsure if I've provided enough details. I can provide the HTTP headers if that will help.
Addendum
This is the RAW response of the HTTP OK reply. As far as I can tell, there is nothing there about expecting extra content.
HTTP/1.0 200 OK
Date: Mon, 01 Jul 2013 15:01:34 GMT
nServer: Linux/2.x UPnP/1.0 Avtech/1.0
Expires: 0
Pragma: no-cache
Cache-Control: no-cache
Connection: close
Content-Type: video/mpeg4
Content-Length: 5
0
OK
Why then is it continuing to receive TCP data and then getting a HTTP
Continuation or non-HTTP traffic?
By default, when you make a request, the body of the response is downloaded immediately.
So in this case once the successful POST request is made the DVR will immediately start sending the video data over TCP - most probably as a H.264 Byte Stream. That would account for the non-HTTP traffic you are seeing.
This code gets the HTTP 200 OK response, how do I get to the
Continuation or non-HTTP traffic?
You can override this default behaviour and defer downloading the response body until you access the Response.content attribute with the stream parameter: You could then use something like r.iter_content to iterate over the response data in chunks and then write them to a file. e.g.
import requests
url = "http://192.168.0.200/cgi-bin/supervisor/NetworkBk.cgi"
dc = {"action":"download", "start_time":"2013 7 1 13 59 00", "end_time":"2013 7 14 3 0", "num":"255", "ch":"5"}
r = requests.post(url, data=dc, auth=(username, password), stream=True)
if r.status_code == 200:
with open(path, 'wb') as f:
for chunk in r.iter_content():
f.write(chunk)
Related
When looking at client->server interaction for fetching images, I see the following HTTP GET request from client where the packet contains 2 HTTP GET requests and I am not sure how the server would respond to such requests?
Will the server ignore the second GET request?
Will the server send the response one by one to each GET request?
This doesn't seem to be HTTP pipelining. Please advise if it is.
Transmission Control Protocol, Src Port: 59649 (59649), Dst Port: 8080 (8080), Seq: 1, Ack: 1, Len: 648
Source Port: 59649
Destination Port: 8080
[Stream index: 86]
[TCP Segment Len: 648]
Sequence number: 1 (relative sequence number)
[Next sequence number: 649 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
Header Length: 32 bytes
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: *******AP***]
Window size value: 683
[Calculated window size: 43712]
[Window size scaling factor: 64]
Checksum: [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
Timestamps: TSval 6345, TSecr 6344
Kind: Time Stamp Option (8)
Length: 10
Timestamp value: 6345
Timestamp echo reply: 6344
[SEQ/ACK analysis]
[iRTT: 0.000099000 seconds]
[Bytes in flight: 648]
Hypertext Transfer Protocol
GET HTTP/1.1\r\n
[Expert Info (Chat/Sequence): GET HTTP/1.1\r\n]
[GET HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]enter code here
Request Method: GET
Request URI:
Request Version: HTTP/1.1
Host: \r\n
sent: \r\n
User-Agent: \r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-GB,*\r\n
Connection: keep-alive\r\n
\r\n
[Full request URI: ]
[HTTP request 2/2]
[Prev request in frame: 1254]
[Response in frame: 1272]
Hypertext Transfer Protocol
GET \r\n
[Expert Info (Chat/Sequence): GET HTTP/1.1\r\n]
[GET HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: GET
Request URI:
Request Version: HTTP/1.1
Host: \r\n
sent: \r\n
User-Agent: \r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-GB,*\r\n
Connection: keep-alive\r\n
\r\n
[Full request URI: ]
[HTTP request 2/2]
[Prev request in frame: 1254]
[Response in frame: 1272]
Are there any online tool that I can use to test such requests?
It is perfectly acceptable for multiple HTTP requests to be in a single TCP packet, if they fit.
What you are seeing is indeed HTTP pipelining, which is covered in RFC 2616 Section 8 and RFC 7230 Section 6.3.2 of the HTTP 1.1 spec. The client is sending a new GET request without first waiting for a response to a previous GET request. That is the very definition of pipelining:
HTTP requests and responses can be pipelined on a connection. Pipelining allows a client to make multiple requests without waiting for each response, allowing a single TCP connection to be used much more efficiently, with much lower elapsed time.
TCP is just optimizing things by using a single TCP packet for both HTTP requests. The client likely has send coalescing (aka the "Nagle algorithm") enabled (which most socket libraries do by default) to reduce network traffic.
In order for the server to respond to pipelined requests, a persistent connection MUST be used, which is another requirement of pipelining, and is clearly visible in your example (the Connection: keep-alive request header).
TCP is a byte stream, the lower level TCP framing does not matter to the higher level protocol layers. A properly written HTTP receiver will be able to separate the individual HTTP messages regardless of the TCP framing used, and process them individually as needed. The HTTP 1.1 spec requires all requests to be responded to in the same order that they were received (HTTP 2.0 changes that, but that is a much more involved process to handle - multiplexing - which I won't get into).
So, to answer your questions:
Will the server ignore the second GET request? - NO
Will the server send the response one by one to each GET request? - YES
This doesn't seem to be HTTP pipelining. Please advise if it is. - IT IS, but not for the reason you are thinking.
I have captured a HTTP request and response in n a pcap file by running curl script from my Linux server. After that i opened it through wireshark. When analyzing packets i saw 404 not found. Does it mean the request i am running is non existent? or the request file ****.html is non existent?
Below is the wireshark data--
9 2017-06-08 17:27:04.035288 x.x.x.x x.x.x.x HTTP 287 GET /****.htm HTTP/1.1
12 2017-06-08 17:27:04.036517 x.x.x.x x.x.x.x HTTP 73 HTTP/1.1 404 Not Found ..
sorry for all the x's and *'s. This is a little confidential..
I am trying to create a MediaServer UPNP program in order to stream video from my phones camera to my PC.
I used Intel device spy to send an M-SEARCH request and used Wireshark to capture the network packets.
Here is the M-SEARCH packet
(Src: 192.168.1.28, Dst: 239.255.255.250; Src Port: 50852, Dst Port: 1900, time 2.09)
M-SEARCH * HTTP/1.1
ST: upnp:rootdevice
MAN: "ssdp:discover"
MX: 5
HOST: 239.255.255.250:1900
Here is the UDP reply
(Src: 192.168.1.23, Dst: 192.168.1.28; Src Port: 53359, Dst Port: 50852)
HTTP/1.1 200 OK
CACHE-CONTROL: max-age=1810
DATE: Wed, 1 Feb 2017 02:07:36 GMT
EXT:
LOCATION: http://192.168.1.23:49156/details.xml
SERVER: Linux/2.x.x, UPnP/1.0, pvConnect UPnP SDK/1.0, TwonkyMedia UPnP SDK/1.1
ST: upnp:rootdevice
USN: uuid:3d64febc-ae6a-4584-853a-85368ca80800::upnp:rootdevice
Content-Length: 0
I do not get a following HTTP GET request to 192.168.1.23. I compared it to other UPNP device responses that worked and could see no difference.
I tried different source ports but with no sucess. Any ideas?
#simonc, Thank you. I did have a \r\n at the end of my message, but I added another one (to the NOTIFY message as well) and now I can see my device.
I have apache2 running on localhost and I want to intercept and modify an http request from my localhost. By modifying I want to change the Accept-Encoding attribute of the header to 'identity'. Using Burp-Suite, it works just fine. However, using my scapy script it seems that the packet is already sent because the http response is still encoded.
The scapy script:
from scapy.all import *
def intercept(pkt):
if pkt.haslayer(Raw):
http_content = pkt.getlayer(Raw).load
http_content = http_content.replace("Accept-Encoding: gzip, deflate", "Accept-Encoding: identity")
pkt[Raw].load = http_content
print pkt.show()
send(pkt)
def main():
sniff(iface='lo', filter='tcp port 80', prn=intercept)
if __name__ == '__main__':
main()
This is what I get back as a response:
<skipped>
###[ Raw ]###
load = 'HTTP/1.1 200 OK\r\nDate: Thu, 11 Aug 2016 09:34:38 GMT\r\nServer: Apache/2.4.23 (Debian)\r\nLast-Modified: Thu, 11 Aug 2016 09:34:25 GMT\r\nETag: "7d-539c878b8f8fd-gzip"\r\nAccept-Ranges: bytes\r\nVary: Accept-Encoding\r\nContent-Encoding: gzip\r\nContent-Length: 103\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xb3\xc9(\xc9\xcd\xb1\xe3\xb2\xc9HML\xb1\xe3RPP\xb0)\xc9,\xc9I\xb5\xf3H\xcd\xc9\xc9W\x08\xcf/\xcaI\xb1\xd1\x87\x08q\xd9\xe8CT\xd9$\xe5\xa7TB\x14g\x18!\xabT\x04\xaa0\x82H\x14#\xc5\x13\xd3\x133\xf3\xf4\xf4\xf4l\xf4\x0b#\x06#t\x02\x95\x81m\x05\x00\x1c\x95F\x1d}\x00\x00\x00'
which is encoded.
Can someone help?
Well as far as I know scapy doesn't give you the ability to modify packets that are already created by your system. Of course you can craft and inspect packets but cannot modify already created packets.
As it is correctly pointed out here Scapy sniffs packets without interfering with the host's IP stack.
But for Linux you could try to combine scapy with the nfqueue module. The nfqueue module lets you modify(using scapy) packets that meet a certain iptables rule.
I am loading a MP3 stream from IceCast 2.3.2-kh29 server in the Android app with MediaPlayer class.
Playing works well, but sometimes stops happen. If see the server responses in IcyStreamMeta class for ID3 tags, there is 404 error for this case.
Also it happens in Windows 7: Firefox and other browsers.
Here are normal headers (some data ***ed):
http://***:14534/***.mp3
GET /***.mp3 HTTP/1.1
Host: ***:14534
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Server: nginx/1.4.1
Date: Tue, 23 Jul 2013 21:22:00 GMT
Content-Type: audio/mpeg
Transfer-Encoding: chunked
Connection: keep-alive
icy-br: 192
ice-audio-info: bitrate=192;samplerate=44100;channels=2
icy-description: MP3 192 Kbps
icy-genre: ***
icy-name: ***
icy-pub: 1
icy-url: ***
Cache-Control: no-cache
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
So, the stream sometimes plays only about a minute or less, sometimes seconds and stops. What's the possible reason of 404 error? In other devices there were tests with stable work. Internet speed is well. Can router cause such things? Also, maybe some special HTTP request headers are needed for IceCast (and if they're not present, it gives 404)? Or it's an internal server error for some cases?
So, from WireShark:
2973 53.630385000 SERVER'S IP 192.168.100.6 TCP 1466 14534 > 59847 [ACK] Seq=1284017 Ack=1 Win=63 Len=1412
2976 53.636352000 SERVER'S IP 192.168.100.6 TCP 1157 14534 > 59847 [PSH, ACK] Seq=1285429 Ack=1 Win=63 Len=1103
2978 53.671606000 SERVER'S IP 192.168.100.6 TCP 60 14534 > 59847 [PSH, ACK] Seq=1286532 Ack=1 Win=63 Len=5
2980 53.678606000 SERVER'S IP 192.168.100.6 TCP 60 14534 > 59847 [FIN, ACK] Seq=1286537 Ack=2 Win=63 Len=0
The issue is your chunked encoding. You're proxying your stream through Nginx, and Nginx is "fixing" the output to be compatible with HTTP/1.0. Don't do that.
You can try turning off chunked encoding in your Nginx config:
chunked_transfer_encoding off