I'm implementing an IPS system, and I'm a little confused when observing the procesure of TCP stream reassembling by wireshark.
For example, the server transfer a HTML page to the client. The page is divided into 4 parts and encapsulated by TCP packet. Then the server push another 4 TCP packets to the client for a JavaScript text.
My question is, I know I can determine their sequences by measuring their Seq and Len, but how can I determine the end of the HTML text? How can I know the HTML contains 4 TCP packets but not 5?
RFC 2616 section 4.4 states that the message length could be given in several ways:
By the Content-Length header if one is defined. (This is probably the case you're seeing, and it's relatively simple. If you know the position (seq+offset within packet) of the start of body and the message length, you can just add to get the position of the end.)
By chunked encoding. The RFC has the details, but it has a similar encoding for each chunk and a way of noting the final chunk.
multipart/byteranges (which you won't see unless the client asked for it, and it probably won't for an HTML document).
Or until the TCP connection is closed. (In particular, until a FIN packet is sent from the server to the client, which only happens on a clean close; you'd see an RST otherwise.)
Related
I'm attempting to make a low-level packet reader, sort of like wireshark, for learning purposes and fun.
Going into http parsing, I quickly found that there are these TCP streams, and that data doesn't all come in one packet.
EDIT: What I consider to be the stream I've been told is not the right term. David Schwartz referred to it as a Query and EJP as a Sequence of Segments.
So I did a little reading on TCP, and I've read everywhere about the sequence and acknowledgment numbers, how they increase and wrap around etc..
Connection establishment and termination I already had knowledge of.
To be honest I've never been too good at interpretation or using google, but I've been at this for a bit, and I just can't seem to understand how the server/client ends a TCP stream.
I'm asking this because, after sending the totality of the http request, the server immediately replies with the page, so, as I've been going back and forward through packets in wireshark and I can't seem to find what delimits this stream (anything indicating length in the first packet or difference between the penultimate and last packets).
Is there no actual delimiter in TCP, and you just get the total length from the http headers's Content-Length?
As seen here, the first 3 packets are connection establishment.
The 3 after those are the sending of the http request (with first being ACK (the selected one), and the latest 2 being PSH & ACK).
The two after those are ACKs from the server, followed by the entirety of the response.
The last one is my ACK and the connection is never closed for the time I "recorded".
Thank you.
The end of a TCP stream is marked by a segment whose FIN bit is sent.
However that isn't the answer to your real question, which is about the end of an HTTP request or response, which is determined by:
The Content-length, if present, or
The final chunk in a chunked transfer, if used, or
The FIN
if there is a body, otherwise the blank line after the headers as pointed out by #Barmar below.
I'm trying to make a simple server/application in Erlang.
My server initialize a socket with gen_tcp:listen(Port, [list, {active, false}, {keepalive, true}, {nodelay, true}]) and the clients connect with gen_tcp:connect(Server, Port, [list, {active, true}, {keepalive, true}, {nodelay, true}]).
Messages received from the server are tested by guards such as {tcp, _, [115, 58 | Data]}.
Problem is, packets sometimes get concatenated when sent or received and thus cause unexpected behaviors as the guards consider the next packet as part of the variable.
Is there a way to make sure every packet is sent as a single message to the receiving process?
Plain TCP is a streaming protocol with no concept of packet boundaries (like Alnitak said).
Usually, you send messages in either UDP (which has limited per-packet size and can be received out of order) or TCP using a framed protocol.
Framed meaning you prefix each message with a size header (usualy 4 bytes) that indicates how long the message is.
In erlang, you can add {packet,4} to your socket options to get framed packet behavior on top of TCP.
assuming both sides (client/server) use {packet,4} then you will only get whole messages.
note: you won't see the size header, erlang will remove it from the message you see. So your example match at the top should still work just fine
You're probably seeing the effects of Nagle's algorithm, which is designed to increase throughput by coalescing small packets into a single larger packet.
You need the Erlang equivalent of enabling the TCP_NODELAY socket option on the sending socket.
EDIT ah, I see you already set that. Hmm. TCP doesn't actually expose packet boundaries to the application layer - by definition it's a stream protocol.
If packet boundaries are important you should consider using UDP instead, or make sure that each packet you send is delimited in some manner. For example, in the TCP version of DNS each message is prefixed by a 2 byte length header, which tells the other end how much data to expect in the next chunk.
You need to implement a delimiter for your packets.
One solution is to use a special character ; or something similar.
The other solution is to send the size of the packet first.
PacketSizeInBytes:Body
Then read the provided amount of bytes from your message. When you're at the end you got your whole packet.
Nobody mentions that TCP may also split your message into multiple pieces (split your packet into two messages).
So the second solution is the best of all. But a little hard. While the first one is still good but limits your ability to send packets with special characters. But the easiest to implement. Ofc theres a workaround for all of this. I hope it helps.
When an application such as a web server sends HTTP data to a web browser, how does the browser know when it has received all of the data so that it can begin using it instead of waiting for more? TCP doesn't specify anywhere how large a segmented message is going to be.
Right now I'm thinking that it's up to the application layer, like HTTP's Content-Length header. But it seems like even that header could be split off into a 2nd or 3rd packet.
TCP/IP is a connection oriented protocol. So, when the browser performs a HTTP connection using TCP/IP, the Network stack guarantees that stream will arrive in the same order the sender intended to.
So, there is no packet concept when you are dealing with TCP. TCP is an ordered stream of bytes arriving through a socket. No need to worry about packets at all. That's the beauty of a protocol stack: each layer does its own work, and abstracts the layer above it of the underlying complications of the problems it resolves.
Content-length indeed, except in the case where the client reads until it gets an end-of-file indication due to the other end closing the connection. Of course, in HTTP, 'RSVP', so that's not going to happen.
Absent content length, it's gotta look for </html> or some other delimiter in the content. The browser doesn't see packets at all. The connection looks like a stream, with no boundaries, and it's up to the two ends to make a protocol.
I am writing an application in C, using libpcap. My program listens for new packets and parses them
according to a grammar. The payload actually is XML.
Sometimes one packet is not enough for an XML file, so the XML buffer is splitted into separate packets.
I want to add code logic in order to handle these cases. However I don't know in advance that a packet does not contain the whole data. How do I know that a packet has more data that will be send next? How to i recognize that a new packet contains the rest of the data?
Do I have to use the TH_FIN flag? Could you please explain it to me?
There's nothing in TCP that defines packets, that's up to the higher layers to define if they need to - TCP is just a stream.
If this is raw XML over a TCP stream, you actually need to parse the xml - you'll know when you have a whole xml document when you've received the end of the document element.
If it's XML packaged over HTTP , you might be able to parse out the Content-Length: header which should contain the length of the body.
Note, reassembling a TCP stream from captured packets is a very hard problem, there's a lot of corner cases, e.g. you'd need to handle retransmission , out of sequence tcp segments and many more. http://libnids.sourceforge.net/ might help you.
As Anon say use a higher level stream library.
But even then you need to know the chunk side before starting to handle it, as you will read from the stream in block's of n bytes.
Thus you want to first send in binary the number of bytes to be sent, then send x bytes, and repeat, thus when you are receiving the chucks via select/read to know went you have all of chunk one to pass to the processor.
If you're using TCP, use a TCP library that gives you the data as a stream instead of trying to handle the packets yourself.
Stream is good. Another option is to store the incoming data in a buffer (eg char*) and search for application messaging framing characters or in the case of Xml, the root end tag. Once you've found a complete xml message at the front of the buffer, pull it out and process.
The XMPP instant messaging protocol, used by Jabber, has means to move XML chunks over a TCP stream. I don't know how exactly it is done myself, but RFC 3290 is the protocol definition. You should be able to work it out from that.
We are trying to implement a proxy proof of concept but have encountered an interesting question: Since a single HTTP connection can, and indeed should, make multiple requests, and the HTTP transactions are sent via multiple packets due to TCP's magic, is it possible for a HTTP request to begin in the middle of a packet?
Bear in mind that this is not a theoretical question regarding possible optimization of the browser, but whether it actually happens in real life. It would be even better if someone could point me to a written reference on whether or not this is possible and if so how often it can occur.
Clarification update: We know that if we work in the HTTP layer alone we would not need to bother with this question, however we're trying to figure out if some advanced technique could be applied by working on the TCP layer first.
Assuming that you are talking about IP packets: Yes, it is possible that HTTP request starts middle of IP packet.
When you are using persistent HTTP connections, that is, using same TCP connection for several HTTP requests, it is fully possible that request boundary is middle of IP packet.
Also there is a TCP protocol between IP and HTTP. TCP contains also some headers so a IP packet may start with some TCP headers and rest of the packet consists of HTTP request.
HTTP request may also consist of several IP packets (in case of file uploads, transmission errors and following retransmissions etc).
However, I wonder why you are interested in packets if you are working at HTTP level. TCP should hide the IP packet details.
First of all, TCP is a stream based protocol and has no concept of packets. HTTP itself might have some kind of message or record delimiter, but TCP doesn't.
This page might be helpful: Structure of HTTP Transactions
From your question it sounds like you think that each read from a TCP socket is a "packet" of data. In reality, each read simply reads as many bytes as are in the buffer up to the maximum that you requested, without any concept of records or packets.
So for instance, lets say you read 2048 bytes from the socket, you could have the tail end of one transaction, followed by the beginning of a second response half way through the data you read, and only get the remainder of your second response on your next read from the socket.
If you're here in Jerusalem or near by maybe I could help you out.
Unless you are implementing your own TCP stack, you should not need to worry about the packets, but rather about the API that the TCP provides, in case of POSIX interfaces it would be the recv() or read(). So I treat the question then as "Can more than one HTTP requests come into a single read(), and can the HTTP request be split between multiple read() requests?" -- The answer to both would be "yes, it is possible".
An example of where this can happen is HTTP pipelining. This not frequent in real life (ironically, at least some of the browsers disable it by default because of "buggy proxies" :-) - but when it happens, can be a bit of a problem for the users to diagnose - especially if they have no access to the proxy.
One very notable place where it does happen by default apt-get in Debian-derived linux systems. Just install a Debian or Ubuntu server and try to use it through your proxy. You can do that by editing the /etc/apt/apt.conf.d/proxy file and placing the following there:
Acquire::http::Proxy "http://your.proxy.address:8080";
Depends of which abstraction layer of a packet you are talking about: there are many layers underneath HTTP.
HTTP --> TCP (byte stream) --> IP (packet) --> (possibly something else) Ethernet (frame) --> (possibly) some other transport
If you are talking about the IP layer, then yes the HTTP layer would start later on... Note that TCP presents a "byte stream interface" to its Client layer hence, no concept of packet here.
I think I understand where you are trying to go with this question.
If you don't use persistent HTTP connections, the HTTP GET request header is always the very first thing which is sent over the TCP connection, so we can be sure that the start of the HTTP GET request header does "not start in the middle of some TCP packet". But keep in mind that there may be one or more TCP packets without any user data, e.g. only a SYN, which may preceed the TCP packet with the start of the HTTP GET request header. And also keep in mind that the HTTP GET request header may not be contained in a single TCP packet.
If you do use persistent HTTP connections, the start of the HTTP GET request header for request number N+1 can start in the middle of a TCP packet, namely after the end of HTTP GET request body of request number N.
If you are asking these questions you are possibly "doing it wrong". As several other responders have already pointed out, in the vast majority of cases you should probably just be a TCP client and deal with a TCP stream of data and let the TCP code worry about the TCP packets. (Unless, of course, you are working on some special hardware which is looking at individual IP packets as they fly by and try to do some processing at the HTTP layer.)