I am trying to simulate "HTTP headers spanning multiple TCP segments" as mentioned here - http://wiki.wireshark.org/HTTP_Preferences.
How can this be done using netcat? Are there any examples you might be able to point me to to get me started?
Netcat isn't really the right tool for this job, but an easy way to make the headers span segments is just to make them long enough. Eventually, they won't fit in a single segment.
The packet size may be 1500 octets (normal Ethernet) or more than 9000 octets (Ethernet with jumbo frames). You'll want some actual network, packet processing with localhost is often optimized.
(For the proper tools for this, you probably want to ask on Severfault or Security.SE, as they're normally used for firewall testing)
Related
I am working on a tool that takes a pcap file (from wireshark in this case), and attempts to parse out data from the TCP packets.
Now in this case, I only care about the data in one direction. So my logic was to sort out each wireshark captured packet into a list by the protocol-destIP-sourceIP-destPort-sorcePort.
So from this point, I now have a list of only packets for one direction on a particular port.
From there I just want to be able to walk through the bodies of the TCP payloads in order. is it as simple as then going in order by Sequence numbers?
I would simply take the first sequence number captured, add the payload size to it and expect that to be the next TCP packet sent? Is there more to this that I am missing?
I was noticing when sorting the interfaces this way, eventually I would come up to a sequence that dosent make sense. I guess I could just assume that is the start of the next stream? I know it becomes more difficult if I have to consider traffic going back and forth... but in this case I only want to watch packets in one direction.
Wireshark captures the packet on the wire. It might happen that the packets don't arrive in sequential order, that packets are corrupt (bad checksum), that there are duplicate packets ... - and this is ignoring possible attacks designed to confuse the analysis. The TCP stack will take care of all these problems so that the application gets the right packets, but Wireshark works outside the TCP stack. Thus, while in most cases your simple procedure will likely work (assuming that you at least check TCP flags for start and end of connection), it might fail in some cases.
I'm working with packets one by one and need to be able to edit both http and http2 contents.
The question is: is there a way to distinguish the two on a single packet basis?
Edit: For some additional info, the point is to read and edit large pcap files, so i'm trying to work with as little memory as possible.
On a per-packet basis, no. A single TCP packet could represent any arbitrary part of the stream. You need to capture (at least) the first part of the stream to work out whether it's HTTP or HTTP/2 (or anything else).
You can use Chrome DevTool > Network > Protocol to see the protocol used in the file transference.
I am wondering about the difference between HTTP and TCP data transfer protocols for online games.
I have heard many people using TCP or UDP to transfer data between client and server for online games.
But can you use http at all? I know http is mostly used for web browsing, but if I could set up web server and let my game applications use GET and POST methods, I can still send data back and forth right? Is it that this way of communicating is too slow or unnecessary?
And just one thing about TCP transmission protocols, if I were to write some gaming application using TCP, is it that the data are usually transferred using something called "sockets" (like Socket classes in Java)? What about UDP?
Thanks very much!
Appreciate any answer!
HTTP is an additional layer on top of TCP that defines what a request looks like, what a response looks like, and how the connection is closed or maintained across requests. You can either use it or not use it, depending on what you actually need to transport. If your game consists of a series of requests that each get a reply, HTTP might make sense. If it's more like unsolicited messages in each direction, making HTTP work is like putting a square peg in a round hole.
Most platforms provide a socket interface that allows you to work with either TCP or UDP depending on the protocol specified when the socket is allocated. Some higher-level APIs look completely different for different protocols.
Wireshark doesn't seem to be able to change the content of filtered packets in real time.
Does anyone know a symilar software which can change packet content that is filtered.
Finding something like this will really be a life saver
Thanks.
At least on Unices and -like where raw sockets are used, this is not possible, since the packet is copied to userspace and you only work on that copy. Furthermore, sending a packet back through the raw socket may be considered an "outgoing" packet so that it is, in fact, not reinjected to the input path where it should be. Raw sockets were — according to the Linux manpage — designed to implement new protocols, IOW, raw sockets are an "endpoint", not a "passthrough station".
For packet modification in the input path (passthrough-like), each OS has its own set of interfaces. In Linux (you were sort of unspecific as to which you target), that would be the nfqueue mechanism, usable through libnetfilter_queue. And of course, that is how wireshark, if it wanted to (I don't see it doing packet alteration last time I checked), would go about doing this.
Please give Burp Suite a try. It includes a repeater that let's you modify HTTP requests.
No wireshark won't let you change the contents of the packets and place them back on the line. However there are ways to change packets as they pass through the machine. Typically the host is setup with two nics bridged together. One nic is connected to one network and the other nic to the other network. Then as packets pass through this point the host can see them. Now you can use iptables/netfilter and write a module that changes data in the packet. For example you can write something that can remap source ip addresses. It's been a while since I've used netfilter/iptables, so I can't provide anymore details, but I have used it in a previous job to do some neat things with packets while they were inflight. It does mean you need a host machine sitting at network junction points though.
The documentation suggests that node.get("nextSibling") and node.get("previousSibling") are what you need.
Yes, it can.
You need to pass this option to the configure script before you build it:
--enable-packet-editor
We are trying to implement a proxy proof of concept but have encountered an interesting question: Since a single HTTP connection can, and indeed should, make multiple requests, and the HTTP transactions are sent via multiple packets due to TCP's magic, is it possible for a HTTP request to begin in the middle of a packet?
Bear in mind that this is not a theoretical question regarding possible optimization of the browser, but whether it actually happens in real life. It would be even better if someone could point me to a written reference on whether or not this is possible and if so how often it can occur.
Clarification update: We know that if we work in the HTTP layer alone we would not need to bother with this question, however we're trying to figure out if some advanced technique could be applied by working on the TCP layer first.
Assuming that you are talking about IP packets: Yes, it is possible that HTTP request starts middle of IP packet.
When you are using persistent HTTP connections, that is, using same TCP connection for several HTTP requests, it is fully possible that request boundary is middle of IP packet.
Also there is a TCP protocol between IP and HTTP. TCP contains also some headers so a IP packet may start with some TCP headers and rest of the packet consists of HTTP request.
HTTP request may also consist of several IP packets (in case of file uploads, transmission errors and following retransmissions etc).
However, I wonder why you are interested in packets if you are working at HTTP level. TCP should hide the IP packet details.
First of all, TCP is a stream based protocol and has no concept of packets. HTTP itself might have some kind of message or record delimiter, but TCP doesn't.
This page might be helpful: Structure of HTTP Transactions
From your question it sounds like you think that each read from a TCP socket is a "packet" of data. In reality, each read simply reads as many bytes as are in the buffer up to the maximum that you requested, without any concept of records or packets.
So for instance, lets say you read 2048 bytes from the socket, you could have the tail end of one transaction, followed by the beginning of a second response half way through the data you read, and only get the remainder of your second response on your next read from the socket.
If you're here in Jerusalem or near by maybe I could help you out.
Unless you are implementing your own TCP stack, you should not need to worry about the packets, but rather about the API that the TCP provides, in case of POSIX interfaces it would be the recv() or read(). So I treat the question then as "Can more than one HTTP requests come into a single read(), and can the HTTP request be split between multiple read() requests?" -- The answer to both would be "yes, it is possible".
An example of where this can happen is HTTP pipelining. This not frequent in real life (ironically, at least some of the browsers disable it by default because of "buggy proxies" :-) - but when it happens, can be a bit of a problem for the users to diagnose - especially if they have no access to the proxy.
One very notable place where it does happen by default apt-get in Debian-derived linux systems. Just install a Debian or Ubuntu server and try to use it through your proxy. You can do that by editing the /etc/apt/apt.conf.d/proxy file and placing the following there:
Acquire::http::Proxy "http://your.proxy.address:8080";
Depends of which abstraction layer of a packet you are talking about: there are many layers underneath HTTP.
HTTP --> TCP (byte stream) --> IP (packet) --> (possibly something else) Ethernet (frame) --> (possibly) some other transport
If you are talking about the IP layer, then yes the HTTP layer would start later on... Note that TCP presents a "byte stream interface" to its Client layer hence, no concept of packet here.
I think I understand where you are trying to go with this question.
If you don't use persistent HTTP connections, the HTTP GET request header is always the very first thing which is sent over the TCP connection, so we can be sure that the start of the HTTP GET request header does "not start in the middle of some TCP packet". But keep in mind that there may be one or more TCP packets without any user data, e.g. only a SYN, which may preceed the TCP packet with the start of the HTTP GET request header. And also keep in mind that the HTTP GET request header may not be contained in a single TCP packet.
If you do use persistent HTTP connections, the start of the HTTP GET request header for request number N+1 can start in the middle of a TCP packet, namely after the end of HTTP GET request body of request number N.
If you are asking these questions you are possibly "doing it wrong". As several other responders have already pointed out, in the vast majority of cases you should probably just be a TCP client and deal with a TCP stream of data and let the TCP code worry about the TCP packets. (Unless, of course, you are working on some special hardware which is looking at individual IP packets as they fly by and try to do some processing at the HTTP layer.)