I need to extract TCP Flows with their content from dump file and then save their flow into other file each flow separately
You definitely want to use Bro, more specifically, its contents.bro policy. For example, given a trace that contains HTTP requests, running the following ...
bro -r http.trace -f 'tcp and port 80' contents
... produces files
contents.[senderIP].[senderPort]-[destIP].[destPort]
contents.[destIP].[destPort]-[senderIP].[senderPort]
for each connection, each containing the unidirectional content of the flow.
The flow reassembly is highly robust, the process scales to very large files, and everything is customizable to your needs.
If you're only doing a few, Wireshark can do this.
Steps:
Open up the capture in Wireshark.
Click on a packet from the TCP connection you're interested in
Analyze -> Follow TCP Stream
Click 'Raw'
Select (from the popup menu) one of 'Entire Conversation' or one of the two directions.
Click 'Save As'
Alternate steps, for HTTP only:
Open up the capture
Select File -> Export -> Objects -> HTTP
A dialog will open showing all the HTTP objects in the capture. You can save some or all of them.
This is with Wireshark 1.2.1 on Linux/GTK. The 'follow TCP stream' option has been moved around between versions, so it may be somewhere else if you have an older version. But its always been called Follow TCP Stream so you should be able to find it.
Quick searching also reveals several other options if Wireshark doesn't work for you: ngrep, tcpick, chaosreader, and tcpflow.
tcpflow -r my_dump_file.pcap -o output_dir/
It will extract each tcp flow, separately, into a file under output_dir. Each flow in its own file.
Here's the manpage with more options
Wire shark maybe? It can be used to filter sessions and I think you can then save them seperatly.
You could also have a look at NetFlow and related tools.
Related
I am trying to split large pcap files containing hundreds of TCP streams into separate files. My current approach (see below) seems quite inefficient to me. My question is: What is the most efficient way of splitting pcap files into separate files by TCP stream?
Current approach
In my current approach, I first use tshark to find out which TCP streams are in the file. Next, for each of these TCP streams, I read the original file and extract the given stream. The code snippet below shows my approach:
#!/bin/bash
# Get all TCP stream numbers
for stream in `tshark -r $file -T fields -e tcp.stream | sort -n | uniq`
do
# Extract specified stream from $file and write it to a separate file.
tshark -r "$file" -Y "tcp.stream eq $stream" -w "$file.$stream.pcap"
done
However, this approach seems inefficient as tshark has to read the pcap file several times (once for each stream). I would ideally like a solution that goes over the original pcap file once and upon finding a packet belonging to a specific connection, append it to that file.
Other approaches
I have looked around for other approaches as well, but they do not seem to suit my situation:
PcapPlusPlus' PcapSplitter has a slightly different definition of a TCP connection. They define 'connection' as the same (protocol, source ip, destination ip, source port, destination port)-tuple, which might show weird behaviour if multiple TCP streams have the same tuple. I believe wireshark/tshark actually base their TCP streams on the SYN:SYN-ACK and FIN:FIN-ACK flags (but please correct me if I am wrong).
Python's Scapy Scapy has the same problem as PcapSplitter in that it does not provide any way of splitting TCP streams apart from the 5-tuple described above. (Of course I could write this myself, but that would be beyond the scope of my current work).
Also for both of these solutions, I am not entirely sure whether they are able to correctly handle erroneous captures.
Question
Therefore, I would like to have some suggestions on how to split pcap files into separate files based on TCP stream in the most efficient way.
Have you looked into Tracewrangler? It's for Windows but the documentation does mention that it can run under wine.
That's probably the best tool I can think of, but you might want to have a look at some others listed on the Wireshark wiki Tools page.
An efficient way (in performance point of view) is obliviously a dedicated program for the task.
libpcap-library may have needed functions for implementing such:
pcap_open_offline for opening a pcap file for reading
pcap_dump_open for opening pcap files for writing
pcap_dump for write packet to target files
And bunch of functions for filtering/handling the input.
You can use pkt2flow:
https://github.com/caesar0301/pkt2flow
Usage: ./pkt2flow [-huvx] [-o outdir] pcapfile
Options:
-h print this help and exit
-u also dump (U)DP flows
-v also dump the in(v)alid TCP flows without the SYN option
-x also dump non-UDP/non-TCP IP flows
-o (o)utput directory```
For example, I have text log with sip messages.
I want to filter these messages such as I can to do this in wireshark display filter, for examples with next filter:
sip.Method ne REGISTER || sip.CSeq.method ne REGISTER
In output I want to make file without messages, that not accepted by filter.
Is it possible?
Wireshark is normally used with PCAP files but it can open a plethora of other formats. The section 5.2 of the manual has a list of some 30 different formats, notably ppd logs, juniper netscreen and apple packet logger. Just open the file File > Open.
Then you type your filter (you can use the keyword not in filters) and Apply the filter. Only the packets matched will be displayed. Now you shall save it with File > Export Specified Packets. In the dialog shown you shall click on Displayed and save the file as one of the several format available (although you should really be using the PCAP format for packets, it is widely supported).
A lot depends on the format of your log file. If you have the bytes from the UDP messages under SIP you likley used some tool to assemble this log. If the log has no info about the actual raw packets, you might as well use a simple grep.
I have a strange one.
A number of data items are being collected by collectd and appear correctly with
collectdctl -s /var/run/collectdctl listval|getval and so forth.
These are then rendered into graphite effectively for most items.
Recently, the collectd-graphite connection ceased to be operational
for several recently added items. While it appears in collectd and
is queryable via collectdctl, it remains not on the graphite page.
I am asking to find out how you would approach this.
Thanks for any comment.
There's probably a number of ways you can troubleshoot this, but I end up almost always resorting to tcpdump, sigh. First enable debug logging in collectd just to make sure it really doesn't spit out an error message (LogLevel "debug" https://collectd.org/wiki/index.php/Plugin:LogFile although often collectd is compiled with debug logging disabled).
Then run tcpdump on the graphite server using the -s0 -X flags to tcpdump so you get the packet contents. (You can also use a more sophisticated network sniffer that prints the tcp data stream.) Check whether you see the data items that are missing the packets and whether they look appropriate (see https://collectd.org/wiki/index.php/Plugin:Write_Graphite). Typically this step allows me to quickly determine whether the problem is the sending collectd or the receiving service.
I have FreeBSD host (some sort of HTTP Proxy) with spikes of retransmitted packets number. Is there any way to track were host loosing them (per incoming connection).
I usually capture a bunch of them with tcpdump or similar; and then post process them elsewhere. In your case that should not be hard - as you just need the header.
Something like tcpdump (without; or a < 200 byte -s fly) would do on the target machine.
Compress/move this file then off to a desktop machine to work on it. I'd start with something like wireshark (simply use the filters).
Beyond that - simple grep-ing/wc-counting or a small perl script may be called for. To save you re-inventing histograms; consider http://snippets.aktagon.com/snippets/62-How-to-generate-a-histogram-with-Perl or do a quick google.
I want to monitor the websocket traffic (like to see what version of the protocol the client/server is using) for debugging purposes. How would I go about doing this? Wireshark seems too low level for such a task. Suggestions?
Wireshark sounds like what you want actually. There is very little framing or structure to WebSockets after the handshake (so you want low-level) and even if there was, wireshark would soon (or already) have the ability to parse it and show you the structure.
Personally, I often capture with tcpdump and then parse the data later using wireshark. This is especially nice when you may not be able wireshark on the device where you want to capture the data (i.e. a headless server). For example:
sudo tcpdump -w /tmp/capture_data -s 8192 port 8000
Alternately, if you have control over the WebSockets server (or proxy) you could always print out the send and receive data. Note that since websocket frames start with '\x00' will want to avoid printing that since in many languages '\x00' means the end of the string.
If you're looking for the actual data sent and received, the recent Chrome Canary and Chromium have now WebSocket message frame inspection feature.
You find details in this thread.
I think you should use Wireshark
Steps
Open wireshark
Go to capture and follow bellow path: capture > interfaces > start capture in your appropriate device.
Write rules in filter tcp.dstport == your_websoket_port
Hit apply
For simple thing, wireshark is too complex, i wanted to check only if the connection can be establish or not. Following Chrome plugin "Simple Web-socket (link : https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en)" work like charm. See image.
https://lh3.googleusercontent.com/bEHoKg3ijfjaE8-RWTONDBZolc3tP2mLbyWanolCfLmpTHUyYPMSD5I4hKBfi81D2hVpVH_BfQ=w640-h400-e365