I am able to read/replay all the headers and fields with tshark until hitting IPv6 header (ethernet header & IPv6 header), but when I am trying to replay the pcap files to read icmpv6 fields, nothing is being displayed of those fields.
Is this a bug of tshark? Is there any alternative tool to read all the fields in all headers of a packet?
The version of tshark I am using is 1.2.11
Bro is a network traffic analysis tool with full IPv6 support, whereas tshark seems to struggle with IPv6. In Bro, you can get connection summaries by running it on a trace as follows:
bro -C -r trace.pcap
and inspect the resulting file conn.log in the same directory. You may find the accompanying tool bro-cut helpful to extract only a subset of the columns, e.g.,
bro-cut id.orig_h id.resp_h id.orig_p id.resp_p proto < conn.log
would extract the connection 5-tuple and print it to STDOUT, so that you can continue processing it with your favorite text munching tool.
Related
I am trying to split large pcap files containing hundreds of TCP streams into separate files. My current approach (see below) seems quite inefficient to me. My question is: What is the most efficient way of splitting pcap files into separate files by TCP stream?
Current approach
In my current approach, I first use tshark to find out which TCP streams are in the file. Next, for each of these TCP streams, I read the original file and extract the given stream. The code snippet below shows my approach:
#!/bin/bash
# Get all TCP stream numbers
for stream in `tshark -r $file -T fields -e tcp.stream | sort -n | uniq`
do
# Extract specified stream from $file and write it to a separate file.
tshark -r "$file" -Y "tcp.stream eq $stream" -w "$file.$stream.pcap"
done
However, this approach seems inefficient as tshark has to read the pcap file several times (once for each stream). I would ideally like a solution that goes over the original pcap file once and upon finding a packet belonging to a specific connection, append it to that file.
Other approaches
I have looked around for other approaches as well, but they do not seem to suit my situation:
PcapPlusPlus' PcapSplitter has a slightly different definition of a TCP connection. They define 'connection' as the same (protocol, source ip, destination ip, source port, destination port)-tuple, which might show weird behaviour if multiple TCP streams have the same tuple. I believe wireshark/tshark actually base their TCP streams on the SYN:SYN-ACK and FIN:FIN-ACK flags (but please correct me if I am wrong).
Python's Scapy Scapy has the same problem as PcapSplitter in that it does not provide any way of splitting TCP streams apart from the 5-tuple described above. (Of course I could write this myself, but that would be beyond the scope of my current work).
Also for both of these solutions, I am not entirely sure whether they are able to correctly handle erroneous captures.
Question
Therefore, I would like to have some suggestions on how to split pcap files into separate files based on TCP stream in the most efficient way.
Have you looked into Tracewrangler? It's for Windows but the documentation does mention that it can run under wine.
That's probably the best tool I can think of, but you might want to have a look at some others listed on the Wireshark wiki Tools page.
An efficient way (in performance point of view) is obliviously a dedicated program for the task.
libpcap-library may have needed functions for implementing such:
pcap_open_offline for opening a pcap file for reading
pcap_dump_open for opening pcap files for writing
pcap_dump for write packet to target files
And bunch of functions for filtering/handling the input.
You can use pkt2flow:
https://github.com/caesar0301/pkt2flow
Usage: ./pkt2flow [-huvx] [-o outdir] pcapfile
Options:
-h print this help and exit
-u also dump (U)DP flows
-v also dump the in(v)alid TCP flows without the SYN option
-x also dump non-UDP/non-TCP IP flows
-o (o)utput directory```
Does anybody know where within tshark or wireshark the code is that I could use to reassemble pcap files? I am working on an app and need to reassemble pcap files, but don't need the other functionality of wireshark / tshark...so, hoping to use this as guidance.
Thanks.
If it's tcpdump file (not ng format), you can throw away the first 24 bytes (file header) of the second file and concatenate the rest to the first file, then you do the same for all the other files.
mergecap (from wireshark suite) will merge two or more pcap files.
I want to write an application layer sniffer (SMTP/ftp/http).
Based on my searchs, first (and perhaps hardest!) step is to reassemble the tcp stream of the sniffed connections.
Indeed, what I need is something like the "follow TCP stream" option of wireshark, but I need a tool which do it on live interface and automatically. As I know, Tshark can extract TCP streams data from the saved pcap files automatically (link) but not from live interfaces. Can Tshark do it on live interfaces???
As I know, TCPflow can do exactly what I want, however, it can not handle IP defragmentation and SSL connections (I want to analyse the SSL content in the case I have the server private key).
Finally, I also try bro network monitor. Although it provides the list of TCP connections (conn.log), I was not able to get TCP connections contents.
Any suggestion about mentioned tools or any other useful tool is welcome.
Thanks in advance, Dan.
perl Net::Inspect library might help you. It also comes with a tcpudpflow which can write tcp and udp flows into separate files, similar to tcpflow. It works on pcap files or can do live captures. The library handles IP fragmenting. It also comes with a httpflow tool to extract HTTP requests and responses (including decompression, chunked encoding..). It does not currently handle SSL.
As the author of this library I don't think that extracting TCP flows is the hardest part, the HTTP parser (exluding decompression, including chunked mode) is nearly twice as big than IP and TCP combined.
This example works for reassembling application data of a single protocol:
tshark -Y "tcp.dstport == 80" -T fields -d tcp.port==80,echo -e echo.data
It captures live http data, reassembles it, and outputs it as raw hex.
I can add a small script to parse the hex into ascii if you like.
I want to analyse the SSL content in the case I have the server private key
TL;DR: This can't be done with a capturing tool alone.
Why not: Because each SSL session generates a new secret conversation key, and you can't decrypt the session without this key. Having the private server key is not enough. The idea behind this is that if someone captures your SSL traffic, saves it, and then a year later he "finds" the private server key, then he still won't be able to decrypt your traffic.
I have FreeBSD host (some sort of HTTP Proxy) with spikes of retransmitted packets number. Is there any way to track were host loosing them (per incoming connection).
I usually capture a bunch of them with tcpdump or similar; and then post process them elsewhere. In your case that should not be hard - as you just need the header.
Something like tcpdump (without; or a < 200 byte -s fly) would do on the target machine.
Compress/move this file then off to a desktop machine to work on it. I'd start with something like wireshark (simply use the filters).
Beyond that - simple grep-ing/wc-counting or a small perl script may be called for. To save you re-inventing histograms; consider http://snippets.aktagon.com/snippets/62-How-to-generate-a-histogram-with-Perl or do a quick google.
I need to extract TCP Flows with their content from dump file and then save their flow into other file each flow separately
You definitely want to use Bro, more specifically, its contents.bro policy. For example, given a trace that contains HTTP requests, running the following ...
bro -r http.trace -f 'tcp and port 80' contents
... produces files
contents.[senderIP].[senderPort]-[destIP].[destPort]
contents.[destIP].[destPort]-[senderIP].[senderPort]
for each connection, each containing the unidirectional content of the flow.
The flow reassembly is highly robust, the process scales to very large files, and everything is customizable to your needs.
If you're only doing a few, Wireshark can do this.
Steps:
Open up the capture in Wireshark.
Click on a packet from the TCP connection you're interested in
Analyze -> Follow TCP Stream
Click 'Raw'
Select (from the popup menu) one of 'Entire Conversation' or one of the two directions.
Click 'Save As'
Alternate steps, for HTTP only:
Open up the capture
Select File -> Export -> Objects -> HTTP
A dialog will open showing all the HTTP objects in the capture. You can save some or all of them.
This is with Wireshark 1.2.1 on Linux/GTK. The 'follow TCP stream' option has been moved around between versions, so it may be somewhere else if you have an older version. But its always been called Follow TCP Stream so you should be able to find it.
Quick searching also reveals several other options if Wireshark doesn't work for you: ngrep, tcpick, chaosreader, and tcpflow.
tcpflow -r my_dump_file.pcap -o output_dir/
It will extract each tcp flow, separately, into a file under output_dir. Each flow in its own file.
Here's the manpage with more options
Wire shark maybe? It can be used to filter sessions and I think you can then save them seperatly.
You could also have a look at NetFlow and related tools.