For example, I have text log with sip messages.
I want to filter these messages such as I can to do this in wireshark display filter, for examples with next filter:
sip.Method ne REGISTER || sip.CSeq.method ne REGISTER
In output I want to make file without messages, that not accepted by filter.
Is it possible?
Wireshark is normally used with PCAP files but it can open a plethora of other formats. The section 5.2 of the manual has a list of some 30 different formats, notably ppd logs, juniper netscreen and apple packet logger. Just open the file File > Open.
Then you type your filter (you can use the keyword not in filters) and Apply the filter. Only the packets matched will be displayed. Now you shall save it with File > Export Specified Packets. In the dialog shown you shall click on Displayed and save the file as one of the several format available (although you should really be using the PCAP format for packets, it is widely supported).
A lot depends on the format of your log file. If you have the bytes from the UDP messages under SIP you likley used some tool to assemble this log. If the log has no info about the actual raw packets, you might as well use a simple grep.
Related
I'm working with Scapy to parse a set of .pcap files. I would like to understand what kind of payload those packets are carrying. If I have for example a pcap file with a lot of UDP packets which payloads has the same starting bytes I don't know what kind of encoding was used, and the first values keep repeating in other packets. Is there any program or python library that could allow me to figure out or try to guess what kind of encoding was used (if for example is an RTP payload or MPEG one and so on)?
UPDATE
I was able to use nDPI on those pcap files and it gave me satisfying results for all the flows except for a set of them that it was not able to recognize. I'm going to share with you the first part of the hex representation of the data:
f1d00404d1002d7c484830320000020080073804610d00007b09040000000000010f000000000000000000000000000000000000000000000000000121e002a22e537fcccb815afafce2361b
The first part f1d004 does not change between previous and successive packets. I have already tried to decode them with different protocols using wireshark's feature "Decode as". I have tried with RTP,RTCP,RTSP,JSON,MPEG. If can be useful, this is the capture related to a camera, that's why I tried the previous protocols.
I'm supposed to duplicate a binary TCP Stream.
So I set up a NiFi 1.9.0 server, put in a ListenTCP processor and a PutTCP processor, configured the proper IPs and Ports and connected them.
So far so good, the packets were received by the ListenTCP processor and also forwareded by the PutTCP processor.
But NiFi seems to mess around with the data somehow, the sent packets aren't exactly the same as received. I expected NiFi to just forward everything 1:1 but something is happening and I cannot find out what.
I've been playing around with the Character Set, Max Batch Size and Batching Message Delemiter settings on the ListenTCP processor and also with the Outgoing Message Delemiter and Character Set on the PutTCP processor.
I also messed around with a MergeContent processor but didn't get it to work properly.
Here you can see the difference between received (red) and sent data (captured using tcpflow).
Link to picture
Another problem is that I don't really know the data I'm processing, it says in the documentation:
These log files are in the machine-readable binary format that is described by the XML file called ebm.xml.
and
The streamed events are in the TCP-based binary format.
I do have access to ebm.xml file, but not sure how I can make use of it.
Anyone an idea how I can get NiFi to simply forward everything?
I'm new to NiFi, so I might have missed some possibilites...
The ListenTCP processor reads data from the stream using a new-line character as a logical message separator. For example, if the stream had:
<chunk1><new-line><chunk2><new-line><chunk3><new-line>
It would result in reading chunk1, chunk2, and chunk3 into an internal queue.
When it writes them back out it uses the outgoing message delimiter. So the outgoing flow file would be:
<chunk1><outgoing-delim><chunk2><outgoing-delim><chunk3><outgoing-delim>
Unfortunately it is more geared towards receiving textual data such as logs which are typically line-delimited. The chunks should be passing through unaltered as byte[], but typically binary data wouldn't have these logical new-line boundaries, so I'm not sure how well it works for that.
Unfortunately I'm not too familiar with Wireshark and in our recent homework we are supposed to create a pcap file which includes a specific number. In order to create that pcap file we are supposed to use the search function of Wireshark to find by string in packet bytes and export the result with the specified number in either the protocol header or the payload. How am I supposed to go about this?
Well, this was way easier than I thought. All I needed to do was to create a connection to a FTP server, listen to that connection in Wireshark and then transfer a textfile with the number in it/named after it, in plain FTP in ASCII mode.
I need to extract TCP Flows with their content from dump file and then save their flow into other file each flow separately
You definitely want to use Bro, more specifically, its contents.bro policy. For example, given a trace that contains HTTP requests, running the following ...
bro -r http.trace -f 'tcp and port 80' contents
... produces files
contents.[senderIP].[senderPort]-[destIP].[destPort]
contents.[destIP].[destPort]-[senderIP].[senderPort]
for each connection, each containing the unidirectional content of the flow.
The flow reassembly is highly robust, the process scales to very large files, and everything is customizable to your needs.
If you're only doing a few, Wireshark can do this.
Steps:
Open up the capture in Wireshark.
Click on a packet from the TCP connection you're interested in
Analyze -> Follow TCP Stream
Click 'Raw'
Select (from the popup menu) one of 'Entire Conversation' or one of the two directions.
Click 'Save As'
Alternate steps, for HTTP only:
Open up the capture
Select File -> Export -> Objects -> HTTP
A dialog will open showing all the HTTP objects in the capture. You can save some or all of them.
This is with Wireshark 1.2.1 on Linux/GTK. The 'follow TCP stream' option has been moved around between versions, so it may be somewhere else if you have an older version. But its always been called Follow TCP Stream so you should be able to find it.
Quick searching also reveals several other options if Wireshark doesn't work for you: ngrep, tcpick, chaosreader, and tcpflow.
tcpflow -r my_dump_file.pcap -o output_dir/
It will extract each tcp flow, separately, into a file under output_dir. Each flow in its own file.
Here's the manpage with more options
Wire shark maybe? It can be used to filter sessions and I think you can then save them seperatly.
You could also have a look at NetFlow and related tools.
I have a file that apparently contains some sort of dump of a keep-alive HTTP conversation, i.e. multiple GET requests and responses including headers, containing an HTML page and some images. However, there is some binary junk in between - maybe it's a dump on the TCP or even IP level (I'm not sure how to determine what it is).
Basically, I need to extract the files that were transferred. Are there any free tools I could use for this?
Use Wireshark.
Look into the file format for its dumps and convert your dump to it. Its very simple. Its called the pcap file format. Then you can open it in Wireshark no problem and it should be able to recognize the contents. Wireshark supports many dozens if not many hundred communication formats at various OSI layers (including TCP/IP/HTTP) and is great for this kind of debugging.
Wireshark will analyze on the packet level. If you want to analyze on the protocol level, I recommend Fiddler: http://www.fiddlertool.com/fiddler/
It will show you the headers sent, the responses, and will decrypt HTTPS sessions as well. And a ton more.
The Net tab in the Firebug plugin for Firefox might be of use.