I'm new to LUA and have successfully written a dissector to dissect a custom protocol over TCP (for data being sent over network between 2 modules in my application) for non-fragmented TCP case.
But this doesn't work when the TCP data is sent split into multiple packets. After some research, found out that the only way to do this in lua script is to use pkt.desegment_len and pkt.desegment_offset. But with modified code, when I open a given pcap in wireshark (with this dissector lua script plugged in) I see those pkt info of those packets as - [TCP segment of a reassembled PDU] and nothing else in the details (tree) section.
My protocol contains few header fields, where 6th and 7th bytes denote the length of given packet.
Here is my code, let me know if I'm missing something or can somebody give me a sample working script for this reassembled TCP packet handling.
function p_Kod_CALLP_MED.dissector (buf, pkt, root)
local desiredPacketLen = buf(5,2):uint()
-- if not enough data indicate how much more we need
if desiredPacketLen > buf:len() then
pkt.desegment_len = desiredPacketLen - buf:len()
pkt.desegment_offset = 0
return
end
-- have more than needed so set offset for next dissection
if buf:len() >= desiredPacketLength then
pkt.desegment_len = DESEGMENT_ONE_MORE_SEGMENT
pkt.desegment_offset = desiredPacketLength
end
-- Remaining logic like adding to the tree etc
Related
When I use TCP I need destination port (to be able to "talk" to other process on the other host) and source port (because TCP is connection oriented so I'll send data back to source like ack, seq and more).
On the other side, UDP which is connectionless needs also source port.
Why is it? (I don't need to send back data)
Probably, two reasons.
First, receivers often need to reply and it is useful to provision a standard tool for that.
Secondly, you may have multiple interfaces (network cards) and using source address, you decide which of them must be used to emit the packet.
You don't need to but there's still the possibility to send a response back (that is very useful actually) however as stated in the RCF 768
Source Port is an optional field, when meaningful, it indicates the port
of the sending process, and may be assumed to be the port to which a
reply should be addressed in the absence of any other information. If
not used, a value of zero is inserted.
https://www.rfc-editor.org/rfc/rfc768
I would like to add to the answers here. Apart from simply knowing what to reply to, the source port can belong to the list of well-known port numbers. These ports specify what kind of data is encapsulated in the UDP (or TCP!) packet.
For example, the source port 530 indicates that the packet contains a Remote Procedure Call, and 520 indicates a Routing Information Protocol packet.
We are doing some load testing on our servers and I'm using tshark to capture some data to a pcap file then using the wireshark GUI to see what errors or warnings are showing up by going to Analyze -> expert Info with my pcap loaded in..
I'm seeing various things that I'm not sure or do not completely understand yet..
Under Warnings I have:
779 Warnings for TCP: ACKed segment that wasn't captured (common at capture start)
446 TCP: Previous segment not captured (common at capture start)
An example is :
40292 0.000 xxx xxx TCP 90 [TCP ACKed unseen segment] [TCP Previous segment not captured] 11210 > 37586 [PSH, ACK] Seq=3812 Ack=28611 Win=768 Len=24 TSval=199317872 TSecr=4506547
We also ran the pcap file though a nice command that creates a command line column of data
command
tshark -i 1 -w file.pcap -c 500000
basically just saw a few things in the tcp.analysis.lost_segment column but not many..\
Anyone enlighten what might be going on? tshark not able to keep up with writing data, some other issue? False positive?
That very well may be a false positive. Like the warning message says, it is common for a capture to start in the middle of a tcp session. In those cases it does not have that information. If you are really missing acks then it is time to start looking upstream from your host for where they are disappearing. It is possible that tshark can not keep up with the data and so it is dropping some metrics. At the end of your capture it will tell you if the "kernel dropped packet" and how many. By default tshark disables dns lookup, tcpdump does not. If you use tcpdump you need to pass in the "-n" switch. If you are having a disk IO issue then you can do something like write to memory /dev/shm. BUT be careful because if your captures get very large then you can cause your machine to start swapping.
My bet is that you have some very long running tcp sessions and when you start your capture you are simply missing some parts of the tcp session due to that. Having said that, here are some of the things that I have seen cause duplicate/missing acks.
Switches - (very unlikely but sometimes they get in a sick state)
Routers - more likely than switches, but not much
Firewall - More likely than routers. Things to look for here are resource exhaustion (license, cpu, etc)
Client side filtering software - antivirus, malware detection etc.
Another cause of "TCP ACKed Unseen" is the number of packets that may get dropped in a capture. If I run an unfiltered capture for all traffic on a busy interface, I will sometimes see a large number of 'dropped' packets after stopping tshark.
On the last capture I did when I saw this, I had 2893204 packets captured, but once I hit Ctrl-C, I got a 87581 packets dropped message. Thats a 3% loss, so when wireshark opens the capture, its likely to be missing packets and report "unseen" packets.
As I mentioned, I captured a really busy interface with no capture filter, so tshark had to sort all packets, when I use a capture filter to remove some of the noise, I no longer get the error.
Acked Unseen sampleHi guys! Just some observations from what I just found in my capture:
On many occasions, the packet capture reports “ACKed segment that wasn't captured” on the client side, which alerts of the condition that the client PC has sent a data packet, the server acknowledges receipt of that packet, but the packet capture made on the client does not include the packet sent by the client
Initially, I thought it indicates a failure of the PC to record into the capture a packet it sends because “e.g., machine which is running Wireshark is slow” (https://osqa-ask.wireshark.org/questions/25593/tcp-previous-segment-not-captured-is-that-a-connectivity-issue)
However, then I noticed every time I see this “ACKed segment that wasn't captured” alert I can see a record of an “invalid” packet sent by the client PC
In the capture example above, frame 67795 sends an ACK for 10384
Even though wireshark reports Bogus IP length (0), frame 67795 is
reported to have length 13194
Frame 67800 sends an ACK for 23524
10384+13194 = 23578
23578 – 23524 = 54
54 is in fact length of the
Ethernet / IP / TCP headers (14 for Ethernt, 20 for IP, 20 for TCP)
So in fact, the frame 67796 does represent a large TCP packets (13194
bytes) which operating system tried to put on the wore
NIC driver will fragment it into smaller 1500 bytes pieces in order to transmit over the network
But Wireshark running on my PC fails to understand it is a valid packet and parse it. I believe Wireshark running on 2012 Windows
server reads these captures correctly
So after all, these “Bogus IP
length” and “ACKed segment that wasn't captured” alerts were in fact
false positives in my case
I just came across this and would like to share my observation of TCP ACKed unseen segment. In my case the client was trying to initiate connection on same source port and destination port it used previously and thus the server was confused and replied with old TCP SEQ number saying I havent seen this new packet
How do TCP knows which is the last packet of a large file (that was segmented by tcp) in the scenario that the connection is kept-established. (like ftp or sending mp3 on yahoo messenger)
I mean how does it know which packet carries data of one.mp3 and which packet carries data of another.mp3 ??
Anyone ?
Thank you
There are at least 2 possible approaches.
Declare upfront how much data you're going to send. Something like a packet that declares Sending a message that's 4008 bytes long
The second approach is to use a terminating sequence (nastier to process)
So the receiver:
Tries to read the declared amount or
Scans for the terminating sequence
TCP is a stream protocol and fragmentation should be transparent to a TCP application. It operates on streams of data, never packets. A stream is assembled to its intended order using the sequence numbers. The sequence of bytes send by application is encapsulated in tcp segments. The stream is recreated on the receiver side before data is delivered to the application.
The IP protocol can do fragmentation.
Each TCP segment goes to the IP layer and may be fragmented there. Segment is reassembled by collecting all of the packets and offset field from the header is used to put it in the right place.
I was configuring IPtable yesterday. My colleague just asked me this question, and I couldn't anwser. I realized that I'm a much better developper than sysadmin and need to improve that.
So what are they? What are they for? Cons/Pros (if it's relevant).
These are like basic questions.
UDP :: User Datagram Protocol
1) No end to end Connection between to machines (may be in local network or somewhere in the internet).
2) The data received at the receiver end is not in stream as in TCP but as a complete block of data.
3) At the transport layer no packet order check is performed. That is in case of any error in the received packet, the receiver will not ask for resending the same packet to the sender.
4) Because of the above behaviour no sending buffers are required at the sender's end.
5) As no end to end connection is estld. and there are no handshakings required, UDP are pretty much faster but less reliable than TCP. Thus mostly used in gaming and DNS etc..
6) No acknowledgement required to be sent after recieiving packets.
TCP :: Transmission control Protocol
1) End to end Connection is maintained between to machines (may be in local network or somewhere in the internet).
2) The data received at the receiver end is a stream in TCP. Thus, when we do network programming for servers we first parse the header first and then depending upon the size mentioned in the header we obtain that much more number of bytes from the buffer.
3) Error checking and sequence number are all done. Thus in case any packet is received out of order (rarely) or is erred than that packet is made to resend. Also, lots of other protocols are involved for flow control (end to end flow control).
4) As connection establishment , handshaking and acknowledgement is to be done TCP are basically slower in operation than UDP.(Not significantly I believe)
5) Lots of protocols uses TCP as underlying transport protocol. HTTP,FTP,TELNET etc..
6) The communication procedure involves:
Server:: 1) Socket Open
2) Socket Bind
3) Socket Listen
4) Socket Accept
5) Socket Send/Recv
Client :: 1) Socket Open
2) Socket Connect
3) Socket Send/Recv
There are lots of other differeces also..but the above being the most common ones.
TCP is a reliable protocol which ensures that your packets reach their destination and is used in applications where all data must me trasfered accurately between parties. TCP requires both parties to negotiate a connection before data transfer can start and it is a resilient protocol since it will repeatedly resend a packet until that packet is received by the intended recipient.
UDP is unreliable in a sense that it allows some packets to be lost in transit. Some applications of UDP are found in movie streaming where you can actually afford to lose a frame and not jeopardize movie quality. UDP does not need binding between the two parties and is often looked at as a light alternative to TCP.
A nice table is found here:TCP vs UDP
P.R.'s answer is mostly correct, but incomplete.
TCP is a reliable, connected stream protocol. Its view of data is that of a bidirectional stream of bytes between hosts: whatever bytes you send will arrive at the other end in the same order, at least as far as the application is concerned (the OS will rearrange packets if needed).
UDP is an unconnected datagram protocol. Its view of data is that of discrete datagrams, or messages, with no guarantee that these messages actually reach their recipient, or that they arrive in the order they were sent. It does guarantee that if a message arrives, it arrives in its entirety and without modification.
This website probably offers the simplest explanation to the actual difference of UDP and TCP. From implementation point of view, see this question.
For short answer: TCP works kind of like registered letter when UDP is kind of like ordinary letter - with the latter you never know whether the recipient got the packet you sent.
There are loads of helpful comparisons
chris is right!
One fancy link dropping out of google is: http://www.skullbox.net/tcpudp.php
im now developing a project using winpcap..as i have known packets being sniffed are usually fragmented packets.
how to reassemble this TCP segements?..any ideas, suggestion or tutorials available?..
this i assume to be the only way i can view the HTTP header...
thanks!..
tcp is a byte stream protocol.
the sequence of bytes sent by your http application is encapsulated in tcp data segments and the byte stream is recreated before the data is delivered to the application on the other side.
since you are accessing the tcp datasegments using winpcap, you need to go to the data portion of the segment. the header of tcp has a fixed length of 20 bytes + an optional part which you need to determine using the winpcap api.
the length of data part in the tcp segment is determined by subtracting the tcp header length (obtained from a field in the tcp segment) and the ip header length (from a field in the ip datagram that encapsulates the tcp segment) from the total length (obtained from another field in the ip datagram).
so now you have the total segment length and the length of the data part within the segment. so you know offset where the http request data starts.
the offset is
total length-length of data part
or
length of ip-header + length of tcp header
i have not used winpcap. so you will have to find out how to get these fields using the api.
also ip datagrams may be further fragmented but i am expecting that you are provided only reassembled datagrams using this api. you are good to go!
There is no such thing as a TCP fragment. The IP protocol has fragments. TCP is a stream protocol. You can assemble the stream to its intended order by following the sequence numbers of both sides. Every TCP Packet goes to the IP level and can be fragmented there. You can assemble each packet by collecting all of the fragments and following the fragment offset from the header.
All of the information you need is in the headers. The wikipedia articles are quite useful in explaining what each field is
http://en.wikipedia.org/wiki/TCP_header#Packet_structure
http://en.wikipedia.org/wiki/IPv4#Header
PcapPlusPlus offers this capability out-of-the-box for all major OS's (including Windows). Please check out the TcpReassembly example to see a working code and the API documentation to understand how to use the TCP reassembly feature
Depending on the whose traffic you're attempting to passively reassemble, you may run into some TCP obfuscation techniques designed to confuse people trying to do exactly what you're trying to do. Check out this paper on different operating system reassembly behaviors.
libtins provides classes to perform TCP stream reassembly in a very high level way, so you don't have to worry about TCP internals to do so.