I am writing a program which can detects protocols used in network packets. For every packet receives it will try to detect protocols in layers such network and transport. Detecting protocols in these two layer was very easy because somewhere in the packet we have some bytes which tell us about protocols. But for application layers, it's much harder as I know. No where in a HTTP packet is not mentioned the protocol (as far as I know). And another difficulty during detecting application layer protocols is that it is possible that a whole HTTP request or response takes more than one packet, and it's lot harder to concatenate multiple packets.
I want to know theoretically how can I detect these protocols.
Unfortunately there's no simple answer. While there aren't that many network and transport layer protocols and the ones that exist are well standardized, the application layer is a lot messier.
One way to guess the application protocol is to look at various "hints" like port number, the presence of certain strings in specific locations (e.g. "HTTP"), packet lengths etc. But it's not bulletproof. I can easily run some custom protocol on port 80 which happens to contain "HTTP" in the payload but isn't HTTP. This is why even dedicated tools like wireshark sometimes fail to detect the correct protocol. By the way, you could use wireshark's source code for the specifics on how it dissects different protocols.
Regarding protocols being sent in more than one packet - that's easier. Your parser has to treat TCP as the stream protocol it is. Individual packets are meaningless in TCP and your parser has to track the stream across multiple packets.
As I understand it TCP is required for congestion control and error recovery or reliable delivery of information from one node to another and its not the fastest of protocols for delivering information.
Some routing protocols such as EIGRP and OSPF ride directly on top of IP. Even ICMP rides directly over IP.
Why is UDP even required at all? Is it only required so that developers/programmers can identify what application the inbound packet should be sent to based on the destination port number contained within the packet?
If that is the case then how is information gathered from protocols that ride directly on top of IP sent to the appropriate process when there is no port number information present?
Why are voice and video sent over UDP? Why not directly over IP?
(Note that I do understand thoroughly the use case for TCP. I am not asking why use UDP over TCP or vice versa. I am asking why use UDP at all and how can some protocols use directly the IP layer. Whats the added advantage or purpose of UDP over IP?)
Your question makes more sense in terms of why is UDP useful (than why is UDP required).
UDP is a recognized protocol by the Internet Assigned Numbers Authority. UDP can be useful if you want to write a network protocol that's datagram based and you want to play more nicely with Internet devices.
Routers can have rules to do things like drop any packet that doesn't make sense. So if you try and send packets using say an unassigned IP protocol number between hosts separated by one or more routers, the packets may well never get delivered as you've intended. The same could happen with packets from an unrecognized UDP protocol but that's at least one less door to worry about whether your packet can make it through.
Internet endpoints (like hosts) may do similar filtering too. If you want to write your own datagram based protocol and use a typical host operating system, you're more likely to need to write your software as a privileged process if not as a kernel extension if you're trying to ride it as its own IP protocol (than if you'll be using UDP).
Hope this answer is useful!
First of all, IP and UDP are protocols on the different layers, IP by definition is Internet layer when UDP is transport layer. Layers were introduced to simplify network protocols architecture and to separate concerns. Application layer protocols are supposed to be based on transport layer (with some exceptions).
Most popular transport protocols (in IP network) are UDP and TCP. While TCP is feature rich but with many tradeoffs UDP is very simple but gives a lot of freedom and so typically is a base for other protocols.
The main feature of UDP is multiplexing: ports that allow multiple protocol instances (aka sockets) to coexist on the same node. This means that implementing your own protocol over IP instead of UDP either you won't be able to have multiple instances of your protocol on the same machine or you'll have to implement multiplexing yourself.
There're other features like segmentation and checksum. These features are not mandatory.
And as was mentioned in another answer there're lots of middleware like routers, NATs and firewalls that can ruin the idea of a custom "right over IP" protocol, but it's more like a collateral damage than a feature of UDP.
I've seen while searching that it is possible to use two different programs on the same computer communicating over the network using the same port and same network interface provided one use UDP and the other TCP. However I didn't get a good explanation, how does it actually work and why this is possible?
Is it also possible for multiple programs to use the same UDP port since UDP does not establish a real connection between the peers, but just sends the packets to an address? I understand it's not possible with TCP as it creates a synchronized connection between the server and the client, but what about UDP?
Please explain in details if possible, or link a good article on the topic.
The other answers are correct but somewhat incomplete.
An IP (aka "INET") socket "connection" (i.e. communication between two processes, possibly on different machines) is defined by a 5-tuple: protocol, source address, source port, destination address, destination port. You can see that this is not limited to a stateful connection such as TCP.
This means that you can bind different processes to any unique instance of that 5-tuple. Because the "protocol" (e.g. TCP and UDP) is part of the differentiating factor, each can have a different process.
Theoretically, you could bind different services to the same TCP port if they bind to different interfaces (network cards, loopback, etc.) though I've never tried it.
It is standard practice, however, to always use the same service on the same port number. If both UDP and TCP are supported, they're just different ways of communicating with that same service. DNS, for example, uses UDP on port 53 for lookup because they are small requests and it's faster than creating a TCP connection but DNS also uses TCP on port 53 for "transfers" which are infrequent and can have large amounts of data.
Lastly, in complete accuracy, it isn't necessarily a 5-tuple. IP uses the "protocol" to pass to the next layer such as TCP and UDP though there are others. TCP and UDP each seperately differentiate connections based on the remaining 4 items. It's possible to create other protocols on top of IP that use completely different (perhaps port-less) differentiation mechanisms.
And then there are different socket "domains", such as the "unix" socket domain, which is completely distinct from "inet" and uses the filesystem for addressing.
The destination isn't identified by IP Addr:Port alone. There is another thing - IP header has a field called Protocol which differentiates the TCP and UDP endpoint. As such it becomes possible for two process to bind to same IP:Port as long as communication protocol is different.
The endpoint of a connection is for UDP and TCP defined by IP, protocol (TCP or UDP) and port. This means as long as you use a different protocol the endpoint of the communication is different too.
Because they are not the only component of the means of address. It's the same as why you can have two houses with the same number on different streets, or why you know John Whorfin is not the same Red Lectroid as John Bigbooté.
Each IP packet contains a field that says which transport-layer protocol is to be used, and within the domain of that protocol is a set of ports that can be the same as in any other protocol because they are actually a completely separate set.
As for the second question, there are answers elsewhere.
I have some questions about WebRTC:
Does WebRTC use TCP or UDP as its peer-to-peer transport? How can I
know?
I read that there are reliability mode and DTLS agreement, how does
these affect?
Is this transport the same for both Media and DataChannel?
How do I switch between TCP and UDP?
I ask this because I know that browsers have a limit on the number of parallel connections (I think they talk over TCP), and maybe UDP connection is not limited.
It can use either. By default, preference is given to UDP, but depending on the firewall(s) in between the peers connecting it may only be able to connect with TCP. You can use Wireshark to capture packets and verify whether TCP or UDP is being used. In Chrome you can also see details on the selected candidate (googActiveConnection) by going to chrome://webrtc-internals.
"Reliability mode" probably refers to the reliability mode of the DataChannel, which can be configured to run in reliable or unreliable mode. DTLS refers to the currently optional, but soon to be default method of exchanging encryption keys (the other deprecated mode is SDES). Firefox only supports DTLS, so for browser interop, you'll currently need to enable it in Chrome.
The RTCPeerConnection (media) will use TCP or UDP, while the DataChannel uses SCTP. The SCTP implementation used by Firefox is implemented on top of UDP: https://code.google.com/p/sctp-refimpl/.
It's possible to filter out TCP or UDP ICE candidates before adding them with addIceCandidate. Generally, you should not try to force the transport used since WebRTC will just "do the right thing". The browser does not limit the number of TCP connections used by WebRTC beyond any limit on the RTCPeerConnection or DataChannel (i.e., if you can have 10 PeerConnections, they can each use TCP without any problem).
I know the difference between the two on a technical level.
But in real life, can anyone provide examples (the more the better) of applications (uses) of TCP and UDP to demonstrate the difference?
UDP: Anything where you don't care too much if you get all data always
Tunneling/VPN (lost packets are ok - the tunneled protocol takes care of it)
Media streaming (lost frames are ok)
Games that don't care if you get every update
Local broadcast mechanisms (same application running on different machines "discovering" each other)
TCP: Almost anything where you have to get all transmitted data
Web
SSH, FTP, telnet
SMTP, sending mail
IMAP/POP, receiving mail
EDIT: I'm not going to bother explaining the differences, since you state that you already know and every other answer explains it anyway :)
UDP is mailing a letter at the post office.
TCP is mailing a letter with a return receipt at the post office, except that the post master will organize the letters in-order-of mailing and only deliver them in-order.
Well, it was an attempt anyway.
TCP:
World Wide Web(HTTP)
E-mail (SMTP TCP)
File Transfer Protocol (FTP)
Secure Shell (SSH)
UDP:
Domain Name System (DNS)
Streaming media applications such as movies
Online multiplayer games
Voice over IP (VoIP)
Trivial File Transfer Protocol (TFTP)
REAL TIME APPLICATION FOR TCP:
Email:
Reason: suppose if some packet(words/statement) is missing we cannot understand the content.It should be reliable.
REAL TIME APPLICATION FOR UDP:
video streaming:
* **Reason: ***suppose if some packet(frame/sequence) is missing we can understand the content.Because video is collection of frames.For 1 second video there should be
25 frames(image).Even though we can understand some frames are missing due to our imagination skills. Thats why UDP is used for video streaming.
The classic standpoint is to consider TCP as safe and UDP as unreliable.
But when TCP-IP protocols are used in safety critical applications,
TCP is not recommended because it can stop on error for multiple reasons.
Whereas UDP lets the application software deal with errors, retransmission timers, etc.
Moreover, TCP has more processing overhead than UDP.
Currently, UDP is used in aircraft controls and flight instruments,
in the ARINC 664 standard also named AFDX (Avionics Full-Duplex Switched Ethernet).
In ARINC 664, TCP is optional but UDP is used with the RTOS (real time operating systems) designed for the ARINC 653 standard (high reliability control software in civil aircrafts).
For more information about real time controls using IP and UDP in AFDX,
you can read the pages 27 to 50 in
http://www.afdx.com/pdf/AFDX_Training_October_2010_Full.pdf
TCP
I will not send data anymore until i get an acknowledgment.
this process is slow
It is used for security purpose
example: web, sending mail, receiving mail etc
UDP
Here i have no headache with acknowledgment.
this process is faster but here data can be lost .
example : video streaming , online games etc
TCP + UDP = SMTP(example : mobile,telephone)
TCP guarantees (in-order) packet delivery. UDP doesn't.
TCP - used for traffic that you need all the data for. i.e HTML, pictures, etc.
UDP - used for traffic that doesn't suffer much if a packet is dropped, i.e. video & voice streaming, some data channels of online games, etc.
TCP is a connection oriented protocol, It establishes a path, or a virtual connection all the way through switches routers proxies etc and then starts any communication. Various mechanisms like routing djikstras shortest path algorithm exist to establish the virtual end to end connection. So it finds itself used while browsing HTML and other pages, making payments and web applications in general.
UDP is a connectionless protocol - it simply has a destination and nodes simply pass it along if it comes as best as they can. So packets arriving out of order, along various routes etc are common. So Instant messengers and similar software developers think UDP an ideal solution.
In real life if you want to throw data in the net, without worrying about time taken to reach, order of reaching use UDP. If you want a solid path before you start throwing packets, and want same order and latency for your data packets use TCP - I will use UDP for Torrents and TCP for PayPal!
TCP :
Transmission Control Protocol is a connection-oriented protocol, which means that it requires handshaking to set up end-to-end communications. Once a connection is set up, user data may be sent bi-directionally over the connection.
Reliable – Strictly only at transport layer, TCP manages message acknowledgment, retransmission and timeout. Multiple attempts to deliver the message are made. If it gets lost along the way, the server will re-request the lost part. In TCP, there's either no missing data, or, in case of multiple timeouts, the connection is dropped. (This reliability however does not cover application layer, at which a separate acknowledgement flow control is still necessary)
Ordered – If two messages are sent over a connection in sequence, the first message will reach the receiving application first. When data segments arrive in the wrong order, TCP buffers delay the out-of-order data until all data can be properly re-ordered and delivered to the application.
Heavyweight – TCP requires three packets to set up a socket connection, before any user data can be sent. TCP handles reliability and congestion control.
Streaming – Data is read as a byte stream, no distinguishing indications are transmitted to signal message (segment) boundaries.
Applications of TCP
World Wide Web, email, remote administration, and file transfer rely on TCP.
UDP :
User Datagram Protocol is a simpler message-based connectionless protocol. Connectionless protocols do not set up a dedicated end-to-end connection. Communication is achieved by transmitting information in one direction from source to destination without verifying the readiness or state of the receiver.
Unreliable – When a UDP message is sent, it cannot be known if it will reach its destination; it could get lost along the way. There is no concept of acknowledgment, retransmission, or timeout.
Not ordered – If two messages are sent to the same recipient, the order in which they arrive cannot be predicted.
Lightweight – There is no ordering of messages, no tracking connections, etc. It is a small transport layer designed on top of IP.
Datagrams – Packets are sent individually and are checked for integrity only if they arrive. Packets have definite boundaries which are honored upon receipt, meaning a read operation at the receiver socket will yield an entire message as it was originally sent.
No congestion control – UDP itself does not avoid congestion. Congestion control measures must be implemented at the application level.
Broadcasts – being connectionless, UDP can broadcast - sent packets can be addressed to be receivable by all devices on the subnet.
Multicast – a multicast mode of operation is supported whereby a single datagram packet can be automatically routed without duplication to very large numbers of subscribers.
Applications of UDP
Numerous key Internet applications use UDP, including: the Domain Name System (DNS), where queries must be fast and only consist of a single request followed by a single reply packet, the Simple Network Management Protocol (SNMP), the Routing Information Protocol (RIP) and the Dynamic Host Configuration Protocol (DHCP).
Voice and video traffic is generally transmitted using UDP. Real-time video and audio streaming protocols are designed to handle occasional lost packets, so only slight degradation in quality occurs, rather than large delays if lost packets were retransmitted. Because both TCP and UDP run over the same network, many businesses are finding that a recent increase in UDP traffic from these real-time applications is hindering the performance of applications using TCP, such as point of sale, accounting, and database systems. When TCP detects packet loss, it will throttle back its data rate usage. Since both real-time and business applications are important to businesses, developing quality of service solutions is seen as crucial by some.
Some VPN systems such as OpenVPN may use UDP while implementing reliable connections and error checking at the application level.
TCP is appropriate when you have to move a decent amount of data (> ~1 kB), and you require all of it to be delivered. Almost all data that moves across the internet does so via TCP - HTTP, SMTP, BitTorrent, SSH, etc, all use TCP.
UDP is appropriate when you have small messages which you can afford to lose, and would like to send them as efficiently as possible. One reason you might be able to afford to lose them is because you can re-send them if they get lost. The main example on the internet is DNS - DNS consists of small queries saying things like "what is the IP number for stackoverflow.com?", and the responses are correspondingly small. Computers make a lot of these queries, so they should be made efficiently, but if they get lost en route, it's easy to time out and re-send them.
TCP guarantees packet delivery AND order. Order is almost as important as the delivery in the first place when reconstructing data for files such as executables, etc.
UDP does not guarantee delivery NOR order. Packets can arrive (or not!) in any order.
Common uses for TCP include file transfer where the integrity of the packets is paramount. Voice/video applications can afford to lose some data while still maintaining acceptable quality, and so usually use UDP.
One additional thought on some of the comments above that talks about ordered delivery.... It must be clarified that the destination computer may receive packets out of order on the wire, but the TCP at the destination is responsible for "rearranging out-of-order data" before passing it on to the upper layers of the stack. When you say TCP guarantees ordered packet delivery, what that means is it will deliver packets in correct order to the upper layers of the stack.
SCTP vs TCP vs UDPServices/Features SCTP TCP UDP
Connection-oriented yes yes no
Full duplex yes yes yes
Reliable data transfer yes yes no
Partial-reliable data transfer optional no no
Ordered data delivery yes yes no
Unordered data delivery yes no yes
Flow control yes yes no
Congestion control yes yes no
ECN capable yes yes no
Selective ACKs yes optional no
Preservation of message boundaries yes no yes
Path MTU discovery yes yes no
Application PDU fragmentation yes yes no
Application PDU bundling yes yes no
Multistreaming yes no no
Multihoming yes no no
Protection against SYN flooding attacks yes no n/a
Allows half-closed connections no yes n/a
Reachability check yes yes no
Psuedo-header for checksum no (vtags) yes yes
Time wait state vtags 4-tuple n/a
Since tcp usages are pretty straightforward from other answers, I'll mention some interesting UDP use-cases:
1)DHCP - Dynamic Host Configuration Protocol, which is being used in order to dynamically assign IP address and some other network configuration to the connecting devices. In simple words, this protocol allows you just connect to the network cable(or wifi) and start using the internet, without any additional configurations. DHCP uses UDP protocol. Since the settings request message is being broadcasted from the host and there is no way to establish a TCP connection with DHCP server(you don't know it's address) it's impossible to use TCP instead.
2)Traceroute - well-known network diagnostic tool which allows you to explore which path in the network your datagram passes to reach it's destination(and how much time it takes). By default, it works by sending UDP datagram with unlikely destination port number(ranging from 33434 to 33534) to the destination with the ttl(time-to-live) field set to 1. When the router somewhere in the network gets such datagram - it finds out that the datagram is expired. Then, the router drops the datagram and sends to the origin of the datagram an ICMP(Internet Control Message Protocol) error message indicating that the datagram's ttl was expired and containing router's name and IP address. Each time the host sends datagrams with higher and higher TTL, thus increasing the network part which it succeeds to overcome and getting new ICMP messages from new routers. When it eventually reaches it's destination(datagrams TTL is big enough to allow it),- the destination host sends 'Destination port unreachable' ICMP message to the origin host. This way, Traceroute knows that the destination was reached. Since the TCP guarantees segments delivery it would be at least inefficient to use it instead of UDP which, in turn, allows datagram to be just dropped without any resend attempts(resend is implemented on the higher level, with continuously increasing TTL as described above).
TCP: will get there in meaningful order
UDP: god knows (maybe)
UDP is applied a lot in games or other Peer-to-peer setups because it's faster and most of the time you don't need the protocol itself to make sure everything gets to the destination in the original order (UDP does not garantee packet delivery or delivery order).
Web traffic on the other hand is over TCP. (I'm not sure here but I think it has to do with the way the HTTP protocol is built)
Edited because I failed at UDP.
Real life examples of both TCP and UDP
tcp -> a phone call, sms or anything specific to destination
UDP -> a FM radio channel (AM), Wi-Fi.