If so, do you know examples of what can go wrong in a non-TCP network?
Learning about MQTT I came across several mentions of the fact that MQTT relies on TCP/IP stack. For example, from mqtt.org:
MQTT for Sensor Networks is aimed at embedded devices on non-TCP/IP
networks, whereas MQTT itself explicitly expects a TCP/IP stack.
But if you read the reference documents, you won't finding anything like that. Moreover, there's QoS field that can be used for reliable delivery and whose values other than 0 are essentially useless in TCP/IP networks. Right now I see nothing that would prevent me from establishing an MQTT connection using UNIX pipe, domain or UDP socket rather than TCP socket.
MQTT just needs a delivery that is ordered and reliable, it doesn't have to be TCP. SCTP works just fine, for example, but UDP doesn't because there is no way to guarantee your large PUBLISH packet made up of multiple UDP packets will arrive in order and complete.
With regards TCP reliability, in theory what you are saying is true, but in practice when an application calls write() and receives a successful return, it can't guarantee when the data has actually made it out of the computer to the remote host. All write() (or send()) does is copy the data to the kernel buffers, at which point you have no further control.
The only way to be sure that a message reaches the remote host at the application level is to have the remote host reply.
MQTT-SN (for sensor Network ) is the solution for the problem that MQTT was having while running over TCP/IP .
There is a concept of MQTT gateway which is brought in in for MQTT-SN which helps in bringing non-TCP / IP implementation.
http://emqttd-docs.readthedocs.io/en/latest/mqtt-sn.html
For a service which just returns a small number when queried such as 30, or 10, but would have to handle up to 5 or so requests at any instance, would TCP or UDP be a better protocol? I am leaning towards UDP, but I wanted some expert opinions. I am looking for relatively quick reply times as well. Could you tell me what the advantages of each would be for a service like this? Thanks.
TCP is a reliable connection-based protocol. So, you are guaranteed that data is sent/received - the packets are automatically re-sent if they are not verified to be received on the other end. However, there is the overhead of the three-way handshake for establishing the connection.
TCP is used for protocols like HTTP where there is a one-time exchange of information (the HTTP Request and Reply).
UDP is an unreliable connection-less protocol. So you can simply send / receive a packet but you have no (automatic, OS stack-provided) way of verifying that the other end got your message. If you care, you have to implement some kind of ACK yourself.
UDP is used often for more continuous, "streaming" type protocols. For example, many online multiplayer games use UDP to exchange game information to/from the host. They do this on a continual, periodic basis. So if a packet is lost, it's not really a big deal, because another update is just around the corner. It would be far worse for the gameplay if you had to wait for that (now stale) update to be re-transmitted.
DNS is also implemented over UDP.
Ultimately the choice is yours. I would probably default to TCP for most cases, and only use UDP in a scenario like I described.
Since the MQTT protocol flows over TCP the difference between the three QoS levels is quite subtle. In normal conditions TCP will ensure delivery and prevent duplicates.
I intend to use MQTT to send real-time data to mobile devices, but I don't need reliable delivery. Keeping battery consumption as low as possible is really important, so it would be great to avoid sending unnecessary messages (TCP's acknowledgements).
Would it be possible to implement MQTT over UDP, or other unreliable transport layer protocol? Is there any existing library doing this?
MQTT-S can run over UDP, but there aren't any publicly available libraries for it yet. Are you sure the battery usage is that much of a problem? There is a run down of some experiments done on MQTT power usage on Android at http://stephendnicholas.com/archives/219
I think that MQTT itself really makes use of the TCP delivery and ordering guarantees. I believe it would be very difficult to guarantee it working over UDP.
In TCP, you can differentiate between servers and clients, because servers are those who bind and accept (TCP listeners) and clients just connect to those servers. Both can send and receive.
But, in UDP how do you differentiate between servers and clients? There is no special behavior to differentiate between servers or clients in UDP, right? It seems that we can only classify machines involved in a UDP connection as senders and receivers. A server could be either, or both. It can receive data from many clients or it can send data to many clients (e.g. multicast server).
Please correct me if I am wrong and point me to the correct forum if I posted the question in the wrong one.
Thanks.
There isn't a server or a client with UDP. There are just peers.
Think about UDP as a Sender -> Receiver communication instead of Client <=> Server.
Since UDP is a connectionless protocol a response from the Receiver may or may not happen. That (among other things) is why TCP is considered more reliable but slower than UDP.
http://en.wikipedia.org/wiki/Connectionless_protocol
http://www.diffen.com/difference/TCP_vs_UDP
Not an expert with networking, but this is my understanding.
TCP and UDP are network protocols i.e. dealing with how data is to be transferred between nodes. If you were to look at the packet structure for both TCP and UDP, you will find that both have a source node section and a destination node section. Moreover, a physical machine will still exist as the source of information even in UDP. Whether, you call it server or just a client is a decision which the architecture of the system shall decide.
So, I think you are referring to a level above transmission of data i.e. in my understanding an architecture of network application. That is when we talk about client server applications, and may be P2P kind of an architecture where there can be multiple physical machines providing data. So, the terminology depends on which context are you referring.
To answer your question, yes a server and client can exist in both TCP and UDP. Let the architects decide!
Hope it helps!
This might be a silly question:
Does HTTP ever use the User Datagram Protocol?
For example:
If one is streaming MP3 or video using HTTP, does it internally use UDP for transport?
From RFC 2616:
HTTP communication usually takes place
over TCP/IP connections. The
default port is TCP 80, but other
ports can be used. This does not
preclude HTTP from being implemented
on top of any other protocol on the
Internet, or on other networks. HTTP
only presumes a reliable transport;
any protocol that provides such
guarantees can be used; the mapping
of the HTTP/1.1 request and response
structures onto the transport data
units of the protocol in question is
outside the scope of this
specification.
So although it doesn't explicitly say so, UDP is not used because it is not a "reliable transport".
EDIT - more recently, the QUIC protocol (which is more strictly a pseudo-transport or a session layer protocol) does use UDP for carrying HTTP/2.0 traffic and much of Google's traffic already uses this protocol. It's currently progressing towards standardisation as HTTP/3.
Typically, no.
Streaming is seldom used over HTTP itself, and HTTP is seldom run over UDP. See, however, RTP.
For something as your example (in the comment), you're not showing a protocol for the resource. If that protocol were to be HTTP, then I wouldn't call the access "streaming"; even if it in some sense of the word is since it's sending a (possibly large) resource serially over a network. Typically, the resource will be saved to local disk before being played back, so the network transfer is not what's usually meant by "streaming".
As commenters have pointed out, though, it's certainly possible to really stream over HTTP, and that's done by some.
Maybe just a bit of trivia, but UPnP will use HTTP formatted messages over UDP for device discovery.
Yes, HTTP, as an application protocol, can be transferred over UDP transport protocol.
Here are some of the services that use UDP and an underlying protocol for transferring HTTP data and streaming it to the end-user:
XMPP's Jingle Raw UDP Transport Method
A number for services that use UDT --- UDP-based Data Transfer Protocol, which is the a superset of UDP protocol.
The Transport Layer Security (TLS) protocol encapsulating HTTP as well as the above mentioned XMPP and other application protocols does have an implementation that uses UDP in its transport layer; this implementation is called Datagram Transport Layer Security (DTLS).
Push notifications in GNUTella are HTTP requests sent over UDP transport.
This article contains further details on streaming over UDP and its reliable superset, the RUDP: Reliable UDP (RUDP): The Next Big Streaming Protocol?
Of course, it doesn't necessarily have to be transmitted over TCP. I implemented HTTP on top of UDP, for use in the Satellite TV Broadcasting industry.
If you are streaming an mp3 or video that may not necessarily be over HTTP, in fact I'd be suprised if it was. It would probably be another protocol over TCP but I see no reason why you cannot stream over UDP.
If you do you have to take into account that there is no certainty that your data will arrive at the other end, but I can take it that you know about UDP.
To answer you question, No, HTTP does NOT use UDP.
For what you talk about though, mp3/video streaming COULD happen over UDP and in my opinion should never happen over HTTP.
Maybe some change on this topic with QUIC
QUIC (Quick UDP Internet Connections, pronounced quick) is an experimental transport layer network protocol developed by Google and implemented in 2013. QUIC supports a set of multiplexed connections between two endpoints over User Datagram Protocol (UDP), and was designed to provide security protection equivalent to TLS/SSL, along with reduced connection and transport latency, and bandwidth estimation in each direction to avoid congestion. QUIC's main goal is to optimize connection-oriented web applications currently using TCP.
I think some of the answers are missing an important point. The choice between UDP and TCP should not be based on the type of data (e.g., audio or video) or whether the application starts to play it before the transfer is completed ("streaming"), but whether it is real time. Real time data is (by definition) delay-sensitive, so it is often best sent over RTP/UDP (Real Time Protocol over UDP).
Delay is not an issue with stored data from a file, even if it's audio and/or video, so it is probably best sent over TCP so any packet losses can be corrected. The sender can read ahead and keep the network pipe full and the receiver can also use lots of playout buffering so it won't be interrupted by the occasional TCP retransmission or momentary network slowdown. The limiting case is where the entire recording is transferred before playback begins. This eliminates any risk of a playback stall, but is often impractical.
The problem with TCP for real-time data isn't retransmissions so much as excessive buffering as TCP tries to use the pipe as efficiently as possible without regard to latency. UDP preserves application packet boundaries and has no internal storage, so it does not introduce any latency.
(This is an old question, but it deserves an updated answer.)
In all likelihood, HTTP/3 will be using the QUIC protocol, which is described as
multiplexed transport over UDP
So, from a certain point of view, you could say that HTTP/3 will be using UDP.
The answer: Yes
Reason: See the OSI model.
Explaination:
HTTP is an application layer protocol, which could be encapsulated with a protocol that uses UDP, providing arguably faster reliable communication than TCP. The server daemon and client would obviously need to support this new protocol. Quake 2 protocol proves that UDP can be used over TCP to provide a basis for a structured communication system insuring flow control (e.g. chunk ids).
http over udp is used by some torrent tracker implementations (and supporteb by all main clients)
In theory yes it is possible to use UDP for http but that might be problematic. Say for instance in your example a mp3 or a video is being streamed there will be problem of ordering and some bits might go missing as UDP is not connection oriented there is no retransmit mechanism.
HTTP/3 (aka QUIC) uses UDP instead of TCP.
https://http3-explained.haxx.se/en/the-protocol/feature-udp
UDP is the best protocol for streaming, because it doesn't make demands for missing packages like TCP. And if it doesn't make demands, the flow is far more faster and without any buffering.
Even the stream delay is lesser than TCP. That is because TCP (as a far more secure protocol) makes demands for missing packages, overwriting the existing ones.
So TCP is a protocol too advanced to be used for streaming.