I am searching for a protocol on top of TCP/IP that behaves like BLE GATT.
The server should announce which characteristics/properties it offers and the client can read/write/subscribe to them.
Also, it should be very lightweight, since it needs to run on two MCUs connected directly to each other.
Is there anything in that direction?
Related
I am puzzled...
My project is a PC connected to multiple micro-controller boards in an isolated network. So far the protocol has been UDP which is easy to deal with, has no particular client/server but has its obvious shortcomings of lost messages when things get busy.
The micro-controllers have fixed IP-addresses (set by dip switches), the PC SW has a list of them, sends at present UDP messages to each of them and they reply to the address they came from (i.e. the PC) with status and/or data.
My question is now that I switch to TCP instead of UDP, should the PC be the listening server with many clients (could be anything from 1 - 50), or should the micro controllers be listening servers the PC can connect to as client? Note: controllers have fixed/known addresses - the PC does not.
An additional concern is re-connection. The micro-controllers are external and may lose connection, reset or otherwise need to connect again.
Thanks....
should the PC be the listening server with many clients (could be anything from 1 - 50), or should the micro controllers be listening servers the PC can connect to as client?
That is a basic design question that we cannot answer for you. Likely, it's more practical for arbitrary devices to connect to a central server but that's not a given.
controllers have fixed/known addresses - the PC does not.
That might turn the previous question around.
The micro-controllers are external and may lose connection, reset or otherwise need to connect again.
That's something you need to put into your design - have TCP connections time out and reconnect. Usually, a finite-state machine is useful here. You should also consider whether you use a one-shot connect-transmit-disconnect similar to UDP (easier to implement) or a longer TCP session with multiple data transmissions (more efficient).
If so, do you know examples of what can go wrong in a non-TCP network?
Learning about MQTT I came across several mentions of the fact that MQTT relies on TCP/IP stack. For example, from mqtt.org:
MQTT for Sensor Networks is aimed at embedded devices on non-TCP/IP
networks, whereas MQTT itself explicitly expects a TCP/IP stack.
But if you read the reference documents, you won't finding anything like that. Moreover, there's QoS field that can be used for reliable delivery and whose values other than 0 are essentially useless in TCP/IP networks. Right now I see nothing that would prevent me from establishing an MQTT connection using UNIX pipe, domain or UDP socket rather than TCP socket.
MQTT just needs a delivery that is ordered and reliable, it doesn't have to be TCP. SCTP works just fine, for example, but UDP doesn't because there is no way to guarantee your large PUBLISH packet made up of multiple UDP packets will arrive in order and complete.
With regards TCP reliability, in theory what you are saying is true, but in practice when an application calls write() and receives a successful return, it can't guarantee when the data has actually made it out of the computer to the remote host. All write() (or send()) does is copy the data to the kernel buffers, at which point you have no further control.
The only way to be sure that a message reaches the remote host at the application level is to have the remote host reply.
MQTT-SN (for sensor Network ) is the solution for the problem that MQTT was having while running over TCP/IP .
There is a concept of MQTT gateway which is brought in in for MQTT-SN which helps in bringing non-TCP / IP implementation.
http://emqttd-docs.readthedocs.io/en/latest/mqtt-sn.html
I know, how to write a C# application that works through a local network.
I mean I know, how to make my client-side application access my server-side application in a single local network.
But I wonder: How do such apps, as Skype, TeamViewer, and many other connect via global network?
I apologise, if this question is simple or obvious, but I couldn't find any information about this stuff.
Please, help me, I'll be very grateful. Any information is accepted - articles, plain info, books,and so on...
Question is very wide and I try to do short overview.
Following major difference between LAN (Local Area Network) and WAN (Wide Area Network):
Network quality:
LAN is more or less stable, WAN can be with network issues like:
Packet loss (you need use loss-tolerant transport like TCP or UDP with retransmits or packet loss concealment)
Packet jitter (interpacket intervals may differ a lot from sending part). Most common thing is packets bursts.
Packet reordering
Packet duplication
Network connectivity
WAN is less stable than LAN. So you need properly handle all things like:
Connection stale
Connection loss
Errors in the middle of the connection (if you use UDP for example)
Addresses:
In WAN you deal with different network equipment between client and server (or peers in case of peer-to-peer communication). You need to take in account:
NATs - most of the clients are behind NAT and you need to pass them through. According technics are called "NAT traversal"
Firewalls - may ISP has own rules what client can do or can't. So if you do something specific like custom transport protocol you may bump into ISP firewalls.
Routing - especially multicast and broadcast communication. In common case multicast is not possible to route. Broadcasts are never routed. So you need to avail this type of communication if you want to use WAN.
May be I forgot something. But these points are major. You can read many articles about any of them.
I am building a simple peer to peer application where about 8 participants all connect to each other (n*n). I will be using UDP with a reliability and ordering protocol layered on top. Each peer will be broadcasting a few KB of data per second.
It occurred to me that there are two ways of configuring the ports on each peer:
Each peer takes one port and all messages are received via that
Each peer takes a port for each other peer, and only communicates with a peer using its corresponding port
What are the advantages and disadvantages of each approach?
Creating only one port to communicate with each peers is the best option. You create only one socket and use only one port to send/receive data with as many peers as you want. You can distinguish received data by seeing source address of each packet. This way has Less complicated code and is more resource efficient.
Creating multiple port has absolutely no advantage. It will complicate your code and will use much more resource with no benefit. Resource consumption will grow with more peers.
I have never done anything with this model of application, but here are my few thoughts.
1 big thing to consider will be whether any firewalls are in place between the peers. If so, a hole will need to be punched through for each listening port. This can be done either once (eg. router port-forward rule) or dynamically (UPnP, etc), but you may not be able to count on automatic full-cone NAT to do this for you. If you are expecting any firewalls between your peers, I would recommend using a single port for ease of programming on your part and identify your peers strictly by their remote address or some other in-protocol identifier.
Using a single port per user will make identifying communication much simpler, but if you expect your number of participants to grow by n(n-1)/2. If you never expect more than a small number (eg. 20), port per peer will work decently well without much effort.
Another option for you (possibly) may be using multicast. If all of your peers are on the same broadcast domain, this will reduce bus contention and may make your coding somewhat cleaner as well.
Hope this helps. I apologize if this isn't what you're looking for. Good luck!
Each peer takes one port and all messages are received via that
If each peer can get the source IP/source port of the incoming datagram (and I bet it can), this is enough to differentiate the peers.
Each peer takes a port for each other peer, and only communicates with
a peer using its corresponding port
See above, and most importantly this contradict your base idea of broadcasting in the first place. It just add a level of complexity (and is probably not very scalable, even if for now you envision just 8 peers).
In your base requirement I think you may have a dilemma between:
broadcast everything to everyone,
but still you want a peer to be able to "only communicates with a peer", which is inherently unicast.
This raises some problems, as you already realized by asking the question.
I see 2 other problems:
Scalability-wise, the broadcast everything approach whereas you sometime actually need unicast is going to put some useless load on the network. This is not pretty.
The broadcast approach dictates UDP, but still you want reliable data transfer, so as you stated you'll have to add a "reliability and ordering protocol layered on top". This (not so easy) work would not be needed if only we could use TCP.
There is a third approach:
use broadcast UDP for each peer to announce itself on the network, so that other peers can...
...discover it and then establish a unicast TCP connection with this peer. No more reliability and ordering problems + reduced network load.
This approach is used in SSDP (Simple Service Discovery Protocol), part of UPnP. I do not suggest you use SSDP, it's probably bloated for what you want to do, you said you wanted something simple.
All in all, you first have to resolve your dilemma: decide and differentiate the data that really need to be broadcasted vs the unicast part. YMMV.
PS: with broadcast UDP also comes the problem that though OK on a LAN, this will not pass a router unless you use multicast routing. But that's another story.
From a Joel's post on Copilot:
Direct Connect! We’ve always done
everything we can to make sure that
Fog Creek Copilot can connect in any
networking situation, no matter what
firewalls or NATs are in place. To
make this happen, both parties make
outbound connections to our server,
which relays traffic on their behalf.
Well, in many cases, this isn’t
necessary. So version 2.0 does
something rather clever: it sets up
the initial connection through our
servers, so you get connected right
away with 100% reliability. But then
once you’re all connected, it quietly,
in the background, looks for a way to
make a direct connection. If it can’t,
no big deal: you just keep relaying
through our server. If you can make a
direct peer-to-peer connection, it
silently shifts your data onto the
direct connection. You won’t notice
anything except, probably, much faster
communication.
How do they change the server connection to a P2P connection?
It's pretty tricky and interesting. I'm sure I have some details wrong, but the overview is this:
The programs can already talk to each other through Joel's server, so they can exchange information with each other and Joel's server. Further, Joel has their external IP addresses, and they give joel information about their internal IP addresses.
They decide to try this hole punch technique. Computer A initiates a TCP connection with Computer B using B's external IP address. It won't go through, but what it does is tell's A's router that it needs to allow incoming packets from B on a given port.
Computer B does the same thing, but its message gets through to A since A's router opened a port/ip combination that matches what B sent (there's some port magic that happens here - this is non trivial, but doable).
B's router remembers that B initiated a connection with A on a given port and IP, and so A's packets now flow into B past their router correctly as well.
So it's actually pretty straight forward, but the implementation has details, especially regarding how ports are given to new TCP connections, and how NAT routers typically deal with TCP requests and how they map to external ports. These details are the interesting, and difficult, bit.
-Adam
I believe the simple version is that they drop the server connection and replace it with the P2P connection.
Something along the lines of:
Machine1 connects to copilot's servers.
Machine1 connects to copilot's servers.
Machine1 connects to copilot's servers.
Machine2 subsequently connects, and they begin screen sharing.
Machine2 opens a port intended for Machine1 to connect to.
Machine1 tries to connect to the now open port on Machine2.
If this connection is established:
The connection to copilot's servers is severed.
Data is instead transfered over the direct (P2P) connection between the two machines.
There is a technique called "Hole Punching" that works well with "Cone" NAT (Cone is a technical familly of router). That's not an 100% sure technique, today, it works well with UDP on about 80% of the router.
There is some implementations of library to realize Hole Punching: STUN (wikipedia)