I need (to design?) a protocol for communication between a microprocessor-driven data logger, and a PC (or similar) via serial connection. There will be no control lines, the only way the device/PC can know if they're connected is by the data they're receiving. Connection might be broken and re-established at any time. The serial connection is full-duplex. (8n1)
The problem is what sort of packets to use, handshaking codes, or similar. The microprocessor is extremely limited in capability, so the protocol needs to be as simple as possible. But the data logger will have a number of features such as scheduling logging, downloading logs, setting sample rates, and so on, which may be active simultaneously.
My bloated version would go like this: For both the data logger and PC, a fixed packet size of 16 bytes with a simple 1 byte check sum, perhaps a 0x00 byte at the beginning/end to simplify recognition of packets, and one byte denoting the kind of data in the packet (command / settings / log data / live feed values etc). To synchronize, a unique "hello/reset" packet (of all zero's for example) could be sent by the PC, which when detected by the device is then returned to confirm synchronization.
I'd appreciate any comments on this approach, and welcome any other suggestions as well as general observations.
Observations: I think I will have to roll my own, since I need it to be as lightweight as possible. I'll be taking bits and pieces from protocols suggested in answers, as well as some others I've found... Slip,
PPP and HLDC.
You can use Google's Protocol Buffers as a data exchange format (also check out the C bindings project if you're using C). It's a very efficient format, well suited to such tasks.
Microcontroller Interconnect Network (MIN) is designed for just this purpose: tiny 8-bit microcontrollers talking to something else.
The code is MIT licensed and there's embedded C and also Python implementations:
https://github.com/min-protocol/min
I wouldn't try to invent something from scratch, perhaps you could reuse something from the past like ZMODEM or one of its cousins? Most of the problems you mention have been solved, and there are probably a number of other cases you haven't even though of yet.
Details on zmodem:
http://www.techfest.com/hardware/modem/zmodem.htm
And the c source code is in the public domain.
Related
I recently came across one question related to the I2C protocol,
usually we read the data from the I2C slave devices and use it for the further calculations on the master side,
but can I be sure if the data I got is the data I wanted or is that corrupted while transmitting to the bus?
is there any possibilities to do so in I2C protocol?
In many cases, no, you can't be certain from software. You have to verify the design by hardware (with an oscilloscope) and then make sure nothing changes.
However, some slave devices provide checksums or check bits on certain transactions that can be used to detect some faults.
Reading a known value from the ID register at startup is usually good enough for most applications.
Remember that if your data is getting corrupted then maybe your addresses are too, and that will cause ACKs to not be returned when you expect them.
Many things you can do from software will not detect an error instantly though, so you need to think about how important or harmful it could be if an incorrect value is used. There are various standards that help you plan for this, such as ISO 14971 "Application of risk management to medical devices".
I am working on a project which requires sensor information to be obtained from multiple embedded devices so that it may be used by a master machine. The master currently has classes which contain backing fields for each sensor. Data is continuously read on each sensor and a packet is then written and sent to the master to update that sensor's backing field. I have little experience with TCP/UDP so I am not sure which protocol would work better with this setup.
I am currently using TCP to transfer the data because I am worried about data on our rotary encoders being received out of order. Since my experience with this topic is limited, I am not sure if this is this a valid concern.
Does anyone with experience in this area know any reasons that I should prefer one approach over the other?
How much you care about getting know a packet was delivered?
How much you care about getting know a delivered packet was 100% correct?
How much you care about the order of packet delivery?
How much you care about the peer is currently connected?
If the answers were "I care a lot", you'd prefer to keep on using TCP because it ensure all four points.
The counterpart is that UDP could be more lightweight and fast to handle if you manage small packets.
Anyway, it's not so easy choose this or that. Just try.
And read this brief explanation: http://www.cyberciti.biz/faq/key-differences-between-tcp-and-udp-protocols/
I'm no expert but it seems this might be relevant:
Do you can about losing data?
If so, use TCP. Error recovery is automatic.
If not, use UDP. Lost packets are not re-sent. I also believe ordering here is not guaranteed.
I would like to be able to communicated with PLC controllers, so that I can send and receive custom commands on the PLC.
My idea of being able to do this was to have a TCP listener on the PLC that could read TCP incoming packets on a specific port, and execute routines based on the commands in the packets. It could also send information back via TCP/IP.
This would allow me to write software in multiple languages such as C#, PHP, JavaScript, etc. so that software can be used on any platform such as Windows, iOS, Android, etc. to issue commands to the PLC. This would also mean you do not need the PLC software (which can be costly) to view or control the PLC.
I am not a PLC programmer, so I do not know if PLC has the capability of sending and receive custom TCP packets. I would like to know that a) if it is possible b) how feasible it would be to do this and c) what exactly I should research so that I can accomplish this.
Thanks.
It sounds a bit like reinventing the wheel. You want to make something like KepServerEX?
http://www.kepware.com/kepserverex/
There are also two things to consider - one is the ability to interface with the PLC to share data (ie: for a custom HMI) and the other is to program the PLC. For the latter you still need the control software from the manufacturer unless you're willing to reverse engineer and re-write it from the ground up.
Keep in mind, also, that PLCs don't work the same way that other software does. There are no functions or procedures or classes or objects or even really any "commands", per se. A PLC is a system which executes a continuous fixed program of mostly raw logic rules and calculations. A typical interface to an HMI involves reading and writing directly to/from logic bits and word data (ie:hardware memory locations) which represent the current state of the machine. OPC already does this just fine so I'm not quite sure what you're going for.
If you're looking for a cheap/free alternative to a full commercial package, something here may work for you :
http://www.opcconnect.com/freesrv.php
If I understand correctly, when referred to "Run/Stop" you mean for the PLC to 'Start' or 'Stop' scanning the code and updating its I/O. If this is the situation, it would be perfectly suitable to add a Scan_If_On bit (which will be written by a TCP Command) in parallel connection with the "Start" bit controlled by the HMI.
This way, there will be 2 forms of "Starting" the process controlled by the PLC. HMI and TCP.
Are there any libraries which put a reliability layer on top of UDP broadcast?
I need to broadcast large amounts of data to a large number of machines as quickly as possible, and generally it seems like such a problem must have already been solved many times over, but I wasn't able to find anything except for the Spread toolkit, which has a somewhat viral license (you have to mention it in all materials advertising the end product, which I'm not sure our customer will be willing to do).
I was already going to write such a thing myself (because it would be extremely fun to do!) but decided to ask first.
I looked also at UDT (http://udt.sourceforge.net) but it does not seem to provide a broadcast operation.
PS I'm looking at something as lightweight as a library - no infrastructure changes.
How about UDP multicast? Have a look at the PGM protocol for which there are several commercial and open source implementations.
Disclaimer: I'm the author of OpenPGM, an open source implementation of said protocol.
Though some research has been done on reliable UDP multicasting, I haven't yet used anything like that. You should take into consideration that this might not be as trivial as it first sounds.
If you don't have a list of nodes in the target network you have no idea when and to whom to resend, even if active nodes receiving your messages can acknowledge them. Sending to a large number of nodes, expecting acks from all of them might also cause congestion problems in the network.
I'd suggest to rethink the network architecture of your application, e.g. using some kind of centralized solution, where you submit updates to a server, and it sends this message to all connected clients. Or, if the original sender node's address is known a priori, then just let clients connect to it, and let the sender push updates via these connections.
Have a look around the IETF site for RFCs on Reliable Multicast. There is an entire working group on this. Several protocols have been developed for different purposes. Also have a look around Oracle/Sun for the Java Reliable Multicast Service project (JRMS). It was a research project of Sun, never supported, but it did contain Java bindings for the TRAM and LRMS protocols.
This is just a general question relating to some high-performance computing I've been wondering about. A certain low-latency messaging vendor speaks in its supporting documentation about using raw sockets to transfer the data directly from the network device to the user application and in so doing it speaks about reducing the messaging latency even further than it does anyway (in other admittedly carefully thought-out design decisions).
My question is therefore to those that grok the networking stacks on Unix or Unix-like systems. How much difference are they likely to be able to realise using this method? Feel free to answer in terms of memory copies, numbers of whales rescued or areas the size of Wales ;)
Their messaging is UDP-based, as I understand it, so there's no problem with establishing TCP connections etc. Any other points of interest on this topic would be gratefully thought about!
Best wishes,
Mike
There are some pictures http://vger.kernel.org/~davem/tcp_output.html
Googled with tcp_transmit_skb() which is a key part of tcp datapath. There are some more interesting thing on his site http://vger.kernel.org/~davem/
In user - tcp transmit part of datapath there is 1 copy from user to skb with skb_copy_to_page (when sending by tcp_sendmsg()) and 0 copy with do_tcp_sendpages (called by tcp_sendpage()). Copy is needed to keep a backup of data for case of undelivered segment. skb buffers in kernel can be cloned, but their data will stay in first (original) skb. Sendpage can take a page from other kernel part and keep it for backup (i think there is smth like COW)
Call paths (manually from lxr). Sending tcp_push_one/__tcp_push_pending_frames
tcp_sendmsg() <- sock_sendmsg <- sock_readv_writev <- sock_writev <- do_readv_writev
tcp_sendpage() <- file_send_actor <- do_sendfile
Receive tcp_recv_skb()
tcp_recvmsg() <- sock_recvmsg <- sock_readv_writev <- sock_readv <- do_readv_writev
tcp_read_sock() <- ... spliceread for new kernels.. smth sendfile for older
In receive there can be 1 copy from kernel to user skb_copy_datagram_iovec (called from tcp_recvmsg). And for tcp_read_sock() there can be copy. It will call sk_read_actor callback function. If it correspond to file or memory, it may (or may not) copy data from DMA zone. If it is a other network, it has an skb of received packet and can reuse its data inplace.
For udp - receive = 1 copy -- skb_copy_datagram_iovec called from udp_recvmsg. transmit = 1 copy -- udp_sendmsg -> ip_append_data -> getfrag (seems to be ip_generic_getfrag with 1 copy from user, but may be a smth sendpage/splicelike without page copiing.)
Generically speaking there is must be at least 1 copy when sending from/receiving to userspace and 0 copy when using zero-copy (surprise!) with kernel-space source/target buffers for data. All headers are added without moving a packet, DMA-enabled (all modern) network card will take data from any place in DMA-enabled address space. For ancient cards PIO is needed, so there will be one more copy, from kernel space to PCI/ISA/smthelse I/O registers/memory.
UPD: In path from NIC (but this is nic-dependent, i checked 8139too) to tcp stack there is one more copy: from rx_ring to skb and the same for receive: from skb to tx buffer +1copy. You must to fill in ip and tcp header, but does skb contain them or place for them?
To reduce latency in High-performance, you should decline to use a kernel driver. Smallest latency will be achieved with user-space drivers (MX does it, Infinband may be too).
There is a rather good (but slightly outdated) overview of linux networking internals "A Map of the Networking Code in Linux Kernel 2.4.20". There are some schemes of TCP/UDP datapath.
Using raw sockets will make path of tcp packets a bit shorter (thanks for an idea). TCP code in kernel will not add its latency. But user must handle all tcp protocol itself. There is a some chance of optimizing it for some specific situations. Code for clusters don't require handling of long distance links or slow links as for default TCP/UDP stack.
I'm very interested in this theme too.