I have a situation right now when I have after reading the byte stream from COM port in object of QByteArray type exactly and only 1 byte of data. BUT one very non-friendly protocol requires to have 9 bits of data after reading data from COM port.
But according to win32API function: ReadFile(....) I can read from the COM stream ONLY bytes= 1,2,3.....
So - That's why I am reading only 8 bits=1 byte with help of this function and with help of some operations with parity bit I am calculating the value of the 9th bit of generalized data...
So on one hand I have 1 byte (8 bits) of proper(real) data - on another hand I have a value of this 9th bit (0 or 1); 2 objects which in sum must create the value of generalized data.
How I can combine these objects into one & final QByteArray object? Because the global function ReadComData can and must return only QByteArray object.
UARTs cannot "write" 9-bit data. On the wire, your (typically 8-bit) data are usually framed between a start-bit and a stop-bit, so you have 10 bits transmitted for every byte you send. If you have a parity bit, it is transmitted after the last data bit, but before the stop bit. But this is generated by the sending UART, not part of a protocol. A data bus for a typical UART 16550 is only 8-bits wide (you can actually send 5-, 6-, 7-, or 8-bit data).
On the receiving end, the UART has to be configured based on what is on the wire. If your sender is using a parity bit, then you program the UART (via the "COM" port settings) accordingly. The parity bit is just to help check for errors on the wire. It is based on the data bits -- you cannot put another data bit in a parity bit. The receiving UART can be used to check for parity errors (read via the line status register (LSR)), and this can be passed up to you via system calls.
It is possible your protocol is splitting up the data across multiple bytes. If that's the case, then convert two bytes into one 16-bit word and mask the 6 bits you don't want to use.
Related
I am trying to send a mavlink command for instance
GPS_RTCM_DATA ( #233 )
flags uint8_t
len uint8_t
data uint8_t[180] RTCM message (may be fragmented)
https://mavlink.io/en/messages/common.html#GPS_RTCM_DATA
I understand uint8_ would be in a single byte unsigned int.
What does the [180] means?
The uint8_t[180] in the MAVLink GPS_RTCM_DATA message means that the data field can contain up to 180 bytes.
Beware that RTCM messages can be bigger than 180 bytes and be fragmented in
more than one GPS_RTCM_DATA message.
You can check the flags field as stated in the mavlink documentation:
LSB: 1 means message is fragmented, next 2 bits are the fragment ID,
the remaining 5 bits are used for the sequence ID. Messages are only
to be flushed to the GPS when the entire message has been
reconstructed on the autopilot. The fragment ID specifies which order
the fragments should be assembled into a buffer, while the sequence ID
is used to detect a mismatch between different buffers. The buffer is
considered fully reconstructed when either all 4 fragments are
present, or all the fragments before the first fragment with a non
full payload is received. This management is used to ensure that
normal GPS operation doesn't corrupt RTCM data, and to recover from a
unreliable transport delivery order.
I tried every but it doesn't work. Except putting it as a 180 byte arrays. The data might be only 30 bytes for example. But input with the other 150 0x00 bytes in this way, the python program accepts my command. Strangely so. I can't explain why but in this case it works.
I recently cam across an AES implementation in C for embedded systems and have been looking at using it in a wireless point-to-many-points link (or point-to-point for starters).
The code compiles correctly on my platform and it seems to give the correct results when using the test code provided with CTR mode, back to back (this CRT mode is what I require as my packets are of 120 bytes each and cannot be changed).
Essentially the transmitter Tx should use the tiny-AES encryption function to generate a continuous stream of encrypted packets of 120 bytes each, transmit them with a CTR attached and the header etc, and to be received and decrypted on the other end by the receiver Rx.
It is envisaged the transmitter Tx and receiver Rx use exactly the same tiny-AES code with the same key that is known to me only. This key can be set and would be different on different Tx-Rx pairs.
My question relates to the CRT mode in particular and the use of the following functions both on the Tx and the Rx :
/* Initialize context calling (iv can be NULL): */
void AES_ctx_init(AES_ctx *ctx, uint32_t keylen, const uint8_t *key, const uint8_t *iv);
/* ... or reset IV or key at some point: */
void AES_ctx_set_key(AES_ctx *ctx, uint32_t keylen, const uint8_t *key);
void AES_ctx_set_iv(AES_ctx *ctx, const uint8_t *iv);
So
What is the purpose of AES_ctx_init() and is this done only once at the start of an encrypt/decrypt session of a secure link ?
What is the purpose of the AES_ctx_set_iv() function and again, is this done only once at the start of the secured link session ? The comment "... or reset IV or key at some point" is not clear to me – the reason for my question ... That is when does the IV need to be reset when using AES in CRT mode ?
Do the Tx and the Rx require to use the very same IV and can this be left as is or does it need to be changed with every new secured link session ? I understand that the secret key would normally be chosen for a Tx/Rx pair and only modified by the user on both when required but normally it would stay the same for every session. Is this generally correct ? Is this the case (or not) for the IV, and at what point does one reset the IV ?
In wireless links, due to noise, and if there is no implementation of a FEC, the receiver will be forced to drop packets due to bad CRCs. In my link the CRC is verified for the received 120 bytes packet (one by one as they are received) and the decryption is initiated to recover the original data if the CRC is correct, but the data is dropped if not. What are the implications of this to the stream of encrypted Tx packets as they keep getting transmitted regardless, as there is no hand-shaking protocol to tell the Tx the Rx dropped a packet due to a CRC error, if any ?
If answers to these questions exist in the form of some further documentation on the tiny-AES I would appreciate some links to it as the notes provided assume people are already familiar with AES etc.
So here's some further to the comments/replies already made. Basically the Tx / Rx pair has a pre-defined packet structure with a certain payload, header CRC etc but let’s call this the “packet”. The payload is what I need to encrypt and this is fixed to 120 bytes, not negotiable regardless and it’s in the packet.
So what you are saying is that with each transmission of a packet the Tx and Rx will need to change the nonce with every packet, and both Tx & Rx need to be using the same nonce with each treatment of a packet ?
Say I begin transmission, and have packet(1), packet(2) … packet(n) etc. then with each packet transmitted I need to update the “counter”, and that both transmitter and receiver need to be synchronized so that both use the same nonce in a session ?
This can be problematic when and if, due to noise or interference, the Tx/Rx system loses synchronization as it can happen, and somehow the two independent counters for the nonce are no longer synced and on the same page so-to-speak…
Generally a session would not require more than 2^16 packets on average, so can you see a way around this ? As I said, the option of sending a different nonce with every individual packet is totally out of the question, as the payload is already complete and full.
Another way I thought perhaps you can shed some light if this may work, is through GPS. Say if each Tx and Rx have each a GPS module (which is the case), then timing information could be derived from the GPS clock on both Tx & Rx as both would receive this independently, and some kind of synchronized counter could be rolling to update the nonce on both, say counting from 0 to 2^16 in a session… would you agree ? So even if due to noise/interference, packets are lost by the receiver, the counter would continue updating the nonce with some form of reliable “tick” in the background…
With regards to the source of entropy, well apparently the lampert circuit would provide this for a good PRNG running locally, say for a session of 2^16 packets, but this is not available on my system yet, but could be if we decide to take this further …
What are your thoughts on this as a professional in the field ?
Regards,
I'm assuming the Tiny AES you're referring to is tiny-AES-c (though your code is a little different than that code; I can't find anywhere that AES_ctx_set_key is defined). I assume when you say "CRT mode" you mean CTR mode.
What is the purpose of AES_ctx_init() and is this done only once at the start of an encrypt/decrypt session of a secure link ?
This initializes the internal data structures. Specifically it performs key expansion which creates the necessary round keys from the master key. In this library, it looks like it would be called once at the beginning of the session, to set the key and IV.
In CTR mode, it is absolutely critical that the Key+Nonce (IV) combination is never reused on two different messages. (CTR mode doesn't technically have an "IV." It has a "nonce," but it looks exactly like an IV and is passed identically in every crypto library I've ever seen.) If you need to reuse a nonce, you must change the key. This generally means that you will have to keep a persistent (i.e. across system resets) running counter on your encrypting device to keep track of the last-used nonce. There are ways to use a semi-random nonce safely, but you need a high-quality source of entropy, which is often not available on embedded devices.
If you pass NULL as the IV/nonce and reuse the key, CTR mode is a nearly worthless encryption scheme.
Since you need to be able to drop packets cleanly, what you'll need to do is send the nonce along with each packet. The nonce is not a secret; it just has to be unique for a given key. The nonce is 16 bytes long. If this is too much data, a common approach is to exchange the first 12 bytes of the nonce at the beginning of the session, and then use the bottom 4 bytes as a counter starting a 0. If sessions can be longer than 2^32 blocks, then you would need to reset the session after you run out of values.
Keep in mind that "blocks" here are 16 bytes long, and each block requires its own nonce. To fit in 120 bytes, you would probably do something like send the 4 byte starting counter, and then append 7 blocks for n, n+1, n+2, ... n+6.
Designing this well is a little complicated, and it's easy to do it wrong and undermine your cryptography. If this is important, you should probably engage someone with experience to design this for you. I do this kind of work, but I'm sure there are many people you could find.
So what you are saying is that with each transmission of a packet the Tx and Rx will need to change the nonce with every packet, and both Tx & Rx need to be using the same nonce with each treatment of a packet ?
Correct. The Tx side will determine the nonce, and the Rx side will consume it. Though they could coordinate somehow other than sending it explicitly, as long as it never repeats, and they always agree. This is one of the benefits of CTR mode; it relies on a "counter" that doesn't necessarily have to be sent.
The clock is nice because it's non-repeating. The tricky part is staying synchronized, while being absolutely certain you don't reuse a nonce. It's easier to stay synchronized if the time window for each nonce is large. It's easier to make sure you never reuse a nonce if the time window is small. Combining a clock-derived nonce with a sequential counter might be a good approach. There will still be ongoing synchronization issues on the receiving side where the receiver needs to try a few different counters (or nonce+counters) to find the right one. But it should be possible.
Note that synchronization assumes that there is a way to distinguish valid plaintext from invalid plaintext. If there is a structure to the plaintext, then you can try one nonce and if the output is gibberish, try another. But if there is no structure to the plaintext, then you have a big problem, because you won't be able to synchronize. By "structure" I mean "can you tell the difference between valid data and random noise.
Using the clock, as long as your window is small enough that you would never reuse a starting nonce, even across resets, you could avoid needing a PRNG. The nonce doesn't have to be random; it just can never repeat.
Note that if you have a 4-byte counter portion, then you have 2^32 blocks, not 2^16.
I am working on a project where I have to receive a float value and an integer value. The float has to be saved and the integer value has to be used for another purpose.
Usually an arduino does not receive any integers or floats at all. The usual way an arudino receives something is through the serial interface.
Data is sent as a sequence of bytes.
The meaning of that data is determined by the sender. You can receive monkeys and elephants with an arduino (virtual ones of course) as long as you know the protocol.
Read https://en.wikipedia.org/wiki/Universal_asynchronous_receiver-transmitter
and get some basic knowledge in data types and how to communicate them through a serial interface.
You should define a protocol for your application or simply send the data as strings which you parse on-board.
Let's say we have two machines on a network MA and MB,
MA considers little endian the order of the bits in a byte,
on the contrary MB considers big endian the order of the bits in a byte.
How do MA and MB agree on what "endianess" to use for the bits in a byte during the
communication over the network ?
Is there a standard "network endianess" or what ?
Do socket programmers have to take any actions in ensuring a correct communication ?
For example HTTP is a text protocol, that means that machines send and receive bytes which represent characters,what if the encoding of the characters is different in the endianess of the bits ?
Yes, the hardware protocols specify the bit order of bytes on all network links. This is generally handled automatically by the NIC hardware.
See, for example, this description of Ethernet frame format.
Ethernet transmission is strange, in that the byte order is big-endian (leftmost byte is sent first), but bit order little-endian (rigthmost, or LSB (Least Significant Bit) of the byte is sent first).
Check this page: http://www.comptechdoc.org/independent/networking/protocol/protlayers.html
It suggests that byte ordering is done at the Presentation Layer which is quite high up. This however will relate specifically to the application that you are using. I suspect data at lower levels (wrapping the higher levels) has a predetermined byte and bit order.
I know in a lot of asynchronous communication, the packet begins starts with a start bit.
But a start bit is just a 1 or 0. How do you differentiate a start bit from the end bit from the last packet?
Ex.
If I choose my start bit to be 0 and my end bit to be 1.
and I receive 0 (data stream A) 1 0 (data stream B) 1,
what's there to stop me from assuming there is a data stream C which contains the same contents of "(data stream A) 1 0 (data stream B)" ?
Isn't it more convenient to have a start BYTE and then check the data stream for that combination of bits? That will reduce the possibility of a confusing between the start/end bit.
Great question! Most asynchronous communication also specifies a stop bit, which is the complement of the start bit, ensuring each new symbol begins with a stop-to-start transition.
Example: let's transmit the characters ABC, which are ASCII 65, 66, and 67:
A = 65 = 0x41 = 0100 0001
B = 66 = 0x42 = 0100 0010
C = 67 = 0x43 = 0100 0011
Let's also assume (arbitrarily) that the start bit is 0 and the stop bit is 1, and the data is transmitted from MSB to LSB. The transmitter will be in the stop (1) state when no data is transmitted. So the receiver might see this:
Data: ....1111 0010000011 111 0010000101 0010000111 11111....
(quiet) ^ A $ ^ B $ ^ C $ (quiet)
With apologies for the ASCII graphics, the data consists of a series of stop (1) bits while the channel is idle. When the transmitter is ready to send a character, it sends a start (0) bit (marked with ^), followed by the character code, and ending with a stop (1) bit (marked with $). It continues to send stop bits until the next character is transmitted, beginning with another start bit.
The reason we use start bits instead of bytes is efficiency. The scheme above requires 10 bits (1start + 8data + 1stop) to transmit 8 bits of data, resulting in an overhead of (10 - 8) / 8 = 1/4 = 25%. If we used start and stop bytes, we'd need to transmit 3 bytes for each byte of data, which would be an overhead of (3 - 1)/1 = 2 = 200%. If the start, data, and stop bytes were each 8 bits, we'd have to transmit 24 bits instead of 10 for each character, so it would take almost 2 1/2 times as long to send the data!
One can always define a start byte as an indication that a message is beginning (and the ASCII SOH, STX, and ETX codes were intended for such purposes). However, the standard hardware and protocols for connection to data-transmission equipment (RS232C and later) operate at a lower level, and it is generally neither possible nor desirable to alter that arrangement (especially via software).
High performance synchronous data transmission schemes, such as those used on local-area networks and wide-area transmission systems do use elaborate frame markers. The frame marker is a distinct pattern of bits that never occurs in the stream for message data. There is typically a special rewriting rule that essentially "escapes" any in-data occurrence of a similar bit pattern so that transmission equipment will not see it as a frame marker. These escaped patterns are reconstructed by the recipient so the sender and receiver never have to pay attention to this. These arrangements make specialized hardware even more important, such as in the typical Network Interface Card (these days, motherboard chip) on personal computers.
BACKGROUND ON ASYNCHRONOUS SERIAL COMMUNICATION
It is useful to think of asynchronous serial transmissions as asynchronous between character/data frames and synchronous within the span of the character frame (including the start bits and initial stop/fill).
With this scheme, there is a constant fill signal between the frames and it is usually at least one data-bit wide, although some arrangements require a 1.5-bit or two-bit stop/fill. The stop "bit" uses the same signal level and can be considered the minimum fill period before another start bit will arrive.
When a frame is arriving, it is necessary to synchronize with the predetermined number of bits it is expected to carry. The transition from the fill to an opposite level signal is accomplished by the start bit which is always opposite to the stop/fill level. The sampling of the bits can be timed to happen in the middle of subsequent bit-arrival periods.
Technically, if frames were being sent at the maximum rate, it would not be necessary to send any stop/fill, proceeding to the start bit of the next frame immediately. However, counting on at least one bit worth of fill before the start-bit transition helps to keep the sender and receiver synchronized.
If you think of the asynchronous streams as being encoded from key depressions using a keyboard, you can see the importance of allowing arbitrary fill between character frames. Once it is known what frame to send next, it can be inserted immediately, with its start bit, at the agreed bit rate, after there has been at least one bit worth of preceding stop/fill.
It is also useful to notice that, in typical low-speed asynchronous transmissions, there are only two kinds of bits/levels, so the only way the presence of data as opposed to fill can be distinguished is by a marker scheme like this where the start of the frame is uniquelly detectable and the end of frame is predetermined (unless there is a more-sophisticated variable-length frame structure generally not used in asynchronous serial communication). It is actually rather difficult for a receiver to discover the bit rate of a transmitter without some additional agreement, such as looking for a recognizable data sequence from which one can estimate the bit rate which would have it arrive correctly when it arrives in incorrect form.
Even though high-speed modems now transmit complex analog signals that aren't described in terms of two simple signal levels, the RS232C (and later-mode) digital communication between a computer UART and the data coupling on the modem is pretty much as described.
High-speed modems also have additional capabilities for synchronizing with a distant end-point, as you can tell by listening to the signal audio while a connection starts up. In addition, There are separate signal lines in the serial cable to the computer that are used for pacing between the computer and the modem so that the sending party does not transmit new data frames faster than the receiving party (either computer or modem) can accept them. But a frame, once started, is always started at the agreed synchronous speed.
Wikipedia has a good description of asynchronous serial communication, what computer serial ports use.
There is a common over-simplification that suggests the stop bit determines the length of the data. That's not the case. The stop bit looks just like a level for another data bit. The way the stop bit and the period until the next start bit, are recognized is by knowing the bit rate at which in-frame data and start/stop bits are being transmitted and knowing how many bits a frame contains. Otherwise, there is no way to distinguish a stop bit from just another bit of that polarity as part of the data frame.
Here is the way start and stop bits usually work:
A start bit is sent, say 1. This indicates to the receiver that a specified number of bits of data will be transmitted, say 8.
8 bits of data are sent.
A stop bit is sent, say 0. This indicates that the 8 bits of data have been sent.
If more data is to be sent, each byte must be initiated with a start bit and terminated with a stop bit. The transmitter and receiver must agree on how many bits of data are sent for each start bit so the receiver will be able to distinguish the stop bit from the data. Sometimes the start bit is actually multiple bits or even a byte, but the idea is the same. The receiver recognizes the end of the data frame when it sees the stop bit after receiving the pre-specified number of data bits. Sometimes a parity bit is sent before the stop bit to provide a simple error-detecting mechanism.
It is all protocol dependent. You can say that after start symbol you will expect N symbols or you will read until you encounter the stop symbol.
Where symbol colud be any n-bit sequence (including bit and byte.)
Indeed, your example with bits exactly apply to a protocol which uses bytes instead of bits.
Say you send 00000000 stream A 11111111 00000000 stream B 11111111. In this case you may still confuse it with stream C = stream A 11111111 00000000 stream B.
Usually a start bit is used because a voltage level change can trigger an event (See edge triggering in flip flops.) On the other hand a start symbol with multiple bits will be used to synchronize clocks of two systems in addition to triggering an event. An example of it would be a PAL signal.
The start and stop bits come from the days of the teletypes. They essentially were pulses that took up time to sort of let the mechanical hardware set. Dos text file lines are ended with CR LF which literally caused the carriage to return to column 1, and the platen to advance one line. I think it is in the order because it takes longer for the CR to occur, and the LF can effectively happen in parallel.
Detecting it is a little bit harder. You sort of have to watch the bit stream go by.
Over time you ought to be able to detect it, as the data is normally ASCII with the start/stop bits around it. Normally this isn't an issue, because it is handled by the UART which runs the COM port.