I understand from this discussion that boost::asio::async_write writes data to the kernel buffers only. It does not mean that the peer has received the data. But if I am sending big packets of size let's say 200000 bytes each, and then I pull the network cable to kill the connection abruptly. Will it still keep reporting on and on saying 200000 bytes written into kernel buffers for each async_write? My testing says that it doesn't. It gives up with a large buffer like 200000 bytes and does not report all bytes sent. But if its a small buffer like 30-40 bytes, it keeps reporting okay?
Question:
The primary point of raising this question is: Is there an underlying buffer size which gets filled up at one point for async_write to say that now its not able to write anymore because the earlier scheduled data has not gone out? If yes then what is the size of this underlying buffer? Can I query it from the boost::asio::ip::tcp::socket?
You can query/change the underlying system socket buffer size with send_buffer_size socket option.
The operating system though can dynamically adjust the socket buffer size and limit its maximum size:
tcp_wmem (since Linux 2.4)
This is a vector of 3 integers: [min, default, max]. These
parameters are used by TCP to regulate send buffer sizes. TCP
dynamically adjusts the size of the send buffer from the
default values listed below, in the range of these values,
depending on memory available.
Related
I'm currently working on STM32H747XI (Portenta H7). I'am programming the ADC1 with the DMA1 to get 16bits data at 1Msps.
I'm sorry, I can't share my entire code but I will therefore try to describe my configuration as precisely as possible.
I'm using the ADC1 trigged by a 1MHz timer. The ADC is working in continus mode with the DMA circular and double buffer mode. I tryed direct mode and burst with a full FIFO. I have no DMA error interrupe and no ADC overrun.
My peripheral are running but I'm stuck front of two issues. First issue, I'am doing buffer of 8192 uint16_t and I send it on the USB CDC with the arduino function USBserial.Write(buf,len). In this case, USB transfer going right but I have some missing data in my buffer. The DMA increments memory but doesn't write. So I don't have missing sample but the value is false (it belongs to the old buffer).
You can see the data plot below :
transfer with buffer of 8192 samples
If I double the buffer size, this issue is fixed but another comes. The USB VPC transfer fail if the data buffer is longer than 16384 byte. Some data are cut. I tried to solve this with differents sending and delays but it doesn't work. I still have the same kind of cut.
Here the data plot of the same script with a longer buffer : transfer withe buffer of 16384 sample (32768 byte)
Thank you for your help. I remain available.
For a fast check try to disable data cache. You're probably not managing cache correctly or you haven't disable caching in the memory space where you're using DMA.
Peripherals are not aware of cache so you must manage it manually. You have also to align buffers to cache lines in this case.
Refer to AN4839
Why is TCP receive window considered to be a multiple of MSS Maximum Segment Size?
Wiki states that in order to fully utilize the packet lengths and avoid IP fragmentation , an integral multiple of the Maximum Segment Size (MSS) is generally recommended for the receive window and the value is therefore often only given as a factor/multiple.
Here it is stated that segments exceeding the MSS size is discarded.
For alignment... ergo it fits in a nic buffer on [ROM] # a place in its memory which is then what you read from via a port or DMA
I'm working on a streaming app, I get a frame, encode it to h264 and then send it via tcp to the client, then the client receive the encoded frame, it decode it a display it. What I found is that, when calling several times the write method in smalls intervals of time, the transfer rate is considerably affected.
The interval time is between 12 ms and 17 ms, which is the time it take to grab a frame and encode it. In the client I am counting the time it takes from read one frame and another. Using 12/17 ms the time it takes to arrive to the client is ~400 ms. However, if I add a sleep in the writes, from lets say 12/17 ms to 150 ms,the time in the client reduces to ~150 ms.
So I tried to send one frame, once the client receive it, it send an acknowledgment and then the server grab the next frame. Using this method the latency was the same ~150 ms.
I'm splitting the data into chunks of an specified size (using 512 bytes at the moment), the client receive the chunks and assembly them, using sha256 I confirmed that the info arrives right, the frames size varies (VBR) from 1200 bytes to 65kb. So my conclusion is that if you stress the socket with a lot of writes the transfer rate get affected, is this right or I may be doing something wrong?
And aside, 150 ms is like 6 fps, how VNC and other streaming apps do it? they buffer some frames and then play them? So there would be a latency but the "experience" would be of a higher frame rate?
Thanks
The TCP/IP protocol stack is free to optimize its behavior to trade off latency vs. bandwidth. You have not demonstrated that you lack the bandwidth, only that you get notified of the arriving data less often that you wish - with there being no technical justification for such wish. There's nothing in the protocol itself that guarantees that the data will arrive at the receiving end in chunks of any particular size.
Thus, suppose that you send each frame in a single write on the QIODevice. The receiving end can receive this data in one or more chunks, and it may receive this data after an arbitrary delay. And that's precisely what you're seeing.
The type of data you're sending has no bearing on performance: you should be able to create a 10-line testcase for the sender, and a similarly sized testcase for the receiver, and confirm that indeed you receive all the data you desire, only it comes in larger chunks than you incorrectly expect. And that's fine and it's normal. To maintain streaming, you should pre-buffer a "sufficient" amount of data at the receiving end.
I have a peripheral over USB that is sending data samples at a rate of 183 MBit/s. I would like to send this data over ethernet, which is limited to < 100 Mbit/s. Is it possible to send this data without overflow (i.e missing data) by increasing the TCP socket buffer?
It also depends on the receiver window size. Even if there is 100mbits, sender will push data depending on the window size available on the receiver. TCP window size without scaling enabled can go only upto 64kb. In your case, this size is not sufficient as it needs at least (100-183Mbits)10MB buffer. In Windows 7 & newer Linux OS, TCP by default enables window scaling which can extend the size upto 1GB. After enabling the TCP window scaleing option, you can increase the socket buffer to a bigger size say 50MB which should provide the required buffering.
The short answer is, it depends.
Increasing buffers (at transmitter) can help if the data is bursty. If the average rate is <100MBit (actually less, you need to allow for network contention and overhead), then buffering can help. You can do this by increasing the size of the buffers internally to the TCP stack, or by buffering internally to your application.
If the data isn't bursty, or the average is still too high, you might need to compress the data before transmission. Dependant on the nature of the data, you may be able to achieve significant compression.
Is there a way to change the TCP receiver window size using any of winsock api functions? The RCVBUF value just increases the capacity of the temporary storage. I need to the improve the speed of data transfer and I thought increasing the receiver window size would help but I couldn't find a way to improve it using winsock api. Is there a way to do it or should I modify the registry?
The RCVBUF value just increases the capacity of the temporary storage.
No, the RCVBUF value sets the size of the receive buffer, which is the maximum receive window. No 'just' about it. The receive window is the amount if data the sender may send, which the receiver has to be able to store somewhere ... guess where? In the receive buffer.
On Windows it was historically 8k for decades, which was far too low, and gave rise to an entire sub-industry of 'download tweaks' which just raised the default (some of them also played dangerously with other settings, which isn't usually a good idea).