I am working on a project, where I want to expose an API which will read a large file send that in response.
Since file can be large in size, it is better to send the file in chunks, so not much memory pressure on system.
I evaluated 2 options : chunked transfer (https://en.m.wikipedia.org/wiki/Chunked_transfer_encoding) which is supported by HTTP-1.1, and a server side streaming by grpc.
In grpc approach, client request through a rpc, while server will stream bytes over grpc channel and close the same once done.
My ecosystem can support both, and client also supports grpc.
Can you please suggest which option is better, what are pros and cons of both approaches.
Both approaches will yield similar results. Just choose the stack you are most comfortable with.
HTTP chunking uses length-prefixed batches. gRPC streaming uses length-prefixed messages. The main difference is the batch size is chosen for you with chunking.
gRPC uses chunking to provide its streaming, although chunking is implemented differently between HTTP/1.1 and HTTP/2. When using gRPC the main thing is to use a "reasonable" message size; there's an overhead of ~5-10 bytes per message. Overheads exist in chunking as well, just you didn't have to choose the "reasonable" message size.
Both chunking and gRPC streaming use pipelining, where you send a chunk before the previous chunk has been fully received and processed. This reduces the influence of the chunk size on network performance. As long as you choose a chunk size of at least 1 KB, the chunk size shouldn't matter too much and fine-tuning will rapidly see diminishing returns.
Related
I would like to understand the usage of 'Transfer-encoding: Chunked' in case of HTTP requests.
Is it common for requests to be chunked?
My thinking is no since requests need to be completely read before processing, it does not make sense to be sending chunked requests.
It is not that common, but it can be very useful for large request bodies.
My thinking is no since requests need to be completely read before processing, it does not make sense to be sending chunked requests.
(1) No, they don't need to be read completely.
(2) ...and the main reason to compress it to save bytes on the wire anyway.
For an HTTP agent acting as a reverse proxy or a forward proxy, so taking a message from one side and sending it on the other side, using a chunked transmission means you can send the parts of the message you have without storing it locally. You avoid the 'buffering' problems, slowdown and storage.
You also have some optimizations based on each actor preferred size of data blocks, like you could have an actor which likes sending packets of 8000 bytes, because that's the good number for his own kernel settings (tcp windows, internal http server buffer size, etc), while another actor on the message transmission using smaller chunks of 2048 bytes.
Finally, you do not need to compute the size of the message, the message will end on the end-of-stream marker, that's all. Which is also usefull if you are sending something which is compressed on the fly, you may not know the final size until everything is compressed.
Chunked transmission is used a lot. It is the default mode of most HTTP servers if you ask for HTTP/1.1 mode and not HTTP/1.0.
I've written a small program with the boost asio library to transfer files via TCP from a server to one or more clients.
During testing I found out that the transfer is extremely slow, about 10KiB/s. Nagle's algorithm is already disabled. If I transfer the same file via FileZilla from the same server to the same client, I get about 280KiB/s, so obviously something was very wrong.
My approach so far was to fragment each file into smaller packets of 1024 bytes, send one fragment(each fragment=1 async_write-call) to the client and wait for the client's response. I need to fragment the data to allow the client to keep track of the download progress and speed. In retrospect I suppose this was rather naïve, because the server has to wait for the client's response after each fragment. To check if this was the bottleneck, I've increased the fragment size twice, giving me the following results:
a) Fragment Size: 1024bytes
Transfer Speed: ~10KiB/s
b) Fragment Size: 8192bytes
Transfer Speed: ~80KiB/s
c) Fragment Size: 20000bytes
Transfer Speed: ~195KiB/s
The results speak for themselves, but I'm unsure what to do now.
I'm not too familiar with how the data transfer is actually handled internally, but if I'm not mistaken all of my data is basically added onto a stream? If that's the case, do I need to worry about how much data I write to that stream at once? Does it make a difference at all whether I use multiple write-calls with small fragments as opposed to one write-call with a large fragment? Are there any guidelines for this?
Simply stream the data to the client without artificial packetization. Reenable nagling, this is not a scenario that calls for disabling it. It will cause small inefficiencies to have it disabled.
Typical write buffer sizes would be 4KB and above.
The client can issue read calls to the network one after the other. After each successful read the client will have a new estimation for the current progress that is quite accurate. Typically, there will be one succeeding read call for each network packet received. If the incoming rate is very high then multiple packets tend to be coalesced into one read. That's not ap roblem.
If that's the case, do I need to worry about how much data I write to that stream at once?
No. Just keep a write call outstanding at all times.
I'm dumping data from my server app to my client app in chunks (TCP/IP). At some point the client may wish to abort the transfer and make a new request to the server. The rapid approach to get this done is to kill the TCP connection so that any data already sent by the server and live on the network is dumped. The new connection will handle the new request/transfer so there is no delay in receiving old redundant data.
Is this an acceptable solution?
NB: I did consider breaking the chunks into smaller sizes separated by client ack messages but then you have the problem of fixing a chunk size....too small and too many acks (slowed transfer)....too big and there is still a residual delay in dumping redundant data.
Any ideas or standard design approaches that I should be aware of?
TIA
You can use two TCP connection similar to FTP - one to send control request to the server & other transfers the actual data. If you wish to abort a transfer then just a request to abort it over control channel.
Send the data in chunks but don't acknowledge. When the client wants to abort the transfer make it send a cancellation request to the server. The client now just throws away chunks (which are still arriving). Eventually, the server gets the cancellation request and stops sending data. If you want to you can make the server send an acknowledgement of cancellation.
This way you can have small chunks with minimal overhead. You could have 1KB chunks with a 4 or 8 byte chunk header containing the size of the chunk. That is an extremely small bandwidth and latency overhead.
Note, that a small chunk does not generally result in a small IP packet. TCP streams data. It does not care about your chunk size.
I have several clients that constantly post data to a REST service. REST service is put behind a network load balancer. Each client sends 100 - 500 MB a day and I need to support 500+ clients.
I can POST either very large packets, this will reduce overhead for TCP/IP session set up and HTTP headers. This will, however, firmly tie one client to a particular server and limit my scalability options. Alternatively, I can send small HTTP packets, which I can load balance well, but I will get more overhead for TCP/IP session set up and HTTP headers.
What is the recommended packet size for HTTP POST? Or how can I calculate one for my environment?
There is no recommended size.
While HTTP POST size is not constrained by the RFCs, since HTTP is a commodity protocol implementing request / response type messaging, most of the infrastructure is configured around the idea that TCP connections are not particularly long lasting / does not carry significant amounts of data. i.e. there will be factors outside your control which may impact the service - although HTTP supports range requests for responses, there is no corollary for requests.
You can get around a lot of these (although not all) by using HTTPS. However you still need to think about how you detect/manage outages - are you happy to wait for a TCP timeout?
With 500+ clients presumably using the system quite heavily, the congestion avoidance limits shouldn't be a problem - whether TCP window scaling is likely to be an issue depends on how the system is used. HTTP handshakes should not be an issue unless you restrict the request size to something silly.
If the service is highly dependant on clients pushing lots of data on to your server, then I'd encourage you to look at parsing the data on the client (given the volume, presumably it's coming from files - implying a signed java applet or javascript with UniversalBrowserRead privilege) then sending it over a bi-directional communication channel (e.g. websocket).
Leaving that aside for now, the only way you can find out what the route between your clients and your server will support is to measure it - and monitor it. I would expect that a 2Mb upload size would work pretty much anywhere, while a 10Mb size would work most of the time within the US or Europe - and that you could probably increase this to 50Mb as long as there's no mobile clients.
But if you want to maintain the effectiveness of the service you'll need to monitor bandwidth, packet loss and lost connections.
I am trying to get a handle on what happens when a server publishes (over tcp, udp, etc.) faster than a client can consume the data.
Within a program I understand that if a queue sits between the producer and the consumer, it will start to get larger. If there is no queue, then the producer simply won't be able to produce anything new, until the consumer can consume (I know there may be many more variations).
I am not clear on what happens when data leaves the server (which may be a different process, machine or data center) and is sent to the client. If the client simply can't respond to the incoming data fast enough, assuming the server and the consumer are very loosely coupled, what happens to the in-flight data?
Where can I read to get details on this topic? Do I just have to read the low level details of TCP/UDP?
Thanks
With TCP there's a TCP Window which is used for flow control. TCP only allows a certain amount of data to remain unacknowledged at a time. If a server is producing data faster than a client is consuming data then the amount of data that is unacknowledged will increase until the TCP window is 'full' at this point the sending TCP stack will wait and will not send any more data until the client acknowledges some of the data that is pending.
With UDP there's no such flow control system; it's unreliable after all. The UDP stacks on both client and server are allowed to drop datagrams if they feel like it, as are all routers between them. If you send more datagrams than the link can deliver to the client or if the link delivers more datagrams than your client code can receive then some of them will get thrown away. The server and client code will likely never know unless you have built some form of reliable protocol over basic UDP. Though actually you may find that datagrams are NOT thrown away by the network stack and that the NIC drivers simply chew up all available non-paged pool and eventually crash the system (see this blog posting for more details).
Back with TCP, how your server code deals with the TCP Window becoming full depends on whether you are using blocking I/O, non-blocking I/O or async I/O.
If you are using blocking I/O then your send calls will block and your server will slow down; effectively your server is now in lock step with your client. It can't send more data until the client has received the pending data.
If the server is using non blocking I/O then you'll likely get an error return that tells you that the call would have blocked; you can do other things but your server will need to resend the data at a later date...
If you're using async I/O then things may be more complex. With async I/O using I/O Completion Ports on Windows, for example, you wont notice anything different at all. Your overlapped sends will still be accepted just fine but you might notice that they are taking longer to complete. The overlapped sends are being queued on your server machine and are using memory for your overlapped buffers and probably using up 'non-paged pool' as well. If you keep issuing overlapped sends then you run the risk of exhausting non-paged pool memory or using a potentially unbounded amount of memory as I/O buffers. Therefore with async I/O and servers that COULD generate data faster than their clients can consume it you should write your own flow control code that you drive using the completions from your writes. I have written about this problem on my blog here and here and my server framework provides code which deals with it automatically for you.
As far as the data 'in flight' is concerned the TCP stacks in both peers will ensure that the data arrives as expected (i.e. in order and with nothing missing), they'll do this by resending data as and when required.
TCP has a feature called flow control.
As part of the TCP protocol, the client tells the server how much more data can be sent without filling up the buffer. If the buffer fills up, the client tells the server that it can't send more data yet. Once the buffer is emptied out a bit, the client tells the server it can start sending data again. (This also applies to when the client is sending data to the server).
UDP on the other hand is completely different. UDP itself does not do anything like this and will start dropping data if it is coming in faster then the process can handle. It would be up to the application to add logic to the application protocol if it can't lose data (i.e. if it requires a 'reliable' data stream).
If you really want to understand TCP, you pretty much need to read an implementation in conjunction with the RFC; real TCP implementations are not exactly as specified. For example, Linux has a 'memory pressure' concept which protects against running out of the kernel's (rather small) pool of DMA memory, and also prevents one socket running any others out of buffer space.
The server can't be faster than the client for a long time. After it has been faster than the client for a while, the system where it is hosted will block it when it writes on the socket (writes can block on a full buffer just as reads can block on an empty buffer).
With TCP, this cannot happen.
In case of UDP, packets will be lost.
The TCP Wikipedia article shows the TCP header format which is where the window size and acknowledgment sequence number are kept. The rest of the fields and the description there should give a good overview of how transmission throttling works. RFC 793 specifies the basic operations; pages 41 and 42 details the flow control.