Based on this AWS doc both layer 4 and layer 7 ALBS terminate the client connection and establish a new TCP connection with the back-end instances (connection are reused on both ends).
Is there any info on how these connections are managed? (i.e 1 server
connection per client connection?)
How is TCP backpressure handled? If the server stops reading TCP packets, will the client be made aware of it so it can backpressure?
Related
By definition, HTTP pipelining is a technique in which multiple HTTP requests are sent on a single TCP connection without waiting for the corresponding responses, according to Wikipedia, and some other resources. What is meant by TCP connection, is it the connection that starts when the client and server first communicate doing the 3 way handshake? Is it the same TCP stream? TCP stream meaning that the client and server communicate with each other using the same combination of ports.
Architecture:
We have a bunch of IoT devices connected via an AWS network loadbalancer (NLB) to our backend servers.
This is a bidirectional channel (not a request response style, but messages passed from either party to the other).
Objective:
How to keep connections (both sides of NLB) alive during inactivity.
Description:
Frequently clients go to inactive mode and do not send (or receive) anything to (or from) servers. If this state lasts longer than 350 seconds (connection idle timeout value of NLBs) the LB silently kill the connection. This is bad, because we see a lot of RST packets everywhere.
Questions:
I'm aware of SO_KEEPALIVE feature and can enable it on our backend servers. This keeps the connection between backend servers and NLB alive. But what about clients? Do NLBs forward TCP keep-alive packets to the other party? (Here it says it does not). If it does not, how to keep clients connections open? (At them moment, I'm thinking to send an empty message to keep the connection.)
Is this behavior specific to AWS NLBs or do loadbalancers generally work this way?
AWS docs say that NLB TCP listener has ability to keep connection alive with TCP keep-alive packets: link
For TCP listeners, clients or targets can use TCP keepalive packets to reset the idle timeout.
Based on my tests client is receiving TCP keep alive packets sent by server and correctly responds back.
Server doesn't interrupt connection what means it receives response from client.
It means that NLB TCP listener actually forwards keep-alive packets.
Based on the same docs, NLB TLS listener shouldn't react the same on TCP keep-alive packets.
TCP keepalive packets are not supported for TLS listeners.
But actual tests result shocked me when Wireshark showed keep-alive packets received on client connected through TLS listener.
My previous test results performed 2 months ago don't correspond what I'm experiencing now and I'm thinking behaviour may changed.
(previously server was keeping the connection even after client became unavailable in unexpected manner)
Not an answer, just to document what I found/did:
NELBs do not forward keep-alive packets. Meaning you have to enable them on both server and clients.
NELB's timeout cannot be changed. it's 350 second
I couldn't find any way to forge an empty TCP packet to fool the LB to forward it to the other side of the LB.
At the end, we implemented the keep alive feature at the application layer (sending an empty message to clients periodically.)
Consider a scenario where exists one server and multiple clients. And each client creates TCP connections to interact with the server. There are three usages of TCP alive:
Server-side keepalive: The server sends TCP keepalive to make sure that the client is alive. If the client is dead, the server closes the TCP connection to the client.
Client-side keepalive: Clients sends TCP keepalive to prevent the server from closing the TCP connection to the client.
Both-side keepalive: Both server and clients send TCP keepalive as described in 1 and 2.
Which of the above usages of TCP keepalive are typical?
Actually, both server and client peers may use TCP keepalive. It is useful to ensure that the operating system will eventually release any resource associated with dead connections. Note that if a connection between two hosts get lost because of some issue with a router between them, then both hosts have to independently detect that the connection is dead, and cleanup for themselves.
Now, each host will maintain a timer on each connection indicating when it last received a packet associated with that connection. A host will send a keepalive packet when that timer goes over a certain threshold, which is defined locally (that is, hosts do not exchange information about their own keepalive configuration). So either host with the lowest keepalive time will take the initiative of sending a keepalive packet to the other host. If the packet indeed goes through, the other host (that is, the one with the higher keepalive time) will respond to that packet and reset its own timer; therefore, the host with an higher keepalive time will certainly never reach the need to send keepalive packet itself, unless the connection has indeed been lost.
Arguably, it could be said that servers are generally more aggressive on keepalive than client machines (that is, they will more often be configured with lower keepalive time), because hanging connections often have undesirable effects on server software (for example, the software may accept a limited number of concurrent connection, or the server may fork a new process instance associated with each connection).
Server-side keepalive: The server sends TCP keepalive to make sure that the client is alive. If the client is dead, the server closes the TCP connection to the client.
If the client is dead, the server gets a 'connection reset' error, after which it should close the connection.
Client-side keepalive: Clients sends TCP keepalive to prevent the server from closing the TCP connection to the client.
No. Client sends keepalive so that if the server is dead, the client will get a 'connection reset' error, after which it should close the connection.
Both-side keepalive
Both sides are capable of getting a 'connection reset' due to keepalive failure, as above.
Whuch of the above usages is typical?
Any of them, or none. If a peer is sending regularly it doesn't really need keepalive as well. It is therefore often of more use to a server than a client.
Consider such a scenario, there are client-a and server-b. server-b has TCP keepalive disabled. And server-b does not have any application logic to check whether a TCP connection is open. client-a establishes a TCP connection to server-b. There is no data transfer between client-a and server-b afterward. In such a case, will the server ever close the idle connection? Or the server will have the TCP connection stay open forever.
After reading Longest Open TCP Connection?, I guess that such a TCP connection will stay open forever. Is my guess correct?
There is no limit in the TCP connection itself. Client and server could in theory stay connected for years without exchanging any data and without any packet flow. Problems are usually caused by middleboxes like NAT router or firewalls which keep a state and expire the state after some inactivity. Any new packets sent within the connection cannot be delivered then because no associated state exists anymore in the middlebox.
While a tcp client is connecting to a tcp server, and at same time the Listening port at server side hasn't been established. At this time will the tcp client report connection refused exception or connection time out exception ?
At first I think a connection refused exception will be reported by tcp/ip, because the server could check if the LISTEN port( which is connected by client) is exist. But actually a connection time out instead.
I am wondering if tcp/ip protocol has explained such a situation?
At this time will the tcp client report connection refused exception or connection time out exception ?
If the client gets an active refusal, i.e. a TCP RST segment, this shows up at the client as ECONN. Otherwise you get a connection timeout. Note that this can also include the case where an intermediate firewall prevented the connection, as it is usually considered more secure not to leak the information that the host even exists if firewall rules prevent connecting to that port.