I know that the TCP is connection oriented. But if I set up a forwarding server(syslog server for example) which forwards logs on TCP, is the connection always on or whenever the logs are forwarded to a server.
It depends on the server configuration.
Say you are working on Linux, you can use the command
cat /proc/sys/net/ipv4/tcp_keepalive_time
to check your current keepalive value in seconds.
Related
The goal is to make an http request from the client browser to my server. Simple stuff; however I'm hitting a wall with the networking portion. In order to expose my server to WAN I have used one of my public IPs and NAT to translate to the private ip of my server on inbound traffic and to my public IP on outbound traffic.
The issue is that I can't make a connection. Specifically I can't get the last part of the TCP handshake. Using a test setup with Wireshark on the client and server I can see that the client send the SYN -> the server receives the SYN -> the server sends a SYN/ACK -> the client receives a SYN/ACK -> the the client send an ACK -> the server DOES NOT receive the ACK. It waits for a moment then does a retransmission. Eventually resetting.
I have tried adding various firewall rules even though I don't think it could be the firewall because the first packets make a successful round trip.
I've turned windows firewall off(the server)
I've tried disabling TCP checksum offloading
I've looked for network anti virus settings on the server and on the sonic wall(the router)
I would expect the the tcp connection to complete. I can't for the life of me think of a reason why the ACK would consistently go missing.
That is another thing. The behavior is consistent.
pings also work just fine.
NOTE: The server is actually a VM and the physical server that manages it is in my network.
Any guidance on what to try and where to look would be very much appreciated. Thanks.
UPDATE: I can make connection using port 5000(It's another port I have opened on the firewall). Port 80 still doesn't work though.
In my case this was caused by COX not allowing inbound traffic to port 80. I'm not sure why the first portions of the tcp handshake were getting through. If anyone can explain that part leave a comment.
On its documentation ( https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/service_discovery#logical-dns ) for Logical DNS service discovery, Envoy says:
"only uses the first IP address returned when a new connection needs to
be initiated"
How does envoy decide when a new upstream connection needs to be initiated?
It also says:
"Connections are never drained"
What happens to old connections if an upstream host becomes unreachable? Do health-checks apply to all the upstream hosts that currently have established connections or are they only monitoring the host with the current "first IP address"? If the latter, am I right to assume that Envoy will only remove the failed upstream connection (and consequently stop trying to send traffic to those hosts) once it tries to write to it and the peer ACK times out? If so, is it possible to configure the timeout duration?
After looking into the code and doing some tests this is what I've seen:
How does envoy decide when a new upstream connection needs to be
initiated?
For connection establishment, in the case of the TCP proxy (the filter I was using), there is a 1:1 mapping between downstream and upstream connections, therefore a new upstream connection is established when a new downstream connection is established.
What happens to old connections if an upstream host becomes
unreachable?
It depends on whether the connection was gracefuly terminated (TCP RST packet sent) or not. If it was, then the connection will be destroyed (along with the downstream connection), if it was not, then nothing happens until the TCP connection times out (I believe due to TCP_USER_TIMEOUT or tcp_retries2 retries - it was taking more than 15 minutes on my local machine).
Do health-checks apply to all the upstream hosts that currently have
established connections or are they only monitoring the host with the
current "first IP address"?
They only apply to the current "first IP address".
If the latter, am I right to assume that Envoy will only remove the
failed upstream connection (and consequently stop trying to send
traffic to those hosts) once it tries to write to it and the peer ACK
times out?
Yes. Typically the downstream clients timeouts will kick in first and destroy the connection though.
If so, is it possible to configure the timeout duration?
I couldn't find an option to set the socket's TCP_USER_TIMEOUT in envoy. Changing the OS tcp_retries2 might help, but, according to the documentation, the total time is also influenced by the smoothed round trip time of the TCP connection, so a change to tcp_retries2 wouldn't be able to define an absolute timeout value.
I have configured HAProxy (1.5.4, but I tried also 1.5.14) to balance in TCP mode two server exposing AMQP protocol (WSO2 Message Broker) on 5672 port.
The clients create and use permanent connection to the AMQP Servers, via HAProxy.
I've changed the client and server TCP keepalive timeout, setting net.ipv4.tcp_keepalive_time=120 (CentOS 7).
In HAProxy I've setted timeout client/server to 200 seconds (>120 seconds of the keepalive packets) and used the option clitcpka.
Then I've started wireshark and sniffed all the tcp traffic: after the last request from the clients, the tcp keepalived packets are sente regularly after 120 seconds, but after 200 seconds after the last request from the clients the connection are closed (thus ignoring the keepalived packet).
Below the configuration:
haproxy.conf
global
log 127.0.0.1 local3
maxconn 4096
user haproxy
group haproxy
daemon
debug
listen messagebroker_balancer 172.19.19.91:5672
mode tcp
log global
retries 3
timeout connect 5000ms
option redispatch
timeout client 200000ms
timeout server 200000ms
option tcplog
option clitcpka
balance leastconn
server s1 172.19.19.79:5672 check inter 5s rise 2 fall 3
server s2 172.19.19.80:5672 check inter 5s rise 2 fall 3
TCP keep alive is at the transport layer and is only used to do some traffic on the connection so intermediate systems like packet filters don't loose any states and that the end systems can notice if the connection to the other side broke (maybe because something crashed or a network cable broke).
TCP keep alive has nothing to do with the application level idle timeout which you have set explicitly to 200s:
timeout client 200000ms
timeout server 200000ms
This timeouts gets triggered if the connection is idle, that is if no data get transferred. TCP keep alive does not transport any data, the payload of these packets is empty.
The timeout client detects a dead client application on a responsive client OS. You can always have an application that occupies a connection but doesn't speak to you. This is bad because the number of connections isn't infinite (maxconn).
Similarly, set timeout server for the backend.
These options were for haproxy talking to application. Now, there is a completely separate check where OS talks to OS (without touching the app or haproxy):
With option clitcpka or option srvtcpka or option tcpka you allow the inactive connection to be detected and killed by the OS, even when haproxy doesn't actively check it. This primarily needs OS settings (Linux).
If no data sent for 110 seconds then immediately send the first keep-alive (KA), don't kill connection yet:
sysctl net.ipv4.tcp_keepalive_time=110
Wait for 30 seconds after each KA, once they're enabled on this connection:
sysctl net.ipv4.tcp_keepalive_intvl=30
Allow 3 KAs be unacknowledged, then kill the TCP connection:
sysctl net.ipv4.tcp_keepalive_probes=3
In this situation OS kills the connection 200 seconds after packets stop coming.
I have around 20 clients communicating together with a central server in the same LAN. The clients can make transaction simultaneously with the server. The server forward each transaction to external appliance in the network. Sometimes it works, sometimes my application shows a "time out" message in a client screen (randomly)
I mirrored all traffic and found TCP Retransmission after TCP Reset packets for the first TCP Sequence. I immediately thought about packet loss but all my cables/NIC are fine, and I do not see DUP ACK in the capture.
It seems that RST packets may have different significations.
What causes those TCP Reset?
Where should I focus my investigation: network or application design ?
I would appreciate any help. Thanks in advance.
Judging by the capture, I assume your central server is 137.56.64.31. What's happening is the clients are initiating a connection to the server with a SYN packet and the server responds with a RST. This is typical if the server has no application listening on that particular port e.g. the webserver application isn't running and a client tries to connect to port 80.
The clients are all connecting to different ports on the server, which is unusual for an central server, but not unheard of. The destination ports the clients are connecting to on the server are: 11007, 11012, 11014, 11108, and 11115. Is that normal for the application? If not, the clients should be connecting to whatever port the application server is listening on.
The reason for the retransmits is that instead of giving up on the connection upon receiving a RST from the server, the client tries to initiate the connection again so Wireshark considers it a retransmission.
Does the client remote port changes during an HTTP 1.1 connection exchange?
I am trying to figure out if I can programmaticaly uniquely identify a connection on the server using the request remote port and remote ip address.
This is not as much an HTTP question, as it's a TCP one. And no, the port doesn't change: the ephemeral port stays the same for the duration of the connection.
However, as soon as a new connection is made, the client can (and most probably will) use a different port. This totally depends on the implementation of the client OS and the Network Address Translation of intermediary routers.
Anyway, it is not something you can depend on to build something like a session, because the next request from the same client may very well arrive from a different port (let alone that HTTP does not have to run on top of TCP).
Just use a session-ID which you store in a cookie.