We have a Kafka consumer which is consuming the messages perfectly.
But the thing is , When the Kafka consumer started, the consumer is sending lot of TCP packets to Kafka Broker. I can sniff those packets using Wireshark.
Our Team understanding is like, It should send the TCP packet whenever the message is getting produced by the Producer and the Consumer should be able to consume those messages by sending a request to Broker. (Like Event Driven)
We cannot really understand why the TCP packets are getting transferred between the Broker and Consumer continuously when the consumer is getting started.
Kafka works as a pull based system communicating over TCP. That's why consumers continuously send TCP requests (poll() request and heartbeats, actually). You can check this link for more information about this design decision of Kafka.
Related
Architecture:
We have a bunch of IoT devices connected via an AWS network loadbalancer (NLB) to our backend servers.
This is a bidirectional channel (not a request response style, but messages passed from either party to the other).
Objective:
How to keep connections (both sides of NLB) alive during inactivity.
Description:
Frequently clients go to inactive mode and do not send (or receive) anything to (or from) servers. If this state lasts longer than 350 seconds (connection idle timeout value of NLBs) the LB silently kill the connection. This is bad, because we see a lot of RST packets everywhere.
Questions:
I'm aware of SO_KEEPALIVE feature and can enable it on our backend servers. This keeps the connection between backend servers and NLB alive. But what about clients? Do NLBs forward TCP keep-alive packets to the other party? (Here it says it does not). If it does not, how to keep clients connections open? (At them moment, I'm thinking to send an empty message to keep the connection.)
Is this behavior specific to AWS NLBs or do loadbalancers generally work this way?
AWS docs say that NLB TCP listener has ability to keep connection alive with TCP keep-alive packets: link
For TCP listeners, clients or targets can use TCP keepalive packets to reset the idle timeout.
Based on my tests client is receiving TCP keep alive packets sent by server and correctly responds back.
Server doesn't interrupt connection what means it receives response from client.
It means that NLB TCP listener actually forwards keep-alive packets.
Based on the same docs, NLB TLS listener shouldn't react the same on TCP keep-alive packets.
TCP keepalive packets are not supported for TLS listeners.
But actual tests result shocked me when Wireshark showed keep-alive packets received on client connected through TLS listener.
My previous test results performed 2 months ago don't correspond what I'm experiencing now and I'm thinking behaviour may changed.
(previously server was keeping the connection even after client became unavailable in unexpected manner)
Not an answer, just to document what I found/did:
NELBs do not forward keep-alive packets. Meaning you have to enable them on both server and clients.
NELB's timeout cannot be changed. it's 350 second
I couldn't find any way to forge an empty TCP packet to fool the LB to forward it to the other side of the LB.
At the end, we implemented the keep alive feature at the application layer (sending an empty message to clients periodically.)
I was discussing Websocket overhead with a fellow engineer and we were both uncertain of how a Websocket actually detects the status of a client connection.
Is there a "status" packet sent to the client/server periodically?
Does it have anything to do with ping or pong in the low level API?
What about frames?
How does the Websocket detect that it's disconnected on the client? the server?
I'm surprised I couldn't find this answered on SO, but that may be my mistake. I found this answer addresses scalability, but that's not what I'm asking here. This answer touches on implementation, but not at the depth I'm pursuing here.
A webSocket connection is a TCP connection that uses the webSocket protocol. By default, a server or client knows when the connection has disappeared only when the underlying TCP realizes that the connection has been closed and the webSocket layer is listening for a close event on the connection so it is informed that way.
The webSocket protocol does not, by itself, require heartbeat packets that can regularly test if the connection is still working. The TCP socket may still appear to be alive, but the connection may not actually still work. The other end could have disappeared or been interrupted in between and one or both endpoints might not know that at any given time.
Socket.io which is built on top of webSocket, uses the ping and pong packets to implement a heartbeat which regularly tests the connection and will, in fact, detect a non-functioning connection at the client, close the socket and then reconnect automatically.
Is there a "status" packet sent to the client/server periodically?
Not by default for a regular webSocket connection.
Does it have anything to do with ping or pong in the low level API?
It is up to a client or server if they want to send ping or pong packets themselves to implement some sort of connection validation detection.
What about frames?
webSocket frames are the data format for sending data over a webSocket. They don't have anything to do with this issue.
How does the Websocket detect that it's disconnected on the client? the server?
Described above. Unless the client/server implement their own ping/pong system to detect when a connection has gone awry, they just rely on TCP signaling to know when a connection has been closed by the other end. A webSocket connection can be non-functional without the client or server knowing it until they try to send.
When a browser window/tab opens a webSocket connection and then the window/tab is retargeted to a new URL, the browser will close all resources associated with that window/tab including any webSocket connections. If the link between client and server is functional at that point, then the server will be told the underlying TCP connection (and thus the webSocket) has been closed. If the network link goes down and then the user moves that window/tab to a new URL, the server will not necessarily know that the connection is non-functional without relying on ping/pong type signalling to regularly test the connection.
If the browser crashes, the OS should close any sockets opened by that process.
If the computer/OS crashes, the sockets will likely not get closed normally (though this may be somewhat OS dependent and may be crash-dependent too).
Ive build a local multiplayer game (multiplayer over wlan network). Now, I want to add an online multiplayer feature..
Currently, the network communications consist mostly of "signals" (tcp/udp packets sent from game-host peer to the game-client peers). I would like to use this mostly signal based communication for my online multiplayer (because of performance and efficiency ), too . But, since the host peer is now replaced by a server there will be a lot problems with sending signals (NAT, firewall,...).
So is there good solution to implement these signals?
regards
there will be a lot problems with sending signals (NAT, firewall,...)_
What problems exactly?
Normally, the clients establish a TCP connection to the server and the server uses this TCP connection to communicate with the clients.
For UDP-based communication the clients use Internet Gateway Device Protocol to forward ports on the router, so that the server can send UDP datagrams to the clients.
Assuming your server is in public internet, not behind any NAT. All the clients must initiate the connection. Otherwise the server can't know clients credential and can't connect. As the server has no NAT it will accept connection from client. And this connection client must keep alive. So when server needs to send some data there should be no problem.
This will work for both UDP and TCP.
I am using apache netty module to receive udp packets.
Under load it does not seem to accept all sent udp packets.
I tested it by running a dummy udp server (non-camel, just counts number of received packets) and sent 1000 packets per second from localhost, this dummy udp server accepted and counted all. However sending 1000 packets per second from localhost to my camel udp port, I can see that some (1-5%) never show up.
What camel netty configuration should I play with to resolve this issue?
BTW I am sending udp packets in tight loop
Update: I am now providing following default threadProfile.
<threadPoolProfile id="myDefaultProfile"
defaultProfile="true"
poolSize="20"
keepAliveTime="25"
maxPoolSize="50"
maxQueueSize="100000"/>
Still get packet loss.
I am trying to send data from a router to the kafka brokers. Since a router can only be configured in a way that it knows the IP and port number of the Kafka server.
I do not want to introduce a layer of java to consume messages from the router because it will cause latency.