can't get notification after reconnect to Kaa server - kaa

Question

We have a 1 Kaa (version 0.9) cluster with 3 nodes.
We found sometime the endpoint reconnect to Kaa node and it can't get any notification.
The endpoint had subscribed all topics and we also confirm it in admin website.
after it connect to Kaa and the console just show below message:
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - KaaSync message (zipped=false, encrypted=true) received for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.DefaultOperationDataProcessor - Received Sync response: {"requestId": 2, "status": "SUCCESS", "bootstrapSyncResponse": null, "profileSyncResponse": null, "configurationSyncResponse": null, "notificationSyncResponse": {"responseStatus": "NO_DELTA", "notifications": [], "availableTopics": null}, "userSyncResponse": {"userAttachResponse": null, "userAttachNotification": null, "userDetachNotification": null, "endpointAttachResponses": [], "endpointDetachResponses": []}, "eventSyncResponse": {"eventSequenceNumberResponse": null, "eventListenersResponses": [], "events": null}, "redirectSyncResponse": null, "logSyncResponse": null, "extensionSyncResponses": null}
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.transports.DefaultNotificationTransport - Processed notification response.
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.transports.DefaultUserTransport - Processed user response
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Channel [default_operation_tcp_channel] is reading data from stream using [1024] byte buffer
[pool-6-thread-2] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Executing ping task for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - PingResponse message received for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Channel [default_operation_tcp_channel] is reading data from stream using [1024] byte buffer
[pool-6-thread-2] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Executing ping task for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - PingResponse message received for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Channel [default_operation_tcp_channel] is reading data from stream using [1024] byte buffer
[pool-6-thread-2] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Executing ping task for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - PingResponse message received for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Channel [default_operation_tcp_channel] is reading data from stream using [1024] byte buffer
[pool-6-thread-2] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Executing ping task for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - PingResponse message received for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Channel [default_operation_tcp_channel] is reading data from stream using [1024] byte buffer
[pool-6-thread-2] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - Executing ping task for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.channel.impl.channels.DefaultOperationTcpChannel - PingResponse message received for channel [default_operation_tcp_channel]
[pool-6-thread-1] INFO org.kaaproject.kaa.client.cnhannel.impl.channels.DefaultOperationTcpChannel - Channel [default_operation_tcp_channel] is reading data from stream using [1024] byte buffer
And then, we try to stop the endpoint and reconnect to Kaa server again.
After connect to Kaa, the old notification be shown.
But, we try to send new notification to this endpoint and it still can't get new notification.
We also found if we disconnect to Kaa server and reconnect it and we can get notification again.
But, endpoint can't get notification condition seems still happen in other endpoints. We are not sure this issue relate to our cluster or not.
Sometimes, we will restart kaa-node service and endpoints will connect to other 2 server at that time.

Answer 1

We seems found the root cause. From the log, we saw the notification should be sent to endpoint successfully. But, it seems the performance very bad and endpoint need a lot of time to get the notification. Finally, we found the root cause should be the swap file of our server. Ours server set the swap file before and it seems impact the performance of sending notification. We try to disable the swap space and send notification and the endpoint can get in reasonable time.
We set the swap cause we found the memory usage of Kaa-node will over 80% of our server in one day before. Currently, we try to use different java gc to replace original G1 gc while starting Kaa-node. And the memory usage seems better than use G1 gc. We will continue to monitor it.