Grpc Error on Long Connections: Too Many Pings - grpc

Question

I am building a system with a microservice architecture that communicates between services using grpc. I have a long running request in the beginning that hits a central endpoint that makes a bunch of requests to other services. The first request to the central service in turn waits until the other services finish computing their requests before receiving a response from the central endpoint. This may take minutes to complete. The problem is that I keep getting a grpc error saying "Too many pings". I have set keepalive params on my go servers the following way:
ka_params := keepalive.ServerParameters{
Time: 10 * time.Second,
Timeout: 5 * time.Second,}
opts := []grpc.ServerOption{
grpc.KeepaliveParams(ka_params),
}
s = grpc.NewServer(opts...)
And in my python servers like this:
opts = [("grpc.keepalive_time_ms", 10000),
("grpc.keepalive_timeout_ms", 5000),
("grpc.keepalive_permit_without_calls", True),
("grpc.http2.max_pings_without_data", 0)]
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10), options=opts)
I'm not sure why I get an error with too many pings. Aren't the pings expected because of the keepalive?

Answer 1

I think I found a solution to this. The crux of the issue was that the python and golang grpc versions have different default settings and python grpc is very poorly documented.
To solve the issue you have to set the max_ping_strikes on the python server to 0. The python server should have the following options:
opts = [("grpc.keepalive_time_ms", 10000),
("grpc.keepalive_timeout_ms", 5000),
("grpc.keepalive_permit_without_calls", True),
("grpc.http2.max_ping_strikes", 0)]

Answer 2

On the python server side, to configure the acceptable keepalive time period, you would want to set "grpc.http2.min_ping_interval_without_data_ms" to 10 seconds (maybe a bit higher to account for network latency).
The default setting for this parameter is 5 minutes and if the client is sending pings every 10 seconds, then it would result in a GOAWAY frame being sent with "too_many_pings".
(Also, the "grpc.keepalive_time_ms" on the server side results in the server also sending keepalive pings every 10 seconds. This might not be what you want.)
Reference:
https://github.com/grpc/grpc/blob/master/doc/keepalive.md