Using gRPC in Production -- connection keepalive

Posted on | Comments:
Symbols count in article: 3.9k | Reading time ≈ 4 mins.

The gRPC quickstart guides do a very good job at helping the developers to get started using gRPC in their projects, but deployments to production is not discussed and available resources are sparse.

In this article we discuss a common deployment scenario and how to navigate around the pitfalls.

Infrastructure Overview

In our example, we have a Ruby on Rails monolithic application that is running production and actively maintained by a team of developers. To develop a new feature, we chose to build a microservice in Python and leverage its readily available machine learning libraries. Rails app communicates with the Python microservice via gRPC. Each instance of the Rails app maintains one instance of active gRPC connection with the Python microservice (a channel, in gRPC’s terminology.)

We chose to use a standard Layer 3 load balancer to scale the Python microservices horizontally. (Note: This might make sense for all use cases, if your application is data transfer heavy, a proxy-like Layer 3 LB might be a bottleneck.)

The Python microservice is built as a Docker container and we use red-black deployment strategy to retire older versions safely.

Connection Keepalive

GRPC uses keepalive ping as a way to check if a channel is currently working by sending HTTP2 pings over the transport. It is sent periodically, and if the ping is not acknowledged by the peer within a certain timeout period, the transport is disconnected [1]. However the default interval of keepalive ping is set as 2 hours and is only sent on the server side as a sane default. This means when an old gRPC server is retired by a newer deployment or simply becames faulty, the client only finds out when trying and failing to send a request. As a default value, it works well as it prevents the network from being flooded by useless pings when the services don’t need them. However in practice, we often want to detect interruptions earlier, and having a long keepalive interval degrades the service for the first few users that try to access the service while gRPC client is unaware of the server failure.

When this happens, usually this error appears on the client side

1
GRPC::Unavailable: 14:OS Error

or

1
GRPC::Unavailable: 14:Connect Failed

We can make gRPC detects unavailable services quicker by adjusting the keepalive options. On the server side we can change the server initializer code to include keepalive options: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
server = grpc.server(
    futures.ThreadPoolExecutor(max_workers=10),
    options=(
        ('grpc.keepalive_time_ms', 10000),
        # send keepalive ping every 10 second, default is 2 hours
        ('grpc.keepalive_timeout_ms', 5000),
        # keepalive ping time out after 5 seconds, default is 20 seoncds
        ('grpc.keepalive_permit_without_calls', True),
        # allow keepalive pings when there's no gRPC calls
        ('grpc.http2.max_pings_without_data', 0),
        # allow unlimited amount of keepalive pings without data
        ('grpc.http2.min_time_between_pings_ms', 10000),
        # allow grpc pings from client every 10 seconds
        ('grpc.http2.min_ping_interval_without_data_ms',  5000),
        # allow grpc pings from client without data every 5 seconds
    )
)

On the client- side, we change the client initializer code to include keepalive options:

1
2
3
4
5
6
7
8
9
10
11
12
13
def connect!
  @stub = Stub.new(
      'localhost:50051', :this_channel_is_insecure,
      channel_args: {
        'grpc.keepalive_time_ms': 10000,
        'grpc.keepalive_timeout_ms': 5000,
        'grpc.keepalive_permit_without_calls': true,
        'grpc.http2.max_pings_without_data': 0,
        'grpc.http2.min_time_between_pings_ms':10000,
        'grpc.http2.min_ping_interval_without_data_ms': 5000,
      }
  )
end

These options allows the client and server to discover connection interruptions quicker and re-establish another connection through the load balancer.