grpc performance benchmarking and optimization

Benchmarking is a crucial step in optimizing GRPC performance. By understanding the baseline performance of your GRPC service, you can identify areas for improvement.

GRPC's built-in latency and throughput metrics can be used for benchmarking. These metrics provide a clear picture of your service's performance.

To get started with benchmarking, you'll need to set up a test environment that simulates real-world traffic. This can be done using tools like Apache Bench or Locust.

By analyzing the results of your benchmarking tests, you can identify bottlenecks in your GRPC service and optimize them for better performance.

For another approach, see: C# Grpc Service

Several languages are under continuous performance testing as both clients and servers, including C++, Java, Go, C#, Node.js, Python, and Ruby.

These languages are tested in various combinations, such as Python against a C++ server and C++ against a Python client, to provide an upper bound of performance for each language's client or server implementation.

Here are some key facts about the languages being tested:

C++ is a versatile and powerful programming language that can be used for a wide range of applications, from high-performance servers to low-QPS services. For performance-sensitive servers, it's best to avoid using the Sync API due to its potential impact on resource consumption.

The Sync API is indeed the simplest to implement, but it may not be the best choice for high-QPS workloads. Favor the callback API instead, as it's easier to use and can help avoid blocking operations. This approach is particularly useful when the application can avoid blocking operations or move them to a separate thread.

The callback API is currently slower for truly high-QPS workloads, but it's a good trade-off for most RPCs. If you need to use the async completion-queue API, consider using numcpu's threads for the best scalability trade-off. As of gRPC 1.41 (Sept 2021), using 2 threads per completion queue seems to give the best performance.

Check this out: When to Use Grpc

To get the most out of the async completion-queue API, make sure to register enough server requests for the desired level of concurrency. This will help avoid the server getting stuck in a slow path, resulting in essentially serial request processing.

Here's a quick reference guide to help you choose the right API for your C++ project:

Note that the generic stub, gRPC::GenericStub, can be useful in certain cases when there is high contention or CPU time spent on proto serialization. This class allows you to directly send raw gRPC::ByteBuffer as data, which can help improve performance in specific scenarios.

Java is a popular programming language that offers various techniques to improve performance. You can use non-blocking stubs to parallelize Remote Procedure Calls (RPCs).

To manage threads efficiently, Java provides a custom executor feature. This allows you to limit the number of threads based on your workload. For example, you can use a cached executor as the default, or choose from other options like fixed or forkjoin.

Here are some thread management options in Java:

By choosing the right executor for your workload, you can optimize your Java application's performance and efficiency.

Python is a popular choice for gRPC, but it has some quirks. Streaming RPCs create extra threads for receiving and possibly sending messages, making them much slower than unary RPCs.

Using asyncio can improve performance, which is a good thing to know if you're working with Python. This is because asyncio helps handle tasks asynchronously, reducing the overhead of threads.

Avoid using the future API in the sync stack if possible, as it results in the creation of an extra thread. This can lead to performance issues, especially in streaming RPCs.

If you're looking for a performance boost, you might want to try the experimental single-threaded unary-stream implementation. This is available via the SingleThreadedUnaryStream channel option, which can save up to 7% latency per message.

Explore further: Golang vs Python Performance

In this article, we're taking a closer look at the programming languages under test. C++ is one of the languages being continuously tested as both a client and a server.

The following languages are currently being tested: C++, Java, Go, C#, Node.js, Python, and Ruby. These languages are being tested as both clients and servers.

These languages are also being tested against a C++ server and a C++ client, which helps to provide the current upper bound of performance for each language's client or server implementation. This test does not involve testing the other side of the performance.

Here are the languages under test in a list:

PHP and mobile environments are not supported for gRPC servers, but their client-side performance can be benchmarked using a proxy WorkerService written in another language. This code is implemented for PHP, but it's not yet in continuous testing mode.

In the world of programming languages, two dominant protocols stand out for implementing communications between microservices: REST and gRPC.

REST executes over HTTP, typically using JSON as a format to transport request payloads. This can result in larger payload sizes, making data transport less efficient. In fact, in benchmark tests, Protocol Buffers used by gRPC produced payloads that were approximately 1/3rd the size of the same data formatted with JSON.

A fresh viewpoint: Grpc vs Rest Performance

gRPC, on the other hand, executes over HTTP/2 and formats payloads using Protocol Buffers. This cross-platform data format is used to serialize structured data, making it a more efficient choice for data transport.

The choice between REST and gRPC ultimately depends on the specific needs of your project. If you're working on a high-performance application, gRPC's Protocol Buffers may be the better choice. However, if simplicity and ease of implementation are more important, REST may be the way to go.

Here's a quick comparison of the two protocols:

Reusing gRPC channels is a simple yet effective way to improve performance. This allows calls to be multiplexed through an existing HTTP/2 connection, reducing the number of network round-trips required to complete a call.

gRPC channels are safe to share and reuse between gRPC calls, making it easy to implement this optimization. Multiple gRPC clients can be created from a channel, and clients created from the channel can make multiple simultaneous calls.

Here's an interesting read: Grpc Channel

To take advantage of this optimization, consider using a gRPC client factory, which automatically reuses underlying channels. This can help reduce the overhead of creating new channels for each gRPC call.

By reusing gRPC channels, you can significantly reduce the time it takes to complete gRPC calls and improve the overall performance of your app.

Here are some key benefits of reusing gRPC channels:

Flow control is a feature that prevents apps from being overwhelmed with data. This can happen when receiving large messages, especially if the buffer window is smaller than the incoming message payloads or there's latency between the client and server.

Flow control activates if the buffer window is filled up, causing the sending app to pause sending more data. Once the receiving app has processed data, space in the buffer window is available, and the sending app resumes sending data.

To fix flow control performance issues, you can increase the buffer window size. In Kestrel, this is configured with InitialConnectionWindowSize and InitialStreamWindowSize at app startup. It's recommended to set the connection window size to be equal to or greater than the stream window size.

Discover more: Why Is It Important to Perform Data Analysis on Datasets

If a gRPC service often receives messages larger than 768 KB, Kestrel's default stream window size, consider increasing the connection and stream window size. For example, if you're receiving large messages, you might want to increase the connection window size to 1024 KB and the stream window size to 1024 KB as well.

Here's a summary of the recommended buffer window sizes:

Increasing Kestrel's window size allows Kestrel to buffer more data on behalf of the app, which possibly increases memory usage. Avoid configuring an unnecessarily large window size.

Load balancing is a crucial aspect of performance optimization, especially when working with gRPC. gRPC uses HTTP/2, which can make it challenging for traditional L4 load balancers to distribute traffic effectively.

gRPC load balancers operate at a connection level, distributing TCP connections across endpoints, but this approach doesn't work well with gRPC's multiplexing feature. This is because gRPC sends all calls over a single TCP connection to one endpoint.

Take a look at this: Grpc Load Balancing

There are two effective options for load balancing gRPC: client-side load balancing and L7 proxy load balancing. Client-side load balancing involves the client selecting a different endpoint for each gRPC call, which can be beneficial when latency is important.

Client-side load balancing requires each client to keep track of available endpoints, which can be cumbersome. Lookaside client load balancing is a technique that stores load balancing state in a central location, allowing clients to query for information when making decisions.

Using a proxy is a simpler approach to load balancing, but it adds extra latency to gRPC calls. L7 proxies work at a higher level than L4 proxies, understanding HTTP/2 and distributing gRPC calls across multiple backend endpoints.

Some popular L7 proxies include Envoy, Linkerd, and YARP: Yet Another Reverse Proxy. These options can help simplify the load balancing process and improve overall performance.

Here are some key differences between client-side load balancing and proxy load balancing:

Hardware limitations can be a major bottleneck in performance optimization. Our testing was limited to 2 CPUs, which is a significant constraint.

We had to use 9 CPUs on the client side just to generate enough traffic for our benchmarks. This highlights the importance of considering hardware limitations when planning performance optimization efforts.

The physical machine used for testing had a single Intel Xeon CPU E5-1650 v3 @ 3.50GHz processor. This processor is a powerful one, but it's still limited by its architecture.

The machine had 32GB of DDR4 2400 MHz memory, which is a decent amount for most tasks. However, it's not enough to take full advantage of the processor's capabilities.

Here's a summary of the machine's specs:

Inter-process communication can be a game-changer for gRPC performance, especially when the client and service are on the same machine.

Using a transport like Unix domain sockets or named pipes can significantly improve efficiency, as they are designed specifically for inter-process communication.

This approach can lead to faster communication and better performance, making it a great option for applications that require frequent communication between processes on the same machine.

Client-side load balancing is a technique where the client selects a different endpoint for each gRPC call. This approach is a good choice when latency is important, as the call is sent to the service directly without a proxy.

The client must keep track of the available endpoints to use. This can be a challenge, as each client needs to maintain its own list of endpoints.

Lookaside client load balancing is a technique where load balancing state is stored in a central location. Clients periodically query this central location for information to use when making load balancing decisions.

For gRPC, client-side load balancing is a viable option when working with HTTP/2. This is because gRPC multiplexes multiple calls on a single TCP connection, making it harder for L4 load balancers to work effectively.

You might enjoy: Cilium Grpc Load Balancing

gRPC calls between a client and service are usually sent over TCP sockets, but this can be inefficient when the client and service are on the same machine.

TCP is great for communicating across a network, but it's not the best choice for inter-process communication (IPC) between processes on the same machine.

Consider using a transport like Unix domain sockets or named pipes for gRPC calls between processes on the same machine.

For more information, see Inter-process communication with gRPC.

The RPC (Remote Procedure Call) being tested is a simple unary call, also known as an echo service. This means that each request simply echoes back the data it received.

The RPC is designed to facilitate the implementation of gRPC/Proto layer testing. This approach allows developers to start with a variety of technologies by simply grabbing the implementation from their respective examples.

Here are the key characteristics of the tested RPC:

The metrics from the RPC testing represent the gRPC/Proto layer only, and do not account for the service logic. This means that the actual performance of a real-world service may be more resource-intensive and time-consuming.

Several important scenarios are under test, including contentionless latency, QPS (messages/second rate), and scalability for selected languages. These scenarios are displayed in the dashboards and most performance testing is using secure communication and protobufs.

The benchmark results show that servers with higher throughput have lower latency, but exceptions are Scala, Kotlin, and Go, which have a significant spike in p99 latency. This means 99% of requests should be faster than the given latency.

The full benchmark results can be found on the GitHub wiki, and they illustrate the performance of different languages and protocols. The results show that gRPC and HTTP/protobufs provide significantly greater throughput under increasing loads compared to REST.

In fact, with 500 client threads, the large payload size for REST provides approximately 1% of the throughput of the small input size. In contrast, gRPC's performance is most resilient, outperforming HTTP/protobufs by around 25-30% and REST by more than 9x for large payload sizes.

Explore further: Grpc over Http

Here's a breakdown of the performance of the three test configurations:

The stress test revealed little difference between gRPC and HTTP/protobufs, but REST response times were approximately 5x the other two servers, with a p99 of 30 seconds. This means that 99% of requests should be faster than 30 seconds.

The test setup and execution for gRPC performance is quite fascinating. To run the benchmark, you'll need Docker, as it's the only prerequisite mentioned.

The entire benchmark is based on Docker, and it's recommended to run it on a Linux machine to avoid introducing virtual machine uncertainty, which is present on Docker Desktop for Mac and Windows.

For simulating the client-side, we use containerized ghz, a gRPC benchmarking and load testing tool written in Go. This tool allows us to parametrize the client side with various options, including the number of connections to use, number of requests to send concurrently, CPU usage limit, and payload size.

The server side runs the containerized service in each programming language/gRPC library, with a simple protobuf contract as the common ground. We can limit the server side resources, such as CPU and RAM.

Tests are run sequentially with an optional warm-up phase to compensate for non-optimal startup, like before JIT kicks in. The stats from Docker and ghz are then collected, parsed, and combined by a Ruby script to be presented in a table with the language/library name, requests per second, average latency, and 90/95/99 percentiles.

Here's an overview of the test setup:

In alternative scenarios, the benchmark results can vary significantly depending on the workload. We've run two additional tests to demonstrate this.

The first test, simulating a workload with many clients making small requests, used 250 client connections instead of 50 and a concurrency value of 500 instead of 1000. The test duration was 60 seconds, and the payload size remained at 100B.

The second test, which emulates an on-demand routine with large payloads, used 5 client connections instead of 50 and a concurrency value of 25. The payload size was increased to 1KB, and the test duration was reduced to 30 seconds.

A unique perspective: Grpc New Client