Cilium Grpc Load Balancing on Kubernetes for Scalable Applications

Cilium provides a powerful solution for load balancing gRPC traffic in Kubernetes environments.

Cilium's eBPF-based load balancing solution can handle tens of thousands of connections per second, making it an ideal choice for scalable applications.

Cilium's load balancing solution is highly scalable and can handle a large number of connections without sacrificing performance.

With Cilium, you can achieve high availability and reliability for your gRPC services, even in the presence of node failures or network partitions.

Cilium's load balancing solution is also highly customizable, allowing you to define complex routing rules and policies to suit your specific use case.

Broaden your view: Grpc Load Balancing

To get started with Cilium GRPC load balancing, you'll need to meet a few prerequisites.

First, make sure Cilium is configured with NodePort enabled, either by setting nodePort.enabled=true or by enabling the kube-proxy replacement with kubeProxyReplacement=true.

Cilium must also be configured with the L7 proxy enabled, which is enabled by default.

You'll need an environment that supports LoadBalancer type Services, or you can change this to NodePort or expose the Cilium L7 proxy on the host network, which is possible since Cilium 1.16+.

Suggestion: Grpc Proxy

Cilium is designed to work seamlessly with Kubernetes, providing advanced networking and security features.

Cilium's Kubernetes integration allows for the creation of network policies that can be enforced across the entire cluster, giving developers fine-grained control over traffic flow.

This integration also enables Cilium to automatically detect and manage the network connectivity between pods.

Cilium Service Mesh is a powerful tool for Kubernetes environments. It provides connectivity, load-balancing, security, and observability by operating at both the networking and application protocol layer.

Cilium uses eBPF as its highly efficient in-kernel datapath for all network processing, including protocols like IP, TCP, and UDP. This is a game-changer for network performance.

One of the key benefits of Cilium is its support for Kubernetes Ingress and Gateway API. This allows for seamless integration with existing infrastructure.

Here are some of the key features that make Cilium stand out:

These features make Cilium a robust and flexible solution for Kubernetes environments.

Kubernetes is an open-source container orchestration system for automating the deployment, scaling, and management of containerized applications.

It was created by Google, with contributions from many other companies and individuals.

Kubernetes is designed to work with a wide range of container runtimes, including Docker.

It supports multiple cloud providers, including Amazon Web Services, Microsoft Azure, and Google Cloud Platform.

Kubernetes provides a lot of flexibility and customization options.

You can choose from a variety of networking and storage solutions to suit your needs.

Cilium integrates seamlessly with Kubernetes, allowing you to manage and secure your network traffic with ease.

Cilium provides network policy and visibility features that are tightly integrated with Kubernetes.

This allows you to define policies that control traffic flow and ensure the security and integrity of your applications.

Kubernetes provides a robust set of APIs and tools for managing and scaling your applications.

You can use these APIs to automate the deployment and scaling of your applications, as well as monitor their performance and health.

Related reading: Google Documents Not Loading

Cilium complements Kubernetes by providing advanced network policy and visibility features.

These features allow you to define fine-grained policies that control traffic flow and ensure the security and integrity of your applications.

Kubernetes and Cilium work together to provide a powerful and flexible infrastructure for managing and securing your applications.

To create a gRPC service on servers, you'll need to create a file named grpc-server-svc.yaml with specific content.

The file should contain the apiVersion, kind, metadata, labels, and spec sections, with the name of the service starting with "grpc" and the port number set to 9996.

Here are the key details to include in the grpc-server-svc.yaml file:

spec:

selector:

After creating the file, you can apply it to the cluster using the command kubectl apply -f grpc-server-svc.yaml.

To run XDP on AWS, you need to set up an EKS cluster using the Cilium Quick Installation guide. Make sure to create a node group with SSH access, as you'll need it for additional setup steps.

Use a larger instance type that supports the Elastic Network Adapter (ena), such as m5n.xlarge. This is specified in the config nodegroup-config.yaml.

Each node needs the kernel-ng and ethtool package installed. The former is required for eBPF and native XDP support on the ena driver, while the latter is needed to configure channel parameters for the NIC.

The kernel version should be 5.4.58-27.104.amzn2.x86_64 or similar after the nodes come back up. You can check this using uname -r.

The ena driver version must be at least 2.2.8 to run XDP on ena. You can inspect the driver version using ethtool -i eth0, and it should report 2.2.10g for the given kernel version.

To operate with XDP, you need to lower the MTU to 3498, as the default MTU is 9001 on the ena driver. This is because XDP buffers are linear and operate on a single page.

You also need to set the number of combined channels to at most 1/2 of the value from Combined. This can be done as follows.

Finally, to enable XDP in Cilium, you need to upgrade and roll out the deployment with the loadBalancer.acceleration=native setting.

A different take: Grpc Version

To configure Cilium for gRPC load balancing, you'll need to create a CiliumNetworkPolicy. This policy defines the network traffic flow rules for your gRPC services.

CiliumNetworkPolicy uses a combination of labels and selectors to identify the pods and services that should be included in the policy. The example policy shown in the article uses labels to match the pods and services.

To enable gRPC load balancing, you'll also need to configure the CiliumEnvoy configuration. This involves setting the envoy.filters.http.grpc and envoy.filters.network.grpc settings to enable the gRPC filters.

In Step 4, we'll deploy the gRPC service and the eight Deployments.

To start, we need to create a namespace named grpc-best in the ACK cluster. This is done by running the command `k create ns grpc-best`.

We then enable automatic sidecar injection for the namespace by running `k label ns grpc-best istio-injection=enabled`.

Next, we deploy the gRPC service and the eight Deployments by running the following commands:

These commands apply the configurations for each of the Deployments, making them ready for use.

For high-scale environments, you can configure Cilium's BPF maps to have higher limits on the number of entries.

You can override Helm options to tweak these limits, specifically the bpf.lbMapMax option.

The default value of this LB map size is 65536, which is a good starting point for most setups.

In some cases, you may need to increase the number of entries in Cilium's BPF LB service, backend, and affinity maps.

To do this, you'll need to override the bpf.lbMapMax Helm option.

You can verify load balancing among gRPC servers by using an ingress gateway.

To create an ingress gateway, expose the port 9996 and use the following YAML code to create an Istio gateway: apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

namespace: grpc-best

name: grpc-gateway

spec:

selector:

istio: ingressgateway

servers:

- port:

number: 9996

name: grpc

protocol: GRPC

hosts:

- "*"

You can also create a virtual service to route requests to the gRPC servers, using the following YAML code: apiVersion: networking.istio.io/v1alpha3

Recommended read: Grpc Gateway

kind: VirtualService

metadata:

namespace: grpc-best

name: grpc-vs

spec:

hosts:

- "*"

gateways:

- grpc-gateway

http:

host: grpc-server-svc.grpc-best.svc.cluster.local

port:

number: 9996

To verify load balancing among the gRPC servers, run the following commands to obtain the IP address of the ingress gateway: INGRESS_IP=$(k -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Here's a step-by-step process to verify load balancing among the gRPC servers:

1. Run the following command to obtain the IP address of the ingress gateway: INGRESS_IP=$(k -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

2. Use a FOR loop to verify load balancing among the gRPC servers, as shown in the following example: for ((i = 1; i <= 100; i++)); do

docker exec -e GRPC_SERVER="${INGRESS_IP}" -it "$client_node_container" node kube_client.js >> kube_result

done

sort kube_result | grep -v "^[[:space:]]*$" | uniq -c | sort -nrk1

The expected output indicates that the four gRPC servers where the gRPC service is deployed receive an approximate number of requests.

Here's a comparison of the load balancing results using different methods:

Cilium's advanced features enable seamless integration with popular frameworks like gRPC.

Cilium provides native support for gRPC load balancing, allowing for efficient routing of requests.

With Cilium, you can take advantage of features like connection pooling, which reduces the number of connections required for a given service. This results in improved performance and reduced latency.

Annotation-based load balancing algorithm selection is a powerful feature that allows you to choose the load balancing method for your services.

By default, if no service annotation is provided, the logic falls back to use the global load balancing method specified through loadBalancer.algorithm.

The global load balancing method supports two values: random and maglev, with random being the default if not explicitly set.

To add a new service that must use random as its load balancing algorithm, you can simply omit the service annotation.

Worth a look: When to Use Grpc

XDP Acceleration is a game-changer for high-scale environments. It enables acceleration for NodePort, LoadBalancer services, and services with externalIPs by operating directly in the networking driver at the XDP layer.

Cilium's built-in support for XDP acceleration was introduced in version 1.8, where eBPF is operating directly in the networking driver instead of a higher layer. This allows for faster packet processing and reduced latency.

To enable XDP acceleration, set loadBalancer.acceleration to option native. The default setting is disabled, which disables the acceleration. Most drivers supporting 10G or higher rates also support native XDP on a recent kernel.

For cloud-based deployments, most drivers have SR-IOV variants that support native XDP as well. On-prem deployments can use Cilium XDP acceleration in combination with LoadBalancer service implementations for Kubernetes, such as MetalLB.

In a multi-device environment, XDP acceleration is enabled on all devices, but each underlying device's driver must have native XDP support on all Cilium managed nodes. If some devices support XDP but others do not, you can have XDP enabled on the supported devices by setting loadBalancer.acceleration to best-effort.

A list of drivers supporting XDP can be found in the XDP documentation. To check if XDP acceleration has been enabled successfully, use the cilium-dbgstatus CLI command, which will show "Native" if it has been enabled.

Expand your knowledge: Nextjs Loader

Traffic and Topology Aware Hints allow for more efficient routing of traffic within a Kubernetes cluster. This is achieved through Kubernetes Topology Aware Routing and Traffic Distribution features.

The kube-proxy replacement implements both of these features, which work by setting hints on EndpointSlices to enable Cilium to route to endpoints residing in the same zone. This is done by setting the loadBalancer.serviceTopology=true configuration.

To take advantage of this feature, you'll need to deploy a client pod to start traffic. By doing so, you can see firsthand how Traffic and Topology Aware Hints improve the routing of traffic within your cluster.

Worth a look: Azure Traffic Manager vs Load Balancer

External access to ClusterIP services is restricted by default in Cilium's eBPF kube-proxy replacement. This means that access to a ClusterIP service from outside the cluster is not allowed.

However, this restriction can be lifted by setting the `bpf.lbExternalClusterIP` flag to `true`. This allows external access to the ClusterIP service.

If you're looking to verify load balancing among gRPC servers, you can use an ingress gateway. This involves creating an ingress gateway, an Istio gateway, and a virtual service, as well as running a command to obtain the IP address of the ingress gateway.

Suggestion: Azure Gateway Load Balancer

Cilium's Ingress and Gateway API support is tightly integrated with the CNI, which sets it apart from other Ingress controllers.

This means that Ingress and Gateway API are part of the networking stack, behaving differently compared to other Ingress or Gateway API controllers, even those running in a Cilium cluster.

If this caught your attention, see: Azure Load Balancer vs Application Gateway

Other Ingress or Gateway API controllers are typically installed as a Deployment or Daemonset in the cluster, exposed via a Loadbalancer Service or similar.

Cilium's Ingress and Gateway API config, on the other hand, can be exposed with a Loadbalancer or NodePort service, or optionally on the Host network.

This affects client IP visibility, which works differently for Cilium's Ingress and Gateway API support compared to other Ingress controllers.

Cilium's Network Policy engine can also apply CiliumNetworkPolicy to traffic bound for and coming from an Ingress, which is a unique feature.

LoadBalancer services can be configured with spec.loadBalancerSourceRanges to restrict access from outside the cluster to white-listed CIDRs.

This means that traffic from outside the cluster will be allowed only if it's coming from one of the specified CIDRs. If the field is empty, no restrictions will be applied.

Traffic from inside the cluster, however, is not restricted by this field, regardless of whether it's set. Any pod or host process in the cluster will be able to access the LoadBalancer service internally.

By default, the specified white-listed CIDRs only apply to the LoadBalancer service, but not to the corresponding NodePort or ClusterIP service. However, this can be changed by enabling the option to propagate the white-listed CIDRs to all externally exposed service types.

External access to ClusterIP services is a bit tricky by default in Cilium. As per k8s Service, Cilium's eBPF kube-proxy replacement disallows access to a ClusterIP service from outside the cluster.

This means you can't just access a ClusterIP service from outside the cluster. To allow external access, you need to set the bpf.lbExternalClusterIP=true flag.

This flag tells Cilium to allow external access to ClusterIP services, which can be useful in certain scenarios.

A unique perspective: C# Grpc Service

You can use Hubble to trace socket LB related datapath events and inspect network flows with the CLI guide. This will print datapath events before and after socket LB translation between service and selected service endpoint.

Socket LB tracing with Hubble requires the cilium agent to detect pod cgroup paths. If you encounter a setup failure, you can trace packets using cilium-dbg-monitor instead.

To troubleshoot socket load-balancing setup failures, you can file a GitHub issue with the cgroup path for any of your pods, obtained by running the command `sudocrictl inspect p-o=json $POD_ID | grep cgroup`. This will help you identify the client pod using its printed cgroupid metadata.

To verify the result of load balancing among gRPC servers, you can use either pods or an ingress gateway. The output indicates that the four gRPC servers receive an approximate number of requests, with a load balancing result that shows ASM routing external requests to the servers based on a policy.

Using pods, you can run a FOR loop to send requests from the pods of the gRPC clients to the gRPC service on the four gRPC servers. The expected output is a list of the number of requests each server received, with the results indicating that the load balancing is working correctly.

Here's a breakdown of the expected output:

This output shows that the four gRPC servers are receiving an approximately equal number of requests, indicating that the load balancing is working as expected.

Alternatively, you can use an ingress gateway to verify the load balancing result. This involves creating an ingress gateway that exposes the port 9996 and using a FOR loop to send requests to the gateway. The expected output is the same as when using pods, with the four gRPC servers receiving an approximately equal number of requests.

Observability is crucial for troubleshooting issues in your cluster. You can use Hubble to trace socket LB related datapath events and inspect network flows.

To do this, follow the Hubble Inspecting Network Flows with the CLI guide. The Hubble output will print datapath events before and after socket LB translation between service and selected service endpoint. This will help you understand the network flow and identify potential issues.

If you encounter an error message "Failed to setup socket load-balancing tracing with Hubble", you can use cilium-dbgmonitor instead. This will allow you to trace packets and continue troubleshooting.

When you see a message about socket load-balancing setup failure in the logs, it's essential to file a GitHub issue with the cgroup path for any of your pods. To obtain the cgroup path, run the command "sudocrictl inspect p-o=json $POD_ID | grep cgroup" on a Kubernetes node in your cluster.

As you dive into troubleshooting, it's essential to be aware of potential issues that might arise. Service backend entries could be leaked in the BPF maps in clusters deployed with Cilium version 1.11.14 or earlier.

This can happen due to race conditions between the deletion of a service backend while it's terminating, and the simultaneous deletion of the service it's associated with. This could lead to duplicate backend entries that fill up the cilium_lb4_backends_v2 map.

You might see error messages like these in the Cilium agent logs: "Error: duplicate backend entry". The leak was fixed in Cilium version 1.11.15, but affected clusters might not see the leaked backends cleaned up after restarting the Cilium agent.

The fixes to clean up leaked duplicate backend entries were backported to older releases, and are available as part of Cilium versions v1.11.16, v1.12.9, and v1.13.2. Fresh clusters deploying Cilium versions 1.11.15 or later don't experience this leak issue.

Let's dive into some examples and background information on Cilium gRPC load balancing.

Cilium's Ingress features are used to route traffic to gRPC servers.

You can leverage Cilium's Ingress features in various ways, such as with HTTP, network policies, and external lock-down policies.

A different take: Ingress Load Balancer Azure

Here are some examples of how to use Cilium's Ingress features:

In a gRPC service, four clients can use different programming languages to call the grpc-server-svc.grpc-best.svc.cluster.local service.

This service is specified by the GRPC_SERVER variable, and ASM routes the internal requests evenly to the four gRPC servers.

You can also configure an ingress gateway to route external requests to the four gRPC servers based on a load balancing policy.