Azure Karpenter on AKS: A Guide to Optimization

Azure Karpenter on AKS is a game-changer for optimizing your containerized workloads. By leveraging the power of Karpenter, you can automate the scaling of your Kubernetes clusters on Azure Kubernetes Service (AKS).

Karpenter is designed to work seamlessly with AKS, allowing you to create highly available and scalable clusters with minimal configuration. With Karpenter, you can automate the provisioning of nodes and scaling of your cluster based on demand.

To get started with Karpenter on AKS, you'll need to create a Karpenter cluster configuration, which involves specifying the node configuration, scaling policies, and other settings. This configuration will serve as the foundation for your automated scaling and provisioning process.

Intriguing read: Azure Kubernetes Service vs Azure Container Apps

To get started with Azure Karpenter, you'll need an active Azure subscription with Owner permissions. Ensure you have the necessary prerequisites in place, including Azure CLI, kubectl, and kubelogin installed on your system.

Before diving into the next steps, make sure you have the following prerequisites:

You'll also need to download the azure-vote-all-in-one-redis.yaml file from the Azure-Samples repository on GitHub.

To get started, you'll need to have certain prerequisites in place.

You'll need an active Azure subscription with Owner permissions to create and manage resources in Azure Kubernetes Service (AKS).

You can install Azure CLI by following the instructions at Microsoft's official documentation.

Kubectl and kubelogin are also required, which can be installed by following the instructions in the provided documentation.

Visual Studio Code is optional but recommended for editing files and managing your project.

Here are the prerequisites you'll need to have in place:

To get started with Karpenter on Azure Kubernetes Service (AKS), you'll need an active Azure subscription with Owner permissions. This will allow you to create and manage resources in AKS.

You'll also need to install Azure CLI, kubectl, and kubelogin to interact with Azure services from the command line and log in to AKS using Entra ID credentials. Visual Studio Code is optional but can be helpful for editing files and managing your project.

For your interest: Azure Aks Architecture

To implement Karpenter in a demo AKS environment, you can install Managed Karpenter (also known as Node Auto Provisioning). This feature runs Karpenter as a managed addon similar to Managed Cluster Autoscaler.

Note that there are specific limitations to consider when using Node Auto Provisioning, including the requirement for Cilium + Overlay + Azure network configuration and the inability to use Windows node pools or apply custom configuration to the node kubelet.

Before diving in, make sure you have the necessary prerequisites in place, including an active Azure subscription, Azure CLI, kubectl, and kubelogin. You can also refer to the official GitHub repository for Karpenter for more information on getting started.

Here are the specific steps to follow:

By following these steps, you'll be well on your way to getting started with Karpenter on AKS.

Karpenter on AKS is a powerful combination that can help you scale your cluster on demand. You can install Self Hosted Karpenter on an Azure AKS cluster using the official GitHub repo.

To get started, you'll need to install the necessary tools, including Azure CLI, kubectl, Helm, jq, and yq. Once you have these tools installed, you can create a new AKS cluster with the required configuration, ready to run Karpenter using workload identity.

The Karpenter Helm chart requires some configuration via values to work with a specific AKS cluster. You can use the configure-values.sh script to generate karpenter-values.yaml with the required configuration.

Here are the key environment variables you'll need to set:

With these variables set, you can install Karpenter using Helm. A single Karpenter NodePool is capable of handling many different pod shapes, making scheduling and provisioning decisions based on pod attributes such as labels and affinity.

To scale up your deployment, you can create a default NodePool using the Karpenter Helm chart. This will configure Karpenter to reduce cluster cost by removing and replacing nodes.

For your interest: Karpenter Azure

Node Autoprovisioning is an efficient way to optimize performance and cost in Kubernetes clusters. It can scale up and down worker nodes faster than Cluster Autoscaler.

Karpenter is an efficient node autoscaler for Kubernetes clusters, designed to optimize performance and cost. It can scale up and down worker nodes faster than Cluster Autoscaler and can launch individual nodes without creating traditional node groups in AKS.

A single Karpenter NodePool is capable of handling many different pod shapes. Karpenter makes scheduling and provisioning decisions based on pod attributes such as labels and affinity.

Key Features of Karpenter include:

Karpenter is now active and ready to begin provisioning nodes. It can reduce cluster cost by removing and replacing nodes when they are underutilized. This behavior can be disabled by setting consolidateAfter to Never.

There are two modes to use Karpenter provider for AKS:

Configuring Karpenter is a crucial step in getting the most out of this efficient node autoscaler for Kubernetes clusters. You can customize Karpenter config by leveraging new resource types in Kubernetes, such as Nodepools.

To customize Nodepools, you can specify specific VM series, VM family, or even specific CPU or Memory ratio. You can also select nodes based on feature sets like GPU enable or Network Acceleration. Additionally, you can define the architecture of CPU type, either ARM or AMD, based on the capability of specific workloads.

Take a look at this: Kubernetes Azure

To configure Nodepools, you can use the default Nodepool YAML for Karpenter (NAP), which has configurations on Node SKU types and capacity, as well as limits on nodes CPU:Memory along with Weight in case of multiple Nodepools.

To install Self-Hosted Karpenter on an Azure AKS cluster, you'll need to follow the official GitHub repository instructions. This involves installing utilities like Azure CLI, kubectl, Helm, jq, and yq.

Once you have the necessary tools installed, you can create a new AKS cluster with the required configuration and enable workload identity. You'll also need to create a federated credential linked to the Karpenter service account for auth usage.

When using the Helm chart, you'll need to configure values to work with a specific AKS cluster. You can use the `configure-values.sh` script to generate a `karpenter-values.yaml` file with the required configuration.

Here are some key values to include in the `karpenter-values.yaml` file:

With these values in place, you can install Karpenter using Helm and create a default NodePool. Karpenter will then begin provisioning nodes based on pod attributes like labels and affinity.

Cost Optimization is a crucial aspect of Azure Karpenter, and one way to achieve it is by utilizing Reserved Instance VM's. This can be done by creating a separate NodePool with the highest weight value and specifying all Reserved Instance VM SKU families or explicit SKU names.

To do this, you can use the karpenter.azure.com/sku-name or karpenter.azure.com/sku-family parameter in the spec section of your NodePool configuration. This will allow you to specify different VM series along with multiple VM SKUs.

For example, you can specify the following Reserved Instance VM SKU families or explicit SKU names: Standard_D2s_v3, Standard_D4s_v3, Standard_D8s_v3, Standard_D16s_v3, Standard_D32s_v3, Standard_D64s_v3, Standard_D96s_v3

To enable Node Auto Provisioning (NAP) on Azure Kubernetes Service (AKS), you'll need to install the Az CLI with a preview extension version greater than 0.5.17.

First, ensure your existing AKS cluster has the 'Azure' network plugin with Cilium as Network Policy. This is crucial for NAP to work correctly.

To enable NAP on an existing AKS cluster, you'll need to run a command with the feature flag '--node-provisioning-mode Auto', which sets NAP as the default Node Autoscaler.

You can verify Karpenter enablement by checking the AKS cluster's configuration.

Here's a quick rundown of the requirements for enabling NAP on AKS:

Note that enabling NAP on a cluster with node pools that have the cluster autoscaler enabled is not supported.

Utilizing Reserved Instance VMs can help you save cost. This is particularly useful when you have a consistent workload that requires a specific set of VMs.

By creating a separate NodePool with a high weight value, you can specify all Reserved Instance VM SKU families or explicit SKU names. This allows you to reserve a set of VMs for your workload.

To do this, use the karpenter.azure.com/sku-name or karpenter.azure.com/sku-family parameter in your NodePool configuration. For example, you can specify the following VMs: Standard_D2s_v3, Standard_D4s_v3, Standard_D8s_v3, Standard_D16s_v3, Standard_D32s_v3, Standard_D64s_v3, and Standard_D96s_v3.

Here are some examples of Reserved Instance VMs you can use:

By reserving these VMs, you can ensure that your workload has access to the resources it needs while also saving cost.

Using the CLI is another option for creating a Kubernetes cluster and enabling necessary addons, such as Cilium, to run Karpenter.

You can use the Azure CLI to create a cluster and enable Cilium, just like the Terraform configuration does.

Related reading: Azure Cli vs Azure Powershell

The Azure CLI can be used to create the cluster, enable Cilium, and add Azure as the plugin, which is similar to the Terraform configuration.

Here's an example of how to do it: you can use the Azure CLI to create the cluster and enable Cilium, just like the example provided.

To deploy an app with Azure Karpenter, you need to scale replicas of your application to trigger scale out events. This will automatically trigger the auto-scaling of nodes.

You can scale replicas of the Vote Application to see this in action. This will help you understand how Azure Karpenter auto-provisions nodes.

After scaling the replicas, you can verify the auto-scaling of nodes by reading via Karpenter using a specific kubectl command. This command will help you confirm that the nodes are being scaled as expected.