Getting Started with Elk Stack Kubernetes

Author

Reads 1.3K

Modern data center corridor with server racks and computer equipment. Ideal for technology and IT concepts.
Credit: pexels.com, Modern data center corridor with server racks and computer equipment. Ideal for technology and IT concepts.

The Elk Stack, comprising Elasticsearch, Logstash, and Kibana, is a powerful logging and analytics toolset that can be deployed on Kubernetes for scalable and efficient data processing.

Kubernetes, a container orchestration system, is ideal for managing the Elk Stack's complex components.

To get started, you'll need to install a Kubernetes cluster, such as Minikube or a cloud-based service like Google Kubernetes Engine (GKE).

With your Kubernetes cluster up and running, you can then deploy the Elk Stack components, including Elasticsearch, Logstash, and Kibana, using a Helm chart or a Kubernetes Deployment YAML file.

Prerequisites

Before we dive into the world of ELK Stack on Kubernetes, let's make sure you have the necessary tools in place. To get started, you'll need a running Kubernetes cluster. This can be set up using Minikube, EKS, GKE, or any other variant.

You'll also need to have the kubectl command-line tool configured to interact with your cluster. This will allow you to manage and deploy applications on your Kubernetes cluster.

Engineer fixing core swith in data center room
Credit: pexels.com, Engineer fixing core swith in data center room

Additionally, you'll need to install Lens software to connect and see all the information related to your cluster.

Here are the specific prerequisites you'll need:

  • A running Kubernetes cluster (Minikube, EKS, GKE, etc.)
  • kubectl command-line tool configured to interact with your cluster
  • Lens software to connect and see all the information related to the cluster

Setting Up the ELK Stack

To set up the ELK Stack, you'll need to start with a Kubernetes cluster. You can use Minikube to create a single-node cluster, but make sure to allocate extra resources as you'll be deploying a multi-node Elasticsearch cluster.

First, you'll need to deploy Elasticsearch, which stores your log data. You can do this by applying a YAML file, such as "elasticsearch.yaml", to your Kubernetes cluster using Lens or the Terminal. This will create a ClusterIP Service for your Elasticsearch cluster.

Next, you'll need to get the Elasticsearch credentials, which are stored in a Kubernetes secret. You can find these credentials by looking at your cluster's secrets in Lens. The default user is "elastic", and the password is stored in the secret.

Two elk clashing antlers in a field, captured in their natural habitat.
Credit: pexels.com, Two elk clashing antlers in a field, captured in their natural habitat.

To access Elasticsearch, you can use the Elasticsearch endpoint, which is available at "https://localhost:9200". However, be aware that the self-signed certificate is not trusted by your browser, so you may need to temporarily acknowledge the warning.

Once you have Elasticsearch up and running, you can deploy Kibana, which provides a web interface for visualizing your log data. You can create a YAML file, such as "kibana.yaml", to deploy Kibana and associate it with your Elasticsearch cluster.

To access Kibana, you can use the Kibana endpoint, which is available at "https://localhost:5601". Again, be aware that the self-signed certificate is not trusted by your browser, so you may need to temporarily acknowledge the warning.

Alternatively, you can use Helm charts to deploy the ELK Stack on Kubernetes. This involves adding the Elastic Helm repository to your Helm configuration, installing Elasticsearch and Kibana, and configuring any additional components, such as Logstash.

Here's a summary of the steps to deploy the ELK Stack using Helm:

  • Add the Elastic Helm repository to your Helm configuration
  • Install Elasticsearch and configure its parameters
  • Install Kibana and associate it with your Elasticsearch cluster
  • Install Logstash, if required
  • Configure an Ingress resource, if desired
  • Access Kibana using the Kibana endpoint or port-forwarding

ELK Stack Components

Close-up of a detailed architectural floor plan focusing on technical elements and design specifications.
Credit: pexels.com, Close-up of a detailed architectural floor plan focusing on technical elements and design specifications.

The ELK Stack Components are the building blocks of a powerful logging and analytics toolset. Elasticsearch is the core component, responsible for storing and indexing log data.

The ELK stack consists of three main components: Elasticsearch, Kibana, and Logstash. Elasticsearch stores and indexes log data, Kibana provides a user-friendly interface for exploring and analyzing log data, and Logstash is an optional component used for log ingestion, processing, and enrichment.

Here's a brief overview of each component:

Each component plays a crucial role in the ELK stack, and understanding their functions is essential for deploying and configuring the stack on Kubernetes.

Logstash

Logstash is an optional component of the ELK stack used for log ingestion, processing, and enrichment. It can be installed using the Elastic Helm chart.

You can install Logstash using the following command: helm install logstash elastic/logstash --namespace. This will deploy Logstash to your Kubernetes cluster.

If you have specific log processing requirements, such as parsing structured logs or applying filters, you can configure Logstash pipelines to ingest and process your logs. You can define Logstash configuration files and mount them as ConfigMaps or use other methods for configuration management.

Cut Logs on Green Field Surrounded With Tall and Green Trees
Credit: pexels.com, Cut Logs on Green Field Surrounded With Tall and Green Trees

To create a minimal Logstash deployment, you'll need to create a YAML file with the necessary configuration. This file is typically named "logstash.yaml" and contains the necessary settings for Logstash to listen to a Beats agent or Elastic Agent.

Here is an example of what the logstash.yaml file might look like:

```yaml

elasticsearch:

hosts: ["quickstart"]

port: 9200

input:

beats:

port: 5044

output:

elasticsearch:

hosts: ["quickstart"]

index: "logstash-%{+yyyy.MM.dd}"

```

This configuration tells Logstash to listen to port 5044 for incoming Beats data and write the output to an Elasticsearch cluster named "quickstart".

Metricbeat

Metricbeat is a key component of the ELK Stack, and it's used to collect and index metrics from your Kubernetes cluster.

Deploying Metricbeat with Helm is a straightforward process that can be completed in just a minute or two.

To set up a data pipeline, you'll need to deploy the Metricbeat Helm chart, which will display Metricbeat pods running alongside your Elasticsearch and Kibana pods.

If you curl Elasticsearch, you'll see that metrics have already begun to be indexed in Elasticsearch.

Credit: youtube.com, Elasticsearch: Collecting Metrics with Metricbeat | ELK Stack Tutorial

All that's left to do now is define the index pattern in Kibana and begin analyzing your data.

Kibana will automatically identify and display the Metricbeat index when you create a new index pattern.

To finalize the creation of the index pattern, select the @timestamp field.

Once you've created the index pattern, you can hop over to the Discover page and see all the metrics being collected from your Kubernetes cluster by Metricbeat.

Filebeat and Metricbeat can be used to add extra monitoring and log collection to your Kubernetes cluster, but this step is optional.

To start Filebeat and Metricbeat daemons, you'll need to apply a directory to launch them.

Note that if you're running in GKE, you'll need to give yourself a cluster-admin role before you can fully deploy the daemons.

To collect all metrics, you'll also need to install kube-state-metrics on your cluster.

Set Up Index Templates

Set up Index Templates is an essential step in configuring your Elasticsearch cluster.

Credit: youtube.com, pfELK - Creating Logstash Indexing Template

Index templates are pre-defined configurations that determine how data is stored in Elasticsearch.

You can configure index templates using Jobs, which are scripts that run on your Elasticsearch cluster.

In the example, the author uses Filebeat to collect logs and Metricbeat to collect metrics from Kubernetes, and provides four config files for Jobs that configure index templates or install Kibana dashboards.

These Jobs will typically run once, unless they do not exit cleanly.

To get started, you can run the Jobs individually or apply the folder to run them all at once.

Once the Jobs are successful, your Elasticsearch cluster is ready to use.

Log Management

Log Management is a crucial aspect of any Kubernetes cluster, and the ELK Stack is a popular choice for managing logs. Filebeat is a lightweight shipper that can be used to forward and centralize log data.

To get started with Filebeat, you'll need to deploy it to your Kubernetes cluster. This can be done by creating a YAML file called "filebeat.yaml" and applying the following specification.

Credit: youtube.com, What is the ELK Stack | ELK | Elasticsearch | Logstash | Kibana

Filebeat can be used to collect logs from all containers running in your Kubernetes cluster. You can access these logs by navigating to the Discover tab in Kibana and selecting the Filebeat index pattern.

There are two options for accessing the logs ingested by Filebeat:

  • Follow the Elasticsearch deployment guide and run the command to deploy Filebeat.
  • Create a YAML file "filebeat.yaml" and apply the specification to deploy Filebeat.

Once Filebeat is running and shipping logs, you can access Kibana to visualize the logs. This can be done by navigating to the Discover tab in Kibana, selecting the Filebeat index pattern, and exploring your logs.

You can also use Filebeat to monitor the Pods and retrieve details about the Filebeat instance. Additionally, you can access the logs of the Filebeat Pods and the logs ingested by Filebeat.

Elastic Cloud

Elastic Cloud is built on the Kubernetes Operator pattern, which extends Kubernetes orchestration capabilities to support the setup and management of Elasticsearch and Kibana on Kubernetes.

This means you can easily manage your Elasticsearch and Kibana clusters on Kubernetes, without having to worry about the underlying infrastructure.

Our commitment to cloud native technologies is evident in our decision to bring our products to platforms like Kubernetes, the go-to open source choice for containerized architectures.

Here's an interesting read: Cloud Foundry vs Kubernetes

Elastic Cloud

Credit: youtube.com, Getting Started with Elasticsearch Service and Elastic Cloud

Elastic Cloud is built on a robust foundation that extends Kubernetes orchestration capabilities to support the setup and management of Elasticsearch and Kibana on Kubernetes.

Our Elastic Cloud offering is designed to bring our products to platforms like Kubernetes, which is the go-to open source choice for containerized architectures.

This means you can easily manage and scale your Elasticsearch and Kibana environments on Kubernetes, without worrying about the underlying infrastructure.

By leveraging the Kubernetes Operator pattern, our Elastic Cloud offering provides a seamless and efficient way to deploy and manage your Elasticsearch and Kibana clusters on Kubernetes.

Cluster Setup

To set up an Elasticsearch cluster, you'll need to start by deploying it with Helm, a package manager for Kubernetes. This involves adding Elastic's Helm repository and downloading the Helm configuration for installing a multi-node Elasticsearch cluster on Minikube.

You can verify your Elasticsearch pods status with a simple command, which will show you the status of your pods. It might take a minute or two, but eventually, three Elasticsearch pods will be shown as running.

Credit: youtube.com, Setup Elasticsearch Cluster + Kibana 8.x

To deploy Elasticsearch, you'll need to set up port forwarding, which allows you to access your Elasticsearch cluster from outside the cluster. This is an essential step in making your Elasticsearch cluster accessible.

You can deploy a large Elasticsearch cluster using Kubernetes and a few simple scripts, making it easy to create a scalable Elasticsearch cluster at a low cost. This is a significant advantage of using Kubernetes with Elasticsearch.

To start the Elasticsearch cluster, you can apply the directory 2_elasticsearch with kubectl, which will start the Elasticsearch cluster and take a little while to complete.

Cluster Setup

To set up an Elasticsearch cluster in Kubernetes, you'll need to start by deploying the different components of the ELK Stack.

You can use Elastic's Helm repository to make this process easier.

First, add the Helm repository:

You'll then need to download the Helm configuration for installing a multi-node Elasticsearch cluster on Minikube.

Next, install the Elasticsearch Helm chart using the configuration you just downloaded.

Credit: youtube.com, How to Install or Setup Elasticsearch Filebeat Logstash Kibana (ELK Stack) in Kubernetes

It might take a minute or two, but eventually, three Elasticsearch pods will be shown as running.

To verify your Elasticsearch pods status, use the command shown in the output.

Before deploying Elasticsearch, you should set up a Kubernetes Namespace for your Elasticsearch related resources.

You'll also need to define a Kubernetes StorageClass for persistent volumes used by the Elasticsearch data nodes.

Once you've done this, you can start the Elasticsearch cluster by applying the directory 2_elasticsearch with kubectl.

This will start the Elasticsearch cluster and might take a little while.

Advanced Configuration

In the Elk Stack Kubernetes setup, you can configure the ELK components to run as separate pods, each with its own configuration file. This allows for more flexibility and customization.

The Elasticsearch pod can be configured using the elasticsearch.yml file, where you can specify settings such as the cluster name, node name, and index templates. The default configuration is stored in the /etc/elasticsearch directory.

Credit: youtube.com, Zero to Hero: Deploy Elastic Stack on Kubernetes with ECK (TLS)

Kibana can be configured using the kibana.yml file, where you can specify settings such as the server host and port, as well as the Elasticsearch URL. The default configuration is stored in the /etc/kibana directory.

The Logstash pod can be configured using the logstash.yml file, where you can specify settings such as the input and output plugins, as well as the pipeline configuration. The default configuration is stored in the /etc/logstash directory.

You can also use environment variables to pass configuration settings to the ELK components. This allows for more dynamic and flexible configuration.

Kubernetes Integration

You can deploy the ELK stack on Kubernetes using Helm charts, which provide a convenient way to package and manage applications. This approach is outlined in the "Deploy the ELK Stack on Kubernetes with Helm Charts" section.

To get started, ensure you have a Kubernetes cluster running and Helm installed on your local machine, along with Tiller deployed in your Kubernetes cluster. You'll also need to add the Elastic Helm repository to your Helm configuration.

Discover more: Setting up Elk Stack

Credit: youtube.com, Filebeat + Elk Stack Tutorial With Kubernetes

The official Elastic Helm charts for deploying the ELK stack components are available, and you can access them by adding the Elastic Helm repository. This step is crucial for deploying Elasticsearch, Kibana, and Logstash.

Elasticsearch is the core component of the ELK stack, responsible for storing and indexing log data. You can customize parameters such as the number of replicas and JVM heap size when installing Elasticsearch with Helm. This flexibility allows you to tailor the deployment to your specific needs.

Kibana is the visualization and dashboarding component of the ELK stack, offering a user-friendly interface for exploring and analyzing log data stored in Elasticsearch. You can quickly deploy and configure Kibana to connect to your Elasticsearch cluster using Helm.

Logstash is an optional component used for log ingestion, processing, and enrichment. If you have specific log processing requirements, you can install Logstash using the Elastic Helm chart. This component allows for more advanced log management and processing.

To access Kibana, you can configure an Ingress resource or use port-forwarding to access it. Once you've accessed Kibana, you can define index patterns that specify which Elasticsearch indices to query. This step is essential for visualizing log data in Kibana.

Credit: youtube.com, Day-5 | Logging with EFK Stack | Elastic Search, FluentBit and Kibana

The Kubernetes Operator for Elasticsearch provides a more comprehensive solution for deploying and managing Elasticsearch and Kibana on Kubernetes. This Operator offers features such as autoconfiguration, snapshots, and custom plugins, making it an attractive option for larger-scale deployments.

Here are some key features of the Kubernetes Operator for Elasticsearch:

  • Deploy clusters in a single Kubernetes environment and connect them with clusters running in another.
  • Deploy fully-loaded clusters with all free proprietary features, including Canvas and Maps.
  • Autoconfigure features like native authentication, TLS encryption, and role-based access control (RBAC).
  • Schedule snapshots and send them to Amazon S3, Google Cloud Storage, and Azure Blob Storage.
  • Natively deploy common Elasticsearch architectures for logging, metrics, and other time-series use cases.
  • Upgrade terabyte clusters in minutes with rolling upgrades and built-in StatefulSets.
  • Add custom plugins, dictionaries, and bundles to run it your way.
  • Upgrade to an Enterprise subscription for access to premium features like machine learning, alerting, and advanced security features.

By leveraging the Kubernetes Operator for Elasticsearch, you can take advantage of these features and simplify the process of deploying and managing your ELK stack on Kubernetes.

Getting Started

To get started with the ELK stack on a Kubernetes cluster, you'll first need to define a Kubernetes Namespace for your Elasticsearch related resources. This will help keep your resources organized and separate from other applications.

Before deploying the ELK stack, you'll also need to create a Kubernetes StorageClass for persistent volumes used by the Elasticsearch data nodes. This will ensure that your data is stored securely and persistently.

You can then proceed to spin up an Elasticsearch cluster with Kubernetes, which will take just a second to give Kubernetes all the information it needs to create the cluster.

Introduction

Computer server in data center room
Credit: pexels.com, Computer server in data center room

In today's world of complex systems, logging and monitoring are crucial for maintaining application health and performance. The ELK stack is a powerful solution for aggregating and analyzing log data.

The ELK stack comprises Elasticsearch, Logstash, and Kibana, which work together to provide a comprehensive logging and monitoring system.

Just the Beginning

Now that you've set up Elasticsearch on Kubernetes, you can start exploring its full potential.

Elasticsearch can handle a wide range of data types, including network data, infrastructure logs, and documents with tons of text.

You can also run hot-warm architectures with Elastic Cloud on Kubernetes, making it easy to scale and manage your data.

This flexibility is what makes Elasticsearch on Kubernetes a game-changer for many use cases.

Claire Beier

Senior Writer

Claire Beier is a seasoned writer with a passion for creating informative and engaging content. With a keen eye for detail and a talent for simplifying complex concepts, Claire has established herself as a go-to expert in the field of web development. Her articles on HTML elements have been widely praised for their clarity and accessibility.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.