docker log to logstash with ELK Stack

The ELK Stack is a powerful tool for managing and analyzing logs from Docker containers. It's a combination of Elasticsearch, Logstash, and Kibana that works together to collect, process, and visualize log data.

ELK Stack is particularly useful for Docker logs because it can handle large volumes of log data and provide real-time insights into container activity.

To get started with ELK Stack and Docker logs, you'll need to install Elasticsearch, Logstash, and Kibana on your system.

Recommended read: Logstash

Configuring Logstash is a crucial step in setting up a log pipeline with Docker.

To accept messages in Gelf format, you'll need to create a logstash.conf file that specifies the UDP port 12201 for communication.

The hostname for Elasticsearch should be specified as 'elasticsearch' instead of 'localhost' to ensure Docker services can communicate with each other.

The index key in logstash.conf should be set to 'logstash-' followed by a date string that changes every day.

A fresh viewpoint: Elasticsearch Logs

Logstash agents expose the UDP port 12201 to receive logs from Docker containers.

A logstash.conf file is mounted in the logstash-agent directory to configure the agent to send data to a Redis instance.

The central Logstash instance takes logs from Redis and pushes them to Elasticsearch for indexing and availability.

The logstash.conf file is mounted in the logstash-central directory to configure the central instance.

Take a look at this: Azure Self Hosted Agent Docker

To set up the ELK Stack, you'll need to ensure that the ports 5601, 9200, and 5044 are free on your machine. This is because these ports are used by Kibana, Elasticsearch, and Logstash respectively.

The recommended ELK Stack Docker image is deviantony/docker-elk, which has great documentation and is fully up to date with the latest versions of Elasticsearch, Logstash, and Kibana.

Before running the stack, make sure that the vm_max_map_count kernel setting is set to at least 262144.

To run the stack, simply use the deviantony/docker-elk image and it will start all three ELK services: Elasticsearch, Logstash, and Kibana. The image also persists the Elasticsearch data directory as a volume.

Here's a quick rundown of the ports used by each ELK service:

After running the stack, you'll be required to enter an index pattern, which can only be done once you've indexed some logs.

There are a few ways to ship Docker logs into ELK, but the method you choose depends on how your logs are outputting.

You can use Filebeat to collect logs if you're using the default JSON-file logging driver. This is a relatively easy way to go, and Filebeat is a lightweight shipper that can be configured to export to either your Logstash container or directly to Elasticsearch.

Filebeat can be deployed on your host or as a container alongside your ELK containers, and there are various Filebeat Docker images available with configurations for running Filebeat and connecting it to Logstash.

If you're using a different logging driver, you may want to consider using a logging driver like the syslog driver, which can forward logs to a NAT server or other destinations, and then configure Logstash to parse and enhance the logs.

For another approach, see: Install Logstash

Filebeat is a lightweight shipper that can be used to collect logs from your Docker environment. It's written in Go and supports encryption.

Filebeat can be deployed either on your host or as a container alongside your ELK containers. You'll need to add linkage to the ELK containers if you choose the latter.

There are various Filebeat Docker images available, some of which include configurations for running Filebeat and connecting it to Logstash.

To use Filebeat, you'll need to specify the path to the JSON log file, located in /var/lib/docker/containers/.

Logging drivers were introduced by Docker in version 1.12 and allow you to run a container while specifying a third-party logging layer to which to route the Docker logs.

You can forward logs to AWS CloudWatch, Fluentd, GELF, or a NAT server, but remember that logging drivers need to be specified per container and will require additional configuration on the receiving ends of the logs.

Using the syslog logging driver is probably the easiest way to ship logs into ELK, as it provides a straightforward pipeline of Docker container logs being outputted into your syslog instance.

These logs will then be forwarded to the Logstash container for parsing and data enhancement, and from there into Elasticsearch.

Log Processing is a crucial step in the Docker-to-Logstash pipeline.

Logs are pulled from Docker containers and hosts by Logstash, the stack's workhorse that applies filters to parse the logs better.

The Logstash configuration file is where you'll configure the input, filter, and output sections. This file is located at /usr/share/logstash/pipeline if you're running Logstash 5.x.

The input section depends on the log shipping method you're using, such as Filebeat or syslog. If you're using Filebeat, you'll need to specify the Beats input plugin.

The filter section contains all the filter plugins you wish to use to break up the log messages. These will greatly depend on the type of container you're logging and the generated log messages for that particular container.

There is no easy way to configure this section because every container is outputting a different type of logs. There is plenty of trial and error involved, but there are online tools like the Grok Debugger to help you along the way.

The output section will specify the Logstash output, in this case, the Elasticsearch container. A basic Logstash configuration example for Docker logs being shipped via syslog uses a series of filters, including grok, date, mutate, and the if conditional.

To apply the new configurations, don't forget to restart the Logstash container.

Visualizing logs in Kibana is a crucial step in processing and analyzing log data.

To view our service logs in Kibana, we first need to load the Logstash index.

We can do this by navigating to the left sidebar, selecting the Management menu, and then clicking on Index Patterns link.

Our Logstash index will have a name like logstash-2020.01.01, so we need to use a wildcard pattern to load it.

Next, we need to specify a field name that will act as a time filter.

The @timestamp field in our Logstash index is a suitable choice for this.

Once we've created the index pattern, our Logstash index will be loaded into Kibana, and we can explore it further.

We can now analyze our service logs for further processing.

Shipping logs into ELK is a crucial step in setting up a logging system. You can use Filebeat to collect logs from a smallish Docker environment if you're outputting logs to a JSON file.

There are different methods for shipping logs depending on your logging driver. If you're using a different logging driver, you may want to consider a different method.

To ship logs into ELK, you can use Filebeat to collect logs from a smallish Docker environment. If you're using a different logging driver, you may want to consider using a different method.

You can use the Gelf driver for Docker service logging, which sends messages to Elasticsearch through Logstash. You'll need to specify the gelf-address and tag for the options.

Here are the steps to use the Gelf driver:

The Gelf driver uses the UDP protocol and port 12201 to send messages from Docker services. You can specify the tag to distinguish between different Docker services.

Here are the ports used by the ELK stack:

Logstash Configuration is a crucial step in setting up a Docker log to Logstash pipeline. We'll need to configure Logstash to accept messages in Gelf format through port 12201 via UDP connection.

To do this, we create a logstash.conf file in the demo2 directory, which will store our Logstash configuration. This file specifies elasticsearch as a hostname instead of localhost, allowing Docker services to communicate with each other.

The logstash.conf file also specifies an index key with a prefix of logstash- and a suffixed date string, which will change every day.

We can then mount a directory containing our logstash.conf file to the Logstash agent, which will configure the Logstash instance to send incoming data to Redis.

To configure Logstash to accept GELF format messages, you'll need to create a logstash.conf file. This file will store the Logstash configuration to accept messages with GELF format through port 12201 via UDP connection.

The Logstash agent exposes the UDP port 12201, which your application's Docker container will use to send its logs. You'll mount a directory called logstash-agent containing your logstash.conf file.

Within the logstash.conf file, you'll specify the hostname as elasticsearch instead of localhost, as Docker will recognize elasticsearch while communicating with each other. This is important for Docker services to send messages to Elasticsearch with port 9200.

The index key in the logstash.conf file should be specified with a prefix of logstash- and suffixed by a date string that changes every day. For example, 2020–04–20, 2020–04–21, 2020–04–22, and so on.

Logstash configuration involves specifying the versions of various components.

Elasticsearch, Logstash, and Kibana are three key components of the ELK stack.

Elasticsearch 5.3.0 is a version of Elasticsearch that can be used with Logstash. Logstash 5.3.0 is a version of Logstash that can be used with Elasticsearch 5.3.0. Kibana 5.3.0 is a version of Kibana that can be used with Elasticsearch 5.3.0 and Logstash 5.3.0.

Here are some specific version combinations:

Scalability is a top concern when it comes to handling large volumes of logs. We can have multiple central Logstash instances ingesting logs from the Redis database, which can itself be scaled to multiple nodes with replication.

This setup allows for a good extent of scalability. Elasticsearch itself can be set up to run in a cluster of multiple nodes to aid with high availability.

The Redis buffer ensures some amount of rate limiting on the incoming logs. This gives the consumer Logstash instances time to pick up the logs at their own pace, allowing the Elasticsearch engine sufficient time to index the data.

This approach helps prevent log ingestion from overwhelming the Elasticsearch engine. By giving it time to catch up, we can maintain a smooth and efficient logging process.

Orchestrating ELK can be a bit tricky, but don't worry, I've got you covered.

You can use Docker Compose to simulate centralized logging management for Docker services using the ELK stack. This is a great way to get a feel for how it works, and you can even configure the ELK stack on your container management or as a separate instance within your cloud service.

To get started, you'll need to write a docker-compose.yml file under your demo directory. This file will specify the configuration for your ELK stack.

Here's a rundown of what you'll need to include in your docker-compose.yml file:

When you run your Docker Compose, you'll be able to see your ELK stack running on your local machine, along with your Flask web application and Redis. However, you won't be able to see logs from Kibana, app, and Redis using the docker-compose logs command, because you've activated the Gelf driver to log container messages and send them to the ELK Stack.

Here's a summary of the ELK stack configuration:

By following these steps, you'll be able to get your ELK stack up and running, and start shipping your Docker logs into ELK.