Elasticsearch Logs: Complete Guide to Configuration and Management

Elasticsearch logs are a crucial component of any Elasticsearch setup, providing valuable insights into the performance and behavior of your cluster.

By default, Elasticsearch logs are configured to store a maximum of 20 days' worth of logs, which can be adjusted to suit your specific needs. This setting can be found in the Elasticsearch configuration file, specifically in the `index.lifecycle.policies` section.

Managing Elasticsearch logs effectively is essential to ensure optimal performance and prevent log file growth issues. Elasticsearch provides a built-in log rotation feature, which can be configured to rotate logs daily or weekly.

Log rotation is a simple yet effective way to manage log files and prevent them from consuming too much disk space. By rotating logs regularly, you can ensure that your logs remain manageable and easy to analyze.

Elasticsearch provides comprehensive logging capabilities to help you monitor and troubleshoot your cluster.

To get started with Elasticsearch logs, you'll first need to understand the basics of logging. Elasticsearch, as well as OpenSearch, provides comprehensive logging capabilities to help you monitor and troubleshoot your cluster.

You can start by checking out the guide on how to activate and use Elasticsearch slow logs, which is a great resource for beginners.

To enable logging in Elasticsearch, you'll need to consider log levels, log formats, and log rotation.

Related reading: Azure Monitor vs Log Analytics

Elasticsearch logs are configured using the Log4j 2 configuration file, typically located in the `config` directory. You can modify this file to change the default log level from "INFO" to "DEBUG" or "TRACE" for more detailed logs.

To change the log level, locate the line that starts with `logger.action.level` and modify it to the desired level. For example, to set the log level to "DEBUG", change the line to `logger.action.level = DEBUG`. Save the changes and restart Elasticsearch for the new log level to take effect.

The `log4j2.properties` file can also be used to configure log rotation settings. You can set the maximum number of log files to keep, the log rotation interval, and the action to take when rolling logs. For example, to keep up to 30 log files, set `appender.rolling.strategy.max` to 30. To rotate logs every 12 hours, set `appender.rolling.policies.time.type` to `TimeBasedTriggeringPolicy` and `appender.rolling.policies.time.interval` to 12 hours.

Here's a summary of the log level and rotation settings:

Configuring levels in Elasticsearch allows you to customize the amount of detail in your logs. You can change the default log level from "INFO" to "DEBUG" or "TRACE" for more detailed logs.

The default log level is set to "INFO", but you can change it to "DEBUG" or "TRACE" for more detailed logs. To configure log levels, you need to modify the `log4j2.properties` file located in the `config` directory of your Elasticsearch installation.

You can change the log level by modifying the line that starts with `logger.action.level` in the `log4j2.properties` file. For example, to set the log level to "DEBUG", modify the line as follows: `logger.action.level = DEBUG`.

To save the changes, simply save the `log4j2.properties` file and restart Elasticsearch for the new log level to take effect.

Here are the built-in log levels in Log4j 2, from least to most verbose: OFF, FATAL, ERROR, WARN, INFO, DEBUG, and TRACE. The default log level is INFO.

You can change log levels using the cluster update settings API, or by modifying the `log4j2.properties` file. For example, to set the log level for the discovery module to "DEBUG", add the following line to the `log4j2.properties` file: `logger.discovery.level = debug`.

Here are the ways to change log levels:

Note that messages logged at higher verbosity levels (DEBUG and TRACE) are only intended for expert use.

To configure Elasticsearch logs, you need to modify the `log4j2.properties` file. This file is where you can change the log format from plain text to JSON.

Elasticsearch logs are formatted as plain text by default, but you can configure them to use JSON format if needed. You can do this by changing the layout type to “JsonLayout” in the `log4j2.properties` file.

To use JSON log format, you need to configure the Log4j 2 layout property `appender.rolling.layout.type` to `ECSJsonLayout`. This layout requires a `dataset` attribute to be set, which is used to distinguish log streams when parsing.

Recommended read: Elasticsearch Use Cases

The `dataset` attribute is a flag that populates the `event.dataset` field in the ECSJsonLayout. This makes it easier to distinguish different types of logs when parsing them.

You can configure the rolling file appender to roll logs to a specific directory, such as `/var/log/elasticsearch/production_server.json`. You can also use a time-based roll policy to roll logs on a daily basis.

Here's a summary of the configuration options for the rolling file appender:

Note that you can replace `.gz` with `.zip` in the `appender.rolling.filePattern` to compress the rolled logs using the zip format.

Configuring rotation is a crucial step in managing Elasticsearch logs. Elasticsearch automatically rotates log files when they reach a certain size or age, but you can customize this setting to suit your needs.

By default, log files are rotated daily or when they reach 100 MB in size. You can configure log rotation settings in the `log4j2.properties` file. To do this, locate the line that starts with `appender.rolling.policies.type` and modify the `appender.rolling.strategy.max` parameter to set the maximum number of log files to keep.

You can also set the log rotation interval by modifying the `appender.rolling.policies.time.type` and `appender.rolling.policies.time.interval` parameters. For example, to rotate logs every 12 hours, change the lines as follows:

Don't forget to save the changes and restart Elasticsearch for the new log rotation settings to take effect.

Configuring Elasticsearch logs is a straightforward process that can be customized to suit your needs. The default log level is set to "INFO", but you can change it to "DEBUG" or "TRACE" for more detailed logs.

To configure log levels, you need to modify the `log4j2.properties` file located in the `config` directory of your Elasticsearch installation. This file is used by Log4j2 for logging.

You can change the log level to "DEBUG" or "TRACE" by modifying the `logger.action.level` line in the `log4j2.properties` file. For example, to set the log level to "DEBUG", modify the line as follows: `logger.action.level = DEBUG`.

To configure log rotation, you need to modify the `appender.rolling.strategy.type` line in the `log4j2.properties` file. This will allow you to set the maximum number of log files to keep and the log rotation interval.

Here are the steps to configure log rotation:

Some common log rotation settings include:

Remember to save the changes and restart Elasticsearch for the new log level and rotation settings to take effect.

Multiple configuration files can be loaded as long as they are named `log4j2.properties` and have the Elasticsearch config directory as an ancestor. This is useful for plugins that expose additional loggers.

The logger section contains the Java packages and their corresponding log level. The appender section contains the destinations for the logs. Extensive information on how to customize logging and all the supported appenders can be found on the Log4j documentation.

You can also configure logging levels using the cluster update settings API to change the related logger's log level. Each logger accepts Log4j 2's built-in log levels, from least to most verbose: OFF, FATAL, ERROR, WARN, INFO, DEBUG, and TRACE. The default log level is INFO.

Elasticsearch uses Log4j2 for logging, which allows you to configure log levels for different components. The default log level is set to “INFO,” but you can change it to “DEBUG” or “TRACE” for more detailed logs.

To configure log levels, you need to modify the `log4j2.properties` file located in the `config` directory of your Elasticsearch installation. This file contains configuration settings for log levels, appenders, and other logging-related options.

The `log4j2.properties` file is where you can configure log levels for different components. For example, to set the log level for the action module to “DEBUG,” you would modify the line that starts with `logger.action.level` as follows: `logger.action.level = DEBUG`.

You can also configure log levels for other components by adding new lines to the `log4j2.properties` file. To set the log level for the discovery module to “DEBUG,” you would add the following line: `logger.discovery.name = org.elasticsearch.discovery logger.discovery.level = debug`.

Log rotation settings can be configured in the `log4j2.properties` file as well. The default log rotation settings rotate logs daily or when they reach 100 MB in size. To change the log rotation settings, you would modify the lines that start with `appender.rolling.strategy.max` and `appender.rolling.policies.time.interval`.

Here is an example of how to configure log rotation settings:

By configuring log levels and rotation settings, you can customize your Elasticsearch logs to better suit your needs.

Inspecting and searching logs can be a bit of a challenge, but Elasticsearch makes it relatively easy. You can view and query logs using Discover, which is a powerful tool for exploring your logs in detail.

To get started, you'll need to create a data view to make your logs visible in Discover. This will allow you to navigate to Discover in the Analytics sidebar of Kibana and select the data view you created.

In Discover, you can narrow down your queries by adjusting time and date ranges as required. You can also search for field names and filter by type, which is super useful when dealing with large datasets.

One of the most useful features of Discover is the ability to customize and save your searches for future use. You can also put them in a dashboard, which makes it easy to access your most frequently used searches.

If you're working with multiple indices or data streams, you can filter your log queries using the event.dataset field. This will help you narrow down your results to a relevant subset of data. For example, if you're looking at enterprise_search.crawler, you'll only see results from the App Search web crawler logs.

Here are some of the key benefits of using Discover:

Elasticsearch logs can be configured to rotate and compress files using the RollingFile appender. This allows you to specify the maximum number of log files to keep, the log rotation interval, and the file pattern for rolled logs.

To configure the RollingFile appender, you can modify the `appender.rolling.strategy.max` parameter to set the maximum number of log files to keep. For example, to keep up to 30 log files, change the line as follows: `appender.rolling.strategy.max = 30`.

You can also specify the log rotation interval by modifying the `appender.rolling.policies.time.type` and `appender.rolling.policies.time.interval` parameters. For example, to rotate logs every 12 hours, change the lines as follows: `appender.rolling.policies.time.type = TimeBasedTriggeringPolicy` and `appender.rolling.policies.time.interval = 12h`.

The configuration for old style pattern appenders is deprecated and will be removed in the future. These logs will be saved in *.log files and if archived will be in *.log.gz files.

Here's a summary of the RollingFile appender configuration options:

Data Management is a crucial aspect of Elasticsearch logs, and it's where Elastic truly shines. With their architecture designed to handle petabytes of logs, you can have all the telemetry you need without breaking the bank.

Elastic's indexing, compression, and efficient data tiering allow you to store more logs without sacrificing speed or performance. In fact, Elasticsearch logsdb index mode can store high-volume logging with smarter index sorting and denser compression, resulting in up to 70% smaller footprint.

You can filter, transform, join, correlate, and aggregate logs with ease using ES|QL's piped syntax. This means you can get faster queries and more precise answers, with less noise and complexity.

Elastic machine learning automatically detects anomalies, categorizes logs, and surfaces patterns, all without requiring setup. This means you can focus on what matters most – understanding and fixing issues, rather than manually digging through logs.

Elastic AI Assistant takes it to the next level by interpreting log messages, investigating anomalies, and pulling answers from your organizational knowledgebases using retrieval augmented generation (RAG). This gives you the root cause, remediation steps, and even the right runbook to fix issues quickly.

By using Elasticsearch logsdb index mode, you can move logs automatically across storage tiers without losing searchability. This keeps performance high where you need it and cost low where you don't, so you can store more logs and scale without the sprawl.

Broaden your view: Remove Elasticsearch Index

To create a data view in Elasticsearch, you'll need to navigate to Stack Management → Data Views. This is where you can create a data view by entering an index pattern that matches one or more data sources.

You can allow hidden and system indices under Show advanced settings. This gives you more flexibility when creating your data view.

To use your data view, you'll need to select a timestamp field for the global time filter. This ensures that your logs are properly timestamped.

A data view is essential for making your logs visible in Discover.

Related reading: Elasticsearch Create Index with Mapping

Observability is a powerful tool for turning messy logs into operational answers. Companies like Comcast ingest massive amounts of data daily with Elastic to monitor services and accelerate root cause analysis, ensuring a top-notch customer experience.

Comcast is just one example of a company that's seen the benefits of Elastic Observability. They ingest 400 terabytes of data daily with Elastic, which helps them provide a great experience for their customers.

Informatica cut costs and reduced MTTR by migrating its entire logging workload to Elastic for 100+ applications and 300+ Kubernetes clusters. This is a significant achievement that highlights the potential of Elastic Observability.

By implementing a centralized logging platform with Elastic, companies can reduce storage costs by 50% and improve data retrieval times. This can be a game-changer for businesses that need to process large amounts of data quickly.

Here are some impressive stats from companies that have used Elastic Observability: