Elasticsearch Force Merge: Understanding the Merge Process and Impact

A force merge in Elasticsearch can significantly impact your cluster's performance, especially if not done properly.

This process involves merging multiple segments into a single segment, which can lead to a significant reduction in memory usage.

The number of segments in your index can grow exponentially over time, causing fragmentation and slowing down your queries.

Elasticsearch's default merge policy is designed to optimize for search performance, but it may not always be the best choice for your specific use case.

A force merge can help alleviate these issues, but it's essential to understand the merge process and its impact on your cluster.

Intriguing read: Force Onedrive Sync

Elasticsearch's force merge operation is a resource-intensive process that can temporarily degrade cluster performance.

It's essential to use it sparingly and during off-peak hours, if possible, to minimize disruptions. This operation consolidates smaller segments into fewer, larger segments, significantly improving search performance and reducing disk usage.

The force merge operation is executed on each shard of a node sequentially, and the storage for the shard temporarily increases so that all segments can be rewritten into a new segment. When max_num_segments is set to 1, the storage for the shard temporarily doubles.

Curious to learn more? Check out: How Do You Force Icloud Backup

You can call the Force Merge API on multiple indexes, data streams containing multiple backing indexes, index aliases pointing to multiple indexes, or all data streams and indexes in a cluster.

Here are the scenarios where a force merge is triggered:

It's also worth noting that force merging will be performed by the node hosting the shard, usually the node's role matches the data tier of the ILM phase that the index is in.

To perform a force merge, you can use the `_forcemerge` API. You can start a force merge operation on a single index, like `my_index`, by sending a request to the API. This will start a force merge operation on the index.

You can also specify a target number of segments using the `max_num_segments` parameter. For example, you can reduce the number of segments to five, if your index is smaller than 25GB, otherwise the number of segments might be slightly over 5.

To force merge multiple indexes, you can call the Force Merge API on the following index combinations: multiple indexes, one or more data streams containing multiple backing indexes, one or more index aliases pointing to multiple indexes, or all data streams and indexes in a cluster.

Here are some things to keep in mind when force merging multiple indexes: the merge operation is executed on each shard of a node sequentially, and the storage for the shard temporarily increases so that all segments can be rewritten into a new segment.

Recommended read: Elasticsearch Spring Data

To perform a force merge, you can use the `_forcemerge` API. This will start a force merge operation on the specified index, reducing the number of segments to a default of 5GB or a target number of segments specified using the `max_num_segments` parameter.

You can also force merge multiple indexes, which is executed on each shard of a node sequentially. This temporarily increases storage for the shard to rewrite all segments into a new segment.

To use the forcemerge action in the hot phase, the rollover action must be present. If no rollover action is configured, ILM will reject the policy. This is a resource-intensive operation that can negatively impact your cluster if too many force merges are triggered at once.

If you experience a force merge task queue backlog, you might need to increase the size of the force merge threadpool so indices can be force merged in parallel. To do this, configure the `thread_pool.force_merge.sizecluster` setting.

Here are some options for where to perform a force merge:

It's recommended to target merges against SSD disks for faster performance. If using best_compression, ILM will close and then re-open the index prior to the force merge, making it unavailable for read or write operations.

Query parameters can greatly impact the force merge operation in OpenSearch.

The allow_no_indices parameter determines whether the request returns an error if any wildcard expression or index alias targets any closed or missing indexes. By default, it's set to true.

You can specify the types of indexes to which wildcard expressions can expand using the expand_wildcards parameter. It supports comma-separated values, including all, open, closed, hidden, and none. The default value is open.

A flush operation is performed on the indexes after the force merge if the flush parameter is set to true, which is the default value. This ensures that the files are persisted to disk.

If you want OpenSearch to ignore missing or closed indexes, you can set the ignore_unavailable parameter to true. Otherwise, it returns an error if the force merge operation encounters missing or closed indexes.

The max_num_segments parameter determines the number of larger segments into which smaller segments are merged. You can set this parameter to 1 to merge all segments into one segment.

Here are the available query parameters with their default values:

Default is open.flushBooleanPerforms a flush on the indexes after the force merge. A flush ensures that the files are persisted to disk. Default is true.ignore_unavailableBooleanIf true, OpenSearch ignores missing or closed indexes. If false, OpenSearch returns an error if the force merge operation encounters missing or closed indexes. Default is false.max_num_segmentsIntegerThe number of larger segments into which smaller segments are merged. Set this parameter to 1 to merge all segments into one segment. The default behavior is to perform the merge as necessary.only_expunge_deletesBooleanIf true, the merge operation only expunges segments containing a certain percentage of deleted documents. The percentage is 10% by default and is configurable in the index.merge.policy.expunge_deletes_allowed setting. Prior to OpenSearch 2.12, only_expunge_deletes ignored the index.merge.policy.max_merged_segment setting. Starting with OpenSearch 2.12, using only_expunge_deletes does not produce segments larger than index.merge.policy.max_merged_segment (by default, 5 GB). For more information, see Deleted documents. Default is false.primary_onlyBooleanIf set to true, then the merge operation is performed only on the primary shards of an index. This can be useful when you want to take a snapshot of the index after the merge is complete. Snapshots only copy segments from the primary shards. Merging the primary shards can reduce resource consumption. Default is false.

Discover more: Elasticsearch Updating Documents

Force merging is a resource-intensive operation that can negatively impact your cluster if too many are triggered at once. This can happen when you modify an existing ILM policy's phase min_age, causing indices to trigger the force merge at a faster rate.

To prevent this, you can increase the min_age or set index.lifecycle.origination_date to change how the index age is calculated. You can also run the ILM Move Step API against multiple indices, but be aware that this can also trigger force merges.

If you experience a force merge task queue backlog, you might need to increase the size of the force merge threadpool so indices can be force merged in parallel. To do this, configure the thread_pool.force_merge.sizecluster setting.

Here are some best practices to follow when using force merge:

When performing a force merge, it's recommended to target SSD disks and use the `only_expunge_deletes` option for active indices to reduce the impact on performance and disk usage.

In Elasticsearch 8.11 and later, the data streams lifecycle feature is available, which can handle merging the long tail of recently created small segments as soon as the write index gets rolled over.

However, for other index types, regular merging is still the way to go, and it's essential to understand when to use force merge. Force merge is most beneficial in scenarios like read-only indices, time-based indices, and disk space management.

If you're dealing with read-only indices, force merging can improve search performance and reduce disk usage. Once an index is no longer receiving updates, it's a good idea to force merge it.

Time-based indices, particularly those using rollover, can also benefit from force merging. Each index only receives indexing traffic for a certain period, and once it receives no more writes, its shards can be force-merged to a single segment.

To force merge multiple indexes, you can call the Force Merge API on various index combinations, including multiple indexes, data streams containing multiple backing indexes, index aliases pointing to multiple indexes, and all data streams and indexes in a cluster.

See what others are reading: Elasticsearch Indices

Here's a summary of the best practices for force merging:

Remember, force merging can temporarily increase storage for the shard being merged, so be sure to plan accordingly.

Avoid force merging active indices, as it can lead to a large number of small segments. This negates the benefits of the force merge, so it's best to wait until the index is no longer receiving updates.

Limit concurrent force merges to prevent degrading cluster performance. Running multiple force merges at the same time can cause issues, so it's essential to space them out.

Use the `only_expunge_deletes` option when force merging active indices. This reduces the impact on performance and also helps reduce disk usage by removing deleted documents.

When force merging, it's a good idea to target SSD disks instead of HDD disks. This can improve the performance of the merge operation.

To prevent a force merge task queue backlog, you may need to increase the size of the force merge threadpool. However, be cautious when adjusting this setting, as it can cause cascading performance impacts.

Here are some key settings to monitor when dealing with force merges:

By following these best practices, you can ensure that your force merge operations run smoothly and efficiently.

A force merge can block until the merge is complete, and if the client connection is lost, the process will continue in the background.

Calls to this API block until the merge is complete, which can cause issues if the client connection is lost before completion.

If a force merge is already in progress, any new requests to force merge the same indices will also block until the ongoing force merge is complete.

The force merge process will continue in the background if the client connection is lost before completion, but it's essential to be aware of this behavior.

If you only want to expunge segments containing document deletions, you can set the optional boolean parameter to true, which defaults to false.

Primary shards are the foundation of your index's data structure, and understanding how to manage them is crucial for optimal performance.

Force merging primary shards can be done to reduce the number of shards and improve query efficiency, as seen in the example where an index's primary shards are force merged.

Additional reading: Elasticsearch Shards

Force merging can negatively impact your cluster if too many force merges are triggered at once, especially when modifying an existing ILM policy's phase min_age or applying an ILM policy to existing indices.

The force merge task queue backlog can be caused by running multiple force merges concurrently, which can degrade cluster performance. To manage this, you can increase the size of the force merge threadpool by configuring the thread_pool.force_merge.sizecluster setting.

Avoid force merging active indices, as it can lead to a large number of small segments, negating the benefits of the force merge. Instead, consider using the only_expunge_deletes option to reduce the impact on performance and disk usage.

To minimize the impact of force merging, it's recommended to target merges against SSD disks, as they can handle the increased load better than HDD disks.

See what others are reading: Elasticsearch Performance

Monitoring Operations is crucial to understanding the impact of merge operations on your Elasticsearch cluster. You can monitor the progress of a force merge operation using the `_cat/segments` API.

See what others are reading: How Do I Merge Email Accounts

The `_cat/segments` API returns a list of segments for a specific index, including their size and the number of documents they contain. This information can be helpful in tracking the growth of your index over time.

You can also use the `_cat/tasks` API to show the force merge tasks that are currently running. This can be particularly useful when monitoring the progress of a force merge operation.

Deleted documents are not immediately removed from the Lucene segment, but are instead marked to be deleted.

This can lead to increased disk space usage and performance degradation if not properly managed.

To remove these deleted documents, you need to merge the segments, which can be done periodically or forced using the Force Merge API.

The Force Merge API is a powerful tool, but it requires careful consideration to avoid blocking write requests or producing large segments.

You should only call the Force Merge API after all write requests sent to the index are completed, to prevent performance issues.

If your connection is lost during a force merge operation, the process will continue in the background, and new requests will be blocked until the merge is complete.

Data Streams are a crucial aspect of managing data, and understanding their behavior is essential for efficient data management. Force merging data streams can be useful to manage a data stream's backing indexes, especially after a rollover operation.

Time-based indexes receive indexing requests only during a specified time period. This means that once that time period has elapsed and the index receives no more write requests, you can force merge segments of all index shards into one segment.

Forcing a merge can significantly improve search efficiency on single-segment shards, as they use simpler data structures. This can lead to faster search results and better overall performance.

A different take: Elasticsearch Search Count

Force merging can be a resource-intensive operation, so it's essential to be mindful of its impact on your cluster. If too many force merges are triggered at once, it can negatively impact your cluster.

This can happen when you modify an existing ILM policy's phase min_age, causing indices to trigger the force merge at a faster rate. Alternatively, applying an ILM policy that includes a force merge action to existing indices can also cause issues.

To prevent these problems, you can increase the min_age or set index.lifecycle.origination_date to change how the index age is calculated. This will slow down the force merge process and give your cluster some breathing room.

If you experience a force merge task queue backlog, you might need to increase the size of the force merge threadpool so indices can be force merged in parallel. To do this, configure the thread_pool.force_merge.sizecluster setting.

However, be aware that thread_pool.force_merge.size is an advanced setting and adjusting it can cause cascading performance impacts. Monitor cluster performance and increment the size of the thread pool slowly to reduce the backlog.

Here are some key considerations for force merging:

It's also recommended to target merges against SSD and not HDD disks for optimal performance. If using best_compression, ILM will close and then re-open the index prior to the force merge, making the index unavailable for read or write operations during this time.

To force merge multiple indexes, you can call the Force Merge API on specific index combinations.

The Force Merge API can be used on multiple indexes, one or more data streams containing multiple backing indexes, one or more index aliases pointing to multiple indexes, or all data streams and indexes in a cluster.

The merge operation is executed on each shard of a node sequentially.

As a result, the storage for the shard temporarily increases so that all segments can be rewritten into a new segment.

This temporary increase in storage can be significant, especially when max_num_segments is set to 1, in which case the storage for the shard temporarily doubles.

Intriguing read: Elasticsearch Tiered Storage