Elastic Cross Cluster Search Configuration and Best Practices

Elastic cross cluster search configuration is a powerful feature that allows you to search across multiple clusters. This is particularly useful for large-scale deployments where data is spread across multiple clusters.

To enable cross cluster search, you need to configure a cluster as a search cluster and set up a minimum of one data cluster. The search cluster will serve as the central hub for all searches.

A search cluster can be configured to use a specific data type, such as a node or a node type, to store the search index. This is done by setting the index.store.type property in the elasticsearch.yml file.

To ensure efficient search performance, it's essential to optimize the search index by setting the index.store.type to "fs" and the index.number_of_replicas to 1. This will help reduce the load on the search cluster and improve search results.

Curious to learn more? Check out: Cross Tenant Azure

Cross-cluster search allows users to execute a query across multiple clusters, performing one search request between one or more remote clusters.

Users can filter and analyze logs stored on clusters in several data centers using cross-cluster searches.

Cross-cluster search enables unified search queries throughout all the configured clusters, while allowing users to configure several remote clusters across various geographical locations and providers.

Send a search request from a local cluster and receive responses from all connected remote clusters.

Cross-cluster search is possible in both Elasticsearch and OpenSearch.

To set up cross-cluster search, you'll need to configure remote clusters on Elasticsearch Service or on your own hardware. This requires running Elasticsearch on your own hardware or using Elasticsearch Service.

The local coordinating node must have the remote_cluster_client node role, which is a crucial requirement for cross-cluster search.

If you use sniff mode, the local coordinating node must be able to connect to seed and gateway nodes on the remote cluster. We recommend using gateway nodes capable of serving as coordinating nodes, and seed nodes can be a subset of these gateway nodes.

Here are the key requirements for setting up cross-cluster search:

Cross-cluster search also requires different security privileges on the local cluster and remote cluster.

To set up cross-cluster search, you'll need to meet some prerequisites. First, you need to have remote clusters set up, which can be done on Elasticsearch Service by following the instructions to configure remote clusters. If you're running Elasticsearch on your own hardware, you'll need to set up remote clusters manually.

The local coordinating node must have the remote_cluster_client node role to support cross-cluster search. This is a crucial requirement, so make sure to assign the correct role to your node.

If you're using sniff mode, the local coordinating node must be able to connect to seed and gateway nodes on the remote cluster. We recommend using gateway nodes that can serve as coordinating nodes, and seed nodes can be a subset of these gateway nodes.

In proxy mode, the local coordinating node must be able to connect to the configured proxy_address. The proxy at this address must be able to route connections to gateway and coordinating nodes on the remote cluster.

To ensure your remote cluster configuration supports cross-cluster search, refer to the Supported cross-cluster search configurations section. You'll also need to configure different security privileges on the local cluster and remote cluster for cross-cluster search to work.

To set up multiple remote locations, you'll need to configure remote clusters in Elasticsearch. This involves using the cluster update settings API request to add remote clusters to your local cluster settings.

For example, you can add three remote clusters, such as cluster_one, cluster_two, and cluster_three, to your local cluster settings.

You can also search multiple remote clusters by using the search API request. This allows you to search the same index on multiple clusters, including your local cluster and remote clusters like cluster_one and cluster_two.

Worth a look: Elasticsearch Index Settings

Here's an example of how this works:

Remember, each cluster is identified by its name, which is used in the search API request. This way, you can easily track which documents come from which cluster.

To get started with elastic cross-cluster search, you'll need to set up two single-node clusters on the same network. Save a file as docker-compose.yml and run docker-compose up to get them running.

The clusters will both run on localhost, so you'll use the port number as an identifier. For example, use port 9200 for the remote cluster and port 9250 for the coordinating cluster.

To test cross-cluster search, first identify the container ID of the remote cluster, then get its IP address. On the coordinating cluster, add the remote cluster name and the IP address with port 9300 for each seed node.

Let's take a look at some real-life examples that illustrate how example walkthroughs can be applied in different situations.

In a sales training program, example walkthroughs can be used to teach new sales representatives how to effectively handle customer objections. This is done by presenting a hypothetical scenario where a customer is hesitant to make a purchase, and then walking through the steps to address their concerns.

A company can use example walkthroughs to train its employees on how to properly use new equipment or software. For instance, a manufacturing company might create an example walkthrough to show employees how to operate a new machine safely and efficiently.

Example walkthroughs can also be used to onboard new employees and get them up to speed quickly. This can be done by creating a step-by-step guide that shows the new employee how to perform their job tasks and responsibilities.

A small business owner can use example walkthroughs to teach their employees how to handle customer complaints. This can be done by presenting a hypothetical scenario where a customer is dissatisfied with a product or service, and then walking through the steps to resolve the issue.

By using example walkthroughs in these ways, businesses can save time and money by reducing the amount of time it takes to train employees and resolve customer issues.

To get started with cross-cluster search, you need to save a file as docker-compose.yml and run docker-compose up to start two single-node clusters on the same network. This will give you two clusters running on localhost, which is perfect for testing.

The important thing to note is the port number, as it will be used to identify each cluster. In this case, use port 9200 (odfe-node1) as the remote cluster and port 9250 (odfe-node2) as the coordinating cluster.

To get the IP address for the remote cluster, first identify its container ID, which can be done using the command docker ps. Then, get that container's IP address using the command docker inspect.

Once you have the IP address, you can add the remote cluster name and the IP address (with port 9300) for each "seed node" on the coordinating cluster. This is done using the command curl -XPOST 'http://localhost:9250/_cluster/remote' -H 'Content-Type: application/json' -d '{"seed_nodes": ["cluster_two:9300"]}'.

If this caught your attention, see: Elasticsearch Port

After adding the remote cluster, cross-cluster search should work, and you can test it using the admin user. To do this, use the command curl -XPOST 'http://localhost:9250/_search' -H 'Content-Type: application/json' -d '{"query": {"match": {"title": "Data Mining"}}}'.

To continue testing, create a new user on both clusters using the command curl -XPOST 'http://localhost:9250/_security/user/_create' -H 'Content-Type: application/json' -d '{"username": "new_user", "password": "new_password"}'.

Note that both clusters must have the user, but only the remote cluster needs the role and mapping. The coordinating cluster handles authentication, and the remote cluster handles authorization.

Curious to learn more? Check out: Elasticsearch Create User

To minimize network roundtrips, you send a cross-cluster search request to your local cluster, which is then parsed by a coordinating node. This node sends a single search request to each cluster, including the local cluster.

Each cluster performs the search request independently, applying its own cluster-level settings to the request. This allows for more efficient processing of search queries.

Here's a step-by-step breakdown of the process:

Cross-cluster search involves sending requests to remote clusters, and any network delays can impact search speed.

By default, Elasticsearch reduces the number of network roundtrips between remote clusters, which reduces the impact of network delays on search speed. However, this approach may not work well for large search requests.

Elasticsearch can't reduce network roundtrips for large search requests, such as those including a scroll or inner hits. Instead, it sends multiple outgoing and ingoing requests to each remote cluster.

You can also choose this option by setting the ccs_minimize_roundtrips parameter to false. This approach may work well for networks with low latency, but it's typically slower.

Here's how cross-cluster search works when you minimize network roundtrips:

During an upgrade, it's good to know you can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local coordinating node's "upgrade from" and "upgrade to" version must be compatible with the remote cluster's gateway node.

You can't run multiple versions of Elasticsearch in the same cluster beyond the duration of an upgrade. This means you'll need to finalize the upgrade process before reverting to a previous version.

For more information about upgrades, be sure to check out the Upgrading Elasticsearch section, which provides more details on the process.

Security is a top concern for cross-cluster search, so make sure to enable and configure security on both local and remote clusters. This includes establishing Transport Layer Security (TLS) on each node at the transport level.

To search across clusters, users can now configure remote clusters using any deployment template in Elastic Cloud, starting with stack version 6.7 or higher. The cross-cluster search deployment template has been retired, but existing deployments are not impacted.

Running the same version of Elasticsearch in each cluster ensures a smooth cross-cluster search experience, but it's not the only option. Users can also keep an exclusive cluster for cross-cluster searches and update it to the latest version.

Discover more: Elasticsearch Search Template

OpenSearch offers a more flexible configuration process for remote clusters. Users must have permission for the index indices:admin/shards/search_shards, in addition to READ or SEARCH permissions.

To configure remote clusters in OpenSearch, users must have READ or SEARCH permission for the remote index, and also indices:admin/shards/search_shards in case they use ccs_minimize_roundtrips=false. This is a more granular approach compared to Elasticsearch.

One notable difference between OpenSearch and Elasticsearch is the way users configure privileges. In OpenSearch, privileges are configured before registering remote clusters, whereas in Elasticsearch, they are configured after registering remote clusters.

Here's a comparison of the two:

These differences highlight the unique configuration requirements for each platform.

To ensure cross-cluster search works smoothly, security needs to be enabled and configured on both local and remote clusters. This includes establishing Transport Layer Security (TLS) on each node, at least at the transport level.

To minimize network round trips, it's recommended to update cluster-level parameters that restrict searches, such as action.search.shard_count.limit, pre_filter_shard_size, and max_concurrent_shard_requests.

You might like: Securing Elasticsearch

Running the same version of Elasticsearch in each cluster is the simplest way to guarantee that clusters enable cross-cluster search. Users can keep an exclusive cluster for cross-cluster searches if they need to have clusters run various versions.

To search other clusters, keep this cluster updated to the most current version. If, for instance, users have both 7.17 and 8.x clusters, a separate 7.17 cluster used as the local cluster for cross-cluster searches can be used.

The security plugin authenticates the user on the coordinating cluster when reaching a remote cluster from a coordinating cluster using a cross-cluster search. This ensures that user permissions on the remote cluster are assessed.

To create roles in OpenSearch, users can use OpenSearch Dashboards, the REST API, or the roles.yml configuration file. Unless users need to create new reserved or hidden users, it's highly recommended to use OpenSearch Dashboards or the REST API to create new users, roles, and role mappings.

Here are some key considerations for cross-cluster search:

To add a remote cluster, you'll need to add the remote cluster IP address and name (with port 9300) for each seed node. This will allow the coordinating cluster to communicate with the remote cluster.

To set up a Cross-Cluster search, you must fulfill the following prerequisites. The local cluster must be defined, which is the set used to initiate the cross-cluster search. This is typically done by defining the master nodes of the local cluster in the Kibana Config file.

To perform a cross-cluster search, you must have at least one remote cluster, which can be registered using sniffing mode or proxy mode. Sniff mode requires the gateway node to have its broadcast addresses accessible to the local cluster.

The APIs that support cross-cluster search include SearchAsync searchMulti searchSearch templateMulti search templateField capabilitiesEQL search (in technical preview)SQL search (in technical preview)Vector tile search (in technical preview)

Here's an interesting read: Searching Incognito Mode

You can configure cross-cluster search to work with different versions of Elasticsearch, but it's essential to understand the supported configurations. Elasticsearch supports searches from a local cluster to a remote cluster running the same version or the previous minor version.

To ensure cross-cluster search support, you can maintain a dedicated cluster for cross-cluster search, keeping it on the earliest version needed to search the other clusters. This is especially useful when you have clusters with different versions.

Elasticsearch also supports searches from a local cluster running the last minor version of a major version to a remote cluster running any minor version in the following major version. For example, a local 7.17 cluster can search any remote 8.x cluster.

Here's a table summarizing the supported cross-cluster search configurations:

The EQL search API requires the local and remote clusters to use the same Elasticsearch version. This means that if you're using the EQL search API, you'll need to ensure that your local and remote clusters are on the same version.

To query indices on remote clusters, users need to have the right permissions in place.

You'll need to create a role on the remote cluster with the necessary permissions, which include READ or SEARCH permissions for the index.

This role will give you the access you need to query the index without any issues.

Note that you'll also need to map the user to this new role, so make sure to do that as well.

By following these steps, you'll be able to query the index without any permission errors holding you back.

Cross-cluster search is supported by several APIs, including Search, Async search, and Multi search. These APIs allow for flexible and efficient searching across multiple clusters.

The Search API supports cross-cluster search, making it a great option for complex search queries.

Async search and Multi search APIs also support cross-cluster search, enabling asynchronous and multi-search capabilities.

Field capabilities are also supported, allowing for the retrieval of field information.

Here is a list of supported APIs:

Keep in mind that some of these APIs are still in technical preview and may be subject to change or removal in future releases.