Elasticsearch Percolate Best Practices for Optimal Performance

To get the most out of Elasticsearch Percolate, it's essential to follow best practices for optimal performance. This includes setting a reasonable number of queries, as too many can cause performance issues.

The ideal number of queries is around 1000 to 2000, depending on the complexity of the queries and the size of the index. This allows for efficient processing and minimizes the risk of timeouts.

Elasticsearch Percolate is particularly useful for real-time analytics and monitoring, where speed is of the essence. By setting the right number of queries, you can ensure that your application remains responsive and efficient, even under heavy loads.

Regularly monitoring and indexing data is also crucial for optimal performance. This ensures that your data is up-to-date and accurate, reducing the risk of errors and discrepancies.

Here's an interesting read: Elasticsearch Performance

Percolation is a powerful feature in Elasticsearch that allows you to store searches and use documents to ask the index about those searches.

The basic idea is to store queries with parameters, like brand:bmw AND model:330e AND price run documents against these stored queries.

This concept is similar to many other use cases, such as dating platforms, job platforms, and online marketplaces, where a notification mechanism is required once a new document has been added to your data that matches the requirements of a user.

To use the percolate query feature, you need to create an index with a mapping that includes a field of type “percolator”, like in the example of creating an index called “my-index” with two fields: “query” of type “percolator” and “content” of type “text”.

The percolation process involves indexing queries, like storing a query that matches the condition for a Tesla Model 3 for less than 50k, and then running documents against these stored queries to find the queries that match.

Here's a step-by-step overview of the process:

Under the hood, this is implemented in about the way you would expect: indexes with percolate fields keep a hidden (in memory) index, and documents listed in your percolate queries are first put in that index, then a normal query is executed against that index to see if the original percolate-field-bearing document matches.

For another approach, see: Elasticsearch Index Api

To use the percolate query feature, you need to create an index with a mapping that includes a field of type “percolator”. This is done by creating an index with a percolator field, as shown in Example 4: "Create an index with a percolator field".

You'll need to add a percolator field to your index, along with other fields as needed. For example, in the "my-index" index, a "content" field of type "text" was added.

Once you have your index set up, you can start indexing queries. This involves storing a query as a document in the percolator field, as shown in Example 5: "Index queries".

The query you index will contain parameters like brand, model, price, and region, as seen in Example 1: "Why do I need percolation?".

Here's a step-by-step overview of setting up percolation:

By following these steps, you'll be well on your way to setting up percolation in Elasticsearch.

Percolation is a powerful feature in Elasticsearch that allows you to store queries and run documents against them, flipping the relationship between document and query around.

This concept can be tricky to understand at first, but let's break it down with some use-cases. For example, you can use percolation to deliver ads with every article that you are rendering on a news website, or to notify users when a new document matches their requirements on an online marketplace.

You store a query with parameters like brand:bmw AND model:330e AND price < 35000 AND region:Munich, and then run documents against it to find matches.

Percolation is often referred to as "flipping around query and documents" because it requires storing queries and running documents against them, rather than storing documents and running queries against them.

You can use the percolate query to find the queries that match a given document, and the response will include the IDs of the indexed queries that match the document.

A fresh viewpoint: Document in Elasticsearch

Here's an example of how to percolate a document: you can provide a single document or a collection of documents to be percolated, and the response will include the IDs of the indexed queries that match the document.

You can also reference existing documents in an index to be used as input for the percolate query, like this: "my-data-index" with "id" equals to 2.

To highlight the query results, you can use the highlight feature, like this: the highlighted "Elasticsearch" term in the highlight part of the response.

Percolation has many use cases, including monitoring and alerting, document classification, and real-time search-as-you-type suggestions.

Here are some examples of how percolation can be used in different scenarios:

Percolation in the document pipeline is a distinct operation from indexing documents. It requires additional queries to trigger alerts.

Your pipeline needs to be aware of this and run additional queries to trigger such an alert. This is especially true when you're running a news website and want to deliver ads with every article that you're rendering.

Additional reading: Elasticsearch Ingest Pipeline

For example, if an article is about cars, you might want to show ads that have something to do with mobility. This is where percolation comes in - it allows you to store a query with parameters like brand:bmw AND model:330e AND price < 35000 AND region:Munich.

Here are some common use cases for percolation:

To implement percolation, you need to store queries as documents in the percolator field. This is different from storing documents and running queries against it, which is what percolation requires.

Recommended read: Elasticsearch Updating Documents

To optimize percolation, consider wrapping the percolation in a constant_score query to get a speed bump, as mentioned in Example 2. This can help improve performance.

You can also filter percolations by not running them against your in-memory index, as shown in Example 3. This can be achieved by using filters, such as a field named type, to determine which queries to run against the in-memory index.

Explore further: Elasticsearch Index Template

To implement percolation, create an index with a percolator field, index queries as documents in the percolator field, and then use the percolate query to find matching queries, as outlined in Example 5.

Here's a step-by-step guide to implementing percolation:

You can get a little speed bump by wrapping the percolation in a constant_score query if you're not interested in the score.

Not running all percolations against your in-memory index can also give you a speed boost.

Using filters is a great way to achieve this. For example, if you have several commercial tiers and only a certain group should be included, you can use a filter to exclude them.

By adding another field named type to the percolator document, you can immediately reuse it in the query. This way, only queries of the type platinum will be considered to be run against the in-memory index.

This approach returns the following results, as both documents match the percolation.

Consider reading: Elasticsearch Index Format

Implementing Elasticsearch Percolator is a crucial step in optimizing and implementing percolation. To get started, you need to create an index with a percolator field, which is a field of type "percolator".

This field will store the queries that you want to match against your documents. To index a query, you need to store it as a document in the percolator field, which in our example is the "query" field.

Here are the steps to create an index with a percolator field:

Once you have indexed your queries, you can use the percolate query to find the queries that match a given document.

Advanced percolation features can help you get the most out of this powerful tool. One way to speed up your percolation is to wrap it in a constant_score query, which can give you a small performance boost.

You can also use this feature to ignore the score of the percolation, which can be useful if you're only interested in whether a document matches a query or not. This is especially useful for use cases like online marketplaces, where you want to notify users when a new car is added that matches their search criteria.

If you're working with a large number of queries, you can use the constant_score query to improve performance. This is because it eliminates the need to calculate scores for each query, which can be a time-consuming process.

Here are some examples of how you can use the constant_score query with percolation:

This can be a useful feature to have in your toolkit, especially when working with complex queries or large datasets. By using the constant_score query, you can improve the performance of your percolation and get the results you need more quickly.

You can enrich a document based on the response of a percolation. This can be useful for the tagging use-case, where you need to collect information for running ads based on the tags field.

The percolation queries contain a tags field, which you can use to add information to the document's tags field. This allows you to easily find the ads supposed to be delivered with the article.

Suggestion: Elasticsearch Document Search

To add this additional step, you'll need to set up your ingestion pipeline. This involves adding a step to collect the percolation results and add them to the document before indexing.

You can then use the terms lookup query to find the ads stored in Elasticsearch. This makes it easy to retrieve the relevant ads based on the tags field.

Let's take a closer look at how Elasticsearch percolate works. In Example 1, we see a mapping that implements a percolate index to support saved term and price searches for toys. This allows users to put in a search term and a max price, then get notified as soon as something matching that term goes below this price.

The document that represents a stored search only stores data inside the search object field, while the mappings for price and description are just there to support percolate queries. This is a key point to note, as it shows how percolate queries can be used to filter and match documents based on specific criteria.

You might like: Elasticsearch Term Query

At query time, we want to use both the plain object fields and the "special" percolator field to check which currently-enabled searches match the document. This query combines percolate matching of a document against the queries stored in the field with regular term queries to limit which documents we test based on their enabled state and the user id.

In Example 2, we see three use cases for Elasticsearch percolate query: Monitoring and Alerting, Document Classification, and Real-time Search-as-you-type Suggestions. These use cases highlight the flexibility and power of percolate queries in Elasticsearch.

Here are the three use cases in more detail: