Elasticsearch Hybrid Search: A Comprehensive Guide

Elasticsearch hybrid search is a powerful tool that allows you to combine multiple search engines to create a unified search experience. This approach is particularly useful for large-scale applications with diverse data sources.

By leveraging the strengths of different search engines, you can improve the accuracy, relevance, and speed of your search results. Elasticsearch hybrid search can be used to search through various data types, including documents, images, and videos.

One of the key benefits of Elasticsearch hybrid search is its ability to handle complex queries and provide more accurate results. This is achieved by combining the indexing and querying capabilities of multiple search engines.

Elasticsearch hybrid search also allows for real-time indexing and querying, making it an ideal solution for applications that require high performance and scalability.

See what others are reading: Telephone Hybrid

Elasticsearch hybrid search is a powerful information retrieval strategy that combines two or more search techniques into a search algorithm.

Hybrid search typically combines keyword search and semantic search, utilizing advanced machine learning techniques.

It retrieves results based on the meaning of the text, while full-text search focuses on exact word matches.

Hybrid search improves search precision by combining the strengths of semantic search and traditional search.

Elasticsearch provides an alternative way to perform semantic search using sparse vector models, such as Elastic Learned Sparse EncodeR (ELSER).

ELSER is a pre-trained, out-of-domain sparse vector model that doesn't require fine-tuning and was trained on a vocabulary of approximately 30,000 terms.

At indexing time, sparse vectors containing term/weight pairs are generated using the `inference` ingest processor and stored in fields of type `rank_features`.

At query time, a specific DSL query called `text_expansion` replaces the original query terms with terms available in the ELSER model vocabulary that are known to be the most similar to them given their weights.

For another approach, see: Elasticsearch Text Search

Elasticsearch integrates the RRF algorithm into the search query to combine results from full-text and vector searches.

RRF works fairly well for short lists of results without any configuration, but there are some parameters that can be tuned to provide the best results.

Hybrid search blends keyword and vector search to deliver comprehensive search results, allowing users to search by what they mean, even if they can't recall a precise description or exact keyword.

Vector embeddings convert data, like sentences or photos, into numbers that capture their meaning and relationships, and are commonly used in modern machine learning.

Sparse vectors handle traditional keyword-based indexing and are sparsely populated with information, while dense vectors handle semantic understanding and contextual queries.

Hybrid search uses reciprocal rank fusion (RRF) to combine multiple result sets into a single result set, delivering comprehensive search results that balance specificity and relevance.

A hybrid search query is a combination of a lexical search query and a vector search query, and can be run using either Convex Combination (CC) scoring or RRF ranking.

RRF ranking requires a commercial license (Plasticum or Enterprise), but CC scoring is available without a license.

Hybrid search with dense models runs a lexical search query mixed with an approximate k-NN search to improve relevance, and can be combined using a disjunction (logical OR condition) where the score of each document is computed using Convex Combination.

Additional reading: Elasticsearch Licensing

To implement a hybrid search strategy, the search() method must receive both the query and knn arguments. This is necessary to enable a combined search that returns results from both full-text and vector search methods.

The full-text search logic used earlier in the handle_search() function needs to be brought back to implement this hybrid search strategy. This involves modifying the search() method to receive both query and knn arguments.

The rank section is added to combine the results into a single ranked list, allowing the best results from each search method to be combined.

To implement a hybrid search strategy, the search() method must receive both the query and knn arguments, each requesting a separate query.

The handle_search() function needs to be updated to combine the results from both full-text and vector search methods. This involves bringing back the full-text search logic used earlier in the function.

The search() method must be modified to receive both query and knn arguments, allowing for separate queries to be executed.

This change enables the function to return a single ranked list of results, combining the best results from each search method.

The rank section is added to the code to facilitate this combination of results.

With this version of the handle_search() function, the best results from each search method are combined into a single ranked list.

Optimizations can significantly reduce the size of your documents and disk space. By configuring your index mapping to remove vector fields from your source documents, you can save a substantial amount of space. This can be achieved by excluding the vector fields from the `_source`, as shown in the code below.

The most obvious optimization is to exclude dense vector fields from the source, which can divide the index size by 3. Excluding rank feature fields can reduce the index size by almost 3.5 in some cases.

The size of the index can vary greatly depending on the nature and size of the unstructured data you're indexing, as well as the dense or sparse models you're using. For example, an index with dense vector fields can be up to 376 MB, while an index without dense vector fields can be as small as 119 MB.

Suggestion: Elasticsearch Vector Db

Here are the results of a test with the msmarco-passagetest2019-top1000 data set:

Keep in mind that excluding vectors from the source means you won't be able to use your index as a source index to be reindexed into another one. However, you can still use the original ingest pipeline to regenerate the embeddings vectors.

You might like: Elasticsearch Use Cases

Elasticsearch hybrid search combines keyword, lexical, and BM25 search with semantic search and vector search. This powerful combination allows for more accurate and relevant results.

Semantic search focuses on understanding the meaning and context of a query, using natural language processing, machine learning, knowledge graphs, and vectors to deliver results that match the user's intent. It also takes into account the user's geographical location, so searching for "football" in the USA would yield different results than in other parts of the world.

Vector search uses numeric representation or vectors to represent items like text, images, or audio and retrieves data based on similarities, capturing the underlying meaning or features of these items. This is achieved through algorithms like the kNN algorithm, which matches vectors of existing documents to the query vector.

Elasticsearch provides two ways to perform semantic search: dense vector models and sparse vector models. The dense vector model produces vectors that usually contain non-zero values and represent the meaning of unstructured data in a multi-dimensional space.

For your interest: Elasticsearch Create User

The Anatomy of Hybrid Search in Elasticsearch is a complex process that involves leveraging dense vector models to represent the meaning of unstructured data in a multi-dimensional space.

Dense vector models produce vectors that usually contain essentially non-zero values, making them a viable option for semantic search. This is achieved through the use of the dense_vector field type in Elasticsearch.

Elasticsearch also provides an alternative way of performing semantic search using sparse vector models, such as the Elastic Learned Sparse EncodeR (ELSER) model. ELSER is a pre-trained, out-of-domain sparse vector model that doesn't require fine-tuning.

Sparse vector models like ELSER are generated using the inference ingest processor and stored in fields of type sparse_vector at indexing time. This allows for efficient storage and retrieval of sparse vectors.

At query time, a specific DSL query called sparse_vector replaces the original query terms with terms available in the ELSER model vocabulary that are known to be the most similar to them given their weights. This enables accurate semantic search results.

For more insights, see: Azure Semantic Search

The ELSER model was pre-trained on a vocabulary of approximately 30,000 terms and has most of its vector values as zeros, making it a sparse model. This characteristic allows for efficient computation and storage.

Elasticsearch provides a way to evaluate the performance of ELSER through the Elastic relevance workbench, which compares it to a normal BM25 lexical search. This allows developers to assess the effectiveness of ELSER in their specific use cases.

Elasticsearch provides a field type called `dense_vector` that is used to store dense vector models, which contain essentially non-zero values that represent the meaning of unstructured data in a multi-dimensional space.

At indexing time, sparse vectors containing term/weight pairs are generated using the `inference` ingest processor and stored in fields of type `rank_features`.

The `dense_vector` field type has a counterpart called `sparse_vector`, which is used to store sparse vector models like Elastic Learned Sparse EncodeR (ELSER).

ELSER is a pre-trained, out-of-domain sparse vector model that does not require any fine-tuning and was pre-trained on a vocabulary of approximately 30,000 terms.

Discover more: Elastic Search by Field

Most of the vector values in a sparse model like ELSER are zeros, with only ~0.1% of the non-zero values.

Sparse vectors are queried via the inverted index, similar to lexical search, whereas dense vectors are indexed in specific graph-based or cluster-based data structures that can be searched using approximate nearest neighbors algorithms.

It's possible to mix both dense and sparse data inside the same index, allowing for hybrid search queries that combine both types of data.

A hybrid query can contain a lexical search query, a vector search query, and a semantic search query, all of which can be specified in a single query payload.

The `sub_searches` section can be used to specify an array of lexical and semantic queries that will be independently executed and whose results will be ranked together.

The `rank` section is mandatory when using `sub_searches` to run hybrid queries, and is only available with a commercial license or a trial license valid for one month.

Hybrid search combines keyword, lexical, and BM25 search with semantic search. BM25 is a ranking algorithm that determines relevance.

Semantic search focuses on understanding meaning and context, understanding the intent behind the words in a query rather than just matching keywords. It leverages natural language processing (NLP), machine learning, knowledge graphs, and vectors to deliver results that are more relevant to the user's intent and incorporate context.

Vector search is a technical search method that uses numeric representation or vectors to represent items like text, images, or audio and retrieves data based on similarities. These vectors capture the underlying meaning or features of these items.

Semantic search and vector search have a lot in common, and in fact, semantic search is powered by vector search.

KNN scores are used to determine the relevance of lexical candidates. This is done by matching vectors of existing documents to the query vector using an algorithm like the kNN algorithm.

The kNN algorithm generates results based on conceptual relevance, which is a key aspect of semantic search. Semantic search focuses on understanding meaning and context, rather than just matching keywords.

To determine the context, semantic search may use known user data, location, or past search history to determine relevant results. This means that searching "football" in the USA would glean different results than the same search in other parts of the world.

In a hybrid search, semantic search and vector search work together to handle complex queries, including multi-language searches and searches that require unstructured data. This is made possible by transforming the query into vector embeddings and matching them to existing documents.

By combining semantic search and vector search, platforms can deliver results that are more relevant to the user's intent and incorporate context. This is achieved by leveraging natural language processing (NLP), machine learning, knowledge graphs, and vectors.

Elasticsearch hybrid search allows you to combine multiple data sources into a single search index, making it easier to retrieve relevant results.

With Elasticsearch hybrid search, you can query multiple indices simultaneously, reducing the need for complex queries and improving search performance.

This approach enables you to store different types of data in separate indices, such as structured data in a relational database and unstructured data in a document store.

You can then use Elasticsearch's query DSL to combine the results from these different indices, creating a unified search experience for your users.

Elasticsearch hybrid search supports various query types, including full-text search, filtering, and aggregation, allowing you to tailor your search results to specific use cases.

By leveraging Elasticsearch's scalability and performance, you can handle large volumes of data and provide fast, accurate search results to your users.

Elasticsearch hybrid search also supports data re-ranking, which enables you to re-order search results based on additional criteria, such as relevance or freshness.

For another approach, see: Elasticsearch Performance

The ELSER sparse model has a limitation of only supporting up to 512 tokens when running text inference, which can be a challenge if you need to search longer text excerpts.

This means you have two options: use another model that supports longer text or split your text into smaller segments.

Splitting your text into smaller segments might seem like a viable solution, but storing each segment in a nested array of your document isn't possible due to another limitation of dense vectors in Elasticsearch.

This limitation is related to Apache Lucene, which is currently fixed and will be lifted in Lucene 9.8.

In the meantime, your only option is to store each text segment in a dedicated document.

Recommended read: Document in Elasticsearch

Elasticsearch AI Playground is a great resource to dive into hybrid search, where you can roll up your sleeves and explore its capabilities.

If you're looking to combine full-text and kNN results, there are resources available that can guide you through the process. Elasticsearch and Go can be used for gopher hunting with hybrid search, making it a powerful combination.

aNN and kNN are two types of vector search algorithms with distinct differences and roles. Understanding these differences is crucial for choosing the right algorithm for your needs.

Here are some key resources to get you started with hybrid search: