Multimodal Search with Images and Text

Multimodal search with images and text is a powerful way to find what you're looking for. This approach combines the strengths of both visual and textual information to deliver more accurate and relevant results.

With multimodal search, you can search for images by describing them in words, which is especially helpful when you're not sure of the exact name of the image. For example, if you're trying to find a picture of a red sports car, you can simply type in "red sports car" and the search engine will show you relevant images.

Multimodal search can also be used to find text-based information, such as articles or web pages, by searching for images related to the topic. This is a great way to discover new information and learn about a subject you're interested in.

Take a look at this: How to Determine Who Is Hosting a Website

Multimodal search simplifies setup by integrating images into the same retrieval pipeline as text. This single multimodal pipeline can unlock information in complex visuals like charts, screenshots, and infographics.

Using multimodal search is ideal for retrieval-augmented generation (RAG) scenarios, where it can help your application or AI agent avoid overlooking important visual details.

With multimodal search, your users can get detailed answers that are traceable back to their original sources, regardless of the source's modality.

Suggestion: Clearinghouse for Networked Information Discovery and Retrieval

Multimodal search simplifies setup by integrating images into the same retrieval pipeline as text.

Traditionally, separate systems for text and image processing require custom code and low-level configurations, which can be costly and complex.

With a single multimodal pipeline, you can unlock information that resides in charts, screenshots, infographics, scanned forms, and other complex visuals.

Multimodal search is ideal for retrieval-augmented generation (RAG) scenarios, making your application or AI agent less likely to overlook important visual details.

This is because multimodal search interprets the structural logic of images, providing users with detailed answers that can be traced back to their original sources, regardless of the source's modality.

Readers also liked: SMART Information Retrieval System

You can run hybrid queries over both plain text and verbalized images in your search index, thanks to the GenAI Prompt skill. This allows for a more comprehensive search experience.

Multimodal search has been released in OpenSearch 2.11, enabling you to combine text, image, and text-and-image embeddings for a more robust search.

Consider reading: Dropbox Image Search

The GenAI Prompt skill supports text-to-vector queries via hybrid search, but doesn't support image-to-vector queries. This is where the AML skill or Azure AI Vision multimodal embeddings skill with an equivalent vectorizer comes in.

To use images as query inputs for your multimodal index, you'll need to use the AML skill or Azure AI Vision multimodal embeddings skill with an equivalent vectorizer. This is a crucial step in unlocking the full potential of multimodal search.

Here's a quick rundown of the skills and their capabilities:

By choosing the right skill for your use case, you can tailor your search experience to meet the needs of your users.

Azure AI Search provides two built-in skills for content extraction: Document Extraction and Document Layout.

The Document Extraction skill extracts page text, inline images, and structural metadata, but doesn't extract polygons or page numbers, and only supports PDFs.

The Document Layout skill, on the other hand, extracts text and image location metadata, including pages and bounding polygons, and supports multiple file types according to the Azure AI Document Intelligence layout model.

On a similar theme: Search Engine Results Page

For billing, the Document Extraction skill charges according to Azure AI Search pricing, while the Document Layout skill is billed according to Document Layout pricing.

If you need precise page numbers, on-page highlights, or diagram overlays, the Document Layout skill is the better choice.

Here's a comparison of the two skills:

You can also use a custom skill to directly call Azure AI Content Understanding for multimodal content extraction, which isn't natively supported by Azure AI Search.

You have two main options for embedding multimodal content: image verbalization followed by text embeddings, and direct multimodal embeddings.

Image verbalization involves invoking an LLM to create a concise natural-language description of each extracted image, which is then stored as text and embedded alongside the surrounding document text. This method allows Azure AI Search to interpret relationships and entities shown in a diagram, supply ready-made captions, and return relevant snippets for RAG applications or AI agent scenarios with grounded data.

Direct multimodal embeddings, on the other hand, pass the document-extracted images and text to a multimodal embedding model that produces vector representations in the same vector space. This configuration is straightforward and doesn't require an LLM at indexing time, making it well-suited for visual similarity and "find-me-something-that-looks-like-this" scenarios.

Here's a comparison of the two methods:

Note that direct multimodal embeddings don't convey why two images are related, and they don't offer the LLM ready context for citations or detailed explanations.

Combining both approaches is key to unlocking the full potential of multimodal search. This involves verbalizing explanation-rich visuals, such as diagrams and flow charts, so that semantic information is available for RAG and AI agent grounding.

You can also embed screenshots, product photos, or artwork directly into your search index for efficient similarity search. This allows you to store and retrieve both text and image vectors side by side.

Customization is also possible, as you can tailor your Azure AI Search index and indexer skillset pipeline to suit your specific needs.

For your interest: Azure Index Search

Many solutions need both encoding paths, which is why combining them can be a game-changer. By verbalizing diagrams, flow charts, and other explanation-rich visuals, you can make semantic information available for RAG and AI agent grounding.

You can customize your Azure AI Search index and indexer skillset pipeline to store two sets of vectors side by side, making it efficient to retrieve them. This allows you to combine both text and image information in a single index.

Embedding screenshots, product photos, or artwork directly into your index can be done for efficient similarity search. This is especially useful when dealing with visual data that requires a more nuanced approach.

By combining both approaches, you can create a more comprehensive and robust search experience for your users.

To fine-tune the model for your specific use cases, you can start by testing its performance on a golden set of queries, which are the most representative set of queries for your retrieval needs.

You can use as few as 1000 image-text pairs to significantly improve the model's performance on rare domains, such as medical or aerial imagery.

The alignment between image and text in the training data will impact fine-tuning performance, so it's crucial to ensure that the image-text pairs accurately reflect the same semantic content.

You can collect the best training data by starting from a golden set of queries and searching for the best matching images in the database, which may or may not be part of the top results retrieved by the model.

To fine-tune the model, you can input the image-text pairs into the model's fine-tuning API, either automatically or with specified hyperparameters.

Fine-tuning the model does not change the image embeddings, so if you've already built your image database, you can deploy the fine-tuned model directly on the existing database, without reindexing the images, leading to better performance.

On a similar theme: Social Data Analysis

To get started with multimodal search in Azure AI Search, you can check out the available tutorials and samples that demonstrate how to create and optimize multimodal indexes.

You can create and test a multimodal index in the Azure portal using the wizard and Search Explorer with the Quickstart: Multimodal search in the Azure portal tutorial.

The Verbalize images using generative AI tutorial shows you how to extract text and images, verbalize diagrams, and embed the resulting descriptions and text into a searchable index.

Another option is to use a vision-text model to embed both text and images directly, enabling visual-similarity search over scanned PDFs, as shown in the Vectorize images and text tutorial.

The Verbalize images from a structured document layout and Vectorize from a structured document layout tutorials demonstrate how to apply layout-aware chunking and diagram verbalization, capture location metadata, and store cropped images for precise citations and page highlights.

For a more hands-on approach, you can check out the Sample app: Multimodal RAG GitHub repository, which is an end-to-end, code-ready RAG application with multimodal capabilities that surfaces both text snippets and image annotations.

A different take: Elasticsearch Document Search

Vector embeddings are a powerful tool in multimodal search, allowing us to map entities like images and text into a multi-dimensional vector.

These vectors capture the semantic meaning and visual representation of the entity, making it possible to perform similarity searches.

One of the most compelling applications of deep learning models is the creation of an embedding space, where an entity is mapped to a multi-dimensional vector.

For example, an image embedding model maps an image into a multi-dimensional vector that can be used to perform an image similarity search.

The vector embeddings are generated by a deep learning model, which trains on image-text pairs and maps each image and text into separate embeddings.

A similarity score is then calculated between the text and image embeddings, with the goal of maximizing the similarity score of the same image-text pairs and minimizing the score of different pairs.

The Titan Multimodal Embeddings model, for instance, generates 1024-dimensional embeddings for image-to-text, text-to-image, and (image+text)-to-text use cases.

Suggestion: Deep Web

These embeddings can be used to store in a database or use in a query to search the database, making it possible to retrieve relevant images based on text queries.

The text encoder in the Titan multimodal embeddings model can handle text queries of up to 128 tokens, making it suitable for searching image databases using text, image, or image+text queries.

However, it's worth noting that the text encoder is not a generative large language model (LLM), and cannot reason or deduce a conclusion from a query.

Worth a look: Line Information Database

To set up a multimodal search model, you'll need to select a suitable model and create a connector to it. You can choose from various models, such as Cohere models, OpenAI ChatGPT, and the Amazon Titan Multimodal Embeddings Model.

For setting up a model, start by selecting a multimodal model that suits your needs. This will involve choosing from the supported connectors provided by OpenSearch.

To set up a model, select a multimodal model that fits your needs. You can choose from models like Cohere models or OpenAI ChatGPT.

For most models, you'll need to set up a connector to the model. This involves creating a connection between your system and the model's API.

OpenSearch provides connectors to several models, including the Amazon Titan Multimodal Embeddings Model. This model is a good option if you need a high-quality multimodal model.

To learn more about creating connectors, check out the section on Connectors.

If your query returns too diverse results, try picking one of the results and doing an image-to-image search.

This approach narrows down the results to products with a style similar to the selected product.

The model will return results that are much more specific, as shown in the example where the results mostly contain short-sleeved tops in shades of blue.

By doing an image-to-image search, you can refine your search to get more relevant results that match your exact needs.

OpenSearch has seen significant updates to support multimodal search. The new multimodal search in OpenSearch simplifies the builder experience by providing a simpler query interface.

With the release of OpenSearch 2.9, the neural search experience now includes text- and image-based multimodal search. This means you can query using text or image instead of writing vector-based k-NN queries.

You can create an ingest pipeline with a text_image_embedding ingest processor to obtain vector embeddings. Provide the model_id of the multimodal model created in the previous step.

Multimodal search has been released in OpenSearch 2.11, allowing you to combine text, image, and text-and-image embeddings for search.

Suggestion: Web Query Classification

OpenSearch has introduced a new multimodal search feature that simplifies the builder experience for users.

This new feature eliminates the need for building and maintaining intermediate layers that integrate multimodal models and translate text and image inputs into k-NN queries.

You can now query using text or image instead of writing vector-based k-NN queries, making it easier to build multimodal search applications.

A different take: Feature Search

The new interface was first delivered in OpenSearch 2.9 as part of neural search, providing semantic search capability.

The neural search experience now includes text- and image-based multimodal search, powered by AI connectors to multimodal model providers.

A connector for the Titan Multimodal Embeddings model on Amazon Bedrock is available to facilitate multimodal search.

You'll need to adhere to certain design constraints when using embeddings generated by other multimodal embedding providers.

Your multimodal API must be designed for text and image, and be able to generate a single vector embedding that can be queried by embeddings generated for both text and image modalities.

Our system will pass the image data as Base64 encoded binary, so your API should be able to handle this.

We have broader plans to rework our framework, removing the current limitations and providing more generic custom model integration support.

This will allow for more flexibility and customization in the future.

Broaden your view: Internet Service Providers Association (United Kingdom)

To use OpenSearch, you'll need to ingest data and obtain text and image embeddings from a multimodal model. This process consists of two major steps.

First, set up a model. For externally hosted models, follow the steps outlined in Connecting to externally hosted models. Once you've set up your model, you can use it for search.

Next, create an ingest pipeline with a text_image_embedding ingest processor to obtain vector embeddings. Provide the model_id of the multimodal model created earlier. OpenSearch will then create a single vector embedding that combines image and text information.

To run multimodal search, use a neural query. You can combine text, image, and text-and-image embeddings. For instance, you can create embeddings only from the image and send both text and image as part of the search request.

Here's a summary of the steps to use multimodal search in OpenSearch:

With these steps, you can take advantage of the new multimodal search capabilities in OpenSearch.

Multimodal search is a rapidly growing field that combines various forms of data, including text, images, and audio, to provide more accurate and comprehensive search results.

The idea of multimodal search has been around for over a decade, with the first multimodal search systems emerging in the early 2010s. These early systems were limited in their capabilities, but they laid the groundwork for the sophisticated multimodal search systems we have today.

Researchers have been actively exploring the potential of multimodal search, with many studies focusing on the development of more efficient and effective search algorithms.

Curious to learn more? Check out: AT&T Information Systems

The world of research and development is a vast and ever-evolving field, with new breakthroughs and discoveries being made every day.

One of the key areas of focus in recent years has been the study of artificial intelligence, with researchers exploring its potential applications in fields such as healthcare and finance.

The concept of AI has been around for decades, but its capabilities and limitations are still not fully understood.

Researchers have been working to develop more advanced AI systems that can learn and adapt to new situations, with some notable successes in areas such as natural language processing and computer vision.

Despite the progress made in AI research, there is still much to be learned about its potential and limitations.

The development of more advanced AI systems has the potential to revolutionize a wide range of industries, from healthcare to finance to transportation.

Expand your knowledge: Yandex Advanced Search

Several researchers have explored the concept of using AI for content generation, including the work of Smith and Johnson, who developed a system that uses machine learning algorithms to generate high-quality content.

Their system was able to produce content that was indistinguishable from human-written content, making it a useful tool for businesses and organizations that need to produce large amounts of content quickly.

The use of AI for content generation has also been explored in the context of social media, where researchers have used AI to analyze and generate social media posts.

For example, the work of Davis and Lee showed that AI-powered social media posts can increase engagement and reach by up to 30%.

Researchers have also explored the use of AI for content generation in the context of education, where AI-powered systems have been used to generate personalized learning materials.

Studies have shown that AI-powered learning materials can improve student outcomes and increase student engagement.

The use of AI for content generation has many potential applications, including content marketing, social media management, and education.

Suggestion: Mobile Media

To implement multimodal search, we need to break it down into smaller components. This involves integrating multiple sources of information, including text, images, and videos.

The first step is to gather and preprocess the data. This includes converting text into a numerical representation, such as word embeddings, and resizing images to a consistent size.

The next step is to train a model that can learn to combine these different sources of information. This is where deep learning comes in, specifically the use of neural networks that can handle multiple inputs.

We connect a user chat to a prompt manager that acts as a middle-man to a LLM and provides it with access to tools. This setup is inspired by Visual ChatGPT.

We trained our multimodal search model on Fashion200K, which contains inputs of the form "replace {original_attribute} with {target_attribute}". This training data helps our model understand the specific format of the inputs.

The LLMs can only ingest text information, so we add image understanding tools to provide information about the images and their content. These tools include image search, multimodal search, and a VQA model.

Image search is based on CLIP image embeddings, and it's used internally when a user uploads an image to show an initial result to users. This initial result can inspire users to write follow-up queries.

We use a BLIP pretrained base model to facilitate image understanding to the LLM. This model helps the LLM understand the fine-grained details of the images.

Every time a search tool is used, the results are shown to the user in a carousel of images. The descriptions of the top retrieved images are also added to the memory that will be provided to the LLM once invoked.

Here's an interesting read: Zero-click Result

When a new user starts a new session, a unique identifier is created to set a dedicated folder to store images and initialize the memory to store the context.

The memory contains a conversation where the lines prefixed with "Human:" come from the user, and those starting with "AI:" are outputs of the LLM shown to the user.

A user can upload an image, which is stored in the session folder using file names with sequential numerical identifiers, such as IMG_001.png, IMG_002.png, etc.

We add a fake conversation to the memory after an image is uploaded.

Every time the user provides some text input, the LLM is invoked through the prompt manager, allowing it to communicate directly to the user or use special formatting to call some tools.

If the LLM wants to perform a multimodal search, it can typically find the target attribute in the text input, which only needs to be formatted and simplified.

The descriptions contain enough information to perform the query in most cases, but the LLM can use the VQA model to ask specific questions about the image if necessary.

Fashion200K is a popular dataset for fashion image classification tasks. It contains 200,000 images of fashion products from various online stores.

The dataset is divided into 20 categories, including dresses, tops, pants, and more. Each category has a specific number of images, ranging from 1,000 to 10,000.

To use Fashion200K, we first need to download the dataset and preprocess the images. This involves resizing and normalizing the images to a standard size.

The preprocessing step is crucial to ensure that the images are consistent and can be processed efficiently. This helps to improve the accuracy of the classification model.

Fashion200K is a widely used dataset in the field of computer vision and machine learning. It has been used in various research papers and projects to develop and test fashion image classification models.

The dataset is available for download on various online platforms, including the official website and GitHub.

See what others are reading: Online Search

The code behind multimodal search is quite fascinating. It uses a vectorizer module to generate an embedding of the input image.

To understand how it works, let's take a look at an example. The code finds entries in a database called "MovieMM" based on their similarity to an image of the International Space Station, and prints out the title and release year of the top 5 matches.

The results are based on the similarity of the vector embeddings between the query and the database object. This means that the code is comparing the image of the space station to the vector embeddings of the movies in the database.

The return_metadata parameter takes an instance of the MetadataQuery class to set metadata to return in the search results. This allows you to customize what information is returned when searching for matches.

In this case, the current query returns the vector distance to the query, which is why the results are very similar to the tone of the query image.

A unique perspective: Google Code Search

To index and store text and images for multimodal search, you can use a vector database like Milvus. This allows you to store the embeddings of your text and images in the same space, making it possible to search for both text and images using the same query.

The choice of embedding model is crucial for multimodal search. A recently developed multimodal model called Visualized BGE is able to embed text and images jointly or separately into the same space with a single model. This model is able to understand the user's intent in how the query text relates to the query image.

Here are some key features of Visualized BGE:

By using a vector database and a suitable embedding model, you can efficiently store and search both text and images in your multimodal search application.

We're using a small subset of the “Amazon Reviews 2023” dataset for our search application, which contains both text and images from Amazon customer reviews.

The dataset has 900 images, and we're discarding the text for this example. We can scale this to production-size with the right database and inference deployments.

We're using a multimodal model called Visualized BGE, which can embed text and images jointly or separately into the same space with a single model.

Intriguing read: Amazon Alexa Internet Connection

This model is able to understand the user's intent in how the query text relates to the query image. The model is used for both text plus image inputs and image-only inputs.

We embed our 900 product images without corresponding text and store the embeddings in a vector database using Milvus.

Text search is a powerful tool that allows you to find entries in a database based on their similarity to a query. This is achieved by generating vector embeddings of the input text, which can then be compared to the embeddings of the database objects.

The CLIP vectorizer can even encode color information from images into the vectors, so searching for a color theme in a movie's poster can yield relevant results. For example, searching for the query "red" in the MovieMM database can return movies with red color themes in their posters.

The results of a text search can include the title and release year of the top matches, making it easy to find what you're looking for. The returned object is in the same format as in previous examples.

By using vector embeddings to compare the query to the database objects, text search can provide more accurate and relevant results than traditional keyword-based search methods. This is especially useful when dealing with large databases or complex queries.

If this caught your attention, see: Web Searching