
Elasticsearch upsert is a powerful feature that allows you to update or insert documents in a single operation, making it an essential tool for managing data in Elasticsearch.
By using the upsert feature, you can avoid the overhead of separate update and insert operations, which can be time-consuming and resource-intensive.
In Elasticsearch, an upsert operation can be performed using the `_update` API, which checks if a document already exists in the index before updating or inserting it.
This approach ensures that you don't overwrite existing documents or create duplicates, maintaining data consistency and integrity.
Worth a look: Elasticsearch Updating Documents
Optimizing Update and Insert
Updating documents in Elasticsearch can be a straightforward process. Only POST is allowed when using the _update endpoint and the "doc" property, so make sure to use the correct HTTP method.
To update a document, you'll need to specify the document ID and the new values for the fields you want to update. For example, if you want to update the salary of an employee with ID 2, you can use the following request:
Related reading: Elasticsearch Document Search
POST /employee/_doc/2/_update
{
"doc": {
"salary": 10000000
}
}
ES will automatically increment the _version of the document.
If the document doesn't exist, Elasticsearch will create a new one. However, if you want to specify a document to be inserted if the document doesn't exist, you can use the Update API with the `upsert` parameter.
Here are the steps to perform an upsert operation:
1. Prepare the Update API request with the index, document type, and document ID.
2. Specify the `doc` parameter with the partial document to be updated if the document exists.
3. Specify the `upsert` parameter with the document to be inserted if the document doesn't exist.
4. Send the request to execute the upsert operation.
For example, if you want to update a document with ID 2 and insert a new document if it doesn't exist, you can use the following request:
POST /employee/_doc/2/_update
{
"doc": {
"salary": 10000000
},
"upsert": {
"name": "John Doe",
"age": 30
}
}
For large-scale data modifications, bulk operations are more efficient. The _bulk API allows you to perform multiple update, delete, and upsert operations in a single request.
Intriguing read: Magento 2 Elasticsearch
Performing Operations
To perform an upsert operation in Elasticsearch, you can use the Update API with the `upsert` parameter. This parameter allows you to specify the document that should be inserted if the document does not already exist.
The request should include the index, document type (if applicable), and document ID. You can use the HTTP POST or PUT method for the request.
To specify the document that should be inserted if it doesn't exist, you'll need to define the `upsert` parameter. This parameter contains the document that should be inserted if the document with the specified ID doesn't exist.
You can also use the `doc` parameter to specify the partial document to be updated if the document exists. If you need to define custom update logic, you can use the `script` parameter.
Here's a step-by-step guide to performing an upsert operation:
- Prepare the Update API request with the index, document type (if applicable), and document ID.
- Specify the `doc` parameter with the partial document to be updated if the document exists.
- Specify the `upsert` parameter with the document that should be inserted if the document with the specified ID doesn't exist.
- Send the request to perform the upsert operation.
In practice, this means you can update a document with ID 2, and if it doesn't exist, it will be created.
Error Handling
Error handling is crucial for maintaining data integrity. Handling errors during document updates, deletes, and upserts is essential to prevent data inconsistencies.
One of the most critical aspects of error handling in Elasticsearch upserts is to identify the type of error that occurred. Error Handling is crucial for maintaining data integrity.
When an error occurs during an upsert operation, it's vital to log the error and the affected documents to facilitate debugging and data recovery.
Recommended read: Elasticsearch Spring Data
Updating Documents
Updating documents in Elasticsearch is a crucial aspect of upsert operations. You can modify the fields of an existing document using the _update API.
This API allows you to update specific fields without reindexing the entire document. For instance, if you want to update the salary of an employee, you can use the _update endpoint and the "doc" property.
Here's an example of how to update a document using the _update endpoint:
```html
POST /employee/_doc/2/_update
For another approach, see: Python Api Elasticsearch
{
"doc": {
"salary": 10000000
}
}
```
Only POST is allowed for updating documents, and the _update endpoint will increment the _version of the document. This is useful for tracking changes to the document.
You can also use the Bulk API to update multiple documents at once, which can improve performance. However, this adds latency to your workload.
See what others are reading: Elasticsearch Document
Adding a Field
You can add a new field to a document in Elasticsearch through an index operation. This can be done by partially updating an existing document with the new field using the Update API.
With dynamic mapping enabled, introducing a new field is straightforward. Simply index a document containing that field and Elasticsearch will automatically figure out the suitable mapping and add the new field to the mapping.
If dynamic mapping is disabled, you'll need to use the update mapping API to add the new field.
Consider reading: Elasticsearch Bulk Api
Example Applications
Elasticsearch upserts are useful for applications with frequent updates, such as a search application for a video streaming service like Netflix, which can have millions of updates per minute.
The Netflix Engagement Report shows that users watched ~100 billion hours of content on Netflix between January to July, resulting in an average of 1.3 million views per minute.
Many search and analytics applications can experience frequent updates, especially when built on CDC data.
In such cases, upserts can be used to create or update documents in a single operation, making it efficient for large-scale data modifications.
For example, you can use the upsert operation to increment the views field in a document, which can be used to boost content ranking.
You can also use the _bulk API to perform multiple update, delete, and upsert operations in a single request, making it more efficient for large-scale data modifications.
The upsert operation can be used to create a document if it does not exist, as shown in the example where a document is created with a salary field and a value of 5000000.
Bulk upserts can be used to update multiple documents in a single request, making it more efficient for large-scale data modifications.
You can prepare a bulk request to upsert multiple documents, as shown in the example where a bulk request is prepared to update multiple employee documents.
Broaden your view: Elastic Search by Field
Partial Painless Scripts
Elasticsearch supports scripted updates, allowing you to perform more complex updates using scripts. This is a powerful feature that enables you to update documents based on complex logic.
Painless scripts are a type of scripting language designed for Elasticsearch, which can be used for query and aggregation calculations, complex conditionals, data transformations, and more. They also enable the use of scripts in update queries.
A Painless script can be used to perform an update in a single API call, as in the example where it increments the new view count based on the value of the old view count. This is a convenient way to update documents.
The Painless script is intuitive to understand, incrementing the view count by 1 for every document. This makes it easy to use and maintain.
For another approach, see: Elasticsearch Painless
Query and Path Parameters
Query and Path Parameters are crucial for Elasticsearch upsert operations. You can specify the target index using the index path parameter, which is a string and required.
Additional reading: Elasticsearch _template

The id path parameter is also required and is a string. It uniquely identifies the document to be updated. I've found that using a unique identifier helps prevent conflicts and ensures the correct document is updated.
You can also specify the script language using the lang query parameter, which is a string. This is useful if you're using a specific language for your upsert operation.
Here's a summary of the query and path parameters you can use:
I recommend checking the Elasticsearch documentation for the most up-to-date information on query and path parameters.
Path Parameters
Path Parameters are a crucial part of working with APIs, and they're actually quite straightforward.
The name of the target index is required, and it's a string. This means it's a text value, not a number. By default, the index is created automatically if it doesn't exist.
A unique identifier for the document to be updated is also required, and it's a string. This means it's a text value, not a number.
Here's a quick rundown of the path parameters we've discussed so far:
Both index and id are required, so make sure you include them in your API calls.
Query Parameters

Query parameters play a crucial role in Elasticsearch operations, allowing you to fine-tune your queries and control how they're executed. They can be used to filter out unwanted results, optimize performance, and even retry failed operations.
One of the most useful query parameters is `refresh`, which determines whether Elasticsearch should refresh the affected shards to make the operation visible to search. You can set it to `'true'` to refresh the shards immediately, `'wait_for'` to wait for a refresh, or `'false'` to do nothing with refreshes.
If you're dealing with conflicts, you can use the `retry_on_conflict` parameter to specify how many times the operation should be retried. This can be a lifesaver when working with large datasets or complex queries.
Here's a summary of the query parameters you can use:
These query parameters can make a big difference in how your Elasticsearch queries perform and behave. By using them effectively, you can optimize your queries, reduce errors, and improve overall performance.
Benefits and Overview

Elasticsearch upsert is a powerful tool that can save you time and resources by combining update and insert actions into a single request.
This reduces the number of requests made to the Elasticsearch cluster, which can improve overall performance. You can save time and resources, making your workflow more efficient.
Consistency is key, and upsert operations ensure that your data remains consistent, even if multiple clients are attempting to update or insert the same document simultaneously.
Upsert operations are atomic, which means that they are either fully completed or not executed at all, preventing partial updates or inserts.
Here are the benefits of using upsert operations in Elasticsearch:
- Efficiency: Saves time and resources by combining update and insert actions into a single request.
- Consistency: Ensures data remains consistent, even with multiple clients updating or inserting the same document.
Featured Images: pexels.com

