Azure Cosmos DB Data Migration Tool: A Comprehensive Guide

Azure Cosmos DB offers a data migration tool to help you move your data from other databases to Azure Cosmos DB. This tool is designed to simplify the migration process.

The data migration tool supports various data sources, including Azure SQL Database, Amazon DynamoDB, and MongoDB. It can also migrate data from on-premises databases using Azure Data Factory.

Azure Cosmos DB's data migration tool uses a change data capture (CDC) mechanism to track changes in the source database. This ensures that only new or updated data is migrated, reducing the transfer time and costs.

The tool provides a user-friendly interface to monitor the migration progress and troubleshoot any issues that may arise during the process.

You might like: Schema Conversion Tool on Azure

To prepare for data migration with the Azure Cosmos DB data migration tool, you'll need to create a target database and container on the Azure Cosmos DB for NoSQL account. This involves using either Azure CLI or Azure PowerShell to create a new database and container.

Intriguing read: Azure Cosmos Db Container Conflicts

The target database should be named "cosmicworks" and configured with 400 RU/s of shared throughput. The container should be named "products" and have a partition key path of "/category". You can create this using the az cosmosdb sql database create and az cosmosdb sql container create commands in Azure CLI, or the New-AzCosmosDBSqlDatabase and New-AzCosmosDBSqlContainer cmdlets in Azure PowerShell.

To get the primary connection string for the account, you'll need to use the az cosmosdb keys list or Get-AzCosmosDBAccountKey command. Record the primary connection string value, as you'll use it later when migrating data with the tool.

Intriguing read: How to Connect to Azure Cosmos Db Using Connection String

To prepare your target for data migration, start by creating a target database and container on your Azure Cosmos DB for NoSQL account. This can be done using either the Azure CLI or Azure PowerShell.

You'll need to create a new database, which can be named anything you like, but for this example, we'll use "cosmicworks". Be sure to configure the database with the right amount of throughput, in this case, 400 RU/s of shared throughput.

To create a new database, you'll need to use the `az cosmosdb sql database create` command in the Azure CLI, or the `New-AzCosmosDBSqlDatabase` cmdlet in Azure PowerShell.

Once you've created your database, you'll need to create a new container within it. For this example, we'll create a container named "products" and set the partition key path to "/category".

Here's a list of the steps to create a target database and container:

It's also worth noting that you can use either the Azure CLI or Azure PowerShell to complete these steps. The commands and cmdlets are listed below:

You'll need to estimate the size of your data in Azure Cosmos DB, and it's not as simple as just mapping the source data size. The source data size may not exactly match the data size in Azure Cosmos DB.

To get an accurate estimate, insert a few sample documents from the source into Azure Cosmos DB and check their data size. This will give you a good idea of how your data will be stored in Azure Cosmos DB.

For example, if each document after migration is around 1 KB, and you have 60 billion documents in your source dataset, the estimated size in Azure Cosmos DB would be close to 60 TB.

It's essential to consider the data size to ensure you have enough resources provisioned for the migration. A smaller container size can lead to lower throughput availability, making the migration take much longer to complete.

Creating containers with enough RUs is crucial for a smooth migration process.

Smaller containers have lower throughput availability, which can significantly slow down the migration process. In fact, it's not recommended to start with the smallest container size.

To avoid this, you should pre-create containers with enough RUs to accommodate your entire dataset. For example, if your estimated data size is around 60 TB, a container of at least 2.4 million RUs is required.

This will ensure that your migration workload is fully consuming the provisioned throughput, making the process faster and more efficient.

Here's a rough estimate of the required RUs for different data sizes:

Keep in mind that this is just an estimate, and you should adjust the RUs according to your specific needs. By pre-creating containers with enough RUs, you'll be able to complete the migration process more quickly and efficiently.

Azure Cosmos DB offers various tools to simplify data migration. You can use Azure Data Factory's Azure Cosmos DB connector for smaller datasets.

Azure Data Factory's connector is ideal for validating data modeling, query performance, and partition key choice. It's perfect for testing and experimenting with different approaches.

The Azure Cosmos DB Spark connector is another option for data migration, especially if you're familiar with Spark. This connector can handle data migration tasks efficiently.

The Azure Cosmos DB Desktop Data Migration Tool is an open-source command line tool that facilitates import and export of data. This tool is not limited to Cosmos Database and can be extended to support other databases like Mongo DB and SQL Server.

This tool offers flexibility and can be customized to meet specific data migration needs.

To set up the Azure Cosmos DB Data Migration Tool, you'll need to download the latest release or ensure the project is built. This will give you access to the Extensions folder, which contains the plug-ins available for use in the migration.

The Extensions folder contains the plug-ins available for use in the migration, including the Cosmos DB extension, which is located in the folder named Cosmos. You can remove any folders for extensions that are not required for the migration before running the application.

To configure the data migration, you'll need to update the migrationsettings.json file, which provides an example outline of the settings file structure. This file should be located in the root of the build folder, and you can update the settings as documented in the Extension documentation.

The migrationsettings.json file should contain settings for the Source and Sink, including the FilePath setting, which should be the location where the sample data is extracted, and the ConnectionString setting, which can be found on the Cosmos DB Emulator Quickstart screen as the Primary Connection String.

The Cosmos DB Data Migration Tool's project structure is designed for flexibility and ease of use. The core project is a C# command-line executable that serves as the composition container for the required Source and Sink extensions.

You can add the desired Extension class library assembly to the Extensions folder before running the application. This allows you to customize the tool without modifying the core project.

The core project also comes with a unit test project to exercise the application's behavior. This ensures that the application is thoroughly tested and reliable.

Extension projects contain concrete integration tests that rely on external systems, providing a more comprehensive testing approach.

For more insights, see: Application Migration to Azure

To configure the setup for your project, you'll need to focus on the data migration configuration. Each extension contains a README document that outlines configuration for the data migration, so be sure to locate the configuration for JSON (Source) and Cosmos DB (Sink).

The migrationsettings.json file provides an example outline of the settings file structure, which can be found in the Visual Studio Solution Explorer by expanding the Microsoft.Data.Transfer.Core project. This file is crucial in configuring the SourceSettings and SinkSettings sections.

To configure the SourceSettings section, ensure the FilePath setting is the location where the sample data is extracted. The ConnectionString setting can be found on the Cosmos DB Emulator Quickstart screen as the Primary Connection String. Save the file.

Here are the key settings to configure in the migrationsettings.json file:

By configuring these settings, you'll be able to set up the data migration configuration for your project.

To start the migration process, you'll need to import data from the source to Azure Blob Storage. This can be done by partitioning the source data set into files around 200 MB in size.

The bulk executor library can scale up to consume 500,000 RUs in a single client VM, but you'll need to provision 10 Ubuntu 16.04 VMs in the same region as your Azure Cosmos DB database to take advantage of the available throughput of 5 million RUs.

Here's a step-by-step guide to the migration process:

Once the migration is completed, you can validate that the document count in Azure Cosmos DB is the same as the document count in the source database.

To estimate the migration speed, you need to consider the provisioned throughput of your Azure Cosmos DB API for NoSQL account.

5 RUs are required for writing a 1-KB document.

With 2.4 million RUs, you can transfer 480,000 documents per second, which translates to 480 MB/s.

A complete migration of 60 TB will take around 125,000 seconds or approximately 34 hours.

To complete the migration within a day, you should increase the provisioned throughput to 5 million RUs.

The migration process involves several steps to ensure a smooth transfer of data from the source to the destination. You'll need to import the data from the source to Azure Blob Storage, partitioning it into files around 200 MB in size to increase migration speed.

To do this, you can use the bulk executor library, which can scale up to consume 500,000 RUs in a single client VM. This means you'll need to provision 10 Ubuntu 16.04 VMs in the same region as your Azure Cosmos DB database, each with the migration tool and its settings file.

Once the data is imported, you'll need to run the queue step on one of the client virtual machines, which creates a tracking collection that scans the ADLS container and creates a progress-tracking document for each source data partition.

The import step is then run on all client VMs, with each VM taking ownership of a source partition and ingesting its data into Azure Cosmos DB. The clients can then query for the next available source partition in the tracking collection.

This process continues until all source partitions are processed, at which point the tool is rerun on the error-correction mode on the same tracking collection to identify source partitions that should be reprocessed due to errors.

Some of these errors may be due to incorrect documents in the source data, which should be identified and fixed before rerunning the import step on the failed partitions.

To validate the migration, you can compare the document count in Azure Cosmos DB to the document count in the source database. This will ensure that the migration was successful and that all data was transferred correctly.

On a similar theme: Convert Vmware to Azure

Here's a high-level overview of the migration process:

By following these steps and using the bulk executor library, you can ensure a smooth and efficient migration of your data from the source to the destination.

To migrate data from JSON files to Azure Cosmos DB for NoSQL containers, you'll need to create a new file named migrationsettings.json. This file will contain the necessary settings for the migration process.

You can create this file in an empty directory on your local machine. Within the JSON file, create a new empty JSON object.

To specify the source of the data, create a new property named Source with the value json. You'll also need to create a property named SourceSettings with an empty object as the value.

The SourceSettings object should contain a property named FilePath with the value set to a URI pointing to the JSON data, such as https://raw.githubusercontent.com/azure-samples/cosmos-db-migration-sample-data/main/nosql-data.json.

To specify the sink of the data, create a new property named Sink with the value cosmos-nosql. You'll also need to create a property named SinkSettings with an empty object.

The SinkSettings object should contain a property named ConnectionString with the primary connection string you recorded earlier in this guide as its value.

Here are the key settings you'll need to include in your migrationsettings.json file:

Once you've saved the migrationsettings.json file, you can run the desktop data migration tool using the dmt command. The tool will output the sources and sinks used by the migration.

During the migration process, it's a good idea to minimize indexing to speed things up.

You can turn off the container's indexing policy by changing the indexingMode to none.

This will save time and resources (RUs) that would be spent on creating indexes for each document ingested.

After the migration is complete, you can update the indexing.

Azure Cosmos DB automatically indexes all properties, so it's worth considering turning it off completely for the course of migration.

Large-scale migrations can be challenging due to the limitations of existing tools for migrating data to Azure Cosmos DB. These tools often have limited scale out capabilities, making it difficult to migrate terabytes of data quickly.

One of the biggest issues is the lack of progress tracking and check-pointing, which means that if an error occurs during the migration, the entire process has to be restarted from scratch. This can be frustrating, especially when 99% of the migration is already complete.

A dead letter queue would be a useful feature for migrating large data sets, as it would allow failed documents to be persisted and processed post-migration. This would prevent the entire migration from failing due to issues with parts of the source data or transient issues with the client or network.

The good news is that many of these limitations are being addressed in tools like Azure Data factory and Azure Data Migration services.

The Azure Cosmos DB data migration tool offers a range of features that set it apart from its predecessors. Currently, it supports 11 extensions for data sources, allowing you to sync your data with Azure Cosmos DB.

One of the key advantages of this tool is its flexibility, offering both a command-line interface (CMD) and a user interface (UI) to use. This means you can choose the approach that best suits your needs.

The tool also includes a custom plugin feature, enabling you to integrate it with your own data migration workflow. This level of customization is a major plus, allowing you to tailor the tool to your specific requirements.

Here are some of the key features of the custom tool:

This tool supports 11 extensions for data sources, allowing you to sync your data with Azure Cosmos DB.

Having both a Command-Line Interface (CMD) and a User Interface (UI) makes it easier to use.

You can integrate this tool with your own data migration workflow using the custom plugin feature.

The tool allows you to copy data by querying, which can be a game-changer for certain use cases.

You can import multiple JSON files into Azure Cosmos DB containers, making it a convenient option for large-scale data migration.

Here are some key differences between this tool and others:

Clicking on the dmt.exe file will launch the application and present you with a list of data source plugins to work with.

The custom tool with bulk execution library is a game-changer for large-scale data migrations. It's designed to be easily scalable across multiple instances and resilient to transient failures. This means you can pause and resume migration at various checkpoints, which is a huge advantage.

One of the key features of this tool is its ability to handle transient errors and scale out threads in a single node to consume up to 500 K RUs per node. This is made possible by the bulk executor library, which partitions the source dataset into micro-batches that are operated independently as a form of checkpointing.

Here's a high-level overview of how the custom tool works:

The tracking collection contains documents with metadata for each source data partition, including its location, migration status, and errors (if any). This allows you to easily track the progress of your migration and identify any issues that may arise.

If you have multiple operations to perform, you can add an array of SourceSettings/SinkSettings pairs to the Operations property in the settings file. This allows you to perform various tasks at once, minimizing the need to create separate files and launch the application multiple times.

The Azure Cosmos DB Desktop Data Migration Tool is an open-source tool, which means it's free to use and can be contributed to by anyone.

You can find the repository for the tool by visiting the link mentioned in the article, where you can explore its capabilities and even contribute to its development.

This tool is designed to make data migration from Firebase to Cosmos DB easier and more efficient.

The tool is a significant improvement over the old Azure Cosmos DB Data Migration tool, offering new capabilities that can streamline the migration process.