
The outbox pattern in Kafka is a game-changer for reliable message delivery. It ensures that messages are processed and sent to the next stage in the pipeline, even if the original message processing fails.
By implementing the outbox pattern, you can guarantee that messages are not lost in transit. This is especially important in mission-critical systems where data integrity is paramount.
The outbox pattern works by storing messages in a separate topic, which is then processed independently of the original message processing. This decouples the message processing from the message sending, making the system more robust and fault-tolerant.
In this setup, messages are stored in the outbox topic until they are successfully sent to the next stage, at which point they are marked as "sent" and removed from the outbox topic.
Components and Setup
The outbox pattern for Kafka involves several key components that work together to ensure reliable message delivery. The outbox pattern is comprised of events, the outbox table, and a publisher.
The events are generated by the application at runtime and are stored in the Outbox table. This data store is used to hold the messages until they are processed by the publisher.
The publisher is responsible for processing messages from the outbox table and then dispatching them to their intended destinations.
Here are the key components of the outbox pattern:
- Events: At runtime, an application generates events.
- Outbox table: This data store is used to store the messages until they are processed by the publisher.
- Publisher: This component is used to process messages from the outbox table and then dispatch them to the appropriate destinations
Message Broker Introduction
Apache Kafka is a distributed, high-performance, open source, scalable, and versatile stream-processing software.
It's written in Java and Scala.
Kafka is a messaging platform based on the publish/subscribe model.
It comes with built-in features for replication, partitioning, fault tolerance, and improved throughput.
Kafka is a good option for applications that need vast data processing capabilities.
The main use of Kafka is in creating real-time streaming data pipelines.
Kafka includes stream processing and fault-tolerant storage functionalities.
Create Message Class
To create the message class, you'll need to start by creating a new C# class named OutboxMessage in a file named OutboxMessage.cs. This class will serve as the foundation for your message structure.
Check this out: Outbox Not Sending

The OutboxMessage class is a crucial component in your system, as it will hold the necessary information for sending messages to Apache Kafka. You'll want to create this class in a file named OutboxMessage.cs.
To implement the message class, you'll need to write the code for the OutboxMessage class, which involves defining the properties and fields that will be used to store message data. This class will be used in conjunction with the IKafkaProducer interface to send messages to Kafka.
The OutboxMessage class will be used to store message data, which will be sent to Kafka using the IKafkaProducer interface. You can create this class by following the code example provided in the relevant article section.
In addition to creating the OutboxMessage class, you'll also need to create the IKafkaProducer interface, which will be used to send messages to Kafka. This interface will define the necessary methods for sending messages to Kafka.
The IKafkaProducer interface will be used to send messages to Kafka, and you'll need to create a class that implements this interface to send messages to Kafka. This class will be used in conjunction with the OutboxMessage class to send messages to Kafka.
Explore further: Outlook Not Sending Outbox

By creating the OutboxMessage class and the IKafkaProducer interface, you'll have the necessary components in place to send messages to Kafka using the KafkaProducer class. This class will be used to send messages to Kafka using the IKafkaProducer interface.
The KafkaProducer class will be used to send messages to Kafka using the IKafkaProducer interface, and you'll need to create an instance of this class to use it in your application. This instance will be registered with the service collection, along with the instance of the ImessageRepository interface.
By following these steps, you'll have the necessary message class and IKafkaProducer interface in place to send messages to Kafka using the KafkaProducer class. This will enable you to send messages to Kafka using the IKafkaProducer interface and the KafkaProducer class.
Related reading: You Have Messages in Your Outbox Waiting to Be Sent
Setting Up
To implement the Outbox Pattern, we first need to set up the OutboxMessages table and a stored procedure to insert messages into the table.

The OutboxMessages table is created using a SQL script, as shown in the example, which includes specific columns for message properties.
We also need a stored procedure to insert messages into the OutboxMessages table, which is typically done using a SQL script that defines the procedure's parameters and logic.
The stored procedure is used to encapsulate the logic of inserting messages into the OutboxMessages table, making it easier to manage and maintain the code.
This setup is a crucial step in implementing the Outbox Pattern, as it allows us to track and manage messages in a centralized and efficient way.
Send Messages
To send messages to Apache Kafka, you need to create an interface named IKafkaProducer in a file named IKafkaProducer.cs and write the code in there. This interface will serve as the contract for any class that wants to produce messages to Kafka.
Create a new class named KafkaProducer in a file with the same name and a .cs extension, and write the code given in Listing 6 in there. This class will be responsible for producing messages to Kafka.
You should also register the instances of type ImessageRepository and IKafkaProducer with the service collection, as shown below. This will allow the application to use the IKafkaProducer instance to send messages to Kafka.
Implementation and Considerations
Implementing the outbox pattern in Kafka requires careful consideration of the producer configuration. The producer must be configured to use a specific partitioner, such as the murmur2 partitioner, to ensure that messages are distributed evenly across the cluster.
The outbox pattern relies on a separate topic for storing messages, which can lead to increased storage costs if not managed properly. To mitigate this, consider implementing a message expiration policy, such as the one described in the "Message Expiration" section.
The producer configuration also plays a crucial role in determining the performance of the outbox pattern. As mentioned in the "Producer Configuration" section, increasing the batch size can improve throughput, but may also increase latency.
Create Database Table
To create a database table, start by creating a new database. This can be done using a database script to create a database called Outbox_Pattern_Demo.
You'll need a database script to create the Outbox database table. The script to create the Outbox_Pattern_Demo database is the first step in this process.
A new database table called OutboxMessage needs to be created to store messages. To do this, use the following database script.
Sequence Diagram of the Complete Flow
The sequence diagram of the complete flow is a crucial aspect of our implementation. It's shown in Figure 8, which provides a clear visual representation of the entire process.
This diagram helps us understand the order of events and how each component interacts with others. The sequence diagram is a valuable tool for debugging and optimizing the system.
By examining Figure 8, we can see the complete flow of our implementation in action. It's a great way to visualize the system's behavior and identify potential bottlenecks.
The sequence diagram of the complete flow is a key component in ensuring the system's efficiency and effectiveness. It helps us anticipate and resolve issues before they arise.
Implementation Approaches
The outbox pattern requires a reliable mechanism to move messages from your outbox table to Kafka. This is a crucial step to ensure data consistency and integrity.
There are two primary approaches to implementing this mechanism, and they involve practical examples. The first approach uses a message broker, which is a reliable way to move messages from the outbox table to Kafka.
The second approach uses a scheduled task, which can be a simple way to implement the outbox pattern. However, it requires careful planning to ensure that the task runs reliably and without errors.
A message broker provides a reliable and scalable way to move messages from the outbox table to Kafka. It can handle high volumes of messages and provides features like message persistence and delivery guarantees.
A scheduled task, on the other hand, can be a simple and lightweight way to implement the outbox pattern. However, it requires careful planning to ensure that the task runs reliably and without errors.
Advanced Implementation Considerations
As you dive deeper into implementation, consider the scalability of your system. This means thinking about how it will handle increased data and user volume without compromising performance.
A good example is the need for load balancing, which helps distribute traffic across multiple servers. This ensures that no single server becomes overwhelmed and slows down the entire system.

Your system's architecture should be modular, allowing for easy updates and maintenance. This is crucial for avoiding downtime and keeping your system running smoothly.
For instance, a microservices architecture can help you achieve this modularity. By breaking down your system into smaller, independent services, you can update and maintain each one separately.
Regular monitoring and logging are essential for identifying and fixing issues before they become major problems. This helps you catch potential issues early on and prevent them from causing downtime.
A good logging system can help you track user behavior and identify areas for improvement. By analyzing your logs, you can gain valuable insights into how your system is being used and make data-driven decisions.
Transactional Alternatives
The Two-Phase Commit and the Saga Pattern are two popular alternatives to the Transactional Outbox Pattern.
Both of these alternatives aim to address the limitations and downsides of the outbox pattern.
The Two-Phase Commit, for instance, ensures that database commits and messaging systems are synchronized, eliminating the risk of inconsistent system state.
This is particularly important when dealing with distributed transactions, as seen in the example of dual writes failing due to the lack of distributed transaction support between databases and messaging systems.
The Saga Pattern, on the other hand, provides a more flexible approach by breaking down complex transactions into smaller, more manageable pieces.
This can help mitigate the risk of edge cases that can't be fully addressed by retry mechanisms, such as service crashes between operations.
In summary, the Two-Phase Commit and Saga Pattern offer viable alternatives to the Transactional Outbox Pattern, providing a more reliable and consistent way to handle distributed transactions.
Here's a comparison of the three approaches:
DB and Event Dual Write
DB and event dual write is a common pattern in microservice architectures where an application updates two separate systems without transactional guarantees between them.
This pattern appears frequently in microservice architectures where services need to maintain their own state while communicating changes to other services.
The dual write approach can be implemented in two ways: publishing an event to a messaging system like Kafka after saving data in the database, or saving data in the database after publishing an event to Kafka.
Publishing an event to Kafka after saving data in the database can lead to inconsistent system state if the event publishing fails after the database update.
Saving data in the database after publishing an event to Kafka can lead to inconsistent system state if the database update fails after the event is published.
The critical problem with dual writes is the lack of distributed transaction support between databases and messaging systems.
This limitation creates two primary failure modes: database commits but messaging fails, and message is sent but database update fails.
These failure modes result in inconsistent system state, violating one of the core principles of reliable systems.
Even with retry mechanisms, there are edge cases that canโt be fully addressed, such as service crashes between operations.
Execute the Application

To execute the application, you'll need to run the producer and consumer applications separately as they're part of different solutions. Start by launching ZooKeeper in a terminal window.
To set up the environment, start Kafka in another terminal window. This will allow the applications to communicate with each other.
Before running the applications, make sure to set appropriate breakpoints in the source code of both applications so that you can debug them as needed. This will help you identify and fix any issues that may arise.
To run the producer application, follow the steps outlined below. Start by executing the producer application, and then execute the consumer application.
You'll also need to launch the Postman Http Debugger tool. This will allow you to send an HTTP POST request to the producer API using Postman.
Here are the steps in a concise list:
- Start ZooKeeper in a terminal window.
- Start Kafka in another terminal window.
- Execute the producer application.
- Execute the consumer application.
- Launch the Postman Http Debugger tool.
- Send an HTTP POST request to the producer API using Postman.
Common Issues and Solutions
Transaction rollbacks can create orphaned messages, which refer to a state that never actually existed in the database. This is because the message is sent before the database transaction is committed, and then the transaction rolls back.
Network partitions can cause partial updates, allowing the database update to succeed but preventing the message from being sent. This creates an inconsistent state thatโs difficult to detect and resolve.
Service outages can create backlogs, where updates continue in the database while messages queue up or fail entirely. This leads to temporary or permanent inconsistency.
Ordering problems can occur without careful management, potentially violating business logic that depends on the sequence of events.
To avoid these issues, it's essential to implement a reliable messaging system that can handle failures and inconsistencies. Here are some common issues and their solutions:
By understanding these common issues and implementing effective solutions, you can build a reliable and consistent messaging system that meets the needs of your application.
Pros and Cons
The outbox pattern with Kafka is a widely adopted solution for achieving reliable message sending in distributed systems. It ensures reliable message sending by leveraging a persistent storage mechanism.
The outbox pattern decouples the sending process from the application's immediate response, minimizing the risk of message loss and improving system resilience. This is achieved by storing messages in the outbox and then sending them asynchronously.
Here are the key benefits of the outbox pattern with Kafka:
Despite these benefits, the outbox pattern with Kafka also has some downsides. Increased complexity is one of the main concerns, as it can introduce additional complexity to your architecture and development process.
Latency is another issue, as the outbox pattern may introduce a delay from the time an event occurs and when it's published to other services. This delay can be a problem in situations where immediate or real-time communication is needed.
Database coupling is also a challenge, as the outbox pattern depends on the database to store events before they're published. This can make future changes more challenging.
Scalability, resiliency, operational overhead, and integration challenges are also potential drawbacks of the outbox pattern with Kafka.
Notifications and Real-Time Data
Real-time notifications are a must-have in many applications, sending instant alerts and messages to users. You can use the outbox pattern to implement notifications in real-time.
The outbox pattern allows events to be processed asynchronously, triggering real-time notifications to logged-in users. This is particularly useful for applications that require instant updates.
In a real-world use case, the Notification service can listen for events in a Kafka topic, such as "order-created", and trigger notifications accordingly. This can be implemented using a standard Spring Kafka consumer.
The Notification service can immediately consume messages from Kafka and log notifications, as seen in the example where it logs a line: โNotifying user for created order 1 and product โCoffeeโ."
Discover more: Azure Service Bus vs Kafka
Real-World Example
In a real-world example, microservices-based applications often maintain their own data store to persist and retrieve data. This helps improve scalability and promotes agility in the deployment process.
These applications take advantage of messaging to facilitate asynchronous communication between services. Messaging solutions like Kafka, Azure Service Bus, and RabbitMQ are popular choices.
By using a data store, these applications can handle large amounts of data efficiently. This is especially useful for applications that require real-time data processing.
In practice, this means that each service in the application has its own data store, allowing for more flexibility and autonomy. This approach also helps reduce the risk of data inconsistencies.
Real-Time Notifications
Real-Time Notifications are crucial in modern applications to send instant alerts and messages. You can use the outbox pattern to implement notifications in real-time.
The outbox pattern allows events to be processed asynchronously, triggering real-time notifications to logged-in users of the application. This is particularly useful for applications that require immediate feedback, such as order confirmations or message notifications.
To implement real-time notifications, you can use a standard Spring Kafka consumer to listen for events in a Kafka topic. For example, the Notification service can listen for events in the "order-created" topic.
The Notification service will immediately consume the message from Kafka and log a line with the notification details. This can be seen in the example code, where the NotificationApplication logs a line with the order and product details.
Legacy System Integration
Integrating legacy systems with Kafka can be a challenge, but it's not impossible. You can use Change Data Capture (CDC) to capture raw data changes from these systems.
The CDC approach works well for unmodifiable systems, but it does come with some challenges. Maintaining transactional consistency can be more difficult, and you'll need to consider the Dual Write Problem.
To mitigate these challenges, you can use the Outbox Pattern. This involves applying stream processing to transform raw data changes into higher-level events. These events can then be published to Kafka topics for consumption.
The Outbox Pattern has several key principles and implementation considerations. It's essential to understand these to ensure a smooth integration process.
Here are some key principles of the Outbox Pattern:
- Dual Write Problem
- Consistency Challenge
- Testing
- Principles of the Outbox Pattern
- Outbox Pattern Implementation
- Advanced Implementation
- CDC Outbox Checklist
Ordering and Idempotence
Idempotence is crucial in distributed systems, and consumers need to handle potential duplicates by using a monotonically increasing value, like Postgres's LSN, instead of arbitrary IDs.
To ensure idempotence, consumers can track the highest processed LSN and ignore any messages with lower values. This approach requires storing only a single value rather than an ever-growing set of processed IDs.
In error scenarios, events published to Kafka may not be in the same order as they were generated in the application. This is because Spring Modulith can publish a previously incomplete event while events that came later are already published.
If ordering is important for your use case, CDC with Debezium is the best option as it guarantees ordering.
Idempotency in Consumers
Idempotency in Consumers is crucial when dealing with message delivery in distributed systems. At-least-once semantics can lead to duplicate messages being delivered, and consumers need to handle this.
To ensure idempotency, you can use a monotonically increasing value, like Postgres's LSN (Log Sequence Number). This value is generated automatically and is guaranteed to increase with each new log entry.
Consumers can track the highest processed LSN and ignore any messages with lower values. This approach requires storing only a single value rather than an ever-growing set of processed IDs.
Here's a summary of the steps to achieve idempotency in consumers:
- Use a monotonically increasing value (like Postgres's LSN)
- Track the highest processed LSN and ignore lower values
Two-Phase Commit
Two-Phase Commit can be a viable option for direct dual-write approaches, thanks to upcoming support for two-phase commit transactions in Kafka (KIP-939).
It increases the availability requirements, meaning both the database and Kafka must be available.
This approach brings additional overhead to synchronous request processing.
It may not provide significant advantages over the outbox pattern for most use cases.
Publish Event then Save to DB
Publishing an event to Kafka and then saving the order to the database seems like a straightforward approach, but it has its drawbacks. This approach is also known as the Dual Write Approach 1.
The pseudocode for this approach is: sendEventToKafka(); startTransaction(); saveOrderInDB(); commitTransaction();. However, if the event is published to Kafka but the local transaction to the database fails, the published event can't be rolled back, leading to an inconsistent system.
The creation of the Order is now dependent on the availability of Kafka, which means if Kafka is down, the request can't be fulfilled, even if the Order service itself is available.
What About Ordering?
Ordering is not guaranteed when using Spring Modulith to publish events to Kafka, especially in error scenarios where a previously incomplete event might be published while others that came later are already published.
If ordering is crucial for your use case, consider using CDC with Debezium, as it guarantees ordering.
In error scenarios, Spring Modulith can publish events out of order, making it a less reliable option for applications that require strict ordering.
Background Services and Scheduling
To implement background services and scheduling in an outbox pattern with Kafka, you can use the IHostedService interface in .NET Core.
You can create a background service by implementing the IHostedService interface, which contains two methods: StartAsync and StopAsync.
The BackgroundService class is an abstract helper class that already implements the IHostedService interface, making it easier to create a background service class.
The PublishOutboxMessagesAsync method of the OutboxMessageProcessor class sends outbox messages to Kafka by iterating through messages in the Outbox database table that are yet to be dispatched.
For your interest: Free Website Background Patterns
You can also use Quartz.NET to schedule a job that periodically processes unsent messages and sends them to Kafka.
Create a new folder named "Jobs" and add two new classes named "MessageSendingJob.cs" and "QuartzConfigurator.cs" to implement the Quartz.NET library.
The CreateUserJob class implements the IJob interface from the Quartz library and takes instances of the IOutboxRepository and IKafkaMessageSender interfaces in its constructor.
The Program class is responsible for configuring and scheduling the Quartz.NET job, creating a scheduler, starting it, creating a job of type CreateUserJob, and scheduling it to run every even 2 minutes.
To create a message consumer to consume messages stored in Kafka, create another ASP.NET Core application and create a new interface named IKafkaConsumer.
Create a new class named KafkaConsumer that implements the IKafkaConsumer interface and a service that runs in the background and consumes messages residing in Kafka by creating a class named KafkaMessageProcessor that extends the BackgroundService class.
Frequently Asked Questions
What is the difference between 2PC and outbox pattern?
2PC ensures strong consistency, but can be slow at scale, whereas the Outbox Pattern provides a more scalable solution with eventual consistency by decoupling transactions from event publishing
What is the difference between outbox and Kafka?
The outbox pattern stores event details in a database table, whereas Kafka is a message broker that sends events to a queue. This difference affects how events are processed and stored, with implications for system design and scalability.
Featured Images: pexels.com

