Event-Driven Architecture in Golang with Watermill and Docker

Author

Reads 468

Music Band Performs on Stage during Nighttime
Credit: pexels.com, Music Band Performs on Stage during Nighttime

Event-driven architecture in GoLang can be a powerful tool for building scalable and maintainable systems. This is especially true when using libraries like Watermill, which simplifies the process of building event-driven systems.

Watermill is a Go library that provides a simple and efficient way to build event-driven systems. With Watermill, you can easily create and publish events, as well as subscribe to and process them.

One of the key benefits of using Watermill is that it allows you to easily decouple different components of your system, making it easier to develop and test individual components independently. This is particularly useful when working with microservices architecture.

By using Watermill, you can also take advantage of Docker containers to isolate and deploy individual components of your system, making it easier to manage and scale your application.

Benefits and Principles

Event-driven architecture in GoLang offers several benefits, including improved scalability and reduced tight coupling between services. This is achieved by decoupling services through events, allowing them to communicate asynchronously and evolve independently.

Credit: youtube.com, Event-Driven Architecture: Explained in 7 Minutes!

By using events instead of direct calls, services become more flexible and independent. For example, an Orders service can emit an OrderCreated event, and any service that cares can subscribe and react on its own terms.

Decoupling through events reduces coordination and failure propagation, making it easier to build robust and scalable applications. Prefer event-based communication between systems to reap these benefits.

The benefits of event-driven architecture include:

  • Improved scalability
  • Reduced tight coupling between services
  • Increased flexibility and independence of services

By understanding these benefits and principles, you can design and build more robust and scalable applications using event-driven architecture in GoLang.

System Design

In an event-driven architecture, the system design plays a crucial role in ensuring reliability and scalability. A key aspect of this design is the use of a transactional outbox, which allows for strong consistency in database operations.

The typical workflow in this system involves a single database transaction that inserts both the order and the event into separate tables. This transaction is committed only if both inserts succeed. This approach ensures that either both operations are successful or neither is, maintaining the integrity of the data.

Explore further: Event Web Page Design

Credit: youtube.com, Unlock the Power of Event-Driven Architecture: How Netflix & Uber Handle Billions of Events

The system also employs a background consumer process that scans the outbox table periodically, querying for entries where the published_at field is NULL. This process then publishes each event to Kafka (Redpanda) and marks the event as sent in the database.

Here's a summary of the system design:

See what others are reading: Domain-driven Design with Golang

Design for Failure

Design for Failure is a crucial concept in system design. It means accepting that failures will happen and building systems that can still function correctly even when something goes wrong.

Reliable systems should be designed to isolate failure domains, so a broken dependency doesn't take down the entire service. This is known as graceful degradation.

Kafka is a great example of a system that can fail, but your service should still be able to accept new orders by queuing the event in the DB to be published later.

Isolating failure domains is key to building robust systems. It allows your service to continue functioning even when a dependency is unavailable.

Designing for failure requires a mindset shift from building perfect systems to building systems that can recover from failures.

Check this out: Golang Design

Streams

Credit: youtube.com, What's Stream Processing + When Do We Use It? | Systems Design Interview 0 to 1 with Ex-Google SWE

Event streams are a key concept in system design, allowing consumers to read events starting with the earliest event or beginning with new events as they are added.

Unlike message queues, event streams continue to grow indefinitely until events are removed, which can be done by configuring a maximum stream length or archiving based on age.

Event streams provide retention and the ability to replay events, making them a better choice than message queues when you need these features.

An event stream will continue to grow indefinitely until outside forces intervene, as shown in the example provided in Figure 1.6 – Event stream.

Using event streams instead of message queues can help you avoid the issue of message queues returning to their default empty state.

Event streams are particularly useful when you need to replay events, such as in the Outbox pattern, which will be introduced in Chapter 6, Asynchronous Connections.

Carried State Transfer

Designing systems that can handle failures is crucial, and a robust design ensures the system still behaves correctly even when failures happen.

Credit: youtube.com, Event Carried State Transfer: Keep a local cache!

Reliable systems should embrace the idea that failures are normal, and a good example of this is using event-carried state transfer, which is an asynchronous cousin to REST.

Event-carried state transfer is a push model where data changes are sent out to be consumed by any components that might be interested, making it a great way to handle complex workflows across components.

In event-carried state transfer, data changes are sent as events, and components can create their own local cached copies, negating any need to query the originating component to fetch any information.

This approach is seen in the example of PaymentReceived as an event-carried state transfer, where the event is sent with additional information, changing how downstream services might react to it.

Design tip: Use event-carried state transfer to create an asynchronous workflow that can handle complex operations, and remember to include all necessary details in the event to avoid any need for querying the originating component.

Golang CQRS, Metrics, AMQP with Watermill v0.3.0

Credit: youtube.com, CQRS that you should have used but may not have heard of | Advanced Development Concepts

Golang is particularly suited to building event-driven architectures due to its efficient concurrency model and performance.

The go.cqrs library provides a framework for implementing CQRS and event sourcing in Golang, defining the core abstractions for commands, events, and aggregates.

Watermill v0.3.0 is a significant release after 54 days of work, with 12,909 lines of code, 47 Monsters, and 42 KFC Twisters.

Watermill is a Go library for building event-driven applications the easy way, like an HTTP router but for messages.

It's a library, not a framework, so there's no vendor lock-in, and it's close to 9k GitHub stars and 79 contributors.

Watermill is excellent for AI applications due to its built-in concurrency, making orchestrating parallel API calls much easier than languages like Python.

Career transitions within tech are easier internally, and starting simple is often the best approach, adding complexity only when necessary.

Intriguing read: Golang Test Framework

Docker Compose: Postgres, Kafka, and Services

Docker Compose makes it easy to run multiple services together.

Statue of Kafka in Czech Republic
Credit: pexels.com, Statue of Kafka in Czech Republic

In our system, we use Docker Compose to manage Postgres and Kafka services.

Postgres is our database, and we use the official Postgres image in our Docker Compose file.

Kafka is our message broker, and we use the official Confluent Kafka image in our Docker Compose file.

In our Docker Compose file, we define a Postgres service that uses the official Postgres image and exposes port 5432.

We also define a Kafka service that uses the official Confluent Kafka image and exposes port 9092.

By using Docker Compose, we can easily start and stop both services together with a single command.

Our Kafka service depends on the Postgres service, so we make sure to start Postgres before starting Kafka.

This allows us to create a Kafka topic and producer that can write to the Postgres database.

We can then use Docker Compose to scale our services up or down as needed.

Here's an interesting read: Azure Event Hub vs Service Bus

Distributed Workflows

Golang's efficient concurrency model makes it well-suited for building event-driven architectures.

Credit: youtube.com, System Design Interview - Distributed Message Queue

Complex workflows across components can be achieved using events, making the workflow entirely asynchronous.

This approach creates eventual consistency, where each component may not have the final state of the application when queried, but it will eventually.

The UX and collaboration of components involved in the operation need to be evaluated on their own to determine the correct solution for the problem.

Each component may need to be queried separately to determine its state, due to eventual consistency.

Asynchronous event consumers can be used to add new services, like the Analytics service, to the application.

This approach allows for more flexibility and scalability in the system design.

Components and Tools

At the heart of event-driven architecture in Go is the event pattern, which consists of four core components: event, queue, producer, and consumer. These components work together to enable efficient communication between different parts of an application.

A key component of the event pattern is the message queue, which is useful for simple publisher/subscriber scenarios when subscribers are actively running or can retrieve events quickly enough. Message queues can be used to handle a high volume of events.

Watermill is a Go library that enables efficient working with message streams, making it a valuable tool for building event-driven applications. It's designed to support event sourcing, CQRS, RPC over messages, and sagas.

Additional reading: Golang Go

Implementation and Examples

Credit: youtube.com, How to build an event-driven backend application in Go with Encore

In a Go-based event-driven architecture, the event bus is a crucial component that enables loose coupling between services. The event bus is an abstraction layer that allows services to publish and subscribe to events without knowing about each other.

A key advantage of using an event bus is that it enables services to be developed and deployed independently, without affecting other services in the system. This decoupling makes it easier to scale and maintain the system over time.

To implement an event bus in Go, you can use a library like Gorilla's Bolt or a message broker like RabbitMQ. These libraries provide a simple and efficient way to publish and subscribe to events in a distributed system.

If this caught your attention, see: Golang vs Go

Go CQRS Implementation Basics

In Go, aggregates represent the core of your domain logic and encapsulate the state, ensuring all changes to that state are consistent and valid. This is achieved through aggregates listening for commands, executing the associated business logic, and applying state changes through events.

Credit: youtube.com, How to Implement the CQRS Pattern in Clean Architecture (from scratch)

Aggregates in Go.cqrs listen for commands and execute the associated business logic, applying state changes through events.

Commands and events are the core of the communication in CQRS. A service or user interface sends commands, and the aggregates process them, resulting in events that propagate through the system.

A key strength of Go.cqrs is its integration with an event store, which keeps a log of all events. This log can then be used to rebuild the current state of the application.

Here's a breakdown of the key components in Go CQRS implementation:

  • Aggregates: Represent the core of your domain logic and encapsulate the state.
  • Event Handlers: Publish and subscribe to events asynchronously.
  • Commands and Events: Core of the communication, where commands are sent and processed, resulting in events that propagate through the system.
  • Event Store: Integrates with the event store, keeping a log of all events to rebuild the current state of the application.

Note that Go.cqrs provides a framework for implementing CQRS and event sourcing in Go, defining the core abstractions for commands, events, and aggregates.

Orders Service

The Orders Service is a crucial part of any e-commerce system. It's responsible for saving new orders to a database and notifying other systems that an order was created.

Imagine you're ordering a new book online. Your request to create an order is sent to the Orders Service, which then saves the new order to the database and sends an event to a message queue like Kafka.

Credit: youtube.com, The Saga Pattern in Microservices (EDA - part 2)

The typical workflow of the Orders Service involves a client sending a request to create an order, which is then saved in a single database transaction along with an event. This ensures strong consistency and prevents partial state changes.

Here's a breakdown of the typical workflow:

The Orders Service uses an asynchronous event-driven approach with a transactional outbox to ensure that the order is stored in the DB and an event is queued in the outbox, but nothing has been sent to Kafka yet. This approach provides a fast response to the client without waiting for Kafka delivery.

Tools and Libraries

Watermill is a Go library for building event-driven applications efficiently. It's designed to work with message streams and supports event sourcing, CQRS, RPC over messages, and sagas.

There's no standard messaging library in Go, which is why Watermill was created. This library helps to fill that gap and make building event-driven applications easier.

Credit: youtube.com, Gopher in an Event-Driven playground | Tamimi | Conf42 Golang 2022

Watermill is a library, not a framework, which means you don't need to change your architecture to use it. It's as easy to use as building an HTTP server in Go.

The library is designed to be event-driven, making it perfect for building message-driven applications. With Watermill, you can think of it like an HTTP router but for messages.

Watermill has been around for a while, with the first release being version 1.0 five years ago. Since then, there have been several updates, including version 1.4 and version 1.3.

Watermill is open-source, which means it's free to use and modify. This is great for developers who want to contribute to the library or use it in their own projects.

The latest release, version 1.4, was a big deal, marking five years since the first release. The library continues to be actively maintained and updated with new features and improvements.

On a similar theme: Golang Version Manager

Best Practices and Patterns

In event-driven architecture, it's essential to separate concerns and keep handlers focused on a single responsibility.

Credit: youtube.com, A Beginner's Guide to Event-Driven Architecture

This means using a single handler function per event type, as shown in the example of the `EventHandler` struct. This approach makes it easier to maintain and update code.

By keeping handlers simple and focused, you can avoid complex conditional statements and unnecessary logic. For instance, in the `EventHandler` struct, each handler function is responsible for a specific event, making the code more modular and efficient.

User Experience (UX)

User experience (UX) is all about making things easy and convenient for the user. With the rapid growth of IoT devices and people constantly having their phones in hand, users expect instant updates on the latest news and events.

Millions of people rely on event-driven applications to send updates on orders, shipment notifications, and more. This type of application is already sending updates in real-time, making it easier for organizations to extend this functionality to users.

Users expect seamless interactions with applications, and event-driven applications are well-suited to meet this expectation.

Consistency

Credit: youtube.com, Journey to EDA: Patterns, Best Practices & Practical Tips • David Boyne • GOTO 2023

Eventual consistency can be a challenge for distributed applications, where changes may not be immediately available and queries may produce stale results until the change has been fully recorded.

This can be particularly problematic for event-driven applications, which are more likely to deal with eventual consistency issues.

To ensure data consistency across boundaries, consider using a pattern like Transactional Outbox, which lets you save both the core data and the outbound event in a single DB transaction.

This ensures that they succeed or fail together, guaranteeing consistency between your database and event streams.

Avoid naïve "double-writes" to different systems, as this can lead to inconsistencies.

Use atomic patterns like Outbox to guarantee consistency between your database and event streams.

Dual writes, where you're changing the application state in two or more places during an operation, can also lead to consistency issues.

For event-driven applications, this can mean making a change locally to a database, and then publishing an event either about the event or the event itself.

Credit: youtube.com, How to design great Aggregate Roots in Domain-Driven Design

If the events you intend to publish never make it to the event broker, then your state changes cannot be shared, and post-operation operations will never happen.

To address this challenge, consider publishing your events into the database alongside the rest of the changes to keep the state change atomic.

This allows a second record of to-be-published events to be created, adding additional resiliency on top of what you got from using an event broker between components.

By following these best practices and patterns, you can ensure that your event-driven application remains consistent and reliable.

Maintainable Code with Layered Separation

Organizing code into layers makes systems easier to test, change, and scale. This approach keeps business logic clean and independent of tech details.

By separating code into layers like domain, use case, and infrastructure, you can change or upgrade one part without rewriting everything. This is especially useful when working with infrastructure like Postgres or Kafka.

Credit: youtube.com, Layered Architecture Explained: Best Practices & Design Patterns | layered architecture tutorial

This layered approach also makes it easier to plug in new technologies as needed. For example, if you need to use a different database, you can simply swap out the infrastructure layer without affecting the rest of the system.

Business logic stays clean and independent of tech details, making it easier to maintain and update. This is a key benefit of layered separation.

By structuring your system around well-defined contracts and interfaces, you can easily add new features or functionality without a full rewrite. This is a key principle of designing for extensibility.

Expand your knowledge: Google Analytics Key Events

Transactional Outbox Pattern

The Transactional Outbox Pattern is a game-changer for ensuring data consistency across boundaries. It lets you save both the core data and the outbound event in a single DB transaction, ensuring they succeed or fail together.

Avoiding naïve “double-writes” to different systems is crucial. Use atomic patterns like Outbox to guarantee consistency between your database and event streams.

Credit: youtube.com, What is the Transactional Outbox Pattern? | Designing Event-Driven Microservices

A Transactional Outbox pattern like the one in Example 3 is where you explicitly begin a transaction and do two inserts: one to orders and one to outbox. Only after both succeed do you commit.

If any step fails, you roll back so the database isn’t left in a partial state. This approach keeps your database clean and prevents inconsistencies.

In a Clean Architecture implementation, the OrderRepository.Create(order, tx) could handle the SQL, and similarly an OutboxRepository.Save(event, tx). But the core idea remains: wrap the operations in one atomic transaction.

You can store the entire order as JSON in the OutboxEvent.Payload, like in Example 3, to ensure the event has all necessary info. This approach lets other services rely solely on the event without calling back to the order service.

However, storing the needed data in the outbox event can introduce coupling. It’s a design choice – here we went with self-contained events.

The Transactional Outbox Pattern is a great way to prepare for future scalability and extensibility. It lets you publish events into the database alongside the rest of the changes to keep the state change atomic.

Credit: youtube.com, Outbox Pattern: Fixing event failures in an event-driven architecture

This approach adds additional resiliency on top of what you got from using an event broker between components, as illustrated in the diagram in Example 4.

By using the Transactional Outbox Pattern, you can ensure that your system is event-driven and scalable, and that you can change or upgrade one part without rewriting everything.

This pattern makes change cheap and favors clear boundaries, reusable logic, and loosely coupled components. It’s a great way to design for clarity, resilience, and growth.

For another approach, see: Gcloud Api Using Golang

Conclusion

Event-driven architecture in GoLang can be a game-changer for building scalable and efficient systems.

By using goroutines and channels to handle events, developers can write concurrent code that's both readable and maintainable.

In our example, we saw how to create a simple event-driven system using goroutines and channels. This approach allowed us to handle multiple events simultaneously without blocking the main thread.

Event-driven architecture in GoLang is particularly useful for handling real-time data streams, as shown in our example where we used a goroutine to process incoming data.

Credit: youtube.com, Patterns of Event Driven Architecture - Mark Richards

With event-driven architecture, developers can write code that's more modular and reusable, making it easier to maintain and update over time.

By leveraging the power of goroutines and channels, developers can build high-performance systems that can handle large volumes of data and events.

In the end, event-driven architecture in GoLang is a powerful tool for building scalable and efficient systems that can handle real-time data streams and events.

Ismael Anderson

Lead Writer

Ismael Anderson is a seasoned writer with a passion for crafting informative and engaging content. With a focus on technical topics, he has established himself as a reliable source for readers seeking in-depth knowledge on complex subjects. His writing portfolio showcases a range of expertise, including articles on cloud computing and storage solutions, such as AWS S3.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.