Golang Workflow Automation and Orchestration

Golang Workflow Automation and Orchestration is a powerful combination that can streamline your workflows and save you time.

Golang's concurrency features, such as goroutines and channels, enable efficient parallel processing of tasks.

With Golang's workflow automation capabilities, you can create complex workflows by combining simple tasks.

As we'll see in the following examples, this makes it easy to automate repetitive tasks and free up resources for more important things.

Golang's workflow orchestration features, on the other hand, allow you to manage and coordinate the execution of multiple tasks.

This is particularly useful when working with distributed systems or microservices architectures.

Recommended read: Azure Workflow

In Go, a workflow is made up of a series of tasks, each with its own set of inputs and outputs.

A workflow in Go can be represented using a directed acyclic graph (DAG), where each node represents a task and the edges represent the flow of data between tasks.

Each task in a Go workflow is executed concurrently, allowing for efficient processing of multiple tasks simultaneously.

Readers also liked: Golang vs Go

Activities are a crucial part of workflows, and they're surprisingly easy to work with.

Activities receive a plain context.Context as their first argument, and they're automatically retried by default, so it's good practice to make them idempotent. This means you don't have to worry about handling retries in your code.

To execute an activity, you'll need to register it first. You can register activities using a struct, which is useful if you want to provide some shared state to activities, like a database connection.

In Go programs, it's common to use context.Context to pass around request-scoped data.

Context propagation is supported in this library, allowing you to pass a ContextPropagator to the backend to propagate context values.

To use context propagation, you can refer to the context-propagation sample, which shows an example of how to use this feature.

By using context propagation, you can easily share data between activities and workflows.

Check this out: Golang Use Cases

Workflow execution is a crucial part of any workflow system, and Temporal makes it easy to manage. You can start a sub-workflow by calling `workflow.CreateSubWorkflowInstance`, which returns a Future that resolves once the sub-workflow has finished.

To execute a workflow, you'll need to start a worker, which listens to a specific task queue and executes workflows and activities registered to it. Workers can be scaled out horizontally, making them a great option for large-scale workflows.

The worker listens to the same task queue used to start the workflow execution, and it begins polling the queue for tasks to process. Once a task is found, the worker communicates the event back to the server, which sends activity tasks to the task queue for execution.

Here's a summary of the workflow execution process:

This process is recorded in the event history, which can be audited in the Temporal Web UI under the History tab. After a workflow completes, the history persists for a set retention period before being deleted.

The Worker plays a crucial role in executing workflows and activities. It's responsible for polling the Task Queue for tasks to process.

There are three different types of workers: the default worker, which listens to both workflow and activity queues, a workflow worker that only listens to workflow queues, and an activity worker that only listens to activity queues.

All workers have the same simple interface, allowing you to register workflows and activities, start the worker, and wait for pending tasks to finish.

A Worker listens only to the Task Queue it's registered to, and can only execute workflows and activities registered to it. It knows which code to execute from tasks it receives from the Task Queue.

Here are the key characteristics of a Worker:

After a Worker executes code, it returns the results back to the Temporal Server.

Executing activities is a crucial part of workflow execution. You can execute activities by calling workflow.ExecuteActivity, which returns a Future[T] that you can await to get the result or any error it might return.

The call to ExecuteActivity is straightforward, and it's essential to understand that activities are automatically retried by default. This means that if an activity fails, it will be retried until it succeeds or a maximum number of retries is reached.

Activities receive a plain context.Context as their first argument, and it's good practice to make them idempotent. Idempotence ensures that an activity's output is the same regardless of how many times it's executed with the same input.

Here are some key things to keep in mind when executing activities:

By following these guidelines, you can ensure that your activities are executed correctly and efficiently, which is essential for successful workflow execution.

Error handling is a crucial aspect of workflow management in Go. You can access the original type of an error via the err.Type field, and if a stacktrace was captured, you can access it via err.Stack(). This is useful when dealing with custom errors returned from activities and subworkflows.

If this caught your attention, see: Golang Create Error

To handle errors in workflows and activities, you can use the workflow.Error type, which is marshalled/unmarshalled by the library to wrap the original error. This allows you to access the original error type and stacktrace.

Temporary workflows can recover from unknown errors in activities by retrying the failed activity. This is demonstrated in the "Recover from an unknown error in an Activity" example, where a bug in the Deposit() Activity function causes the activity to fail, but the workflow continues to run and retry the failed activity.

Here are some key error handling concepts:

Error handling is a crucial aspect of any application, and Temporal provides robust features to handle errors in workflows and activities. You can access the original error type via the err.Type field, and if a stacktrace was captured, you can access it via err.Stack().

Custom errors returned from activities and subworkflows need to be marshalled/unmarshalled by the library, which wraps them in a workflow.Error. This allows you to handle errors in a more structured way.

To handle errors in workflows and activities, you can use the RetryPolicy specified when the Workflow first executes the Activity. This means that the Workflow will keep retrying the failed Activity until it succeeds.

If you want to avoid automatic retries when hitting certain error types, you can wrap an error with workflow.NewPermanentError. This will prevent the Workflow from retrying the failed Activity.

Here's a summary of the error handling options in Temporal:

When debugging errors, you can use the Temporal Web UI to view the state of the Workflow and explore the location and reason for failed Activity Tasks. This allows you to fix issues while the Workflow is running, without losing its state or restarting the transaction.

Cancellation is a crucial aspect of error handling in workflow management. You can cancel a workflow by passing a cancelable context to CreateSubWorkflowInstance.

If a sub-workflow is canceled, it's essential to react to the cancellation in a way that's similar to canceling a workflow via the Client. This is done by using the same mechanisms as described in Canceling workflows.

Canceling a workflow is as simple as creating a new context with workflow.NewDisconnectedContext, especially when you need to run activities or make calls using workflow.Context. This is necessary because the original context is canceled at this point.

To cancel a sub-workflow, you can pass a cancelable context to CreateSubWorkflowInstance and then cancel the sub-workflow. This approach is similar to timer cancellation and allows for a more efficient way of handling errors.

If your Update request is retried by the SDK Client until the calling context is cancelled, it's likely because there is no Workflow Worker polling the Task Queue.

Update failures are similar to Workflow failures, and issues that cause a Workflow failure in the main method also cause Update failures in the Update handler.

A Workflow Task Failure causes the server to retry Workflow Tasks indefinitely, and the outcome of your Update request depends on its stage.

Check this out: How to Update a Github Using Golang

If the Workflow finished while the Update handler execution was in progress, you'll receive a ServiceError "workflow execution already completed".

Here are some scenarios where Update issues might occur:

In the first scenario, the handler's failure will cause the Workflow Task to fail, leading to a retry of the Workflow Tasks indefinitely. In the second scenario, the Workflow finishing while the Update handler execution was in progress will result in a ServiceError "workflow execution already completed".

Concurrent processes can interact in unpredictable ways, making it crucial to manage concurrency correctly. This is especially true when multiple handler instances run simultaneously.

To prevent concurrent handler execution, use workflow.Mutex to lock access to specific sections of code. This ensures that only one handler instance can execute that code at a time.

Incorrectly written concurrent message-passing code may not work correctly, but coordinating access with workflow.Mutex corrects this issue. Locking makes sure that only one handler instance can execute a specific section of code at any given time.

Consider reading: Golang Code Comment Specifications

To handle Signal messages, receive them from their channel using GetSignalChannel. This allows the Workflow to change its state and control its flow. Before completing the Workflow or using Continue-As-New, make sure to do an asynchronous drain on the Signal channel to prevent Signals from being lost.

Temporal does not support Continue-as-New functionality within Update handlers, so complete all handlers before using Continue-as-New. Use Continue-as-New from your main Workflow function, just as you would complete or fail a Workflow Execution.

Worth a look: Gcloud Api Using Golang

You can cancel sub-workflows by passing a cancelable context to CreateSubWorkflowInstance.

This allows you to cancel the sub-workflow in a similar way to canceling a workflow via the Client.

Canceling a sub-workflow is the same as canceling a workflow via the Client.

You can achieve this by using the same methods and techniques as described in Canceling workflows.

This approach enables you to manage and control sub-workflows with precision and flexibility.

ContinueAsNew is a powerful feature that allows you to restart workflow execution with different inputs, keeping the history size small and preventing performance issues. This is particularly useful when you need to run a workflow multiple times with varying inputs.

The purpose of ContinueAsNew is to avoid hitting size limits and running out of memory, which can significantly impact performance. It works by returning a special error from your workflow that contains the new inputs.

Internally, the new execution starts with a fresh history, using the same InstanceID but a different ExecutionID. This means that the caller won't notice any difference, and only when the workflow ends without being restarted will the caller get the result and control will be passed back.

You can restart a sub-workflow without the caller noticing, until it ends without being restarted. At that point, the caller will get the result and control will be passed back.

Workers are responsible for executing workflows and activities, and they need to be registered with the worker. There are three types of workers: the default worker, workflow worker, and activity worker.

The default worker is a combined worker that listens to both workflow and activity queues. The workflow worker only listens to workflow queues, while the activity worker only listens to activity queues.

All workers have the same simple interface, which allows you to register workflows and activities, start the worker, and wait for pending tasks to be finished. This makes it easy to manage and scale your workers.

Here are the key characteristics of a worker:

This means that a worker is tightly coupled to the Task Queue and the Workflows and Activities it is registered to execute.

Queues play a crucial role in concurrency and synchronization, allowing workers to pull tasks from multiple queues. By default, workers listen to two queues: the default queue and the _system_ queue for system workflows and activities.

You can configure other queues for your workers to listen to, and all worker options structs take a Queues option. For example, when starting workflows, creating sub-workflow instances, or scheduling activities, you can pass a queue to specify where the task should be scheduled.

Here are the default behaviors for queue selection:

This allows for flexible and efficient task management, enabling workers to focus on specific tasks and queues as needed.

Signals are a way to send a message to a running workflow instance. You can send a signal to a workflow by calling workflow.Signal and listen to them by creating a SignalChannel via NewSignalChannel.

A Signal is an asynchronous message sent to a running Workflow Execution to change its state and control its flow. It's essential to handle Signal messages by receiving them from their channel.

To handle Signal messages, you can pass the Signal's name to GetSignalChannel to get the Signal Channel that listens for Signals of that type. Alternatively, you might want the Workflow to proceed and still be capable of handling external Signals.

A fresh viewpoint: Golang Channels

Here are some key considerations when working with Signal Channels:

You can send a Signal to a Workflow Execution from a Temporal Client or from another Workflow Execution. However, you can only send Signals to Workflow Executions that haven’t closed.

Concurrency and Synchronization can get messy, especially when dealing with multiple processes interacting at the same time. This can lead to unpredictable behavior, like in the case of concurrent message-passing code that may not work correctly when multiple handler instances run simultaneously.

Incorrectly written concurrent code can cause problems, but there's a solution. Use workflow.Mutex to prevent concurrent handler execution.

Coordinating access with workflow.Mutex is essential to avoid pathological cases. Locking makes sure that only one handler instance can execute a specific section of code at any given time, preventing other instances from interfering. This ensures that your code runs smoothly and predictably.

Broaden your view: Golang Source Code

Testing and Debugging is a crucial part of any workflow system. go-workflows makes this process easier with its built-in support for testing workflows.

You can test workflows using a simple example with mocked activities, where timers are automatically fired by advancing a mock workflow clock. This allows you to test workflow behavior without actually running the workflow.

go-workflows also provides a diagnostic web UI for investigating workflows. You can serve it via diag.NewServeMux to get a simple paginated list of workflow instances.

The diagnostic web UI also includes a way to inspect the history of a workflow instance, making it easier to debug issues. This feature is especially useful when dealing with complex workflows.

A unique perspective: Go Programming Language Web Development

Testing is a crucial part of any development process, and workflows are no exception.

Go-workflows makes testing workflows a breeze, with a simple example that uses mocked activities.

Timers are automatically fired when you advance a mock workflow clock, which is used for testing workflows.

You can also register callbacks to fire at specific times, sending signals, canceling workflows, and more.

This testing approach differs depending on the backend implementation.

Logging is a crucial aspect of testing and debugging, allowing you to understand what's happening in your workflows.

go-workflows supports structured logging using the slog package, which is used by default unless you pass a custom logger with backend.WithLogger when creating a backend instance.

This means you can easily switch to a custom logging solution if needed, without having to rewrite your code.

Structured logging makes it easier to track down issues by providing a clear and organized format for your log messages.

The Analyzer is a crucial tool in the testing and debugging process. It's a simple golangci-lint based analyzer that spots common issues in workflow code.

This tool is specifically designed to catch problems that might have otherwise gone unnoticed. It's a great way to ensure that your workflow code is clean and error-free.

The Analyzer uses golangci-lint, a popular tool for Go code analysis, to identify potential issues in your code. This includes things like formatting errors, dead code, and more.

By using the Analyzer, you can catch these issues early on and fix them before they become bigger problems. This saves you time and frustration in the long run.

The Diagnostics Web UI is a valuable tool for investigating workflows. It's included in the package and can be served via diag.NewServeMux.

This UI provides a simple paginated list of workflow instances. You can easily navigate through the list to find the instance you're interested in.

The UI also offers a way to inspect the history of a workflow instance. This feature allows you to see the step-by-step progression of the workflow, which can be super helpful for debugging.

Check this out: Create a List of Structs Golang

When testing and debugging, it's essential to have a variety of samples to work with. Having a mix of different data sets can help you catch issues that might not be apparent with just one type of data.

A good starting point is to use a small sample of 10-20 items to test your code. This can help you quickly identify any major issues.

Having a diverse set of samples, including edge cases, can also help you catch issues that might not be apparent with just a small sample. For example, a sample that includes extreme values, such as very large or very small numbers, can help you catch issues related to data overflow or underflow.

Using a mix of different data types, such as strings, numbers, and booleans, can also help you catch issues related to data type mismatches. For example, a sample that includes a mix of string and numeric values can help you catch issues related to trying to perform arithmetic operations on a string.

Having a sample that includes duplicate values can also help you catch issues related to data duplication. For example, a sample that includes multiple identical values can help you catch issues related to trying to insert a duplicate value into a database.

A sample that includes missing or invalid data can also help you catch issues related to data validation. For example, a sample that includes a mix of valid and invalid data can help you catch issues related to trying to validate data that is not in the correct format.

You need to register and configure different tasks as well as workflow activities. This involves setting up the main file, Main.go, where you can define and manage these tasks and activities.

Curious to learn more? Check out: Golang Test Main

The next step is to set up the Workflow folder code, which is responsible for executing the workflow and workflow activity. This is where the actual workflow logic comes into play.

Now it's time to focus on the activity.go file, where you can place the actual business logic of the task. This is the heart of the workflow, where the task's functionality is defined.

By following these steps, you'll be well on your way to creating a robust and efficient workflow.

In Go, goroutines are lightweight threads that can be created and managed with ease. They're a fundamental part of the language's concurrency model.

The Go runtime automatically schedules goroutines, allowing you to write concurrent code without worrying about thread management. This makes it easy to write high-performance, concurrent programs.

Context is a key concept in Go's concurrency model, allowing you to cancel or timeout long-running goroutines. It's used extensively in the language's standard library.

You might like: Golang Go

You can create a backend and start a worker in a separate go-routine, allowing for more flexibility in your workflow.

To finish an example, you create a Client instance which you can use to create a new workflow instance, giving you the ability to manage multiple workflow instances from different processes.

With the exception of the in-memory backend, you don't have to start the workflow from the same process the worker runs in, making it easy to create a client from another process.

A workflow instance is just one running instance of a previously registered workflow, allowing you to have multiple instances of the same workflow running at the same time.

You can create/wait for/cancel/... workflow instances from another process, giving you more control over your workflow management.

You can schedule timers to fire at any point in the future by calling workflow.ScheduleTimer. It returns a Future you can await to wait for the timer to fire.

Timers can be named for tracing and debugging purposes, which can be super helpful when you're trying to figure out what's going on in your code.

Scheduling timers is a great way to add some extra functionality to your workflow, like sending a reminder or triggering a specific task.

You can use timers to create more complex workflows by combining them with other features, like conditional statements and loops.

To use timers effectively, you need to await the Future returned by ScheduleTimer, which allows your code to pause and wait for the timer to fire.

For more insights, see: Golang Use .env File

As you work with GoLang workflows, you'll inevitably encounter errors. The Client might not be able to contact the server, or the workflow might not exist.

The Client waits for a response from the Worker for Queries and Updates, but if an issue occurs during handler execution, the Client may receive an exception.

To minimize downtime, make sure your workflow exists and is properly set up. If issues persist, check your network connection to ensure the Client can contact the server.

Here are some common errors to watch out for:

These errors are often the result of a misconfigured workflow or network issue. By being aware of these potential pitfalls, you can take steps to prevent them and keep your workflow running smoothly.

To get started with Golang workflow, you'll need to ensure that Go is installed on your machine. You can download it from the official Go website.

First, you'll need to install the Temporal Go SDK, which you can do by running a command from the official documentation. This will allow you to interact with Temporal.

Next, you'll need to set up a new Go module for your project. This involves creating a new directory and running the command `go mod init`. You'll also need to create a `.env` file, which will contain configuration details such as the Temporal host and task queue.

Here's a list of prerequisites to keep in mind:

By following these steps, you'll be well on your way to starting your Golang workflow project.

The backend is responsible for persisting workflow events, and there are multiple implementations to choose from. You can use an in-memory backend for testing, or opt for a more robust solution like SQLite, MySQL, or Redis.

There are three backend implementations maintained in this repository, each with its own set of options. Some of these options include setting timeouts for sticky tasks, workflow task locks, and activity task locks.

Here are some of the common options that all backend implementations accept:

By choosing the right backend implementation and configuring it to your needs, you can ensure that your workflow events are persisted correctly and efficiently.

Temporal is a powerful tool for building scalable and reliable applications. It allows you to start a Workflow Execution with the input parameters it expects.

You can start a Workflow Execution by connecting to the Temporal Cluster, specifying the Task Queue the Workflow should use, and providing the input parameters. This can be done synchronously or asynchronously.

Temporal uses a Task Queue to manage Workflows and Activities. You define a Task Queue by assigning a name as a string. This name is used when starting a Workflow Execution and when defining Workers.

In Temporal, a Workflow is a sequence of tasks that are executed in a specific order. A Workflow can have multiple tasks, including PrimaryTask and SecondaryTask. If a PrimaryTask fails, the Workflow will fail. If a SecondaryTask fails, it will also mark the Workflow as failed.

Temporal uses a Worker to execute the Activity and Workflow functions. A Worker is a wrapper around your compiled Workflow and Activity code. It communicates the results back to the Temporal Server.

To start a Workflow, you can use the Client.SignalWithStartWorkflow API or the Client.UpdateWithStartWorkflow API. These APIs allow you to start a Workflow Execution and pass the Signal or Update at the same time.

Here are the steps to start a Workflow:

1. Connect to the Temporal Cluster

2. Specify the Task Queue the Workflow should use

3. Start the Workflow with the input parameters it expects

By following these steps, you can start a Workflow Execution and take advantage of Temporal's scalability and reliability features.

Additional reading: Golang Rest Api

Temporal provides a number of features to help you manage your Workflows, including logging and error handling. You can get a logger using workflow.Logger or activity.Logger, which already has the workflow instance or activity ID set as a default field.

Temporal also provides a number of APIs to help you manage your Workflows, including the Client.SignalWithStartWorkflow API and the Client.UpdateWithStartWorkflow API. These APIs allow you to start a Workflow Execution and pass the Signal or Update at the same time.

Here are the API parameters:

By using these APIs and features, you can build scalable and reliable applications with Temporal.

To get started with Temporal, you'll need to meet some prerequisites. Ensure that Go is installed on your machine, and you can download it from the official Go website. You can also run the Temporal server using Docker with a specific command.

You'll need to create a new Go module for your project. This involves creating a .env file, which is a configuration file that contains details like the temporal host and task queue information.

Temporal has a specific constraint: Workflows need to be replayable. This means you can't simply incorporate your logic into the Workflow Definition.

To interact with Temporal, you'll need to install the Temporal Go SDK. The installation command is mentioned in the official documentation. If you're using VSCode, you can configure the .vscode with launch.json.

Here's a summary of the prerequisites:

To get started, you'll need to download the example application.

The application is available in a GitHub repository, which you can access by cloning the repository using git in a new terminal window.

You can use the GitHub Template repository as a foundation for your own Temporal application, as it's easily convertible to a new repository.

Make sure to change the name in the go.mod file if you decide to create a new repository based on the template.

The banking service in the example application simulates an external API call, which you can inspect in the banking-client.go file.

Here's an interesting read: Golang Applications