Golang Task Queue with Redis and Postgres Support

Author

Reads 316

People in masks queue at an amusement park, reflecting new COVID-19 health protocols.
Credit: pexels.com, People in masks queue at an amusement park, reflecting new COVID-19 health protocols.

Implementing a task queue in Golang is a straightforward process, thanks to libraries like Bull and go-redis. With Bull, you can easily create a queue that supports Redis as a backend, allowing for efficient task management and execution.

Bull provides a simple API for adding, processing, and deleting tasks from the queue. For example, you can use the `Add` method to add a task to the queue, and the `Process` method to execute the task.

Redis is a popular choice for task queues due to its high-performance capabilities and ease of use. In a Bull setup, Redis acts as a message broker, storing tasks and handling their execution.

In contrast, Bull also supports Postgres as a backend, making it suitable for use cases that require a more traditional database setup.

Installation

To install a task queue in Go, you'll need to initialize a Go module by running `go mod init github.com/your/repo` in your project folder.

Intriguing read: Go vs Golang

Credit: youtube.com, #49 Golang - Task Scheduler in Go with gocron

Make sure you're running a Go version with modules support, as taskq only supports the last 2 Go versions.

Before installing taskq, you'll need to install it specifically with the version number, like `taskq/v3`, to avoid a common mistake.

To start using Asynq, you'll need to install it with the `go get` command.

You'll also need to ensure you have a Redis server running, either locally or in a Docker container, with version 4.0 or higher.

Here's an interesting read: Golang Go

Producer and Consumer

In a Golang task queue, it's essential to separate the tasks into two distinct processes: the producer and the consumer. The producer accepts requests from customers and adds tasks to queues, while the consumer fetches tasks from the queues and processes them.

One of the key benefits of isolating producers and consumers from each other is that producers will continue working even if consumers start crashing. This allows for greater reliability and fault tolerance in your system.

Discover more: Azure Pipelines Tasks

Three young people working on computers in a creative office with vibrant graffiti walls.
Credit: pexels.com, Three young people working on computers in a creative office with vibrant graffiti walls.

You can scale producers and consumers separately, which means you can add more producers to handle a higher volume of requests without affecting the consumer's processing capacity.

Having different configurations for producers and consumers is also a good practice. For example, you might use large network timeouts in consumers to prevent them from timing out due to slow network connections.

In this approach, you can use a Redis backend, as demonstrated in the redisexampleopen example.

For your interest: Golang Use Cases

Features

The golang task queue offers a range of features that make it a reliable and efficient tool for managing tasks. One of the key benefits is guaranteed at least one execution of a task, ensuring that your tasks are always completed.

You can also schedule tasks, which is especially useful for tasks that need to run at specific times or intervals. The queue also supports retries of failed tasks, so you can be confident that your tasks will be completed even if there are issues along the way.

Additional reading: Onedrive Reporting Task

Yellow letter tiles spelling 'to do' against a vibrant blue background for organizational themes.
Credit: pexels.com, Yellow letter tiles spelling 'to do' against a vibrant blue background for organizational themes.

Here are some of the key features of the golang task queue:

  • Guaranteed at least one execution of a task
  • Scheduling of tasks
  • Retries of failed tasks
  • Automatic recovery of tasks in the event of a worker crash
  • Weighted priority queues
  • Strict priority queues
  • Low latency to add a task since writes are fast in Redis
  • De-duplication of tasks using unique option
  • Allow timeout and deadline per task
  • Allow aggregating group of tasks to batch multiple successive operations
  • Flexible handler interface with support for middlewares
  • Ability to pause queue to stop processing tasks from the queue
  • Periodic Tasks
  • Support Redis Sentinels for high availability
  • Integration with Prometheus to collect and visualize queue metrics
  • Web UI to inspect and remote-control queues and tasks
  • CLI to inspect and remote-control queues and tasks

Message Deduplication

Message Deduplication is a key feature that helps prevent duplicate tasks from being processed. It's like having a quality control system that ensures only unique tasks are executed.

If a Message has a Name, it's used as a unique identifier, and messages with the same name won't be processed again within a 24 hour period. This helps prevent redundant tasks from being executed.

Non-deduplication occurs when the Name is omitted, and each message will be processed. This is useful when you need to process every message, even if it's a duplicate.

OnceWithArgs sets a name based on a consistent hash of the arguments and a quantised period of time, such as 'this hour' or 'today'. This guarantees that the same function won't be called with the same arguments during that period.

Here's a summary of the deduplication options:

  • Name-based deduplication within a 24 hour period
  • Non-deduplication (each message will be processed)
  • OnceWithArgs deduplication based on consistent hash and quantised period

New Redis Client Inspector in v0.25.0

People Working at the Office
Credit: pexels.com, People Working at the Office

In version 0.25.0, a new Inspector can be created from a Redis client using NewInspectorFromRedisClient.

The new function returns a new instance of Inspector given a redis.UniversalClient, but you're responsible for closing the underlying Redis connection pool, as Asynq won't do it for you.

With the introduction of NewInspectorFromRedisClient, you can now create an Inspector instance more easily, but don't forget to manage the Redis connection pool properly.

The new Inspector instance can then be used to inspect and manage tasks in your Redis-based system.

A unique perspective: T Golang

Result Writer 0.19.0

Result Writer 0.19.0 is a significant update that brings a client interface to write result data for a task.

This interface is specifically designed to write data to the Redis instance the server is connected to.

The ResultWriter is a game-changer for developers who need to manage result data efficiently.

Curious to learn more? Check out: Check Type of Interface Golang

Backends

To create a queue factory, you need to choose a backend, such as Redis or SQS.

Redis and SQS are two popular options for a backend, allowing you to create a queue factory.

For Redis, you can use it as a backend for your queue factory, providing a high-performance and scalable solution.

New Redis Client in v0.25.0

Four People Working in Call Center Office
Credit: pexels.com, Four People Working in Call Center Office

In v0.25.0, a new Redis client was introduced, allowing for more flexibility in backend configurations. This new client is created using the `NewClientFromRedisClient` function.

The `NewClientFromRedisClient` function returns a new instance of `Client` given a `redis.UniversalClient`. This function is a game-changer for anyone looking to integrate Redis into their backend.

A warning is issued when using `NewClientFromRedisClient`: the underlying Redis connection pool will not be closed by Asynq, so you're responsible for closing it. This means you need to keep track of your Redis connections to avoid any potential issues.

River: Fast & Robust Postgres Tool

River is a fast and robust Postgres tool that's worth checking out. It's designed to handle high traffic and large datasets with ease.

River uses a connection pool to manage multiple connections to the database, which can significantly improve performance. This is especially useful for applications that need to handle a large number of concurrent requests.

River's connection pool can handle up to 100,000 connections, making it suitable for even the most demanding applications.

Handling Jobs

Credit: youtube.com, #68 Golang - Master Background Tasks with Asynq: Complete Guide

Handling jobs in a Golang task queue is a breeze, thanks to the flexible and robust design of the system. A handler can be a zero-argument function, a function whose arguments are assignable in type from those which are passed in the message, or a function which takes a single *Message argument.

To process tasks, a handler should return nil if the task is successful. If the handler returns a non-nil error or panics, the task will be retried after a delay if retry-count is remaining, otherwise the task will be archived.

Here are the possible return values from ProcessTask, along with their consequences:

To prevent duplicate processing of messages, you can use message deduplication. If a Message has a Name, it will be used as a unique identifier, and messages with the same name will be deduplicated within a 24 hour period. If a period is passed to OnceWithArgs, it guarantees that the same function will not be called with the same arguments during that period.

A fresh viewpoint: Azure Messaging Queue

Job Management

Credit: youtube.com, Fast queue implementation in golang! #golang #programming #coding #softwareengineer #computerscience

Job Management is a breeze in Go thanks to its elegant and tight code. Goroutines and channels make it simple to create a job queue and worker that process jobs off the queue.

A simple queue can be created with a channel of Job objects, with a capacity of 100. This allows for safe and correct code without the need for threads and mutexes.

Producer throttling is another feature that comes in handy, allowing you to control the rate at which jobs are added to the queue.

Recommended read: Golang Source

All Archived Items

Managing archived tasks can be a chore, but it's a necessary one to keep your workflow organized. The DeleteAllArchivedTasks function is a game-changer for this task.

This feature, added in inv0.14.0, allows you to delete all archived tasks from a specified queue and receive a report on the number of tasks deleted.

To take advantage of this feature, you can use the DeleteAllArchivedTasks function to quickly clear out any archived tasks that are no longer needed. This can help declutter your workflow and make it easier to find the tasks that matter most.

Worth a look: Ms Outlook Tasks

All Pending

Professional Man holding Papers
Credit: pexels.com, Professional Man holding Papers

You can delete all pending tasks from a specified queue using the DeleteAllPendingTasks function. This function is available in the Inspector type, added in version inv0.18.0.

Deleting all pending tasks can be a useful feature, especially when you need to clear out a queue and start fresh. The DeleteAllPendingTasks function reports the number of tasks deleted.

The capacity of a job queue can be set when creating the channel, such as a capacity of 100 in the example code. This can help prevent the queue from growing too large and overflowing.

Producer throttling is another feature that can be useful when managing a job queue. It prevents producers from overwhelming the queue with too many jobs at once.

A different take: Golang Func Type

Recommended read: Golang Queue

Scheduler

A Scheduler in the context of a Go task queue is a powerful tool for running tasks at regular intervals based on a user-defined schedule. It's safe for concurrent use by multiple goroutines.

The Scheduler is responsible for kicking off tasks, and it always returns a non-nil handler. This ensures that tasks are executed with a valid handler.

A Close-Up Shot of a Calendar with a Schedule
Credit: pexels.com, A Close-Up Shot of a Calendar with a Schedule

You can create a new Scheduler instance using the `NewScheduler` function, which takes an optional `opts` parameter that defaults to nil if not provided. Alternatively, you can use `NewSchedulerFromRedisClient` to create a Scheduler from a Redis universal client, but be aware that you're responsible for closing the underlying Redis connection pool.

Process

The Process function is a crucial part of the Scheduler.

It returns an option to specify when to process the given task.

One key thing to note is that if there's a conflicting ProcessIn option, the last option passed to Enqueue overrides the others.

Scheduler Entries In v0.14.0

In v0.14.0, the Inspector's SchedulerEntries function returns a list of all entries registered with currently running schedulers.

This function is a valuable tool for monitoring and managing your schedulers. It provides a clear view of all the tasks that are currently being scheduled.

SchedulerEntries returns a list of entries, which can be useful for debugging and troubleshooting purposes.

For your interest: Queue Trigger Azure Function

Server Run 0.8.0

Credit: youtube.com, #50 Golang - Task Scheduler in Go with gocron - Part 2

The Server's Run function starts the task processing and blocks until an os signal to exit the program is received.

Once it receives a signal, it gracefully shuts down all active workers and other goroutines to process the tasks. This ensures a clean shutdown and prevents any potential issues.

The Run function returns any error encountered at server startup time. If the server has already been shutdown, it returns ErrServerClosed.

This means you can rely on the Server to handle shutdowns cleanly, without worrying about leaving tasks hanging.

In practice, this has saved me from headaches when working with Asynq. I've seen it handle shutdowns smoothly, even in high-stress scenarios.

The Run function is a crucial part of the Server's lifecycle, and it's essential to understand how it works to get the most out of Asynq.

By using the Run function, you can ensure that your tasks are processed efficiently and that your server shuts down cleanly.

Take a look at this: Golang Run Debug Mode

Configuration and Error Handling

Credit: youtube.com, Go Background Jobs with River Queue

The "type Config" specifies the server's background-task processing behavior, which is crucial for a golang task queue.

This configuration determines how tasks are processed, and it's essential to get it right to avoid errors.

In the context of a golang task queue, a well-configured server ensures that tasks are executed efficiently and without hiccups.

A different take: Azure Key Vault Task

Stop In 0.8.0

In 0.8.0, the Server's Stop method was added to signal the server to stop pulling new tasks off queues. This allows you to ensure that all currently active tasks are processed before shutting down the server.

Stop does not shut down the server itself, so be sure to call Shutdown afterwards to complete the process. This is important to avoid any issues that might arise from a premature shutdown.

The Stop method is particularly useful when you want to give the server a chance to finish processing its tasks before bringing it down. This can be especially helpful in situations where you need to ensure that all work is completed before shutting down.

Config

Credit: youtube.com, Configuring error handling settings for Lambda asynchronous invocations

The Config section of your application's configuration is crucial for defining its behavior. It specifies the server's background-task processing behavior.

In this section, you'll find the type Config, which is a vital part of the configuration. The type Config specifies the server's background-task processing behavior.

Config is used to determine how your application handles tasks in the background. This can be especially important for applications that rely on asynchronous processing.

The type Config is the only option mentioned in this section, implying that it's the default or recommended choice for most use cases.

On a similar theme: Golang Application

Handle Errors

A Handler function can take a single *Message argument, which can be used to process tasks.

If a task's processing is unsuccessful, the Server will schedule it for a retry. A task will be retried until it gets processed successfully or reaches its max retry count.

To skip a task's retry and archive it immediately, return a SkipRetry error from the Handler function. This error can be returned directly or wrapped in another error.

If this caught your attention, see: Golang Create Error

Credit: youtube.com, My FAVORITE Error Handling Technique

When a task exhausts its retries, it will be moved to the archive and kept there until the archive reaches its max size. At that point, the oldest tasks in the archive will be deleted.

Here's a summary of how errors are handled in different cases:

Directories

In the asynq package, the foundational types and constants are defined in the base package.

The base package is a crucial part of the asynq package, providing a solid foundation for the other packages to build upon.

The errors package defines the error type and functions used by asynq and its internal packages.

This is important for handling errors and exceptions in a consistent and predictable way.

The log package exports logging related types and functions, which can be used to track and debug issues.

This can be especially helpful when things go wrong and you need to figure out what happened.

Here is a list of the main directories in the asynq package:

The testbroker package exports a broker implementation that should be used in package testing.

This is useful for ensuring that the package is working correctly in different testing scenarios.

A fresh viewpoint: Install Golang Package

Nancy Rath

Copy Editor

Nancy Rath is a meticulous and detail-oriented Copy Editor with a passion for refining written content. With a keen eye for grammar, syntax, and style, she has honed her skills in ensuring that articles are polished and engaging. Her expertise spans a range of categories, including digital presentation design, where she has a particular interest in the intersection of visual and written communication.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.