Getting Started with Confluent Kafka Golang

Author

Reads 917

IT professional working on a computer in a modern office setting, focused on coding and tasks.
Credit: pexels.com, IT professional working on a computer in a modern office setting, focused on coding and tasks.

To start working with Confluent Kafka in Golang, you'll need to install the Confluent Kafka Go client library, which can be done using the go get command.

This library provides a Go API for interacting with Apache Kafka, allowing you to produce and consume messages.

First, ensure you have Go installed on your machine, as the Confluent Kafka Go client library is a Go package.

The Go client library supports both synchronous and asynchronous programming models, giving you flexibility in your application design.

Consider reading: Golang vs Go

Using Go Modules

You can use Go Modules to install confluent-kafka-go, which makes it easy to manage dependencies in your Go projects.

Import the kafka package from GitHub in your code, and Go Modules will automatically add a dependency to the latest stable version of confluent-kafka-go to your go.mod file.

If you're building for Alpine Linux (musl), you'll need to specify the -tags musl flag to ensure everything works smoothly.

This approach is supported by Go versions starting from 1.13, making it a reliable option for many developers.

Recommended read: Golang Go

Installing Librdkafka

Credit: youtube.com, Kafka Client Setup In GoLang

Installing librdkafka is a crucial step in setting up confluent kafka golang. You can install it manually on your build and target system using one of the alternatives provided.

For Debian and Ubuntu based distros, install librdkafka-dev from the standard repositories or using Confluent's Deb repository. This will ensure you have the necessary development package for librdkafka.

Redhat based distros require installing librdkafka-devel using Confluent's YUM repository. This package includes the development tools needed for librdkafka.

If you're on MacOS X, you can install librdkafka from Homebrew. You may also need to install pkg-config if you don't already have it.

For Alpine, install librdkafka-dev and pkgconf using apk add.

Note that confluent-kafka-go is not supported on Windows.

Here's a quick rundown of the installation methods for each platform:

After installing librdkafka, you'll need to build your Go application with the -tags dynamic flag. This is required for confluent-kafka-go to work properly. If you're using the master branch of the Go client, you'll also need to use the master branch of librdkafka.

Kafka Producer

Credit: youtube.com, Kafka Producers Explained | Apache Kafka® On The Go

The Kafka Producer is a crucial component of Confluent's Golang client for Apache Kafka. It's responsible for producing messages to a Kafka topic.

There are two main ways to use the Kafka Producer in Go: the Function-Based Producer and the Channel-Based Producer.

The Function-Based Producer is a non-blocking call, meaning it won't wait for the message to be sent before returning. However, if the internal librdkafka queue is full, the call will fail and can be retried.

Here are the key differences between the Function-Based and Channel-Based Producers:

The Channel-Based Producer is designed to handle backpressure by queuing messages in the channel and then inside librdkafka. This approach provides better performance and reliability.

The Producer operates asynchronously, putting messages in the internal transmit queue and returning immediately. To get failed messages, you need to listen to the events channel for kafka.Message type events and check for TopicPartition.Error.

Confluent's Golang client for Apache Kafka offers commercial support and is designed to be future-proof, keeping pace with core Apache Kafka and Confluent Platform components.

Recommended read: Golang Channel

Kafka Consumer

Credit: youtube.com, What is a Kafka Consumer and How does it work?

To poll messages, errors, and events, you can use the consumer.Poll() function, which has a direct mapping to the underlying librdkafka functionality.

This function is used to retrieve messages, errors, and events from the Kafka topic.

The consumer.Poll() function returns events, including rebalance events, which can be handled by creating the consumer with the go.application.rebalance.enable configuration.

To enable rebalance events, you must create the consumer with this configuration and handle AssignedPartitions and RevokedPartitions events by calling Assign() and Unassign() respectively.

The consumer.Poll() function is a key part of working with Kafka in Go, allowing you to retrieve data from the topic in a flexible and efficient way.

Producer and Consumer

In Confluent Kafka Golang, the Producer is a crucial component that sends messages to Kafka topics.

To enable rebalance events in a Kafka consumer, you must create the consumer with the go.application.rebalance.enable configuration.

The Poll() method returns events, including rebalance events, which you can handle explicitly by calling Assign() and Unassign() for AssignedPartitions and RevokedPartitions respectively.

You must handle AssignedPartitions and RevokedPartitions events by explicitly calling Assign() and Unassign().

Rebalance events are exposed as events returned by the Poll() method.

To see these events, you must create the consumer with the go.application.rebalance.enable configuration and handle AssignedPartitions and RevokedPartitions events.

Basic Usage

Credit: youtube.com, #44 Golang - Mastering Kafka with Golang: A Beginner's Guide

A typical Confluent Kafka Golang consumer application is centered around a consume loop that repeatedly calls the Consumer.Poll method to retrieve records one-by-one.

This method returns as soon as a message or event is available, or after the specified timeout (in milliseconds), whichever is sooner. You can specify a timeout of -1 to indicate an infinite timeout, but it's advisable to specify a short timeout to allow your application to be interrupted.

You should always call Consumer.Close after you're finished using the consumer to ensure active sockets are closed and internal state is cleaned up.

For more insights, see: Golang Programs

Channel-Based Producer

The Channel-Based Producer is a great way to produce messages in a Go application. It allows you to write messages to a producer channel, and delivery reports are emitted on the producer's Events() or specified private channel.

Proper channel backpressure is handled if the librdkafka internal queue is full. This means that if the queue is full, the channel will block and prevent further messages from being written, preventing data loss.

A Man Looking at a Computer Screen with Data
Credit: pexels.com, A Man Looking at a Computer Screen with Data

Double queueing occurs in the Channel-Based Producer, where messages are first queued in the channel (size is configurable) and then inside librdkafka. This can help to prevent data loss and ensure that messages are delivered correctly.

Here are the key benefits of the Channel-Based Producer:

  • Proper channel backpressure if librdkafka internal queue is full.
  • Double queueing: messages are first queued in the channel (size is configurable) and then inside librdkafka.

The Channel-Based Producer is a reliable and efficient way to produce messages in a Go application, and its benefits make it a great choice for many use cases.

Basic Usage

A typical Kafka consumer application is centered around a consume loop, which repeatedly calls the Consumer.Poll method to retrieve records one-by-one that have been efficiently pre-fetched by the consumer in background threads.

You can specify a timeout of -1 to indicate an infinite timeout, but since librdkafka is not yet able to respond to SIGINT events, it's advisable to specify a short timeout to allow your application to be interrupted.

The Consumer.Poll method returns as soon as a message or event is available, or after the specified timeout (in milliseconds), whichever is sooner. You should not specify a timeout of 0 to avoid busy looping.

From above crop faceless male developer in black hoodie writing software code on netbook while working in light studio
Credit: pexels.com, From above crop faceless male developer in black hoodie writing software code on netbook while working in light studio

To join a consumer group and specify which topics to consume from, you should call the Consumer.SubscribeTopics method.

You should always call Consumer.Close after you are finished using the consumer to ensure active sockets are closed and internal state is cleaned up.

A latch is added to the example to ensure that the consumer has time to finish closing before finishing shut down, which is a good practice to follow.

Writing to Kafka

Writing to Kafka is a key aspect of working with Confluent's Golang client for Apache Kafka. You can initiate a send by calling the Produce() method, passing a Message object and an optional chanEvent.

To produce asynchronously, you can use a goroutine to handle message delivery reports and optionally other event types concurrently. This is a good idea, as synchronous writes can decrease throughput.

You can make writes synchronous, but it's generally not recommended, by receiving from the delivery channel passed to the Produce() method call. Alternatively, you can wait for all messages to be acknowledged and use the Flush() method.

Suggestion: Golang Message

Credit: youtube.com, Apache Kafka 101: Partitioning (2023)

Important: Flush() will only serve the producer's Events() channel, not application-specified delivery channels.

Here are some key things to keep in mind when writing to Kafka:

  • Asynchronous writes are generally preferred to avoid decreasing throughput.
  • Make sure to process the Events() channel, as it contains client-level errors and control messages.
  • Use a goroutine to handle message delivery reports and other event types concurrently.

Committing to Kafka

Committing to Kafka is a crucial step in ensuring data consistency and reliability. Synchronous commits are provided by the Go client's Commit() method call.

To manually commit offsets, disable the enable.auto.commit configuration. This is essential to avoid conflicts with the auto-commit mechanism.

You can trigger a synchronous commit every MIN_COMMIT_COUNT messages. This ensures that the committed position is updated regularly.

Alternatively, you can trigger a commit on expiration of a timeout. This guarantees that the committed position is updated even if no messages are being processed.

Manual offset commit is a powerful tool, but it requires careful configuration to ensure data integrity. By disabling auto-commit and using manual commits, you can fine-tune your Kafka integration to meet your specific needs.

Delivery Guarantees

Delivery Guarantees are crucial when working with Confluent Kafka in Go. You can get "at least once" delivery by committing a message after processing, but this can be changed to "at most once" delivery by committing before processing.

Credit: youtube.com, Delivery Guarantees & Transactions | Apache Kafka for .NET Developers

Committing on every message can produce a lot of overhead, so a better approach is to collect a batch of messages, execute the synchronous commit, and then process the messages only if the commit succeeded.

To achieve synchronous writes, you can receive the delivery channel passed to the Produce() method call. Alternatively, you can wait for all messages to be acknowledged and use the Flush() method.

Be aware that Flush() will only serve the producer's Events() channel, not application-specified delivery channels. If Flush() is called and no goroutine is processing the delivery channel, its buffer may fill up and cause the flush to timeout.

Installation

Installation is a breeze, and you can choose from a few options. You can use Go Modules to install confluent-kafka-go, which will automatically add a dependency to your go.mod file.

For Debian and Ubuntu based distros, you can install librdkafka-dev from the standard repositories or using Confluent's Deb repository. For Redhat based distros, install librdkafka-devel using Confluent's YUM repository.

If you're on MacOS X, you can install librdkafka from Homebrew, and you may also need to brew install pkg-config if you don't already have it. For Alpine, you can use apk add librdkafka-dev pkgconf. Note that confluent-kafka-go is not supported on Windows.

Related reading: Golang Add to Map

Go Installation

Credit: youtube.com, Go Installation Tutorial: Windows, macOS, Linux - Complete Setup Guide 🐹

To install the Go client, you can use GitHub. The Go client, called confluent-kafka-go, is distributed via GitHub and as confluent-kafka-go to pin to specific versions.

The Go client uses librdkafka, the C client, internally and exposes it as a Go library using cgo. Starting with confluent-kafka-go v1.4.0, the librdkafka client is now included in the Go client and no separate installation of librdkafka is required for the supported platforms.

For other platforms, you'll need to install the C client and its development package as well as a C build toolchain including pkg-config. On Red Hat-based Linux distributions, install the following packages in addition to librdkafka: librdkafka-devel and pkg-config.

On Debian-based distributions, install the following in addition to librdkafka: librdkafka-dev and pkg-config. Next, use goget to install the library.

You can also use Go Modules to install confluent-kafka-go, which was introduced in Go 1.13. This method allows you to import the kafka package from GitHub in your code. A dependency to the latest stable version of confluent-kafka-go should be automatically added to your go.mod file.

For more insights, see: Install Golang Package

Install

Man in White Dress Shirt Analyzing Data Displayed on Screen
Credit: pexels.com, Man in White Dress Shirt Analyzing Data Displayed on Screen

To install the confluent-kafka-go client, you can use Go Modules. This is the recommended approach, as it allows you to easily manage dependencies and keep your code up-to-date.

First, import the kafka package from GitHub in your code: `import "github.com/confluentinc/confluent-kafka-go/kafka"`.

If you're using Alpine Linux (musl), you'll need to specify the `-tags musl` flag.

A dependency to the latest stable version of confluent-kafka-go should be automatically added to your go.mod file.

If you're on a Debian or Ubuntu-based distro, you can install the client by running `go get github.com/confluentinc/confluent-kafka-go/kafka`.

On Red Hat-based Linux distributions, you'll need to install `librdkafka-devel` in addition to `librdkafka`.

Here are the specific packages you'll need to install on different platforms:

If Go Modules can't be used, you can version pin the confluent-kafka-go import to v1 using gopkg.in.

License and Configuration

To use Confluent Kafka with Go, you'll need to install the Confluent Kafka Go client. This client provides a simple way to interact with Kafka topics and brokers.

Clipboard with Statistical Data and Digital Tablet with Stock Market Display on Screen
Credit: pexels.com, Clipboard with Statistical Data and Digital Tablet with Stock Market Display on Screen

The Confluent Kafka Go client supports both the Kafka 0.9 and 2.0 APIs. You can choose the API version that best fits your needs. Confluent Kafka Go client supports both synchronous and asynchronous operations.

To configure the Confluent Kafka Go client, you'll need to set the Kafka broker list, which is a list of host and port pairs that identify the Kafka brokers in your cluster. You can also set the client ID, which is used to identify the client to the Kafka brokers. Confluent Kafka Go client supports SSL/TLS encryption for secure communication with Kafka brokers.

The Confluent Kafka Go client provides several configuration options that can be used to customize its behavior. These options include the number of retries, the amount of time to wait for a response, and the maximum number of concurrent requests.

Check this out: Golang Set Env Variable

Margarita Champlin

Writer

Margarita Champlin is a seasoned writer with a passion for crafting informative and engaging content. With a keen eye for detail and a knack for simplifying complex topics, she has established herself as a go-to expert in the field of technology. Her writing has been featured in various publications, covering a range of topics, including Azure Monitoring.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.