
To start working with Confluent Kafka in Golang, you'll need to install the Confluent Kafka Go client library, which can be done using the go get command.
This library provides a Go API for interacting with Apache Kafka, allowing you to produce and consume messages.
First, ensure you have Go installed on your machine, as the Confluent Kafka Go client library is a Go package.
The Go client library supports both synchronous and asynchronous programming models, giving you flexibility in your application design.
Consider reading: Golang vs Go
Using Go Modules
You can use Go Modules to install confluent-kafka-go, which makes it easy to manage dependencies in your Go projects.
Import the kafka package from GitHub in your code, and Go Modules will automatically add a dependency to the latest stable version of confluent-kafka-go to your go.mod file.
If you're building for Alpine Linux (musl), you'll need to specify the -tags musl flag to ensure everything works smoothly.
This approach is supported by Go versions starting from 1.13, making it a reliable option for many developers.
Recommended read: Golang Go
Installing Librdkafka
Installing librdkafka is a crucial step in setting up confluent kafka golang. You can install it manually on your build and target system using one of the alternatives provided.
For Debian and Ubuntu based distros, install librdkafka-dev from the standard repositories or using Confluent's Deb repository. This will ensure you have the necessary development package for librdkafka.
Redhat based distros require installing librdkafka-devel using Confluent's YUM repository. This package includes the development tools needed for librdkafka.
If you're on MacOS X, you can install librdkafka from Homebrew. You may also need to install pkg-config if you don't already have it.
For Alpine, install librdkafka-dev and pkgconf using apk add.
Note that confluent-kafka-go is not supported on Windows.
Here's a quick rundown of the installation methods for each platform:
After installing librdkafka, you'll need to build your Go application with the -tags dynamic flag. This is required for confluent-kafka-go to work properly. If you're using the master branch of the Go client, you'll also need to use the master branch of librdkafka.
Kafka Producer
The Kafka Producer is a crucial component of Confluent's Golang client for Apache Kafka. It's responsible for producing messages to a Kafka topic.
There are two main ways to use the Kafka Producer in Go: the Function-Based Producer and the Channel-Based Producer.
The Function-Based Producer is a non-blocking call, meaning it won't wait for the message to be sent before returning. However, if the internal librdkafka queue is full, the call will fail and can be retried.
Here are the key differences between the Function-Based and Channel-Based Producers:
The Channel-Based Producer is designed to handle backpressure by queuing messages in the channel and then inside librdkafka. This approach provides better performance and reliability.
The Producer operates asynchronously, putting messages in the internal transmit queue and returning immediately. To get failed messages, you need to listen to the events channel for kafka.Message type events and check for TopicPartition.Error.
Confluent's Golang client for Apache Kafka offers commercial support and is designed to be future-proof, keeping pace with core Apache Kafka and Confluent Platform components.
Recommended read: Golang Channel
Kafka Consumer
To poll messages, errors, and events, you can use the consumer.Poll() function, which has a direct mapping to the underlying librdkafka functionality.
This function is used to retrieve messages, errors, and events from the Kafka topic.
The consumer.Poll() function returns events, including rebalance events, which can be handled by creating the consumer with the go.application.rebalance.enable configuration.
To enable rebalance events, you must create the consumer with this configuration and handle AssignedPartitions and RevokedPartitions events by calling Assign() and Unassign() respectively.
The consumer.Poll() function is a key part of working with Kafka in Go, allowing you to retrieve data from the topic in a flexible and efficient way.
Producer and Consumer
In Confluent Kafka Golang, the Producer is a crucial component that sends messages to Kafka topics.
To enable rebalance events in a Kafka consumer, you must create the consumer with the go.application.rebalance.enable configuration.
The Poll() method returns events, including rebalance events, which you can handle explicitly by calling Assign() and Unassign() for AssignedPartitions and RevokedPartitions respectively.
You must handle AssignedPartitions and RevokedPartitions events by explicitly calling Assign() and Unassign().
Rebalance events are exposed as events returned by the Poll() method.
To see these events, you must create the consumer with the go.application.rebalance.enable configuration and handle AssignedPartitions and RevokedPartitions events.
Basic Usage
A typical Confluent Kafka Golang consumer application is centered around a consume loop that repeatedly calls the Consumer.Poll method to retrieve records one-by-one.
This method returns as soon as a message or event is available, or after the specified timeout (in milliseconds), whichever is sooner. You can specify a timeout of -1 to indicate an infinite timeout, but it's advisable to specify a short timeout to allow your application to be interrupted.
You should always call Consumer.Close after you're finished using the consumer to ensure active sockets are closed and internal state is cleaned up.
For more insights, see: Golang Programs
Channel-Based Producer
The Channel-Based Producer is a great way to produce messages in a Go application. It allows you to write messages to a producer channel, and delivery reports are emitted on the producer's Events() or specified private channel.
Proper channel backpressure is handled if the librdkafka internal queue is full. This means that if the queue is full, the channel will block and prevent further messages from being written, preventing data loss.

Double queueing occurs in the Channel-Based Producer, where messages are first queued in the channel (size is configurable) and then inside librdkafka. This can help to prevent data loss and ensure that messages are delivered correctly.
Here are the key benefits of the Channel-Based Producer:
- Proper channel backpressure if librdkafka internal queue is full.
- Double queueing: messages are first queued in the channel (size is configurable) and then inside librdkafka.
The Channel-Based Producer is a reliable and efficient way to produce messages in a Go application, and its benefits make it a great choice for many use cases.
Basic Usage
A typical Kafka consumer application is centered around a consume loop, which repeatedly calls the Consumer.Poll method to retrieve records one-by-one that have been efficiently pre-fetched by the consumer in background threads.
You can specify a timeout of -1 to indicate an infinite timeout, but since librdkafka is not yet able to respond to SIGINT events, it's advisable to specify a short timeout to allow your application to be interrupted.
The Consumer.Poll method returns as soon as a message or event is available, or after the specified timeout (in milliseconds), whichever is sooner. You should not specify a timeout of 0 to avoid busy looping.

To join a consumer group and specify which topics to consume from, you should call the Consumer.SubscribeTopics method.
You should always call Consumer.Close after you are finished using the consumer to ensure active sockets are closed and internal state is cleaned up.
A latch is added to the example to ensure that the consumer has time to finish closing before finishing shut down, which is a good practice to follow.
Expand your knowledge: Golang Reflect to Call Function in Package
Writing to Kafka
Writing to Kafka is a key aspect of working with Confluent's Golang client for Apache Kafka. You can initiate a send by calling the Produce() method, passing a Message object and an optional chanEvent.
To produce asynchronously, you can use a goroutine to handle message delivery reports and optionally other event types concurrently. This is a good idea, as synchronous writes can decrease throughput.
You can make writes synchronous, but it's generally not recommended, by receiving from the delivery channel passed to the Produce() method call. Alternatively, you can wait for all messages to be acknowledged and use the Flush() method.
Suggestion: Golang Message
Important: Flush() will only serve the producer's Events() channel, not application-specified delivery channels.
Here are some key things to keep in mind when writing to Kafka:
- Asynchronous writes are generally preferred to avoid decreasing throughput.
- Make sure to process the Events() channel, as it contains client-level errors and control messages.
- Use a goroutine to handle message delivery reports and other event types concurrently.
Committing to Kafka
Committing to Kafka is a crucial step in ensuring data consistency and reliability. Synchronous commits are provided by the Go client's Commit() method call.
To manually commit offsets, disable the enable.auto.commit configuration. This is essential to avoid conflicts with the auto-commit mechanism.
You can trigger a synchronous commit every MIN_COMMIT_COUNT messages. This ensures that the committed position is updated regularly.
Alternatively, you can trigger a commit on expiration of a timeout. This guarantees that the committed position is updated even if no messages are being processed.
Manual offset commit is a powerful tool, but it requires careful configuration to ensure data integrity. By disabling auto-commit and using manual commits, you can fine-tune your Kafka integration to meet your specific needs.
Worth a look: Define a Map of Custom Data Type Golang
Delivery Guarantees
Delivery Guarantees are crucial when working with Confluent Kafka in Go. You can get "at least once" delivery by committing a message after processing, but this can be changed to "at most once" delivery by committing before processing.
Committing on every message can produce a lot of overhead, so a better approach is to collect a batch of messages, execute the synchronous commit, and then process the messages only if the commit succeeded.
To achieve synchronous writes, you can receive the delivery channel passed to the Produce() method call. Alternatively, you can wait for all messages to be acknowledged and use the Flush() method.
Be aware that Flush() will only serve the producer's Events() channel, not application-specified delivery channels. If Flush() is called and no goroutine is processing the delivery channel, its buffer may fill up and cause the flush to timeout.
Installation
Installation is a breeze, and you can choose from a few options. You can use Go Modules to install confluent-kafka-go, which will automatically add a dependency to your go.mod file.
For Debian and Ubuntu based distros, you can install librdkafka-dev from the standard repositories or using Confluent's Deb repository. For Redhat based distros, install librdkafka-devel using Confluent's YUM repository.
If you're on MacOS X, you can install librdkafka from Homebrew, and you may also need to brew install pkg-config if you don't already have it. For Alpine, you can use apk add librdkafka-dev pkgconf. Note that confluent-kafka-go is not supported on Windows.
Related reading: Golang Add to Map
Go Installation
To install the Go client, you can use GitHub. The Go client, called confluent-kafka-go, is distributed via GitHub and as confluent-kafka-go to pin to specific versions.
The Go client uses librdkafka, the C client, internally and exposes it as a Go library using cgo. Starting with confluent-kafka-go v1.4.0, the librdkafka client is now included in the Go client and no separate installation of librdkafka is required for the supported platforms.
For other platforms, you'll need to install the C client and its development package as well as a C build toolchain including pkg-config. On Red Hat-based Linux distributions, install the following packages in addition to librdkafka: librdkafka-devel and pkg-config.
On Debian-based distributions, install the following in addition to librdkafka: librdkafka-dev and pkg-config. Next, use goget to install the library.
You can also use Go Modules to install confluent-kafka-go, which was introduced in Go 1.13. This method allows you to import the kafka package from GitHub in your code. A dependency to the latest stable version of confluent-kafka-go should be automatically added to your go.mod file.
For more insights, see: Install Golang Package
Install

To install the confluent-kafka-go client, you can use Go Modules. This is the recommended approach, as it allows you to easily manage dependencies and keep your code up-to-date.
First, import the kafka package from GitHub in your code: `import "github.com/confluentinc/confluent-kafka-go/kafka"`.
If you're using Alpine Linux (musl), you'll need to specify the `-tags musl` flag.
A dependency to the latest stable version of confluent-kafka-go should be automatically added to your go.mod file.
If you're on a Debian or Ubuntu-based distro, you can install the client by running `go get github.com/confluentinc/confluent-kafka-go/kafka`.
On Red Hat-based Linux distributions, you'll need to install `librdkafka-devel` in addition to `librdkafka`.
Here are the specific packages you'll need to install on different platforms:
If Go Modules can't be used, you can version pin the confluent-kafka-go import to v1 using gopkg.in.
License and Configuration
To use Confluent Kafka with Go, you'll need to install the Confluent Kafka Go client. This client provides a simple way to interact with Kafka topics and brokers.

The Confluent Kafka Go client supports both the Kafka 0.9 and 2.0 APIs. You can choose the API version that best fits your needs. Confluent Kafka Go client supports both synchronous and asynchronous operations.
To configure the Confluent Kafka Go client, you'll need to set the Kafka broker list, which is a list of host and port pairs that identify the Kafka brokers in your cluster. You can also set the client ID, which is used to identify the client to the Kafka brokers. Confluent Kafka Go client supports SSL/TLS encryption for secure communication with Kafka brokers.
The Confluent Kafka Go client provides several configuration options that can be used to customize its behavior. These options include the number of retries, the amount of time to wait for a response, and the maximum number of concurrent requests.
Check this out: Golang Set Env Variable
Featured Images: pexels.com


