
Implementing OpenTelemetry in Go can significantly improve the observability of your applications. This is achieved through the use of distributed tracing and metrics.
OpenTelemetry is an open-source observability framework that provides a unified API for collecting and exporting telemetry data. It's designed to work seamlessly with a variety of programming languages, including Go.
The OpenTelemetry Go implementation is built on top of the OpenTelemetry API, which provides a standardized way of collecting and exporting telemetry data. This allows developers to write applications that can be easily instrumented for observability.
By using OpenTelemetry in your Go application, you can gain a deeper understanding of how your application is performing, identify bottlenecks, and make data-driven decisions to optimize its performance.
Additional reading: Opentelemetry Elasticsearch
Getting Started
Getting started with OpenTelemetry in Golang is a straightforward process. You can find a getting started guide on opentelemetry.io.
To start capturing distributed traces and metric events from your application, you'll need to instrument your code. The easiest way to do this is by using an instrumentation library for your code. Be sure to check out the officially supported instrumentation libraries.
A different take: Opentelemetry Collector Golang
Instrumenting your application involves two main steps: instrumenting your code and configuring an exporter. The Go otel package is used to extend the telemetry provided by an instrumentation library or to build your own instrumentation directly.
Here are the officially supported exporters for the OpenTelemetry project:
By following these steps and using the right tools, you'll be able to capture valuable insights from your application and improve its performance.
Here's an interesting read: Portable Air Conditioners
Example Application
OpenTelemetry is a great tool for instrumenting your Go applications, and it's surprisingly easy to get started. You can use it with a basic net/http application, or with other web frameworks like Gin and Echo.
If you're not using net/http, don't worry - OpenTelemetry works with other frameworks too. For a complete list of supported libraries, check out the registry.
You can use OpenTelemetry with Gin and Echo, and even other request routers. The registry will give you a list of supported libraries to choose from.
For more insights, see: Golang Registry
Set up development environment
To get started, you'll need a working Go environment. This is the foundation for any Go project.
You'll also need an application written in Go, which can be a new project or an existing one you're working on.
If you've already stored an API key in a secure location, you can retrieve it from your Environment Settings. This makes it easy to access the key without having to search for it elsewhere.
To set up your development environment, you'll need to meet these two basic requirements: a working Go environment and an application written in Go.
For your interest: Golang Applications
Configure Environment Variables
To configure environment variables, you'll need to set up three key variables: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, and OTEL_SERVICE_NAME.
The OTEL_EXPORTER_OTLP_ENDPOINT variable specifies the Honeycomb endpoint to which you want to send your data. This is where your data will be received and processed by Honeycomb.
You'll also need to set the OTEL_EXPORTER_OTLP_HEADERS variable, which contains a header with your API Key generated in Honeycomb. This is used to authenticate your data and ensure it reaches the correct destination.
The OTEL_SERVICE_NAME variable is used to specify your service name, which will be used as the name of the dataset in Honeycomb where your data is stored.
Here's a summary of the environment variables you'll need to configure:
If you're using Honeycomb Classic, don't forget to specify the Dataset using the HONEYCOMB_DATASET environment variable.
Configure SDK
To configure the OpenTelemetry SDK, you'll need to add the necessary packages to your application. This can be done by including the OpenTelemetry SDK packages in your application.
You'll also need to declare environment variables for configuring OpenTelemetry. This can be done by declaring global variables in the main.go file after the import block. The environment variables used here are OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, and OTEL_SERVICE_NAME.
To send events to Honeycomb, you'll need to configure the OTEL_EXPORTER_OTLP_ENDPOINT variable with the Honeycomb endpoint. The OTEL_EXPORTER_OTLP_HEADERS variable should contain the x-honeycomb-team= header, plus your API Key generated in Honeycomb.
Curious to learn more? Check out: Golang Sdk
Here's a list of the variables you'll need to configure:
You can also look up details for Honeycomb API Keys in your Environment Settings, and use them to retrieve keys from your storage location.
No sends, no errors
No sends, no errors? That's frustrating! If no events arrive in your observability platform and no errors are printed, try increasing the logging verbosity of the OpenTelemetry SDK before initialization. This will give you more insight into what's going on.
The log statements will be more verbose, and you'll see crucial information about OpenTelemetry's configuration. Specifically, you'll see that it's configured to send data to the observability platform.
The table below shows the officially supported exporters for the OpenTelemetry project, which can help you identify potential issues with sending data.
Check the exporter you're using to see if it supports the type of data you're trying to send.
Instrumentation
Instrumentation is a crucial step in getting the most out of OpenTelemetry in Golang. You can add OpenTelemetry instrumentation to your app by following the steps outlined in the OpenTelemetry documentation. This includes adding custom instrumentation to your code using the OpenTelemetry API.
To get started with instrumentation, you can follow the instructions provided in the "Add Custom Instrumentation" section. This involves modifying your code to include custom instrumentation using the OpenTelemetry API. You can also use automatic instrumentation, which is the easiest way to get started with instrumenting your code.
If you're using a Gin request router, you'll need to install the OpenTelemetry exporter and SDK dependencies first. You can do this by running the commands provided in the "Instrumenting your Golang application with OpenTelemetry" section.
Acquire Dependencies
To acquire dependencies, you need to install OpenTelemetry exporter and SDK dependencies. Run the following commands after navigating to the application source folder:
Run the below commands after navigating to the application source folder:
- Install dependencies related to OpenTelemetry exporter and SDK.
- Assuming you are using gin request router, check out the corresponding package if you are using other request routers.
This will help you centralize your OpenTelemetry settings and prepare your application for instrumentation.
Instrumenting Your Application
Instrumenting your application is a crucial step in understanding how it behaves and identifying potential issues. You can start by adding OpenTelemetry instrumentation to your sample app, which will help you capture telemetry at the edges of your system.
To add custom instrumentation, you'll need to write some manual code using the OpenTelemetry API. This will allow you to capture what's going on inside your application. You can follow the instructions in the OpenTelemetry documentation for manual instrumentation in Go.
Automatic instrumentation is another way to get started with instrumenting your code. However, you should also add custom instrumentation where appropriate to get additional insights into your system. To do this, you'll need to follow the instructions below.
Here are the steps to add custom instrumentation to your code:
1. Install the OpenTelemetry dependencies.
2. Declare environment variables for configuring OpenTelemetry.
3. Initialize the tracer in your main function.
4. Add the OpenTelemetry Gin middleware.
5. Set environment variables to send data to SigNoz backend.
You can also use the `GetTracerProvider` function to get the registered global trace provider. If none is registered, it will return an instance of `NoopTracerProvider`.
To create a named tracer, you can use the `Tracer` function, which creates a named tracer that implements the `Tracer` interface. If the name is an empty string, the provider will use a default name.
Here's a summary of the steps to instrument your application:
By following these steps, you'll be able to instrument your application and gain valuable insights into its behavior. Remember to also add custom instrumentation where appropriate to get additional insights into your system.
Spans and Attributes
Spans and Attributes are a powerful tool in OpenTelemetry Golang, allowing you to add context to your traces and gain deeper insights into your application's behavior.
You can add attributes to spans using the `span.SetAttributes()` method, which takes a map of key-value pairs as an argument. This is useful for associating additional metadata with a span, such as a user ID.
For example, you can add a `user.id` field to the current span using `span.SetAttributes(map[string]interface{}{"user.id": "12345"})`. This will allow you to use the `user.id` field in WHERE, GROUP BY, or ORDER clauses in the Honeycomb query builder.
To create spans, you can leverage manual instrumentation to create custom spans that describe what is happening in your application. You can grab the tracer from the OpenTelemetry API using `tracer := otel.Tracer("")`.
You can also add multi-span attributes using the OpenTelemetry concept of baggage. Baggage allows you to add a key with a value as an attribute to every subsequent child span of the current application context.
Here are the steps to add multi-span attributes:
- Install the `baggagetrace` package in your terminal
- Configure the OpenTelemetry SDK tracer provider with the baggage span processor
- Add a baggage entry for the current trace using `baggage.Add("key", "value")`
Note that any baggage attributes you set in your application will be attached to outgoing network requests as a header. So, do NOT put sensitive information in the Baggage attributes.
In addition to adding attributes, you can also add custom events to spans using the `span.AddEvent()` method. This allows you to track specific actions or errors within the span's context.
Here's an example of how to add a custom event to a span:
```go
span.AddEvent("This is a sample event", trace.WithAttributes("pid", 4328, "sampleAttribute", "Test"))
```
This will add an event named "This is a sample event" to the span, with custom attributes "pid" = 4328 and "sampleAttribute" = "Test".
Metrics and Logging
In opentelemetry golang, metrics and logging are crucial for monitoring and debugging your application.
The opentelemetry library offers a simple way to add metrics to your application, starting with the installation and configuration of an Exporter specialized in metrics. This is done by modifying the otel.go file and adding a Provider instantiator for metrics.
You can configure your Exporter and Provider in main.go, and then create a named meter on the global Provider using otel.Meter("info-service"). This allows you to define your measuring instrument through metric.Int64Counter, which records increasing int64 values.
GetMeterProvider in V
GetMeterProvider returns the registered global meter provider. If no global meter provider has been registered, a No-op implementation is returned.
This function is added in version 0.14.0.
The returned GetMeterProvider, and all the Meters it has created or will create, are recreated automatically from the new GetMeterProvider when a global GetMeterProvider is registered for the first time.
Configure Logger
You can configure the logger used internally to opentelemetry with the SetLogger function.
This function was added in version 1.3.0 of the library.
The SetLogger function configures the logger used internally to opentelemetry.
Propagation and Context
In OpenTelemetry Go, we need to propagate the context for trace information to be sent to remote processes. This is done by registering a propagator.
A propagator is used to propagate the context, and we can use the GetTextMapPropagator function to get the global TextMapPropagator. If none has been set, a No-Op TextMapPropagator is returned.
The global TextMapPropagator can be set using the SetTextMapPropagator function, which sets propagator as the global TextMapPropagator.
Context propagation is essential for tracing information to be sent to remote processes, making it a crucial aspect of OpenTelemetry Go.
Error Handling
Error handling is crucial when working with OpenTelemetry in Go. If OpenTelemetry is unable to send spans to Honeycomb, it will print an error to the console.
You can troubleshoot this issue by checking if your Honeycomb API key is missing or incorrect. If that's not the problem, try sending a custom span immediately after initialization as a further test.
To handle errors in a more convenient way, you can use the Handle function, which is a wrapper around GetErrorHandler().Handle(err). This can be particularly helpful when you need to quickly get error handling up and running.
Sending Errors to Honeycomb
If OpenTelemetry can't send spans to Honeycomb, it should print an error to the console. This error can indicate that the Honeycomb API key is missing or incorrect.
The SpanExporter is a crucial component in sending spans to Honeycomb. If it's missing, there's a problem with your exporter configuration.
Try sending a custom span immediately after initialization as a further test. This can help you determine if the issue lies with your exporter configuration or something else.
ErrorHandler in v0.11.0
In v0.11.0, the Handle function was introduced as a convenience function for GetErrorHandler().Handle(err).
The Handle function is a simple way to handle errors, and it's been available since version 0.14.0.
You can use the Handle function to pass an error to the ErrorHandler, which is a powerful tool for managing errors in your application.
The ErrorHandler is a type that was introduced in version 1.0.0, and it's a convenience adapter that allows you to use a function as an ErrorHandler.
If you're experiencing issues with sending spans to Honeycomb, you can check the console for error messages. For example, if the Honeycomb API key is missing or incorrect, you'll see an error message.
This error indicates that there's a problem with your exporter configuration, or it might be a test issue that you can resolve by sending a custom span immediately after initialization.
HTTP and gRPC
When choosing between gRPC and HTTP for OTLP export, keep in mind that some SDKs default to using gRPC, making it the easiest option to start with.
Some firewall policies may not be set up to handle gRPC, requiring the use of HTTP instead. Additionally, gRPC may improve performance but can cause problems with load balancing, especially when using Refinery.
gRPC default export uses port 4317, whereas HTTP default export uses port 4318.
For another approach, see: Grpc Go
Control HTTP Server
You can instrument the HTTP server using the OpenTelemetry SDK and the otelhttp instrumentation library.
To do this, you'll need to modify your main.go file to set up the OpenTelemetry SDK and instrument the HTTP server.
The otelhttp instrumentation library provides a convenient way to enrich your HTTP server with telemetry data.
You can use the Handle function to get the error handler and handle errors, which is added in version 0.14.0.
The Handle function is a convenience function that calls GetErrorHandler().Handle(err).
Instrumenting the entire HTTP server will give you extra information in your collected data, such as the user agent, HTTP method, protocol, and route.
This can be achieved by overwriting the handler with the OTel handler.
gRPC vs HTTP
If you're deciding between gRPC and HTTP for your OpenTelemetry SDK, here's what you need to know. gRPC may improve performance, but its long-lived connections can cause problems with load balancing, especially when using Refinery.
Some SDKs default to using gRPC, which can make it easier to get started. However, some firewall policies may not be set up to handle gRPC, requiring you to use HTTP instead.
The default export port for gRPC is 4317, while the default export port for HTTP is 4318.
Here's a quick summary of the key differences:
- gRPC may improve performance, but can cause load balancing issues.
- Some SDKs default to gRPC, while others default to HTTP.
- Default export ports are 4317 for gRPC and 4318 for HTTP.
Sampling and Monitoring
You can configure the OpenTelemetry SDK to sample the data it generates, which is particularly useful for reducing data volume while keeping the most important information.
To do this, use a TraceIdRatioBased sampler with a ratio expressed as 1/N. This allows Honeycomb to reweigh scalar values, like counts, so they are accurate even with sampled data.
The example shows how to keep approximately half of the data volume by setting the OpenTelemetry sampler argument to 0.5 and the resource attribute SampleRate to 2.
Sampling
Sampling allows you to configure the OpenTelemetry SDK to capture only a portion of the data it generates. This is useful for reducing the volume of data and conserving resources.
To sample data, you can use a TraceIdRatioBased sampler with a ratio expressed as 1/N. This means you would set the sampler argument to 1/2, for example, if you want to keep approximately half of the data volume.
You must also create a resource attribute called SampleRate with the value of N. In the case of keeping half of the data, the resource attribute would contain the denominator, which is 2.
This allows Honeycomb to reweigh scalar values, like counts, so they are accurate even with sampled data. By setting the sample rate correctly, you can ensure that your sampled data is reliable and useful for monitoring and analysis.
Monitor Go App with SigNoz
Monitoring your Go application is a crucial step in ensuring its performance and reliability. With SigNoz, you can visualize your application metrics and tracing data.
You can navigate to your SigNoz cloud account and see your goGinApp service on the service page. This allows you to monitor application metrics like application latency, requests per second, error percentage, and more.
The Metrics tab of SigNoz provides a clear view of your application's performance. You can click on the goGinApp service to be redirected to the metrics page.
OpenTelemetry captures tracing data from your Gin application, which can be analyzed using filters based on tags, status codes, service names, operations, and more. This helps you visualize how user requests perform across services in a multi-service application.
In the Traces tab of SigNoz, you can use flamegraphs and Gantt charts to visualize your tracing data. This provides a detailed view of how your application is performing.
Troubleshooting and Contributing
Troubleshooting can be a challenge, especially when sending data to Honeycomb. Visit Common Issues with Sending Data to Honeycomb for help.
If you're running into issues, don't worry, you're not alone. Many developers have encountered similar problems.
To resolve common issues, check out Common Issues with Sending Data to Honeycomb for step-by-step guidance and solutions.
Contributing to the OpenTelemetry community is a great way to get involved and give back. To get started, read the getting started guide at https://opentelemetry.io/docs/languages/go/getting-started/.
Troubleshooting
Troubleshooting is a crucial step in resolving issues. If you're experiencing problems sending data, check out Common Issues with Sending Data to Honeycomb for solutions.
It's always a good idea to start with the basics. Visit Common Issues with Sending Data to Honeycomb to find answers to common issues.
Troubleshooting can be frustrating, but it's often a matter of finding the right resource. The Common Issues with Sending Data to Honeycomb page is a great place to start.
By following the steps outlined on that page, you can resolve many common issues and get back to work.
Contributing
Contributing to the OpenTelemetry project is a great way to get involved and help others. To read the getting started guide, see https://opentelemetry.io/docs/languages/go/getting-started/.
You can start by following the link to learn more about the process.
Featured Images: pexels.com


