
Observability is crucial for understanding how your Go application behaves in production. It helps you identify issues and optimize performance.
With OpenTelemetry, you can instrument your Go application to collect and analyze telemetry data. This includes metrics, logs, and traces that provide insights into your application's behavior.
OpenTelemetry is an open-source observability framework that supports multiple programming languages, including Go. It provides a standardized way to collect and export telemetry data.
To set up OpenTelemetry in your Go application, you'll need to install the OpenTelemetry SDK and configure it to collect the data you need. This can be done using the `go get` command and by adding the necessary code to your application.
Consider reading: Golang Opentelemetry
What Is Observability?
Observability is the ability to monitor and understand the internal state of a system or application by analyzing the data it generates.
Observability is usually divided into three categories, which are the foundation of effective monitoring and troubleshooting.
Metrics refer to numerical data that measures performance or behavior, providing a snapshot of the system's current state.
Logs are a record of events that occur within the system, offering a detailed history of what's happened.
Traces help identify the flow of data through the system, allowing you to see how different components interact and affect each other.
Logging
Logging is a crucial aspect of Go observability. It helps you understand what's happening in your application by providing a record of discrete events.
Logs are used to diagnose issues and can be thought of as diary entries for your application. They're the first place you look when something goes wrong.
To make the most of logs, it's essential to include as much metadata as possible. This makes it easier to find the details you need when something goes awry.
In Go, you can send logs to the Observability panel using the Slog package. This is a great way to get insights into your application's behavior.
Structured logging is also a must-have in Go. It allows you to include trace information as part of the JSON-formatted logs written to standard output.
To configure structured logging, you can use the Go structured logging package, slog. This package provides a way to extract information from the Context instance and add it as attributes to a log.
Here's a summary of the key attributes you can include in a log:
- logging.googleapis.com/trace: Resource name of the trace associated with the log entry.
- logging.googleapis.com/spanId: The span ID with the trace that is associated with the log entry.
- logging.googleapis.com/trace_sampled: The value of this field must be true or false.
By including these attributes, you can link your logs to your trace data and gain a better understanding of your application's behavior.
Sampling logs can also help alleviate performance bottlenecks. This is especially true when working with Docker and stderr pipes.
In some cases, it's a good idea to include a trace ID in every log message. This can be done by passing the Go Context instance to the logger.
By following these best practices, you can get the most out of your logs and improve your application's observability.
Tracing
Tracing is a powerful tool for performance analysis, and it's essential to understand how it works in Golang observability. Traces help you follow the path of a packet or request from the beginning to the end of the path and easily find out in which part your request has a problem.
Traces are very efficient in large services that consist of several parts and may even have connections with external services. In fact, traces are so efficient that they're often used in conjunction with logs to help diagnose issues.
To instrument your app to collect trace and metric data, you need to follow a series of steps, including configuring the main function, OpenTelemetry, and structured logging. You'll also need to add instrumentation to the HTTP server and client, and link trace spans with logs and metrics.
Here are the steps to instrument your app:
- Configure the main function
- Configure OpenTelemetry
- Configure structured logging
- Add instrumentation to the HTTP server
- Link trace spans with logs and metrics
- Add instrumentation to the HTTP client
- Write structured logs
Logs, metrics, and traces are all types of telemetry data, but they serve different purposes. Logs are records of discrete events, metrics track numerical data over time, and traces track a request as it flows through multiple services.
In particular, traces are made up of spans, each representing an individual operation. This allows you to see how different parts of the system interact and identify performance bottlenecks.
To make tracing more efficient, you can use a common setup with the OTEL libraries for trace emission and Grafana Tempo as the backend. This allows you to deterministically construct trace IDs based on the object being processed.
However, finding specific traces can be a challenge. That's where TraceQL comes in – it introduces a convenient way of searching traces, making it easy to locate "interesting" data.
Here are a few examples of how to use TraceQL:
- If you're using Grafana, don't forget to link your log data source to your traces.
- Optimizing TraceQL queries can be achieved by having common per-process attributes in your "resources".
By following these best practices and using the right tools, you can improve your Golang observability and make it easier to diagnose issues.
Instrumentation
Instrumentation is a crucial step in achieving observability in your GoLang application. To collect trace and metric data, you'll need to instrument your app.
You can do this by configuring the main function to write structured logs and collect metrics and trace data using OpenTelemetry. This involves calling two helper functions, setupLogging() and setupOpenTelemetry(), which configure the logging package and OpenTelemetry respectively.
To instrument your app, you'll need to follow these steps: configure the main function, configure OpenTelemetry, configure structured logging, add instrumentation to the HTTP server, link trace spans with logs and metrics, add instrumentation to the HTTP client, and write structured logs.
Here's a breakdown of the major components of OpenTelemetry: logs, metrics, and traces. These are the building blocks of observability, and they help you understand what's happening in your applications.
To add instrumentation to the HTTP server, use the otelhttp handler to propagate context, and for trace and metric instrumentation. The otelhttp handler uses the global TracerProvider, MeterProvider, and TextMapPropagator instances.
To add instrumentation to the HTTP client, call the otelhttp.Get function to inject the trace context into outgoing HTTP requests and add trace and metric instrumentation.
Here's a summary of the steps to instrument your app:
- Configure the main function
- Configure OpenTelemetry
- Configure structured logging
- Add instrumentation to the HTTP server
- Link trace spans with logs and metrics
- Add instrumentation to the HTTP client
- Write structured logs
By following these steps, you'll be well on your way to achieving observability in your GoLang application.
Telemetry Setup
You can run a sample app to collect telemetry data using vendor-neutral formats like JSON for logs and OTLP for metrics and traces. This sample app uses the OpenTelemetry Collector with Google exporters to route telemetry to Google Cloud.
To set up telemetry in Go, you can use the gotel package, which wraps OpenTelemetry's SDK into a simple interface. This package is available on GitHub and makes it easier to handle logs, metrics, and traces.
In your Go app, you can initialize and use the telemetry system by creating an instance of the Telemetry struct, which implements the TelemetryProvider interface. This allows you to write info-level logs and use the telem variable in other parts of the application.
The telemetry data is sent to an observability backend, such as Grafana's grafana/docker-otel-lgtm, which is an all-in-one OpenTelemetry backend that makes it easy to get started. This container receives OpenTelemetry signals on default ports and automatically forwards logs, traces, and metrics to Loki, Tempo, and Mimir.
Here are the steps to instrument your app to collect trace and metric data:
- Configure the main function
- Configure OpenTelemetry
- Configure structured logging
- Add instrumentation to the HTTP server
- Link trace spans with logs and metrics
- Add instrumentation to the HTTP client
- Write structured logs
Profiling and Tracing Integration
Profiling and tracing integration is a crucial aspect of Go observability. Profiling and tracing can be used together to provide a more comprehensive view of your application's performance.
The Go runtime offers excellent profiling capabilities, and instrumenting your code beyond net/http/pprof is highly recommended. DataDog’s “The Busy Developer's Guide to Go Profiling, Tracing and Observability” is a great resource for learning more about profiler types and low-level implementation details.
Profiling features require a new Go runtime for quick and accurate operation, and upgrading to Go 1.19 is recommended if you plan to use continuous profiling. Once you have the raw pprof file, you'll want to analyze it using tools like go tool pprof -http 5000.
However, go tool pprof -http 5000 has its limits, and storing profiles in a database that supports basic querying and filtering is the ideal solution. Google’s Cloud Profiler is a good option, but be aware that their "Offline" API is heavily rate- and size-limited and appears to be generally unsupported.
To connect profiling with tracing, you can use pprof.Do or pprof.SetGoroutineLabels to create a connection between profiling and tracing. This would enable you to create a link between profiling data and tracing data.
Here's a brief summary of the profiling and tracing integration options:
By integrating profiling and tracing, you can gain a more comprehensive understanding of your application's performance and identify areas for improvement.
Profiling Tools
The Go runtime offers excellent profiling capabilities, but for quick and accurate operation, you'll want to upgrade to Go 1.19 if you plan to rely on profiling data in production.
DataDog's "The Busy Developer's Guide to Go Profiling, Tracing and Observability" is a great resource for learning about profiler types and low-level implementation details.
Many profiling features require a new Go runtime, and storing profiles in a database that supports basic querying and filtering is ideal.
You can store profiles in a database like Google's Cloud Profiler, but be aware that their "Offline" API is heavily rate- and size-limited.
Readers also liked: Golang Profiling Tools
Phlare's development is worth keeping an eye on as a capable replacement for Google's "Offline" API.
You can add arbitrary labels to profiling data, which is especially useful in multi-tenant environments, but be mindful of label cardinality.
Felix Geisendörfer's profiler, fgprof, provides a single view that captures both On-CPU and Off-CPU time.
Continuous profiling has become a trend, with libraries like Pyroscope, DataDog, and Google providing this capability.
Grafana Phlare employs an agent model that periodically scrapes Go's pprof HTTP endpoints located at /debug/pprof/.
A fresh viewpoint: Golang Pprof
Getting Started
To get started with observability in Go, you'll want to understand the building blocks of OpenTelemetry: logs, metrics, and traces. These components help you understand what's happening in your applications.
First, install the Open Telemetry dependencies to get started. You'll only need to install the initial dependencies for now, but be aware that you'll need to install a few more as you move forward.
To centralize your OpenTelemetry settings, create an `otel.go` file. This file will help you configure your pipeline and set up two exporters: one for traces and one for metrics. With this in place, you can start measuring and collecting data.
The Reader's Role

The reader is responsible for receiving and sending information to the exporter with a specific interval, which is every 60 seconds by default.
This reader is crucial in getting information from the required places and updating the metrics at certain intervals.
Its main duty is to get information from the required places, which is a vital part of the entire process.
The reader's role is to send this information to the exporter with the specified interval, ensuring that the metrics are updated regularly.
First Steps
To get started with OpenTelemetry, you'll need to install the necessary dependencies. This can be done by installing the initial dependencies of the project, which will be sufficient for now.
We'll be working with a Pipeline, and to centralize our OpenTelemetry settings, let's create the otel.go file. This file will house our Exporters, which will be responsible for sending telemetry data to an external system.
By modifying our otel.go file, we'll have two Exporters: one specialized in traces and the other in metrics. This will allow us to start measuring and collecting data.

Here's a brief overview of the steps involved in setting up OpenTelemetry:
Once this is done, we can start measuring and collecting data. Our final file will look like this:
Deploy the App
To deploy the app, you'll need to follow these steps. First, activate Cloud Shell in the Google Cloud console. This will start a command-line prompt at the bottom of the console, which can take a few seconds to initialize.
You can then clone the repository using the command `git clone https://github.com/GoogleCloudPlatform/golang-samples`. This will download the necessary files for the app.
Next, navigate to the OpenTelemetry directory by running `cd golang-samples/opentelemetry/instrumentation`. This will change your directory to the correct location for the app.
Finally, build and run the sample using the command `docker-compose up --abort-on-container-exit`. If you're not running on Cloud Shell, you'll need to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to point to a credentials file, and then run the application using `docker-compose up -f docker-compose.yaml -f docker-compose.creds.yaml --abort-on-container-exit`.
Context and Signals
To interact with a Signal, we must create a Provider, as shown in the code example. This Provider will receive the collected information.
A named tracer can be created using the otel.Tracer function, which will create a tracer in the global trace Provider. This is done with the tracer = otel.Tracer("info-service") line of code.
Spans are the building blocks of Traces and represent a unit of work or operation. A new Span can be created using the tracer.Start function.
To avoid memory leaks, we must always close Spans and Providers. This is done by ensuring they are “shutdown” when instantiating a Provider.
See what others are reading: Golang Function with Many Parameters
Viewing and Deploying
To view and deploy observability tools in Go, you'll need to set up a tracing system like Jaeger or Zipkin.
Jaeger is a popular choice for distributed tracing, and it can be integrated with Go using the OpenTracing API.
For logging, you can use a library like logrus or zap, which provide a simple and efficient way to log events in your application.
The logrus library, for example, allows you to log messages at different levels of severity, such as debug, info, warn, and error.
To deploy observability tools, you can use a containerization platform like Docker to package your application and its dependencies.
Docker makes it easy to deploy and manage your application in different environments, from development to production.
By using a combination of tracing, logging, and containerization, you can gain a better understanding of your application's behavior and performance.
Middleware and Analytics
Middleware and Analytics is a crucial aspect of golang observability. You can use middlewares to instrument HTTP requests, logging request duration and counting requests in flight.
In a metrics.go file, a Metrics struct holds references to the metrics you want to collect, making it easy to pass the metrics to middleware and other parts of the application. The Metric struct defines the name, unit, and description of a metric, which is useful for defining metrics in a reusable way.
Using middleware, you can get observability data transparently, for example, with gin-gonic/gin. This can be especially useful for monitoring the health of your application.
Middlewares
Middlewares are a powerful tool for instrumenting HTTP requests, allowing you to collect observability data transparently.
You can define a Metrics struct in a metrics.go file to hold references to the metrics you want to collect, making it easy to pass the metrics to middleware and other parts of the application.
A middleware.go file contains the telemetry middleware, which can be used with the gin web framework to log requests, measure request duration, and count requests in flight.
The telemetry middleware is useful for monitoring the health of the application, and can be used with gin-gonic/gin by following gin's Custom Middleware docs.
Analytics
You can use Prometheus-style metrics in your service, but what do you do when you see a spike on your graph and need to figure out what caused the slowdown?
In multi-tenant environments, you can include user_id or team_id in the labels, which can be a cost-effective solution to the high-cardinality metrics problem.
You can put not only a trace_id but also arbitrary key-value data into the labels, making it easier to jump from the metric directly to the trace of the slow request.
ExemplarAdder and ExemplarObserver are tools that can help you achieve this, allowing you to include user_id or team_id in the labels.
How to Use
To use metrics effectively, you need to understand the different types available. There are three main types: Counter, Gauge, and another type mentioned by Prometheus.
A Counter is a cumulative metric that represents a single monotonically increasing counter. This means its value can only increase or be reset to zero on restart.
You can use a Counter to calculate the success rate of your service by tracking the number of successful requests and unsuccessful requests separately.
A Gauge is a metric that represents a single numerical value that can arbitrarily go up and down. This type of metric is useful for tracking dynamic values, such as the balance of your company’s account.
A fresh viewpoint: Golang Check Type
Here are the main differences between Counter and Gauge metrics:
By understanding these differences, you can choose the right type of metric for your use case and track the data that matters most to your service.
Featured Images: pexels.com


