Golang Tracer with OpenTelemetry and Jaeger Explained

Author

Reads 470

Workplace with modern laptop with program code on screen
Credit: pexels.com, Workplace with modern laptop with program code on screen

Golang tracer is a tool that helps you understand and optimize the performance of your Go applications.

OpenTelemetry is a vendor-agnostic observability framework that provides a standardized way to instrument applications for monitoring and analytics.

Jaeger is a popular open-source distributed tracing system that can be used with OpenTelemetry to visualize and analyze the performance of your Go applications.

By using OpenTelemetry and Jaeger together, you can gain a deeper understanding of how your Go applications are performing and identify areas for improvement.

On a similar theme: Golang Performance

Setting Up

Setting up a GoLang tracer involves several key components. To start, you'll need to set up tracing, which breaks down into several parts.

The tracer provider is a crucial component that manages the creation of tracers, which generate telemetry data by creating spans. It also includes resource attributes that append metadata, such as the service name, to each trace.

To enable tracing for your program, you can use the Start function, which returns an error if tracing is already enabled. This function buffers and writes the trace to a specified writer, and it's used to measure task latency.

Setting Up

Three friends setting up tents in a lush forest campsite by the water.
Credit: pexels.com, Three friends setting up tents in a lush forest campsite by the water.

Setting up tracing involves breaking down its components, which include the exporter and tracer provider. The exporter is crucial for creating telemetry data.

To set up tracing, you need to create an exporter, which is a key component of the setup. This exporter will manage the creation of tracers, which are used to generate telemetry data.

A tracer provider is instantiated once the exporter is created, and it's essential for managing the creation of tracers. The tracer provider includes resource attributes that append metadata to each trace.

When setting up tracing, you should be aware that only one call to trace.Start may be active at any given time. This is because currently, only one flight recorder may be active in the program.

Tracing can be enabled for the current program using the Start function, which returns an error if tracing is already enabled. The Start function also buffers and writes the trace to a specified location.

To mark the end of an operation represented by a Task, you need to call the End method on the Task object. This method is used to measure task latency, which is the time between task creation and when the End method is called.

Additional reading: Golang Time since

Set up Jaeger for local testing

Credit: youtube.com, Telemetry Backend Setup – OpenTelemetry Collector & Jaeger (IEEETSE2025)

Setting up Jaeger for local testing is a breeze, especially with Docker. You can run Jaeger using Docker, which simplifies the setup process.

Jaeger can be up and running on your machine in no time. This lets you focus on integrating and testing tracing within your applications without needing a full-scale deployment.

Tracer Operations

To navigate the tracer, you can use keyboard shortcuts (WASD keys) to move through the timeline. This allows you to see the entire program execution at a glance.

Pressing "?" will give you a list of available shortcuts to help you navigate the trace. The timeline shows the time during execution and the units of time may change depending on your navigation.

You can also use the tracer to find memory leaks and see how much memory the garbage collection is freeing at each run. This is shown in the Heap section of the trace.

The Goroutines section shows how many goroutines are running and how many are runnable at each point in time. A high number of runnable goroutines may indicate scheduling contention.

Credit: youtube.com, The new Trace Flight Recorder in Go 1.25 is awesome!

Here are the key sections you'll see in the trace:

  • Timeline: shows the time during execution
  • Heap: shows memory allocations during execution
  • Goroutines: shows how many goroutines are running and runnable
  • OS Threads: shows how many OS threads are being used and blocked
  • Virtual Processors: shows a line for each virtual processor
  • Goroutines and events: displays where/what goroutine is running on each virtual processor

New Grpc

New Grpc is a crucial component of Tracer Operations. It allows you to send spans via gRPC to a server.

The NewGRPC function returns a new Storer that sends spans. This is a key feature that enables efficient and reliable tracing.

You can use NewGRPC to create a Storer that sends spans to a server. This is a straightforward process that can be easily implemented.

NewGRPC is a powerful tool for Tracer Operations. It provides a robust and scalable way to send spans via gRPC.

For more insights, see: Golang Tcp Server

Stop

The Stop function is a crucial part of tracer operations.

It stops the current tracing, if any, and only returns after all the writes for the trace have completed. This ensures that any data being written to the trace is fully recorded before the tracing is stopped.

The Stop function can be called at any time to end the tracing process. It blocks until any concurrent WriteTo calls complete, guaranteeing that all data is safely recorded.

In practice, this means that you can rely on the Stop function to properly shut down the tracing process, even if there are other operations happening concurrently.

You might enjoy: Golang Generic Function

StartSpan

Credit: youtube.com, Spans vs. Traces in OpenTelemetry: Which is Which?

Starting a span is a crucial part of tracer operations, and we have a function called NewGRPC that returns a new Storer that sends spans via gRPC to a server.

This function is useful for sending spans to a server, which can be helpful for analyzing and understanding the flow of your application.

To start a span, you'll need to have a Storer in place, such as one returned by the NewGRPC function.

Tracer Configuration

Configuring a tracer in GoLang is a straightforward process. You can use the `go tool trace` command to create a tracing configuration file.

To enable tracing, you simply need to set the `go.test_flags` environment variable to include the `-trace` flag. This flag tells Go to generate a tracing configuration file.

The tracing configuration file will contain settings such as the sampling rate and the output file path. You can then use the `go tool trace` command to analyze the generated trace file.

You might enjoy: Golang Go

Enabled

Focused view of programming code displayed on a laptop, ideal for tech and coding themes.
Credit: pexels.com, Focused view of programming code displayed on a laptop, ideal for tech and coding themes.

IsEnabled reports whether tracing is enabled, but the information is advisory only and may have changed by the time this function returns.

There are two ways to check if tracing is enabled. The first is through the IsEnabled function, which is available in Go 1.11 and later.

IsEnabled is a simple way to check if tracing is enabled, but it's not the most reliable method. The tracing status may have changed by the time the function returns.

A more reliable way to check if tracing is enabled is to use the Enabled function of the FlightRecorder. This function returns true if the flight recorder is active.

The Enabled function will return true if Start did not return an error, and Stop has not yet been called. This means you can safely call it from multiple goroutines simultaneously.

Enabled is a safe function to call, even in concurrent scenarios. It's a good choice if you need a reliable way to check if tracing is enabled.

Exporter Initialisation

Close-up of colorful programming code displayed on a computer screen.
Credit: pexels.com, Close-up of colorful programming code displayed on a computer screen.

Exporter Initialisation is a crucial step in setting up a tracing system.

The exporter is responsible for sending telemetry data to one or more backends or a collector.

In a typical setup, the data flow involves application instrumentation, exporter configuration, and collector processing.

Here's a breakdown of the exporter configuration:

  • Define the HTTP headers that the exporter will use when sending data, setting the content-type header to application/json.
  • Configure the OTLP HTTP exporter with the endpoint, headers, and connection type.

The core function involves setting up an OTLP HTTP exporter, which can be done by using the `otlptracehttp.WithEndpoint` method to specify the collector's endpoint.

This is typically set to `localhost:4318`, which is the designated port for HTTP connections receiving OTLP data.

The exporter can also be configured to use insecure connections by calling `otlptracehttp.WithInsecure()`, which is not recommended for production environments.

Write To In 1.25.0

In 1.25.0, the WriteTo function was added to the FlightRecorder. This function snapshots the moving window tracked by the flight recorder, containing data up to date as of the call.

The WriteTo function is not a hard guarantee, meaning it's possible that the data might not be entirely up to date. However, it's still a reliable way to get a snapshot of the flight recorder's data.

Vibrant JavaScript code displayed on a screen, highlighting programming concepts and software development.
Credit: pexels.com, Vibrant JavaScript code displayed on a screen, highlighting programming concepts and software development.

Only one goroutine can execute WriteTo at a time, ensuring that the function is thread-safe. This is important to prevent data corruption and ensure accurate tracing.

An error is returned if another WriteTo call is already in progress or if the flight recorder is inactive. This helps prevent issues and provides valuable feedback for debugging purposes.

Tracer Data

Tracer data is a powerful tool for understanding how your Go program executes. It provides a detailed view of the program's execution, including the path of each operation.

The Go tracer writes tracing output to the standard error output, which can then be read with the go tool trace command. This command is used to analyze the trace data, and it can display various views of the program's execution.

One of the most complex and interactive visualizations is the view trace, which shows a timeline of the entire program execution. This view displays what was running on each virtual processor and what was blocked waiting to run.

Credit: youtube.com, Implementing Distributed Tracing in Golang with OpenTelemetry

The Go tracer also provides a goroutine analysis view, which shows how many of each kind of goroutine were created during the entire execution. This view can be used to see information about each goroutine of a specific kind.

The tracer data contains all the information needed to analyze the trace, making it easier to debug and optimize your program. With the Go tracer, you can gain a deeper understanding of your program's execution and identify performance bottlenecks.

Here are the different views available in the Go tracer:

  • View trace: displays a timeline of the entire program execution
  • Goroutine analysis: shows how many of each kind of goroutine were created
  • Network/Sync/Syscall blocking profile: displays how long goroutines spent blocked on each resource
  • Scheduler latency profiler: provides timing for scheduler level information

Tracer Events

Tracer events are a crucial part of understanding what's happening in your Go program. They show you where and when specific events occur during execution.

You can see the events generated by a goroutine by clicking on it in the Goroutines and events section of the trace. This will give you a list of events, including the stack trace when the event started and its duration.

Credit: youtube.com, Go (Golang) httptrace Tutorial

Each event can be clicked on to view its details, including the stack trace when the event started, its duration, and the goroutines involved in the event. This is a powerful tool for debugging and optimizing your Go code.

Here are the details you can see for each event:

  • The stack trace when the event started
  • The duration of the event
  • Goroutines involved in the event

By examining these details, you can gain a deeper understanding of what's happening in your program and identify areas for improvement.

Sampling

To keep the overhead incurred by tracing low, a sampler is used to only trace a subset of requests. This is achieved by a sampler that may sample requests based on chance, a rate, or possibly other mechanisms.

The default sampler samples all requests, which is useful for testing, and viable for low-traffic systems. This means that if you're testing your system, you'll want to use the default sampler to get a complete picture of what's happening.

Only root spans make sampling decisions, so child spans will inherit the sampling decisions of the root spans. This is an important consideration when designing your tracing system.

Credit: youtube.com, OpenTelemetry Tail Sampling, Explained | The OpenTelemetry Bootcamp

A Sampler determines whether a span should be sampled or not by returning true or false. This is a simple yet effective way to control which spans are included in your tracing data.

The NewProbabilisticSampler function returns a sampler that samples spans with a certain chance, which should be in the range of 0 to 1. This allows you to control the probability of sampling each span.

You can check if a span was sampled using the Sampled method of the Span struct. This method reports whether this span was sampled, giving you a way to analyze your tracing data.

Tracer Export

An exporter is responsible for sending telemetry data to one or more backends or a collector. The exporter is configured within the application and sends the data to a collector, which can perform various functions before exporting it to the final backend system.

The data flow typically looks like this: Application Instrumentation, Exporter, Collector, and finally the backend system. The exporter sends the data to the collector over protocols such as HTTP or gRPC.

Here's an interesting read: Rapid Web Application Development Tools

Credit: youtube.com, Tracing Programs with Go

To initialise an exporter, we define the HTTP headers that the exporter will use when sending data. The content-type header is set to application/json, which tells the receiver to expect the data in JSON format.

The core of the function involves setting up an OTLP (OpenTelemetry Protocol) HTTP exporter. We can configure the exporter to send data to a local collector running on port 4318 using otlptracehttp.WithEndpoint("localhost:4318").

Here are the key components of the exporter configuration:

  • Endpoint: The collector's URL, for example, "localhost:4318" for a local collector.
  • Headers: The HTTP headers that the exporter will use when sending data.
  • Insecure: Whether to use an insecure connection, which means it does not use TLS.

Using an insecure connection is not recommended for production environments but is common in local testing environments. The exporter can be configured to use a secure connection by removing the WithInsecure() method.

Tracer UI

The Tracer UI provides an interactive visualization of your program's execution, allowing you to analyze and understand its behavior in detail. You can run your code with tracing enabled and view the results in a browser window.

One of the most complex and powerful visualizations is the "View trace" timeline, which displays the entire program execution. This view shows what was running on each virtual processor and what was blocked waiting to run. Only works on Chrome.

Credit: youtube.com, Efficient Debugging and Logging with OpenTelemetry in Go - Konstantin Ostrovsky

The timeline is navigable using keyboard shortcuts (WASD keys), and you can also use the "?" button to get a list of available shortcuts. The timeline displays the time during execution, and the units of time may change depending on the navigation.

Here are the main sections of the Tracer UI:

  • Timeline: shows the time during execution and what was running on each virtual processor.
  • Heap: shows memory allocations during execution.
  • Goroutines: shows how many goroutines are running and how many are runnable at each point in time.
  • OS Threads: shows how many OS threads are being used and how many are blocked by syscalls.
  • Virtual Processors: shows a line for each virtual processor.
  • Goroutines and events: displays where/what goroutine is running on each virtual processor.

Viewing Traces on Jaeger UI

Viewing Traces on Jaeger UI is a crucial step in understanding how your application is performing. You can access the Jaeger UI after your application runs.

Each trace shows the path of an operation, which can be a complex process to visualize. In the example of Create Product execution, you can see the entire flow.

To view traces, simply access the Jaeger UI, and you'll see the traces from your app. This will give you a clear picture of how your application is working.

Jaeger UI makes it easy to identify and analyze any issues that may arise during the execution of operations.

View Trace

Side view full length male tracer wearing hoodie jumping over brick building parapet and practicing parkour on clear day
Credit: pexels.com, Side view full length male tracer wearing hoodie jumping over brick building parapet and practicing parkour on clear day

View Trace is a powerful feature of the Tracer UI, allowing you to visualize the entire program execution. It's a complex and interactive visualization that displays a timeline of the program's execution.

The timeline shows what was running on each virtual processor and what was blocked waiting to run. You can navigate the timeline using keyboard shortcuts (WASD keys).

The View Trace feature also provides information about memory allocations during the execution, which can be useful for finding memory leaks and checking how much memory the garbage collection is freeing at each run. This is displayed in the Heap section.

The Goroutines section shows how many goroutines are running and how many are runnable (waiting to be scheduled) at each point in time. A high number of runnable goroutines may indicate scheduling contention.

The Virtual Processors section displays a line for each virtual processor, with the number of virtual processors controlled by the GOMAXPROCS environment variable (defaulting to the number of cores).

Curious to learn more? Check out: Golang Memory Management

Side view full body energetic young male tracer in hoodie and hat jumping over barrier against brick wall on sunny autumn day
Credit: pexels.com, Side view full body energetic young male tracer in hoodie and hat jumping over barrier against brick wall on sunny autumn day

Here are the key sections you'll find in the View Trace feature:

  • Timeline: shows the time during the execution and units of time may change depending on navigation
  • Heap: memory allocations during the execution
  • Goroutines: how many goroutines are running and how many are runnable at each point in time
  • OS Threads: how many OS threads are being used and how many are blocked by syscalls
  • Virtual Processors: a line for each virtual processor
  • Goroutines and events: where/what goroutine is running on each virtual processor, with lines connecting goroutines representing events

By clicking on a particular goroutine, you can see its name, when it started, its duration, the stack trace when it started, the stack trace when it finished, and events generated by this goroutine.

A different take: Golang Stack

OpenTelemetry

OpenTelemetry is a unified way to collect and transport telemetry data. It provides a comprehensive observability solution for maintaining system reliability and performance.

To get started with OpenTelemetry in Go, you'll need to install and configure the OpenTelemetry Go SDK. This will give you access to the OpenTelemetry Tracing API, which you can use to instrument your application. The Go implementation of OpenTelemetry, OpenTelemetry-Go, provides this API.

You can create a tracer by providing the name and version of the library or application doing the instrumentation. This is a crucial step in creating spans, which are used to track the flow of your application's execution. You can have multiple tracers, but usually, one is enough for an app or library.

Credit: youtube.com, OpenTelemetry Golang Tutorial (Tracing in Grafana & Kubernetes & Tempo)

To create spans, you'll need to propagate the context by passing it to a function as the first argument. This is where the magic happens, and you'll start seeing your application's execution flow. With net/http, the active context is propagated in a http.Request.

Here's a quick rundown of the components involved in the OpenTelemetry workflow:

Why OpenTelemetry?

OpenTelemetry provides a unified way to collect and transport telemetry data, which is essential for maintaining system reliability and performance.

The unified approach of OpenTelemetry makes it easier to collect and transport telemetry data, enabling comprehensive observability.

By using OpenTelemetry libraries within your application, you can collect telemetry data from various application instances and forward it to analytical tools like Jaeger.

The Jaeger UI is where users can visualize and query the trace data, making it easier to understand system performance and reliability issues.

The OpenTelemetry architecture consists of several components, including the application, OpenTelemetry Collector, Jaeger Collector, storage, and UI. Here's a breakdown of each component:

  1. Application (OTel library): This is your application code that has been instrumented with the OpenTelemetry libraries.
  2. OpenTelemetry Collector: This component receives data from various application instances and forwards it to analytical tools like Jaeger.
  3. Jaeger Collector: This component receives trace data from the OTel Collector, processes the data, and stores it in a database or in-memory storage.
  4. Storage (Elasticsearch/Cassandra): This is where the processed data is stored, such as in a datastore like Cassandra or Elasticsearch.
  5. jaeger-query: This component retrieves trace data from the storage when requested and serves it to the user interface.
  6. jaeger-UI: This is the graphical user interface of Jaeger where users can visualize and query the trace data.

Jaeger with OpenTelemetry

Credit: youtube.com, Distributed Tracing With Jaeger And OpenTelemetry In Kubernetes

Jaeger with OpenTelemetry is a powerful combination for monitoring and optimizing your application's performance. You can view the traces of your application's operations on the Jaeger UI, which shows the path of each operation, like the Create Product execution.

To get started with tracing, you'll need to install the necessary packages that enable OpenTelemetry. This includes the packages that allow you to implement tracing and track requests within your application.

With OpenTelemetry, you can get a clear picture of your application's performance by tracking requests within your application. This is especially useful for identifying bottlenecks and areas for improvement.

To implement tracing in your Go REST API, you'll need to install the necessary packages that enable OpenTelemetry.

OpenTelemetry API

OpenTelemetry API is the core of instrumenting your application with OpenTelemetry. It provides a unified way to collect and transport telemetry data.

To use OpenTelemetry API, you need to install and configure the OpenTelemetry Go SDK, which you can do by following the steps in the Getting Started guide. OpenTelemetry-Go is the Go implementation of OpenTelemetry, providing the OpenTelemetry Tracing API.

Credit: youtube.com, OpenTelemetry Webinar: What is the OpenTelemetry API

You can create a tracer by providing the name and version of the library/application doing the instrumentation. This tracer will be used to create spans, which are the building blocks of OpenTelemetry tracing.

To start creating spans, you need to wrap the operation with a span using the tracer.Start method. You can also activate the span manually by setting the context.

OpenTelemetry stores the active span in a context.Context, which you can propagate by passing it to a function as the first argument. With net/http, the active context is propagated in a http.Request.

You can record contextual information with attributes, which are key-value pairs that describe the operation. For example, an HTTP endpoint may have attributes such as http.method = GET and http.route = /projects/:id.

Here are some common attributes you can use:

  • http.method
  • http.route
  • http.status_code
  • error

You can also add links to other traces, for example, when a trace creates sub-traces. OpenTelemetry provides a shortcut to record errors, which is usually used together with the SetStatus API.

By default, OpenTelemetry Go SDK logs error messages to stderr. You can discard or redirect those logs by providing an error handler.

For another approach, see: Golang Create Error

Instrumentation

Credit: youtube.com, Instrumenting Go Apps With OpenTelemetry - Alexandre Cabral

Instrumentation is a crucial step in implementing a Go tracer. You'll need to install the necessary packages that enable OpenTelemetry, which include the OpenTelemetry Go Tracing API.

To instrument your application, you'll need to wrap the operation with a span, which is a way to record the execution time of a function. This is done by providing the name and version of the library/application doing the instrumentation to create a tracer.

You can create as many tracers as you want, but usually, you need only one tracer for an app or a library. To record contextual information, you can annotate spans with attributes, such as http.method and http.route. You can also set error and status code to indicate that the operation contains an error.

Here are some common attributes you can use:

  • http.method
  • http.route
  • error
  • status code

Log In 1.11

In 1.11, the Log function was added to emit a one-off event with a given category and message.

A programmer coding on a laptop and monitor in a stylish office setup.
Credit: pexels.com, A programmer coding on a laptop and monitor in a stylish office setup.

You can leave the category empty, and the API will still work as expected.

The Log function assumes there are only a handful of unique categories in the system.

This is a good approach, as it keeps the system organized and easy to manage.

The Log function is a useful tool for tracking events and issues in your system.

Additional reading: Log Golang

Types

The taskType is used to classify task instances, and analysis tools like the Go execution tracer may assume there are only a bounded number of unique task types in the system.

In other words, taskType is a way to categorize tasks for easier analysis.

Analysis tools rely on this classification to function properly, making taskType a crucial part of the system.

TaskType helps ensure that analysis tools don't get overwhelmed by too many unique task types.

This bounded assumption is essential for the efficient operation of analysis tools like the Go execution tracer.

A fresh viewpoint: Web Dev Tools

Type Extracter

Type Extracter is a crucial part of instrumentation, and it's used to extract a SpanContext from a carrier.

An Extracter is a specific type of tool that's designed to do this job, and it's quite efficient at it.

In this context, a SpanContext is the key to understanding the flow of data through a system, and being able to extract it is essential for instrumentation.

On a similar theme: Check Type of Interface Golang

REST API Instrumentation

Credit: youtube.com, Smart® Instrumentation: WebAPI introduction

Instrumenting a REST API is a crucial step in understanding its performance. To do this, you'll need to install the necessary packages that enable OpenTelemetry.

The OpenTelemetry Go Tracing API provides the necessary tools to instrument your application. To get started, you'll need to create a tracer by providing the name and version of the library/application doing the instrumentation.

You can create multiple tracers, but usually, you'll only need one for an app or library. Each tracer can be used to identify the instrumentation that produces the spans.

To record contextual information, you can annotate spans with attributes. For example, an HTTP endpoint may have attributes such as http.method = GET and http.route = /projects/:id.

Here are some common attributes you can use:

  • http.method
  • http.route
  • http.status_code

You can also set the error status code to indicate that the operation contains an error. To do this, you can use the SetStatus API.

By following these steps, you can instrument your REST API and gain a clear picture of its performance. This will help you identify bottlenecks and optimize your application for better performance.

Source Files

Credit: youtube.com, 6.1 Code Instrumentation

Source files are the heart of instrumentation, and they're where all the magic happens. They're the files that contain the code that's used to collect and report data.

In instrumentation, source files are typically written in programming languages like C, C++, or Java. These languages are chosen because they're efficient and can handle the high-speed data collection required for instrumentation.

A good source file should be well-organized and easy to read, with comments that explain what each section of code is doing. This makes it easier for developers to understand and modify the code.

For example, in a recent project, I worked with a source file that used a C++ library to collect data from a sensor. The library was well-documented, making it easy to integrate into our project.

Source files can also include debugging tools, which help developers identify and fix errors in the code. This is especially important in instrumentation, where errors can have serious consequences.

In one case, a colleague used a debugging tool to track down a bug in a source file, and it ended up saving us hours of debugging time.

Related reading: Golang Error

Frequently Asked Questions

Does Uber still use Golang?

Yes, Uber still uses Golang to support its large-scale microservices architecture. With over 46 million lines of Go code, it's one of the largest Go codebases in the world.

Danny Orlandini

Writer

Danny Orlandini is a passionate writer, known for his engaging and thought-provoking blog posts. He has been writing for several years and has developed a unique voice that resonates with readers from all walks of life. Danny's love for words and storytelling is evident in every piece he creates.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.