Golang toml Configuration Management Simplified

Author

Reads 1.3K

Programming Code on Laptop Screen
Credit: pexels.com, Programming Code on Laptop Screen

Golang's toml configuration management is surprisingly straightforward. You can load a toml file into a Go struct with the toml.Unmarshal function.

TOML files are human-readable and easy to edit, making them a great choice for configuration files. They consist of key-value pairs, which can be easily loaded into a Go program.

The toml.Unmarshal function takes a byte slice and a struct pointer as arguments, making it simple to use. This function is available in the standard library, so you don't need to install any additional packages.

Loading a toml file into a Go program is a common use case. You can create a configuration file and load it into your program with just a few lines of code.

Go and TOML Basics

In Go, you can use two major packages to work with TOML: github.com/BurntSushi/toml and github.com/pelletier/go-toml. Both packages serve the same purpose but have different APIs, with the former being easier to implement and the latter being better suited for complex use cases.

Related reading: Golang Use .env File

Credit: youtube.com, How to Unmarshal TOML Config Files with Dynamic Sections in Golang

To get started with TOML in Go, you'll need to use one of these packages. The github.com/pelletier/go-toml package is a Go library for the TOML format, supporting TOML version v1.0.0-rc.3.

The TOML package is a reflection interface similar to Go's standard library json and xml packages, and it's compatible with TOML version v1.0.0.

On a similar theme: Golang Pkg

Go V2

Go V2 offers a Go library for the TOML format, which is a simple and easy-to-read format for configuration files.

The go-toml v2 library specifically supports TOML v1.0.0, making it a reliable choice for working with TOML files in Go.

This library is a great resource for developers who want to work with TOML files in their Go projects, providing a straightforward way to read and write TOML data.

Stdlib Behavior

This library is designed to behave similarly as the standard library's encoding/json, which means you can rely on its consistency and predictability.

The library aims to mimic the standard library's behavior as much as possible, making it easier to switch between the two.

This consistency in behavior will save you time and effort in learning and using the library, as you'll already be familiar with the standard library's encoding/json.

Examples

Credit: youtube.com, Golang Tutorial for Beginners | Full Go Course

A TOML file is simply a list of keys and values, which makes it easy to understand and work with.

You can store data such as the name of a system and its version in a TOML file.

If your struct field name doesn't directly map to a TOML key value, you can use struct tags to make it work.

This is useful when you have existing structs you want to use with TOML, but the field names don't match up perfectly.

Only exported fields are considered when encoding and decoding, so make sure to use those. Private fields are silently ignored, which can be a gotcha if you're not careful.

README

TOML stands for Tom's Obvious, Minimal Language. It's a Go package that provides a reflection interface similar to Go's standard library json and xml packages.

This Go package is compatible with TOML version v1.0.0. You can check the changelog on the releases page, which also includes information in the git tag annotations.

Credit: youtube.com, Everything about TOML format

The library requires Go 1.18 or newer, so make sure to update your Go version if you haven't already. To add it to your go.mod file, use the following command:

It also comes with a TOML validator CLI tool, which is a great feature for testing and validating TOML documents.

Here's a list of the TOML validator CLI tool's commands:

  • tomlv: validates TOML documents and prints each key's type
  • toml-test-decoder: satisfies the toml-test interface for testing TOML decoders
  • toml-test-encoder: satisfies the toml-test interface for testing TOML encoders

Source Files

In Go, you can load configuration data from a .toml file using the ioutil.ReadFile function, which returns a byte array containing the TOML content.

Loading TOML content from a file is useful for decoupling the configuration data, making it easier to manage and update.

You can use a multi-line string as an argument, but loading from a file is a more practical approach, especially for large configurations.

The ioutil.ReadFile function is a simple way to read the contents of a file into a byte array, making it easy to work with TOML files in Go.

The TOML file format is specifically designed for configuration data, and using a separate file for this purpose is a good practice.

You might enjoy: Golang Ioutil

TOML Features

Programming Language on a Screen
Credit: pexels.com, Programming Language on a Screen

Go-toml provides a range of features that make it easy to work with TOML documents. These features include loading TOML documents from files and string data.

You can easily navigate the structure of a TOML document using the Tree feature, which makes it simple to find what you need. This is particularly useful when working with complex TOML files.

The following features are available in Go-toml:

  • Load TOML documents from files and string data
  • Easily navigate TOML structure using Tree
  • Marshaling and unmarshaling to and from data structures
  • Line & column position data for all parsed elements
  • Query support similar to JSON-Path
  • Syntax errors contain line and column numbers

Strict Mode

In strict mode, the decoder can be set to error when some parts of the TOML document are not present in the target structure. This is a great way to check for typos.

Setting the decoder to strict mode is a simple yet effective way to catch errors and typos in your TOML documents. It's a feature that's easy to use and can save you a lot of time and frustration in the long run.

By enabling strict mode, you can ensure that your TOML documents are accurate and complete. This is especially useful when working with large or complex documents.

Versioning

Credit: youtube.com, Software Versioning Explained - Semantic (SemVer), Calendar (CalVer), etc.

Go-toml follows Semantic Versioning, which means it adheres to a specific format for version numbers.

The supported version of TOML is indicated at the beginning of this document, so you can easily check which version you're working with.

The last two major versions of Go are supported, which is in line with the Go Release Policy.

Features

Go-toml provides a range of features that make working with TOML documents a breeze.

You can load TOML documents from files and string data, which is super convenient for developers who need to parse data from various sources.

The Tree feature allows you to easily navigate the TOML structure, making it easy to find and access specific data.

Go-toml also includes marshaling and unmarshaling capabilities, which enable you to convert data structures to and from TOML documents seamlessly.

Line and column position data is provided for all parsed elements, which is helpful for debugging and troubleshooting.

Workplace with modern laptop with program code on screen
Credit: pexels.com, Workplace with modern laptop with program code on screen

Here are some of the key features of Go-toml in a concise list:

  • Load TOML documents from files and string data
  • Easily navigate TOML structure using Tree
  • Marshaling and unmarshaling to and from data structures
  • Line & column position data for all parsed elements
  • Query support similar to JSON-Path
  • Syntax errors contain line and column numbers

With Go-toml, you can query data using a syntax similar to JSON-Path, which is a big time-saver for developers who need to extract specific data from TOML documents.

Working with TOML

Unmarshaling is the process of reading a TOML document and filling a Go structure with its content. This is the opposite of marshaling, which represents a Go structure as a TOML document.

You can use the Unmarshal function to convert a byte array into a nested map, as shown in Example 5. This function is a powerful tool for working with TOML data in Go.

To unmarshal a TOML document into a Go structure, you can use the Decode function, which reads a TOML-encoded value from its input and unmarshals it into the value pointed at by v, as seen in Example 2.

Struct tags can be used to map TOML data to struct members, as demonstrated in Example 3. This allows you to create a Go structure that can be easily populated with TOML data.

Here are some examples of how struct tags can be used:

Commented Config

Credit: youtube.com, Using Centrifugo with TOML-based configuration file (https://toml.io).

TOML is often used for configuration files, and go-toml can emit documents annotated with comments and commented-out values.

go-toml can generate files with commented-out values, making it easier to manage and understand complex configurations.

For example, go-toml can generate a file with commented-out values, allowing you to easily toggle between different settings.

This feature is especially useful when working with large configuration files, as it helps keep the code organized and easy to read.

go-toml's ability to generate commented config files makes it a valuable tool for developers working with TOML.

Positions Not Retrievable

The API for retrieving the position of a specific TOML element no longer exists.

This change was made to minimize the number of concepts introduced by the library and avoid performance hits related to storing positions in the absence of a document model.

In the past, position retrieval seemed to have little use, but errors now have more detailed position information.

Position retrieval is better suited for a document model, which has been removed from the scope of go-toml v2 at the moment.

Loading Data into Struct

Credit: youtube.com, Managing Configurations with TOML using RUST | Stop using .env | env Specific Configuration

Loading Data into Struct is a crucial step when working with TOML. You can load TOML data into a struct using the toml.Decode function, which takes a byte slice and a pointer to a Go data structure.

The toml.Decode function works by decoding the TOML data into the specified Go data structure. This is useful when you have a predefined struct and want to load the TOML data into it.

To make the most of this feature, you can use struct tags to map the TOML data to struct members. For example, you can use the "toml:" tag to specify the key name in the TOML data that corresponds to a struct member.

Here's an example of how you can use struct tags to load TOML data into a struct:

```markdown

type Example struct {

SystemKey string `toml:"system"`

Name string `toml:"name"`

TomlVersion string `toml:"toml_version"`

}

Credit: youtube.com, Liked Pydantic? You'll LOVE Msgspec

```

By using struct tags, you can load the TOML data into the struct members and access the data in a more convenient way.

In some cases, you may need to load TOML data into a struct that has a complex structure. In such cases, you can use the UnmarshalTOML method to implement a struct type that knows how to unmarshal itself.

The UnmarshalTOML method can be used with interfaces in a struct in cases where the actual type is not known until the data is examined. This provides more flexibility when working with complex data structures.

Here's an example of how you can use the UnmarshalTOML method to load TOML data into a struct:

```go

type UnmarshalTOML struct {

// ...

}

func (u *UnmarshalTOML) UnmarshalTOML(data []byte) error {

// ...

}

```

By using the UnmarshalTOML method, you can load the TOML data into the struct and access the data in a more convenient way.

In summary, loading data into a struct is an essential step when working with TOML. By using the toml.Decode function, struct tags, and the UnmarshalTOML method, you can load the TOML data into a struct and access the data in a more convenient way.

Curious to learn more? Check out: Golang Method

TOML Tools and Utilities

Credit: youtube.com, Explaining my standard golangci-lint configuration (.golangci.toml)

Go-toml provides three handy command line tools: tomll, tomljson, and jsontoml. These tools can be installed using the go install command.

You can install tomll by running go install github.com/pelletier/go-toml/cmd/tomll. Then, you can lint a TOML file by running tomll with the --help flag.

Tomljson reads a TOML file and outputs its JSON representation. You can install it by running go install github.com/pelletier/go-toml/cmd/tomljson.

Jsontoml reads a JSON file and outputs a TOML representation. It can also be installed using the go install command.

You can build your own Docker image as usual. Go-toml tools are also available as a Docker image from dockerhub.

Here are the tools and their corresponding installation commands:

TOML Syntax and Semantics

TOML syntax is made up of key-value pairs, tables, and arrays. Key-value pairs are used to define data within a table, where = is used to assign a value to a key.

Tables are used to group related data, and are defined using [] brackets. They can be nested within other tables to create a hierarchy of data, as seen in the example of the server table with a nested development table.

Arrays are used to store multiple values of the same type, and are defined using [] brackets. Values within an array are separated by commas, as shown in the hosts array within the development table, which contains three values.

Local Date and Time

Close-up of colorful programming code displayed on a computer screen.
Credit: pexels.com, Close-up of colorful programming code displayed on a computer screen.

TOML supports native local date/times, which allows you to represent a date, time, or date-time without relation to a timezone or offset.

The go-toml library provides convenient structures for this use-case, including LocalDate, LocalTime, and LocalDateTime, which can be transformed to and from time.Time.

These types are unambiguous structures for their respective TOML representation, making them a great choice for representing local dates and times.

TOML's local date/time support is a powerful feature that simplifies working with dates and times in your code.

Syntax

TOML syntax is made up of key-value pairs, tables, and arrays. Tables are used to group related data.

A table is defined by surrounding it with square brackets [], and can be nested within other tables to create a hierarchy of data.

Key-value pairs are used to define the data within a table, and are assigned using the equals sign =.

Arrays are used to store multiple values of the same type, and are defined by surrounding them with square brackets []. Values within an array are separated by commas ,.

In TOML, arrays can even be used within other arrays, allowing for complex data structures.

TOML Functions and Methods

Laptop displaying code in a dark setting, highlighting programming concepts and digital work.
Credit: pexels.com, Laptop displaying code in a dark setting, highlighting programming concepts and digital work.

The func Marshal returns a TOML representation of the Go value, which is a useful function to know when working with TOML data.

This function was added in version 1.4.0, so if you're using an earlier version, you'll need to upgrade to use it.

To use the Marshal function, you'll need to call it on a Go value, like a struct or a map. This will return a TOML string representation of that value.

The type Marshaler is an interface that's implemented by types that can marshal themselves into valid TOML. This interface was added in version 1.0.0, which is a relatively early version of the TOML package.

Contextualized Errors

Contextualized errors provide a human-readable explanation of decoding errors, making it easier to identify and fix issues. These errors are returned as DecodeError objects.

Go-toml returns DecodeError objects for most decoding errors, which contain a contextualized version of the error. This helps developers quickly understand the nature of the error and where it occurred.

For instance, a DecodeError might contain a message that provides a clear explanation of the error, allowing developers to take corrective action. This contextualized approach can save time and reduce frustration when working with TOML files.

Automatic Field Naming

A thrilling off-road 4x4 SUV adventure through a muddy, snowy field in Poland.
Credit: pexels.com, A thrilling off-road 4x4 SUV adventure through a muddy, snowy field in Poland.

Automatic Field Naming can be a bit tricky in TOML, especially when it comes to case sensitivity. The main difference between v1 and v2 is how they handle automatic field name guessing.

v1 tries multiple variations of the key, which is a bit like trying to guess the correct name. This can lead to unexpected results if you're relying on casing to differentiate between fields.

In v2, case-insensitive matching is used, similar to encoding/json. This means that field names are matched regardless of case, which can be a relief for developers who don't want to worry about casing.

If you're relying on casing to differentiate between fields, and one of them is not using the toml struct tag, you might be impacted by this change. The recommended solution is to be specific about tag names for those fields using the toml struct tag. This will ensure that your field names are matched correctly, regardless of case.

You might like: Golang Case Statement

Query

Futuristic workspace featuring a glowing computer screen with coding displayed, ideal for technology and programming concepts.
Credit: pexels.com, Futuristic workspace featuring a glowing computer screen with coding displayed, ideal for technology and programming concepts.

The go-toml package has a query feature that allows you to run JSONPath-style queries on TOML files. This feature was available in v1 but has been removed in v2.

You can use the dasel package as a replacement for querying TOML files. Dasel is a more complete solution that provides a similar functionality to the removed query feature.

The go-toml/query package was essentially not supported anymore and increased the complexity of the code base, making it a good candidate for removal. The last commit for this package was in May 2020.

JSONPath-like queries are also available in the github.com/pelletier/go-toml/query package, which allows you to quickly retrieve elements of a TOML document using a single expression.

Explore further: Golang Jsonpath

Marshal

The Marshal function is a key part of working with TOML in Go. It returns the TOML encoding of a given value.

You can use Marshal to transform a Go data structure into a TOML representation. The behavior is similar to the Go json encoder, but with some key differences.

Programming Code on Screen
Credit: pexels.com, Programming Code on Screen

For example, Marshal does not support the Marshaler interface or MarshalTOML function for sub-structs. This means that only definite types can be marshaled, such as structs and maps, but not interfaces like `interface{}`.

The Marshal function also supports a range of struct annotations, including the ability to specify a custom key name using the `toml` struct tag.

Here are some examples of how Marshal can be used:

Note that only exported keys are encoded due to the use of reflection, so unexported keys are silently discarded. This can be a gotcha when working with TOML, so be sure to keep it in mind.

TOML Advanced Topics

In the world of TOML, there are some advanced topics worth exploring. You can find more complex examples in the _example/ directory, specifically in the Example UnmarshalTOML and Example Unmarshaler files.

These examples showcase how to implement a struct type that knows how to unmarshal itself, taking full responsibility for mapping values into the struct. This can be particularly useful when working with interfaces where the actual type isn't known until the data is examined.

Close Up Photo of Programming of Codes
Credit: pexels.com, Close Up Photo of Programming of Codes

You can use the Decode method with interfaces in a struct, which may be used with the actual type not known until the data is examined. The Decode method is also used in conjunction with the UnmarshalTOML method.

Here are some key packages and methods to keep in mind:

  • Package (StrictDecoding)
  • Package (UnmarshalTOML)
  • Package (Unmarshaler)
  • Decode
  • Encoder.Encode
  • MetaData.PrimitiveDecode

Note that the encoding process is described in the Encoder package, which is worth checking out for a deeper understanding of how encoding works.

Jsonpath-like Queries

JSONPath-like queries are a powerful tool for quickly retrieving elements of a TOML document using a single expression. This is implemented in the github.com/pelletier/go-toml/query package, which is similar to JSONPath.

The go-toml/query package is no longer available in v2, but you can check out dasel as a replacement. This was because the package was essentially not supported anymore, increased the complexity of the code base, and more complete solutions exist out there.

The github.com/pelletier/go-toml/query package allows you to run JSONPath-style queries on TOML files, making it easy to retrieve specific elements. This can be a huge time-saver when working with large TOML documents.

JSONPath-like queries are a great way to quickly retrieve elements of a TOML document, but keep in mind that pointers are automatically assigned the "omitempty" option, as TOML explicitly does not handle null values.

More Complex Usage

A Man Opening a File
Credit: pexels.com, A Man Opening a File

More complex usage is where things get really interesting. You can find a more complex example in the example directory, specifically in the Example UnmarshalTOML.

This example shows how to implement a struct type that knows how to unmarshal itself. The struct takes full responsibility for mapping the values passed into the struct.

When you need to unmarshal data into a struct, you have two options: UnmarshalTOML or Unmarshaler. UnmarshalTOML is used when the actual type is not known until the data is examined.

Here are the packages you'll need to use for more complex usage:

  • Package (StrictDecoding)
  • Package (UnmarshalTOML)
  • Package (Unmarshaler)

These packages will help you decode TOML strings into your own custom data type, which is exactly what Example Unmarshaler shows.

TOML Development and Contribution

You can contribute to the development of go-toml by reporting bugs and patches using GitHub's pull requests system on pelletier/go-toml.

Any feedback is much appreciated, and it's a great way to get involved in the project.

Development Status

Colorful lines of code on a computer screen showcasing programming and technology focus.
Credit: pexels.com, Colorful lines of code on a computer screen showcasing programming and technology focus.

The next version of go-toml, v2, is currently in active development and nearing completion. It's already more tested, fixes bugs, and is much faster than its predecessor.

If you only need to read and write TOML documents, the features for this are implemented and the API is unlikely to change. This means you can start using v2 without worrying about breaking changes.

The remaining features are expected to be added shortly, but for now, the focus is on getting v2 stable. Pull-requests for v1 are still welcome, but no active development is expected on it.

Here are the features that are currently implemented in v2:

  • Package (Tree)
  • Package (Unmarshal)
  • Encoder (Anonymous)
  • Marshal
  • Unmarshal

TOML v2 is licensed under the MIT License (MIT) + Apache 2.0. You can find more information in the LICENSE file.

Contribute

If you want to contribute to the development of TOML, you can report bugs and patches using GitHub's pull requests system on the pelletier/go-toml repository.

The community appreciates any feedback you can provide, so don't hesitate to share your thoughts.

You can also use the script ./fuzz.sh to run go-fuzz on go-toml, which is a useful tool for testing and debugging.

Ismael Anderson

Lead Writer

Ismael Anderson is a seasoned writer with a passion for crafting informative and engaging content. With a focus on technical topics, he has established himself as a reliable source for readers seeking in-depth knowledge on complex subjects. His writing portfolio showcases a range of expertise, including articles on cloud computing and storage solutions, such as AWS S3.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.