Golang Registry Tutorial and Advanced Features

Author

Reads 823

Close Up Photo of Programming of Codes
Credit: pexels.com, Close Up Photo of Programming of Codes

Golang's registry is a key component of the Go module system, providing a central repository for packages and dependencies.

It's essentially a cache of package versions, allowing for efficient and reliable retrieval of dependencies.

The registry is managed by the Go team and is used by the Go command-line tool to resolve package dependencies.

You can interact with the registry using the `go mod` commands, such as `go mod init` to initialize a new module or `go mod tidy` to clean up the module's dependencies.

The registry also supports private repositories, allowing you to host your own packages and dependencies.

This can be useful for companies or organizations that want to keep their package versions private or have specific dependencies not available in the public registry.

Readers also liked: Create a Package in Golang

Uploading Packages

Uploading packages to a GoLang registry is a straightforward process. You can use the Cloudsmith CLI to upload a Go module with the command `cloudsmith upload --package go --path /path/to/module`.

For your interest: Golang File Upload

Credit: youtube.com, Packages in Go (Go Basics #6)

There are a few ways to package and upload a Go module. The package-go-module gcloud CLI add-on packages your Go modules, allowing you to version and upload to Artifact Registry using the `gcloud` command.

You can also use the Cloudsmith CLI to upload a Go module. For full details on how to install and set up the Cloudsmith CLI, see the Command Line Interface section.

To upload a Go module via the Cloudsmith CLI, use the command `cloudsmith upload --package go --path /path/to/module`. If you're working with an older Go project that's not in the Go Modules format, you may need to add module support using `go mod init` and `go mod tidy` before uploading.

If you're dealing with raw GO binaries, you'll need to upload them as a RAW package format instead. To do this, you'll need to generate a Go module first. You can build a module with standard command-line tooling like `zip` and `git`.

Consider reading: Golang vs Go

Raw Binaries and Proxies

Credit: youtube.com, Golang Malware Development - Windows Registry Shenanigans with Go

Raw Binaries and Proxies are used in the Go module system to download and install packages.

Go modules use the GOPROXY environment variable to specify a proxy server for downloading packages.

A proxy server can be a local server or a remote server, and it can be specified as a URL or a list of URLs.

On a similar theme: Golang Go

Raw Go Binaries

Raw Go Binaries are not supported, and you'll need to upload them as a RAW package format instead. This requires generating a go module first.

To create a go module, you can use standard command-line tooling like zip and git. For example, let's use the logrus library.

You'll need to create the correct directory structure and check out the version of logrus you want to pack. For our example, we'll use v1.4.2.

The directory structure should be set up like this:

  1. First, create the correct directory structure for your go module.
  2. Then, check out the version of logrus you want to pack.

Once you have the correct directory structure, you can clean up and pack the module. Use the `find` command to include only folders with files in them.

You can then upload your go module to a Cloudsmith repository, ready for use.

Explore further: Golang Use .env File

GoProxy Multiple Servers

Credit: youtube.com, GoLab 2019 - Aaron Schlesinger - The Athens Project: A Proxy Server for Go Modules

GoProxy allows you to use multiple servers at the same time, making it a great option for users who need to access multiple proxy servers.

You can set up multiple servers in the GoProxy settings, which makes it easy to switch between them.

Each server can be configured separately, including the server address, port, and other settings.

Having multiple servers can help improve the speed and reliability of your internet connection.

This is because you can use a different server if one of them is experiencing issues or is slow.

You might enjoy: Golang Use Cases

Cloudsmith CLI and Authentication

You can authenticate to Artifact Registry using the Go credential helper to refresh tokens in your netrc file. This simplifies the authentication process.

The Go credential helper can be added to your GONOPROXY export with the command `GONOPROXY=github.com/GoogleCloudPlatform/artifact-registry-go-tools`.

To add your Artifact Registry credentials to your netrc file, run the command `GOPROXY=proxy.golang.org go run github.com/GoogleCloudPlatform/artifact-registry-go-tools/cmd/[email protected] add-locations --locations=LOCATION`, replacing `LOCATION` with the regional or multi-regional location of your repository.

Artifact Registry supports the following authentication methods: using credentials in your environment, or adding an unencrypted service account key to your netrc file.

Recommended read: Golang Test Command

Upload via Cloudsmith CLI

Credit: youtube.com, Cloudsmith CLI Overview

To upload a Go module via the Cloudsmith CLI, you can use the command as shown in the example. This involves installing and setting up the Cloudsmith CLI first, which you can find more details on in the Command Line Interface section.

The command to upload a Go module is straightforward. You can add module support to older Go projects by running go mod init and go mod tidy. This will create new go.mod and go.sum files that you can commit and add a new tag to.

You may experience errors when pushing older Go projects to Cloudsmith due to their format. Adding module support is a simple fix that can resolve this issue.

You might enjoy: Golang Add to Map

With Authentication

You can enable authentication for the Cloudsmith CLI by using the WithHtpasswd or WithHtpasswdFile options.

The WithHtpasswd option allows you to enable authentication with a string representing the contents of a htpasswd file, which will be created temporarily and copied to the container.

Credit: youtube.com, Getting Started with Go and Cloudsmith

You can use the WithHtpasswdFile option to copy a htpasswd file from your local filesystem to the container, which will be copied into the /auth directory.

To use WithHtpasswd, you'll need to pass a string representing the contents of the htpasswd file.

Here's a comparison of the two options:

The htpasswd file will be copied into the /auth directory inside the container, regardless of which option you choose.

By enabling authentication, you'll be able to secure access to your Cloudsmith repository and manage user permissions more effectively.

Repository Management

To create a new repository in the golang registry, you can use the `gcloud artifacts repositories create` command. Run this command to create a new repository, replacing `REPOSITORY` with the desired name, `LOCATION` with the location, and `DESCRIPTION` with a brief description of the repository.

The command output will resemble the following:

Encryption: Google-managed key

Repository Size: 0.000MB

createTime: '2022-06-03T20:20:01.644813Z'

format: GO

mode: STANDARD_REPOSITORY

name: projects/my-project/locations/us-west1/repositories/my-repo

updateTime: '2022-06-03T20:20:01.644813Z'

To view the repository details, you can use the `gcloud artifacts repositories describe` command. This command will display information such as encryption, repository size, and creation time.

Here are the key details you can view using this command:

Create a Repository

Credit: youtube.com, How to Create a Local Repository

To create a repository, you'll need to run a command in the Cloud Console. Run the following command to create a new repository: `gcloud artifacts repositories create REPOSITORY --repository-format=go --location=LOCATION --description=DESCRIPTION`.

This command will create a new repository with a Google-managed key for encryption. The repository size will be 0.000MB, and it will have a standard mode.

To view the repository details, you'll need to run another command. Run the following command to view the repository details: `gcloud artifacts repositories describe --location=LOCATION REPOSITORY`.

The output will show you the encryption method, repository size, and other important details. For example, it will show the encryption method as "Google-managed key" and the repository size as "0.000MB".

Expand your knowledge: How to Run Golang Program

Pkg Registry

The pkg registry is a Docker v2 registry and the OCI distribution specification, designed for use in tests and other situations where a low dependency container registry is needed.

It's focused on being standards compliant and will become stricter over time, so you can expect to be asked to reduce dependencies in your code.

Credit: youtube.com, Discussion/brainstorm on using GitLab generic package registry for corpus management.

The pkg registry has a strict rule: it should only depend on the Go standard library and other packages in go-containerregistry.

This means that if your PR has extraneous dependencies, it might be rejected.

The pkg registry provides several functions, including New, which creates an http.Handler, and TLS, which sets up an HTTPS server.

Here are the available options for creating the registry:

  • func New(opts ...Option) http.Handler
  • func TLS(domain string) (*httptest.Server, error)

The Option type describes the available options for creating the registry.

Tag

Tagging images from the local Registry is a straightforward process. The TagImage method allows you to do just that.

To tag an image, you'll need to provide the Go context, the image reference, and the name of the new tag. The image reference should be in the format my-registry:port/image:tag.

This format is crucial for tagging the image properly.

Check this out: Golang Read Image File

Exists

The ImageExists method is a useful tool for checking if an image exists in the Registry.

It receives the Go context and the image reference as parameters, which should be in the format my-registry:port/image:tag to be pushed to the Registry.

Source Files

Credit: youtube.com, 2025 - subpatch: fearless multi repository management

Managing source files is a crucial aspect of repository management. In a typical repository, source files are the heart of the project, containing all the code, documentation, and assets.

A good repository should have a clear directory structure, with source files organized in a logical and consistent manner. This makes it easier for team members to find and work with specific files.

Source files can be stored in various formats, including text files, images, and videos. Each file should have a unique name and be stored in a location that makes sense for the project.

A Git repository can store source files in a compressed format, reducing storage space and improving performance. This is especially useful for large projects with many files.

In a repository, source files can be managed using version control systems like Git. This allows team members to track changes, collaborate on files, and revert to previous versions if needed.

Source files can be accessed and edited directly from the repository, making it easier for team members to work together on a project. This also allows for easy tracking of changes and updates.

Suggestion: Golang Source Code

Container Registry and Options

Credit: youtube.com, Container Registry

The Registry container can be configured with various options when starting it. You can pass options in a variadic way to customize its behavior.

To pull images from a private registry, you can use the SetDockerAuthConfig function to set the Docker Auth Config for the Registry container. This sets the DOCKER_AUTH_CONFIG environment variable with authentication details.

The DockerAuthConfig helper function returns a map of AuthConfigs, including a base64 encoded Auth field, for the provided authentication details. It also allows you to add more authentication configurations by passing additional host, username, and password triples.

Check this out: Golang Func Type

Docker Auth Config

Docker Auth Config is a crucial aspect of working with private registries. The module exposes a way to set the Docker Auth Config for the Registry container, thanks to the SetDockerAuthConfig function.

This function is useful when you need to pull images from a private registry. It sets the DOCKER_AUTH_CONFIG environment variable with authentication for the given host, username, and password sets. It also returns a function to reset the environment back to the previous state, which is helpful when you need to reset the environment after a test.

The DockerAuthConfig helper function returns a map of AuthConfigs including base64 encoded Auth field for the provided details. It also accepts additional host, username, and password triples to add more auth configurations.

See what others are reading: Azure Docker Registry

Container Registry

Credit: youtube.com, Container Registry

A container registry is a central location where you can store and manage your container images. It's like a library for your containers, where you can easily find and retrieve them.

Docker Hub is a popular container registry that offers a free plan for individuals and small teams. It also has a paid plan for larger teams and enterprises.

Harbor is another container registry that's designed for enterprises and large teams. It offers advanced features like role-based access control and auditing.

Quay is a container registry that's designed for security-conscious teams. It offers features like vulnerability scanning and secret management.

Container registries can be private, meaning they're not accessible to the public, or public, meaning they're accessible to anyone.

Container Options

When starting the Registry container, you can pass options in a variadic way to configure it.

You can configure the Registry container by passing options in a variadic way. This allows for flexible and customizable setup.

Credit: youtube.com, Cheapest Container Registry for you?

The testcontainers package exposes several options that can be used to configure the Registry container. These options enable fine-grained control over the container's behavior.

You can pass options in a variadic way to configure the Registry container. This is made possible by the testcontainers package, which exposes several options for customization.

Check this out: Install Golang Package

With Data

The WithData option allows you to copy a directory from your local filesystem to the container, which will be stored in the /data directory.

You can use this option to initialize the Registry with your own images. The format of the directory should be the same as the one used by the Registry to store images, otherwise, you won't be able to read any images from it.

The WithData option is a powerful tool for customizing your Registry. I've seen it used to great effect in production environments, where users need to store and manage their own images.

Credit: youtube.com, Container Registry

Here are some key features of the WithData option:

  • WithExposedPorts is available since v0.37.0
  • WithEnv is available since v0.29.0
  • WithWaitStrategy is available since v0.20.0
  • WithAdditionalWaitStrategy is available since v0.38.0
  • WithWaitStrategyAndDeadline is available since v0.20.0
  • WithAdditionalWaitStrategyAndDeadline is available since v0.38.0
  • WithEntrypoint is available since v0.37.0
  • WithEntrypointArgs is available since v0.37.0
  • WithCmd is available since v0.37.0
  • WithCmdArgs is available since v0.37.0
  • WithLabels is available since v0.37.0

Security and Verification

Go's sumdb can't record hash values for private repositories, which can cause the local Go command to fail checksum verification after downloading. This can be frustrating, especially if you're working with private modules.

To avoid this issue, you can specify a list of module path prefixes in the GONOSUMDB environment variable, like GONOSUMDB=demo-docs/awesome-repo,cloudsmith-test/acme2. This tells Go to skip checksum verification for those specific modules.

Cloudsmith also supports security scanning for Go modules, which is a critical step in ensuring the integrity of your dependencies.

GONOSUMDB and Checksum Verification

The Go sumdb cannot record the hash value of a private repository, which can cause the local Go command to fail verification after downloading.

GONOSUMDB should be a list of module path prefixes, like for example: demo-docs/awesome-repo,cloudsmith-test/acme2.

There's general agreement against having separate checksum databases, such as a Cloudsmith checksum database.

You can see more details about this by running the go mod download with the -x option.

Security Scanning

Credit: youtube.com, Enabling Docker Security Scanning to Scan Your Images

Security scanning is a crucial step in ensuring the integrity of your code. Cloudsmith supports security scanning for Go modules.

You can find more information about this capability in Cloudsmith's Security Scanning documentation.

This feature helps identify potential vulnerabilities in your code, allowing you to take corrective action before it's too late.

Cloudsmith's security scanning capability is a valuable tool for developers looking to secure their code and prevent common security pitfalls.

Example and Setup

Let's get started with setting up a GoLang registry. Docker Hub is a popular choice for storing and managing container images, and it's free for public repositories.

To use Docker Hub, you'll need to create an account and then create a new repository for your image. This repository will serve as the central location for storing and managing your image's tags and versions.

You can then use the `docker tag` command to associate your local image with the repository on Docker Hub.

Example Project

A vibrant image of freshly picked strawberries being photographed on a smartphone outdoors.
Credit: pexels.com, A vibrant image of freshly picked strawberries being photographed on a smartphone outdoors.

To get a better understanding of what your project should look like, check out our examples repository on GitHub, which provides examples of what to expect for packaging and publishing/uploading.

Our examples repository is a valuable resource that showcases the different aspects of your project, from setup to deployment.

For packaging and publishing, take a look at the examples in the repository to see how to structure your project correctly.

This will help you understand the best practices for your project and ensure a smooth publishing process.

You can find the examples repository on GitHub, where you can explore and learn from the examples provided.

Set up Go environment

To set up your Go environment, you'll need to instruct Go to download modules from Artifact Registry, the public Go module proxy, and then source in that order. This involves exporting the GOPROXY environment variable with the correct location.

You can exclude your module from being checked using the public checksum database by setting the GONOSUMDB environment variable. This is useful if you have a module that you don't want to be checked against the public database. For example, if you have a module at example.com/foo, you can exclude it from being checked by running the command: export GONOSUMDB=example.com/foo.

On a similar theme: Golang Set Env Variable

Credit: youtube.com, Setup your environment for Go - Lesson 03 | Go | Full Course | CloudNative | Golang

Here's a list of steps to set up your Go environment:

  1. Instruct Go to download modules from Artifact Registry by exporting the GOPROXY environment variable: export GOPROXY=https://LOCATION-go.pkg.dev/PROJECT/REPOSITORY,https://proxy.golang.org,direct
  2. Exclude your module from being checked using the public checksum database by setting the GONOSUMDB environment variable: export GONOSUMDB=MODULE_PATH_REGEX
  3. Refresh your OAuth token if you're using short-lived credentials by running the command: GOPROXY=proxy.golang.org \ go run github.com/GoogleCloudPlatform/artifact-registry-go-tools/cmd/[email protected] refresh
  4. Create a directory named "bar" in your home directory by running the command: mkdir bar
  5. Change directories into your module's directory and run go mod init to create a go.mod file for your package: cd bar \ go mod init example.com/bar

Remember to replace the placeholders in the commands with your actual module path or regex.

Registry Operations

To perform registry operations, you'll need to understand the basics of image management. The PushImage method allows you to push an image to the Registry, receiving the Go context and image reference as parameters.

The PushImage method will internally wait for the image to be available in the Registry, querying the Registry API and returning an error if the operation fails. This ensures that the image is successfully pushed before moving forward.

You can also pull images from the Registry using the PullImage method, which downloads the specified image reference and makes it available locally for further operations. The image reference should be in the format my-registry:port/image:tag to be pulled correctly.

Push

Pushing an image to the Registry is a crucial part of Registry Operations. The PushImage method allows you to do just that, taking in the Go context and the image reference as parameters.

Credit: youtube.com, How to View Detailed Docker Push Interactions with Your Registry

The push operation is not instantaneous. It will internally wait for the image to be available in the Registry, querying the Registry API along the way.

If the push operation is successful, the method will return an error in case of any failure. This includes both pushing and waiting for the image, so be prepared for potential issues.

Pull

The Pull operation is a crucial step in working with registries. It allows you to download a specified image reference so it becomes available locally for further operations.

To pull an image, you'll need to use the PullImage method, which receives the Go context and the image reference as parameters. This method pulls the image with the same platform as the registry container's image.

The image reference should be in the format my-registry:port/image:tag to be pulled from the Registry. This is a specific format that you'll need to follow to ensure the image is pulled correctly.

The PullImage method downloads the image, copying it to the local Docker daemon. This makes the image available for further operations such as tagging or pushing.

Delete

Woman in focus working on software development remotely on laptop indoors.
Credit: pexels.com, Woman in focus working on software development remotely on laptop indoors.

Deleting images from the Registry is a straightforward process that requires the image reference in the format image:tag.

To delete an image, you'll need to use the DeleteImage method, which receives the Go context and the image reference as parameters.

The image reference must be in the format image:tag to be deleted from the Registry, so make sure to format it correctly.

You can delete an image from the Registry using the DeleteImage method, which is a convenient way to manage your images.

The DeleteImage method allows you to delete an image from the Registry, giving you more control over your image management.

Advanced Features

The GoLang registry has some amazing advanced features that make it a powerful tool for developers.

It supports multiple registry formats, including Docker, Helm, and OCI, allowing for flexibility and compatibility with different environments.

One of the standout features is the ability to manage private registries, providing an extra layer of security and control for sensitive applications.

This is particularly useful for teams working on large-scale projects, where security and access control are top priorities.

Related reading: Golang Security

Referrers Support in v0.14.0

With the release of v0.14.0, you now have access to the referrers API endpoint, thanks to the WithReferrersSupport feature.

CSS code displayed on a computer screen highlighting programming concepts and technology.
Credit: pexels.com, CSS code displayed on a computer screen highlighting programming concepts and technology.

This feature is specifically designed for OCI 1.1+ environments, where you can leverage the referrers API to enhance your workflow.

The WithReferrersSupport feature is a significant addition to the v0.14.0 release, enabling you to tap into the capabilities of the referrers API endpoint.

To use this feature, make sure your environment meets the minimum requirements of OCI 1.1+.

Consider reading: Rest Api with Golang

New Function

The New Function is a powerful tool in our toolkit. It returns a handler that implements the docker registry protocol.

This handler should be registered at the site root, where it can be easily accessed and utilized.

The New Function is a crucial step in setting up our system, and it's essential to understand its purpose and functionality.

It's a simple yet effective way to get started with advanced features, and I've found it to be a game-changer in my own projects.

Thomas Goodwin

Lead Writer

Thomas Goodwin is a seasoned writer with a passion for exploring the intersection of technology and business. With a keen eye for detail and a knack for simplifying complex concepts, he has established himself as a trusted voice in the tech industry. Thomas's writing portfolio spans a range of topics, including Azure Virtual Desktop and Cloud Computing Costs.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.