Golang Monorepo Best Practices and Setup

A monorepo setup in Golang can greatly simplify your development workflow, but it requires careful planning and execution.

To start, it's essential to use a consistent directory structure, as shown in the "Directory Structure" section, to keep your code organized and easily navigable. This will make it easier to find and manage your codebase.

A popular tool for managing a Golang monorepo is Go Modules, which allows you to manage dependencies and versions with ease. As mentioned in the "Go Modules" section, this can help reduce conflicts and make it easier to collaborate with others.

Using a consistent coding style is also crucial in a monorepo setup, as it makes it easier for developers to understand and work with each other's code. The "Coding Style" section highlights the importance of following a consistent style guide to ensure readability and maintainability.

Expand your knowledge: Golang vs Go

Building a monorepo can be especially appealing when working with a small team, where a few developers collectively maintain multiple software projects. It's a great way to integrate changes across multiple projects at once.

Easier code reviews are another benefit of monorepos. Code reviews are in one place, and the scope of code the team owns is easy to comprehend.

Large organizations like Google, Facebook, and Twitter have successfully employed very large monorepos. This shows that monorepos can be a viable option even for large teams.

Here are some potential pros of using a monorepo:

An efficient build tool for a monorepo should not rebuild components that haven’t changed, nor should it re-run tests that aren’t necessary.

Earthly does this naturally in a local environment, which is super useful in speeding up development.

Shared caching can be used to improve the build process in a CI pipeline, but it requires upload and download steps to sync the cache during each CI run.

This can yield a nice performance boost for compute-heavy steps, such as long-running integration tests which do not always need to be re-run.

For your interest: Golang Run Debug Mode

Releasing and versioning microservices in a monorepo can be a bit tricky. Typically, a repo contains a single module with semantic version numbers specified as Git Tags.

In a monorepo, we may have a couple of options for versioning. One way is to use the replace syntax in go.mod to use libraries within the context of the monorepo, and all consumers of the library are always getting the current copy via local import.

Publishing versions of those libraries are not relevant in this case. However, for microservices in our monorepo, versioning may be more important. Different versions of those containers may need to be available for deployment at any given time.

We can version microservices in a monorepo by storing the version as a file within each service. This allows us to publish the resulting image from our build with a tag based on that version file.

One tool we can use to help with versioning is the open source semver-cli tool. This tool can be used to create a new +release target in our microservice build.

Worth a look: Golang Version Manager

GitHub is a popular platform for hosting and managing open-source projects, and it's a crucial part of the golang monorepo workflow.

GitHub provides a robust set of features for collaboration and code review, including pull requests, code commits, and issue tracking.

The Go team uses GitHub to manage their monorepo, with over 100,000 stars and thousands of contributors.

GitHub Actions automates the build and test process for the Go monorepo, ensuring that code changes are thoroughly tested before they're merged.

The Go team uses a combination of GitHub Actions and Makefiles to automate the build and test process, making it easier to manage complex workflows.

Broaden your view: Simple Http Server Golang Github

To set up a workspace in Go, you first need to create a go.work file to specify the module. This file is crucial in defining the workspace.

The go work init command is used to create a go.work file for a workspace containing the modules in the specified directory. For example, running go work init in the workspace directory will create a go.work file.

In the go.work file, the go directive specifies the version of Go to be used, similar to the go directive in the go.mod file. This ensures that the module is active in any subdirectory of the workspace.

Readers also liked: Golang Go

To create the workspace, start by creating a go.work file that specifies a workspace with the module. This file will be used to manage the dependencies of your project.

The go work init command is used to create the go.work file, which is similar in syntax to the go.mod file. This command tells Go to create a go.work file for a workspace containing the modules in the specified directory.

The go.work file will have a similar structure to the go.mod file, with a go directive that specifies the version of Go to use. This directive is crucial, as it determines how the file is interpreted.

In any subdirectory of the workspace, the module will be active. This is because the go.work file is used to manage the dependencies of the project, regardless of the directory you're in.

Next, you'll need to add a local copy of the golang.org/x/example/hello module to the workspace. This module is stored in a subdirectory of the go.googlesource.com/example Git repository.

Here's an interesting read: Golang Args

To run a program in the workspace, you need to be in the workspace directory.

The Go command includes all the modules in the workspace as main modules.

Running the go run command outside the module or the workspace would result in an error because the go command wouldn’t know which modules to use.

See what others are reading: How to Run Golang File

You can modify existing code in a Go module by cloning a Git repository, adding it to your workspace, and then modifying the code. This is demonstrated by downloading and modifying the golang.org/x/example/hello module.

To add a new function to a module, create a new file in the module directory containing the new function's code. For example, you can add a new function to the golang.org/x/example/hello/reverse package by creating a new file named int.go.

To use the new function, you'll need to modify the code that uses the module. This can be done by importing the new function in the main program and calling it. For example, you can modify the hello program to use the new function by importing it in the main function and calling it with a specific argument.

A monorepo can contain multiple components, such as applications and libraries, which can be organized as distinct Go modules. For example, a monorepo can have a structure with three distinct Go modules, including two backend Microservices and a single shared Library.

For another approach, see: Golang Comments

Modifying a module can be a straightforward process, especially when working with Go. To download and modify a module, you can use the `git clone` command to download a copy of the Git repository containing the module.

You can then add the module to your workspace using the `go work use` command. This will add a new module to the `go.work` file, allowing you to use the new code you'll write in your copy of the module instead of the version of the module in the module cache.

To add a new function to a module, you can create a new file in the module's directory and add the new function to it. For example, you can add a new function to the `golang.org/x/example/hello/reverse` package by creating a new file named `int.go` and adding the following code to it:

```go

package reverse

import "strconv"

// Int returns the decimal reversal of the integer i.

Take a look at this: Golang Test Command

func Int(i int) int {

i, _ = strconv.Atoi(String(strconv.Itoa(i)))

return i

}

```

This will add a new function to the `reverse` package that reverses an integer.

You can then modify the `hello` program to use the new function by importing the `reverse` package and calling the `Int` function. For example:

```go

package main

import (

"fmt"

"golang.org/x/example/hello/reverse"

)

func main() {

fmt.Println(reverse.String("Hello"), reverse.Int(24601))

}

```

This will print the string "Hello" and the reversed integer 24601.

By using the `go work use` command and modifying the module in place, you can make changes to your code and test them immediately without having to worry about conflicts with the module cache.

For more insights, see: Golang Reflect to Call Function in Package

Code Structure by Feature/Service is a crucial aspect to consider when modifying and structuring your code. Organize your code into feature-specific packages to keep things tidy.

A good starting point for each feature/service is to have three main files: service.go, handlers.go, and client.go. The service.go file contains the main logic of the feature/service and interfaces for anything out of its scope.

On a similar theme: Golang Source

Here's a breakdown of what each file typically contains:

Keeping proto files in each service/feature package can also be beneficial for easier tracking and accessibility. In fact, the author recommends packaging these files into the project's root directory for easier access.

Team collaboration and services are crucial in a monorepo setup. By grouping shared packages and microservices into a team or project hierarchy, you can logically organize your system.

With a project hierarchy, you can define a contract with your wider engineering team about what APIs you will provide. This is done through the proto directory in each team, which serves as a gateway into your system for other teams.

This setup creates a healthy team practice where you and your stakeholders can collaborate on API design. The review process in your monorepo ensures that everyone is on the same page.

You might like: Monorepo Nextjs

Service Two was successfully compiled using the strategy developed for Service One. This allowed us to integrate both services into our example.

By using this approach, we were able to create and run unit tests for Service Two. This was a significant milestone in our development process.

Developing and running unit tests for Service Two helped us identify and fix issues early on. This saved us time and effort in the long run.

With Service Two compiled and unit tests in place, we were able to build a solid foundation for our services. This enabled us to focus on further development and improvement.

You might like: Golang Web Development

In a multi-team setup, it's essential to group shared packages and microservices into the team or project that owns them. This logical grouping makes it easier to manage and maintain your system.

Each team or project directory contains its own sub-hierarchy, which includes a pkg directory for shared packages. This allows teams to work independently and reduces conflicts over shared resources.

A proto directory is added to each team, which serves as a gateway for other teams to access your services. This is where you define the contract with your wider engineering team about what services you'll provide.

This setup creates a healthy team practice where teams can collaborate on API design through a review process in the monorepo. Other teams don't need to know which microservices handle specific logic, only that the expected API is available at a specific path.

The root of the monorepo also has a pkg directory, but it's tightly scoped to only include common logic that's necessary for all teams, such as secrets management or service discovery. This helps prevent unnecessary complexity and promotes independent service deployments.

For another approach, see: Gcloud Api Using Golang

Go modules can be a challenge in a golang monorepo, especially when it comes to dependency management. Currently, a single go.mod file at the root is required, which can be painful as it introduces a point of contention when introducing or upgrading dependencies.

In Go 1.18, Workspaces were introduced to support multiple modules, but I've not had the opportunity to evaluate them yet. This could potentially help with dependency management, but for now, it's still a work in progress.

To give you a better idea of the challenges, let's take a look at the advantages of using the "replace" strategy for a local library in a monorepo. Using this strategy allows for faster iteration on code changes, allows checking-in changes in a single commit or pull-request, and updates to a library are immediately used and validated by the service(s) that import it.

Here's an interesting read: Read a Custom Resource Using Cynamic Client Golang

Importing local Go modules in a monorepo can be achieved using the "replace" feature in your go.mod file. This allows you to instruct Go to replace a module found online with a local module at a relative path within the monorepo.

Using the "replace" strategy has several advantages. We can iterate faster on code changes in the library and service(s) that import it. This allows checking-in changes to both the library and service(s) in a single commit or pull-request.

The "replace" syntax works by replacing what would normally be a module found online with a local module. This is particularly useful in a monorepo setup where we have multiple services that import the same library.

Curious to learn more? Check out: Replace Value and Create a Pr Using Golang

Here are some benefits of using the "replace" strategy:

Go Tooling is a crucial part of managing a monorepo in Go, and Earthly is a great tool for the job.

Earthly allows each service or library to independently manage its own build and test cycles.

It can effectively utilize cache so that only the services or libraries which have changed will re-trigger their build.

A parent Earthfile at the root of the monorepo can act as an orchestrator, calling into the more specific Earthfiles lower down in the hierarchy.

Each service or library can have its own Earthfile, and the parent Earthfile can reference the artifacts created by these Earthfiles.

This approach enables a self-contained build process for each service or library, which can be referenced by other services.

Go modules can be a bit tricky to manage, especially in a monorepo setup. Currently, to support this structure, the monorepo must share a single go.mod file at the root.

This can be painful as it introduces a point of contention when introducing or upgrading dependencies. Fortunately, Go 1.18 introduced Workspaces, which support multiple modules.

With Workspaces, you can add multiple modules to a workspace, making it easier to manage dependencies. For example, you can use the "go work use" command to add a new module to the workspace, as shown in Example 2. This allows you to use different versions of dependencies for different modules.

One of the advantages of using Workspaces is that you can iterate faster on code changes in a library and its importing services. By using the "replace" strategy for a local library, as shown in Example 1, you can quickly test changes and validate them in the importing services.

Here's a summary of the advantages of using the "replace" strategy:

Keep in mind that Workspaces are still a relatively new feature, and it's worth noting that the author of Example 3 hasn't had a chance to evaluate them yet.