Building Uber’s Go Monorepo with Bazel

In traditional industries such as automobile or aerospace, engineers first design the products and the manufacturing facilities produce the cars or aircrafts according to the design. In software development, a build system is similar to the manufacturing facilities that take the source code and turn it into services, tools, and applications. Besides facilitating software compilation and linking, build systems often need to generate code, download external packages, or build different installation packages. Some build systems can also manage tools, such as compilers, linkers and code generators, making the build artifacts less dependent on their local environments. When Uber started leveraging Go to develop our back-end services, we used the popular open source build system Make in combination with Go’s default build system go build.  

Upon moving our Android and iOS projects at Uber to a more efficient monorepo model, the Go Developer Experience team made a similar move for Go projects and found that Make and go build no longer met our needs. We finally decided to use Bazel, which provides good support for the Go language and is strengthened by constant contributions from its active open source community.

With a significant portion of our tech stack developed in Go, Uber’s Go monorepo is likely one of the largest Go repositories run on Bazel. We noticed areas where we could improve and contribute to the Bazel ecosystem, enhancing the generation of Bazel rules and integrating Bazel with SubmitQueue, Uber’s system for making sure the master branch of the monorepo can always build and test successfully. 

We hope our experience building a large repository with Bazel and our contributions to the open source Bazel ecosystem will help other engineering teams use Bazel to build their source code repositories. 

Uber writes most of its back-end services and libraries in Go. Before we decided to build a Go monorepo, engineers at Uber developed these Go projects in many small and isolated repositories (some of which we’ve open sourced). We launched our Go monorepo in early 2018, and saw an immediate uptick in build efficiency among early adopter projects. As the Go monorepo matured, we moved more and more projects on to it and usage expanded rapidly, as depicted in Figures 1 and 2, below:  

As of this writing, we have over 70,000 files in our Go monorepo. As we don’t commit generated code in general, these Go files were primarily written manually. The massive growth of our Go monorepo encouraged us to assess new build solutions, like Bazel, to meet our development needs.

Bazel is designed to work at scale and supports incremental hermetic builds across a distributed infrastructure, which is necessary for Uber’s large codebase. With the official Bazel Go ruleset, we are able to manage the Go toolchain and external libraries without depending on locally installed ones. There is also an official Bazel project, Gazelle, that we use to  generate Go and Protocol Buffers rules. With Gazelle, we are able to generate Bazel rules for most Go packages in our Go monorepo with minimal human input. Gazelle can also import the versions of Go modules into Bazel rules so we can conveniently and efficiently build external libraries. 

With Bazel’s remote cache, our build servers can also share their build artifacts. A package is built and tested only when something has changed either in the package or its dependencies.

Out-of-the-box software solutions rarely work for a codebase as large and complex as Uber’s Go monorepo. We have added to and refined Bazel to better suit our needs, improving the rule generator and developing several new Bazel rules and features, as well as tools for building large codebases at scale. Along the way, we fixed numerous bugs in both the Go and Bazel open source projects.

Bazel requires all build targets to be defined explicitly with build rules. There are at least two build targets for each Go package, one to build it as a library so other packages can import it, the other to run unit tests for that package. Creating and maintaining a large number of build rules in a repository the size of Uber’s is a tedious and error-prone task. Fortunately, most of the build configurations of Go and Protocol Buffers can be inferred from its source code, which opens up an opportunity to generate those Bazel rules automatically. This is where Gazelle comes into play.

As stated earlier, our Go monorepo is likely the largest Go repository to utilize Bazel and Gazelle so far, leading to complex scenarios the designers of Bazel and Gazele did not necessarily foresee when using the software at scale. We worked closely with the open source community to remove these roadblocks, fixing bugs, and adding a few new features.

In Bazel, external Go modules are downloaded using go_repository rules. Gazelle generates one such rule for each module in go.mod and go.sum files, which are managed by the Go toolchain. In the Go monorepo, we have over one thousand external modules. 

As part of the development of our Go monorepo, Uber contributed several features to Gazelle to improve how it generates and manages the resulting go_repository rules. For example, Gazelle could only generate all go_repository rules in Bazel’s WORKSPACE file, which also contained manually written and maintained workspace rules and macros. Some of these manual rules and macros must be placed before the generated go_repository rules, others after the generated ones. However, Gazelle could only append new go_repository rules to the end of the WORKSPACE file. We added a feature (#480, #493) to Gazelle so it can now write the go_repository rules to a separate macro file and load it into the WORKSPACE file. As a result, all generated rules are kept outside the WORKSPACE file, making this file smaller and easier to maintain.

Our Go monorepo also lets engineers add or remove external modules. When a module is removed, the Go toolchain removes it from go.mod and go.sum files. However, Gazelle was not able to clean up unnecessary go_repository rules. We added an option to Gazelle to prune unneeded go_repository rules (#514). After a go_repository rule downloads a Go module, it calls Gazelle to generate Bazel rules to build that module. We also added parameters (#603, #649) to the go_repository rule in order to configure Gazelle’s behavior inside the external module. These improvements, along with many smaller features and bug fixes our teams have contributed over the years, propelled two Uber engineers to the number two and three spots on Gazelle’s list of contributors as of April 2020.

Gazelle is designed to support Bazel rules for many different languages. The official extensions of Gazelle can generate Bazel rules for Go and Protocol Buffers. However, Uber leverages many other kinds of rules in our Go monorepo, including rules for Apache Thrift, ThriftRW, Apache Avro, and GoMock. The open source community developed some of these rules, while others were developed internally at Uber. We also developed several Gazelle extensions to cover these additional rules, and plan to open source both our new rules and Gazelle extensions in the near future.

To keep the master branch of our Go monorepo in the green state, meaning all code on the master branch can be successfully compiled and tested at any time, we perform a series of checks before landing a commit to the master. These checks include building and testing all packages changed in that commit, as well as all dependent packages transitively. Depending on how many packages are affected by a commit, the checks can take anywhere from a few minutes to a few hours.

If we land our commits sequentially, each commit must wait until all previous commits land before being checked, which can lead to long landing time. To further complicate matters, the larger our monorepo becomes, the more commits come in, stacking up and creating even longer queues. We knew that eventually the commits would arrive too quickly for us to check them sequentially. 

In order to keep up with our high commit rate, Uber engineers use SubmitQueue to check and land commits in parallel while preventing code conflicts. To do this, SubmitQueue needs to know the list of build targets affected by a given commit. Other repositories at Uber using Buck as their build system are able to run the “buck targets –show-target-hash” command on the revision before and after the commit, find out what targets’ hashes have changed due to the commit, and pass the list of targets to SubmitQueue. Unfortunately, even though Bazel knows what targets and actions it needs to build again internally, it does not expose hash keys of actions or targets from its command line interface.

After talking through the problem with Google’s Bazel team both on Github and offline, we decided to develop our solution outside Bazel. The resultant tool traverses through Bazel’s build graph, calculating the hash for each build target in the graph by combining the rule definition, attributes, input files, and the hash of other targets it depends on. With the hash for each build target, we can identify the list of build targets affected by a commit and pass that list to SubmitQueue. We intend to open source this tool in the future. 

Once SubmitQueue knows which build targets are affected, it needs to call Bazel to build and test those targets to make sure it’s safe to land the commit on to the master branch. The list of targets can be very large when the commit upgrades a core library, e.g., upgrades the Go rule set version. As the monorepo grew, the build target list increased to a point where it became too long to pass it through Bazel’s command line interface. After discussing the issue on Github, we were initially able to work around it by passing the build target list using a specific build configuration instead of the command line. As the monorepo continued to grow, this workaround failed again. 

Eventually, we contributed a feature to Bazel so it can read the build target list from a file. The feature was accepted and merged into Bazel’s master repository, and is available since Bazel 3.1.

Although we successfully adopted Bazel as the build system for Uber’s Go monorepo, there is still work to be done. For example, current Go IDEs do not support Bazel as well as we would like. We have found that IDEs cannot locate Go packages and modules generated and downloaded at build time, and are planning to work with the open source community to close this gap.

Our Go monorepo is also one of the earliest Uber repositories to use Bazel. We are planning to share our experience and technologies with other teams at Uber, helping them migrate to Bazel as their build system. Many of the tools and features we built have this vision in mind, designed to be general and extensible enough that they can be reused outside our Go monorepo or even outside of Uber.