In November 2021 we decided to evaluate arm64 for Uber. Most of our services are written in either Go or Java, but our build systems only supported compiling to x86_64. Today, thanks to Open Source collaboration, Uber has a system-independent (hermetic) build toolchain that seamlessly powers multiple architectures. We used this toolchain to bootstrap our arm64 hosts. This post is a story with how we went about it, our early thinking, problems, some achievements, and next steps.

We started in November 2021 with an infrastructure that was exclusively Linux/x86_64. In January of 2023 we have:

Let’s get into how we achieved this.

Before we dive in, let’s make acknowledgements first: bazel-zig-cc, which we cloned and was the foundation for the cross-compiler tooling, was originally created by Adam Bouhenguel. So, our special thanks to Adam for creating and publishing bazel-zig-cc–his idea and work helped make the concept of using Zig with Bazel a reality.

All major cloud providers are investing heavily in arm64. This, combined with anecdata of plausible platform benefits (power consumption, price, compute performance) compared to the venerable x86_64, makes it feel worthwhile to seriously consider making arm64 a part of our fleet.

So we set out to try and see for ourselves. The first goal could be  phrased as the following:

Run a large-footprint application on arm64 and measure the possible cost savings

A key priority was to minimize the amount of work necessary to run and benchmark a service that consumes many cores. We identified two very different possible approaches:

The first option seemed like the right thing to do when considering our priority around minimizing investment. After all, why invest time and money into something that has a non-trivial chance of being abandoned? We considered running a “parallel zone,” which would have arm64 capacity, but otherwise be decoupled from production (and have much looser quality requirements, allowing us to move fast). 

A bit later an important reason for arm64 got tacked on: if we can run our workloads on arm64, we can diversify our capacity, bringing us in a better position with regards to acquiring capacity. The fact that arm64 is needed for capacity diversification was an early signal to abandon the “quick experimental” route and instead spend more time enabling full support for arm64. Thus the mission statement became (and still remains today):

Reduce Uber’s compute costs, increase capacity diversity, and modernize our platform by deploying some production applications on arm64

Since we originally set out with a prototyping mindset, and now it was being turned π^¹ solidifying a tenet to guide us emerged:

No hacks, all in main (i.e., no long-term branches or out-of-tree patches)

Now that we knew that arm64 needed first-class support in our core infrastructure, the project naturally split itself into two pieces:

How can we compile arm64 binaries? By building natively on arm64 hosts or by cross-compiling, of course. To help guide our decision, let’s understand the differences and requirements for native and cross-compiling.

Some terms out of our way:

The diagram below shows how to turn a source file main.c into an executable by compiling natively (left) and cross-compiling (right).

Native compilation requires less effort and configuration to get started, because this is the default mode for most compiler toolchains. On the surface, we could have spawned a few arm64 virtual machines from a cloud provider and bootstrapped our tools from there. However, all our servers use the same base image, including the build fleet. The base image contains many internal tools that are compiled from our Go Monorepo. Therefore, we had a chicken-and-egg problem: how can we compile the tools for our first arm64 build host?

Let’s compile a C file on a x86_64 Linux host, targeting Linux aarch64:

Note that GCC invokes a target-specific executable (aarch64-linux-gnu-gcc), whereas Clang accepts the target as the command-line argument (-target <…>):

Cross-compiling a C source file with both GCC and Clang seems easy on the surface. But what is behind it?

Which files did “clang” use to build the final executable? Let’ strace(1):

These are the files it touched:

Now that we know what is used for the cross-compiler, we can split the dependencies into two categories:

Uber needs to support the following targets:

At the time of writing neither GCC nor LLVM can cross-compile^³ macOS binaries. Therefore we maintain a dedicated build fleet to compile to macOS. Cross-compiling to macOS targets is highly desired to homogenize our build fleet, but we are not there yet.

Here are the host platforms that we support today:

Here is a picture of host toolchains, sysroots, and their relationships where every host toolchain (left) can use any of the target-specific sysroots (right):

To support these host and target platforms, we need to maintain 8 archives: 3 toolchains (the compiled LLVM for each host architecture) and 5 sysroots for each target architecture. A typical LLVM toolchain takes 500-700 MB compressed, and a typical sysroot takes 100-150 MB compressed. That adds up to ~1.5 GB of archives to download and extract before compiling code, on top of all the other tools. To put this into perspective, the Go 1.20 toolchain for Linux x86_64 is 95 MB compressed, and is the biggest required download to start compiling code.

For completeness sake, let’s look at GCC for a moment. As you may remember from earlier in this section, GCC cross-compiler is aarch64-linux-gnu-gcc. That means a full toolchain is needed for every host+target platform. Therefore, if we were to use a GCC-based toolchain, we would need to maintain 3*5=15 separate toolchains. If we add a new host platform (e.g., Linux aarch64 is realistic in the short term) and two target platforms (linux glibc.2.36 for x86_64 and aarch64 respectively), then the number of maintained archives jumps to 4*7=28!

At the time we were shopping for a hermetic Bazel toolchain, we evaluated both GCC and LLVM-based toolchains. LLVM was slightly more favored due to linear growth of required archives (as opposed to quadratic in case of GCC). But can we do better?

Zig takes a different approach: it uses the same toolchain for all supported targets. Using the previous example:

Which files does it use to compile it? If we strace the above execution we will see that only files from Zig SDK (and in /tmp^⁴ for intermediate artifacts) were referenced. Nothing touched the host system. Which means that Zig is fully self-contained.

How can Zig do that, while Clang cannot? What is the main difference between plain Clang and zig? Zig needs all the same dependencies as Clang. Let’s inspect and discuss them here:

As a result of this, Zig can compile to all supported targets with a single toolchain. So to support our 3 host and 5 target platforms, all we need is 3 Zig tarballs, downloaded from ziglang.org/download:

Andrew Kelley, the creator of Zig, explains what Zig adds on top of Clang in more detail in his blog post. The prospect of requiring just a single toolchain for the host, regardless of how many target platforms we wish to support, was tempting.

Let’s try to do something no other toolchain can do out of the box: cross-compile and link a macOS executable on a Linux machine:

Even though at the end of 2021 Zig was a very novel, unproven technology, the prospect of a single tarball per host platform and ability to cross-compile macOS targets won over the team towards Zig. We collaborated with Zig and started integrating zig cc to our Go Monorepo.

Having a C++ toolchain (in this case, the Zig SDK) is not enough with Bazel: it needs some glue code, a toolchain configuration. In February 2022 the initial support for zig cc in Go Monorepo was added under a configuration flag:

bazel build –config=hermetic-cc <…>

Initially, everything was broken. Most tests could not be built, let alone executed, because C dependencies in our Go code were not hermetic. We started a slow uphill climb of fixing all of these problems. By September 2022 all tests were passing. Since January 2023, the Zig toolchain compiles all of the C and C++ code in Uber’s Go Monorepo for Linux targets.

As Uber has been running binaries produced by Zig since April 2022, we have quite a lot of trust in it. The glue between Bazel and Zig was originally in Adam Bouhenguel’s repository bazel-zig-cc, which was later cloned and further developed by Motiejus Jakštys in his personal repository that was ultimately moved to github.com/uber/hermetic_cc_toolchain.

The collaboration with the Zig Software Foundation allowed us to ask for solutions important to us. Zig folks helped us find and fix issues in both Go (example) and Zig (example). Since the relationship went well in 2021, Uber extended the collaboration for 2023 and 2024. All the work that Zig Software Foundation does is open sourced (until now to Zig or Go), benefiting the wider community. Since the Foundation is a nonprofit, the value of the collaboration will be publicly accessible when the books for 2023Q1 are released.

Once the toolchain was mature enough to compile to arm64, we started cementing arm64 support internally. For example:

Once we were able to compile programs to arm64, we started adopting all the systems that store, download, and execute native binaries. Right now, we have:

Our plans for 2023 include:

Yes and no. For example, at a high level, the launcher in hermetic_cc_toolchain is written by us in Zig. The runtime library (compiler-rt) embedded to nearly every executable, is written in Zig. To sum up, the majority of our Go services have a little bit of Zig in them, and are compiled with a toolchain written in Zig.

Given that, we have not yet introduced any production applications written in Zig into our codebase (where the toolchain is fully set up), because only a few people in the company know the language at the moment.

As of 16th of January, 2023, all C/C++ code in production services built from Go Monorepo are compiled using Zig c++ via hermetic_cc_toolchain. Since Zig is now a critical component of our Go Monorepo, maintenance of hermetic_cc_toolchain will be funded both financially (as of writing, via the collaboration with Zig Software Foundation through the end of 2024) and by Uber employee hours.

While we can run our core infrastructure on arm64 hardware, we are not yet ready to run applications that serve customer traffic. Our next step is to experiment with customer-facing applications on arm64, so that we can measure its performance and decide future direction.

If you like working on the build systems or porting code to different architectures, we are hiring. We will present our work in the upcoming Software You Can Love conference in Vancouver, BC, on June 8, 2023. If you are there and want to chat, either about Uber or about this work, feel free to talk to us in person at the conference.

We’d finally like to thank Loris Cro, Abhinav Gupta, and Gediminas Šimaitis, both for the illustrations and for feedback on the initial drafts of this post.