Faster Together: Uber Engineering’s iOS Monorepo

Over the past few years, Uber has experienced a period of hypergrowth, expanding to service over 550 cities worldwide. To keep up, our mobile team also had to grow. In 2014, we had just over a dozen mobile engineers working on our iOS apps. Today, our mobile team numbers in the hundreds. As a result, our mobile tooling has undergone major changes to meet the demand of this larger and also more dynamic team.

At the @Scale conference last September we showcased how Uber Engineering has grown since those early days. Here, we will focus more deeply on why we eventually had to switch to a single mobile monolithic repository (monorepo)—at first glance, seemingly in contrast to our monolithic migration to a microservice infrastructure architecture—and how this move has changed mobile engineering at Uber for the better.

Before the Monorepo

Let’s take a step back and examine the state of Uber’s iOS development world before the monorepo. On iOS, we relied solely on an open source tool named CocoaPods to construct our applications. CocoaPods is a package manager, dependency resolver, and integration tool all in one. It allows developers to quickly integrate other libraries into their applications without needing to configure complex project settings in Xcode. CocoaPods also resolves dependencies by targeting cycles in dependency graphs that can lead to issues at compile time. Since it is the most popular package manager for Cocoa, many third-party frameworks which we depend on were only available via CocoaPods. Moreover, CocoaPods’ popularity meant that issues with the tool were typically fixed quickly, or acknowledged swiftly by its team of contributors.

While our team and our projects were small, CocoaPods served us well. For Uber’s first few years, we built most of our applications with a single shared library and a number of open source libraries. The dependency graphs were usually small, so CocoaPods could resolve our dependencies very quickly and with minimal pain for our developers. It allowed us to focus on building the product.

Modularizing the Codebase

By late 2014, the Uber Engineering shared library had become a dumping ground for code that lacked any concrete organization. Because everything was potentially just one import away, our shared library had the effect of allowing seemingly independent components and classes to depend on each other. At that point, we had also amassed several years of mission-critical code with no unit tests, and an API that had not aged well. We knew we could do better, so we embarked on an effort to modularize our codebase and simultaneously rewrite the core parts of our apps (such as Networking, Analytics, and others).

To do this, we componentized all the critical parts of our apps into building blocks that any Uber app could use. We called these frameworks ‘modules’ and each module was housed in its own repository. This modularization allowed us to spin up quick prototype apps when needed, as well as bootstrap real production software. For instance, when we decided to spin UberEATS off into its own separate application in late 2015, the UberEATS team heavily leveraged the new modules we built. Engineers were able to spend most of their time working on the product, rather than platform requirements. For example, we built a UI toolkit that implemented the typography, color scheme, and common UI elements that designers across the company used in our mobile apps.

At the start of 2015, we had five modules. As of early 2017, we now have over forty, ranging from framework-level libraries (such as networking and logging) to product-specific libraries (such as mapping and payments), shared between applications. All three apps (Rider, Driver, and EATS) rely on this shared infrastructure. Bug fixes and performance improvements in our modules are immediately reflected in the applications that use them—a huge benefit of this setup. Overall, our modularization effort was a major success.

But as we scaled from five to over forty modules, we ran into some problems. We realized that we had a hard time scaling with Cocoapods due to the amount of modules we had and their interdependencies. At the same time, more than 150 engineers had joined our wider iOS team, which meant that our applications and modules were in a state of constant evolution.

Time for a Change

As the company continued to grow, our engineering needs began to change. Once we started integrating many more libraries into our applications, we quickly hit the limits of CocoaPods’ dependency resolution capabilities. At first, our pod install times were under ten seconds. Then, they numbered in the minutes. Our dependency graph was so complex that engineers were collectively burning hours each day waiting for CocoaPods to resolve the graph and integrate our modules. This time-waste was even worse on our Continuous Integration infrastructure.

We also felt the strain of a multiple-repository approach. Each module was housed in its own repository, and any given module could depend on many other modules. Therefore, a change inside one module required updating the Podfile of the application before the change could appear. Large breaking changes required updating all modules that depended on the one where the change was made.

Since we needed to version all of these modules in order to integrate them, we adopted the semantic versioning convention for tagging our modules. While semantic versioning itself is a simple concept, what constitutes a breaking change in reality could vary with different compiler settings.

As a result, a seemingly innocuous code change in a given module could introduce errors (or warnings, which we treat as errors on CI) in other dependent modules or applications. As an example, consider the following snippet of code (some boilerplate removed for brevity):

public enum KittenType {
   case regular
   case munchkin
}
public protocol KittenProtocol {
   public var type: KittenType { get }
   public var name: String { get set }
}
public struct Kitten: KittenProtocol { }
public protocol Kittens {
   var kittens: Set<Kitten> { get }
   func contains(aKitten ofType: KittenType) -> Bool
}

At first glance, it might seem like adding a new property to the KittenProtocol shouldn’t raise an error outside your module. But it might, depending on the access control level of that protocol. Anyone could conform to the protocol if we made it publicly accessible outside of our module (let’s call this module ‘UberKittens’ because that seems apt). Adding a new property, then, would be a breaking change because properties in a protocol must be implemented in their conforming class or struct.

Even adding a new case to the KittenType enumeration constitutes a breaking change in this setup. Again, since we made this enum public, any existing switch statements that use it will produce a compiler error for the new, missing case not being handled.

The above issues are minimal and can easily be resolved by any consumer of the UberKittens module. But the only way to make these changes safe in a semantic versioning world is to make the patch contain a major version bump. We have hundreds of engineers and hundreds of changes happening on any given day. It’s impossible to catch every potential breaking change. Plus, updating any libraries that your library depends on could introduce dozens of warnings in your code that you would have to resolve.

We really wanted our engineers to be able to move fast and to make the changes they needed to make without having to worry about version numbers. Resolving conflicting versions was frustrating for developers and, as mentioned earlier, CocoaPods had become very slow to resolve our now complex dependencies. We also didn’t want engineers spending days updating modules in the dependency graph in order to see their changes in the applications we ship. Engineers should be able to make any and all changes they need to, in as few commits as possible.

The solution? A monolithic repository.

Planning the Monorepo

Of course, monolithic repositories are not a new idea; many other large tech companies have adopted them with great success. While putting all your code into one repository can have its downsides (VCS performance, breakages that affect all targets, etc.), the upsides can be huge, depending on the development workflow. With a monorepo, our engineers could make breaking changes which spanned across modules atomically, in just one commit. With no version numbers to worry about, resolving our dependency graph would be much simpler. Since we are a company with hundreds of engineers in many disparate teams, we could centralize all our iOS code in one place, making it easier to discover.

We couldn’t pass up these benefits, so we knew we needed a monorepo. However, we weren’t sure what tooling we would need to handle it. When we first began our modularization effort, we considered building a monorepo around CocoaPods. But that would have meant having to build every application and every module for every code change an engineer might put up for code review. We wanted to be smarter about building only what changed, but that would have required investing thousands of engineering hours into a tool that could intelligently rebuild only the parts that changed.

Luckily, a tool came along that can do this (and more), and it’s called Buck.

Buck Saves the Day

Buck is a build tool built for monolithic repositories that can build code, run unit tests, and distribute build artifacts across machines so that other developers can spend less time compiling old code and more time writing new code. Buck was built for repositories with small, reusable modules and it leverages the fact that all the code is in one place to intelligently analyze the changes made and build only what’s new. Since it is built for speed, it also takes advantage of the multi-core CPUs our engineers have in their laptops, so that multiple modules can be built at the same time. Buck can even run unit test targets at the same time!

We’d heard a lot of good things about Buck, but couldn’t utilize it until it publicly supported iOS and Objective-C projects. When Facebook announced Buck for iOS support during the 2015 @Scale conference, we were excited to start testing Buck against our applications.

Our initial tests showed that using Buck could greatly improve our build and test times on CI. Ordinarily, when using xcodebuild, the best approach is to always clean before building and/or testing. Any given CI host could be building commits that are backwards (and forwards) in the commit history, which means that the cache will constantly be in flux. Because of this, the xcodebuild cache can be unstable (and our CI stability is a top priority). But if you have to clean before you build, CI jobs will be unnecessarily slow, since you can’t incrementally build only new changes. So our build times skyrocketed along with our growth. With hundreds of engineers, the collective hours lost waiting for CI to build code changes numbered in the thousands every day.

Buck solves this problem with a reliable (and optionally, distributed) cache. It aggressively caches built artifacts and will also build on as many cores as are available to it. When a target is built, it won’t be rebuilt until the code in that target (or one of the targets it depends on) changes. That means you can set up a repository where the tooling will intelligently determine what needs to be rebuilt and retested, while caching everything else.

Our CI machines benefit greatly from this cache architecture. Today, when an engineer puts up a code change that requires a rebuild, those build artifacts are distributed in future builds on that machine and others. Engineers can save even more time by using the artifacts already built on CI locally for their builds too. We recently open sourced an implementation of Buck’s HTTP Cache API for teams to use.

Buck offers other benefits, too. We can remove a common cause of merge conflicts and developer frustration by using Buck to generate our Xcode project files. This allows every iOS application at Uber to share a common set of project settings. Engineers can easily code review any changes to these settings since they are in easy to read configuration files instead of buried deep in an Xcode project file.

Furthermore, since Buck is a fully fledged tool with support for building and testing, our engineers can check the validity of their code without ever loading Xcode. Tests can be run using one command, which runs the tests in xctool. Even better, if our engineers want to forgo Xcode altogether, they can open Nuclide, which adds debugging support and auto-completion to the Atom text editor.

The Big Mobile Migration

How did we migrate to a monorepo? The answer is: with a lot of dry runs. Much of the work was repeatable and deterministic, so we wrote scripts to do the heavy lifting for us. For example, all  our modules consisted of a CocoaPods podspec file. We published this podspec to an internal, private podspecs repository that CocoaPods used when integrating. Those podspecs could be mapped 1:1 to a corresponding Buck file (named ‘BUCK’), so we wrote a script which generated the BUCK file and replaced the podspec.

We also created a dummy monorepo solely for testing purposes which mimicked the proposed repo structure using symbolic links. This allowed us to easily test the monorepo structure and setup, as well as update the modules as they changed.

However, we noticed that changes in modules quickly made the test repository out of date. We had landed BUCK files in all of the modules, but in the meantime our engineers still had to use the podspec files. So, we needed a way to keep the Buck files and podspecs in sync. To do this, we bootstrapped the test repository on all module changes in CI and signaled to the engineer when their change broke any code or module in the test repository.

This setup helped acquaint engineers with the new ‘Buck world’ that was coming while also keeping the BUCK files up to date. In the final week before the migration, we did this for the applications too so that engineers would know if their code patch broke the applications in the Buck universe.

Constructing the actual monorepo was a challenging endeavor. We knew that we would eventually have to create it, but the question was, when? Originally, we planned on constructing it the same weekend we were going to move all the tooling and infrastructure to the new setup. But if we could do it weeks earlier, it would save us added stress later on. The migration was also something we could script, since the steps were reproducible for all repositories. The general steps we followed to create the monorepo were:

Once we created the monorepo, we had to figure out how to keep it up to date over the coming weeks. We did this by utilizing the file we created in step 7. Since it represented the HEAD sha when the monorepo was last updated from the original repository, we could run a script every hour which would create a git patch from that last update sha to HEAD, and then apply the patch in the monorepo at the correct path.

The migration itself was pretty simple, and we managed to do it over a single weekend. We temporarily blocked all our repositories at the git layer, switched all our CI jobs to use Buck commands instead of xcodebuild, and deleted all our Xcode project files and podspecs. Since we had spent the past few months testing the projects, our CI jobs and our release pipeline, we felt confident when we launched the monorepo in May 2016.

The Results: Centralizing All iOS Code

With the monorepo, we centralized all our iOS code into one place. We organized our repository into this directory structure:

├── apps
│   ├── iphone-driver
│   ├── iphone-eats
│   ├── iphone-rider
├── libraries
│   ├── analytics
│   ├── …
│   └── utilities
└── vendor
   ├── fbsnapshottestcase
   ├── …
   └── ocmock

Now, a breaking API change in the analytics module would require updating all consumers of that module in the same commit. Buck will only build and test analytics and any modules that depend on analytics. Conversely, we no longer have any version conflicts, and we always know that master is green.

The biggest benefits however, came from switching to the Buck cache:

Let builders build: we bucked the trend of previously longer build times under xcodebuild.

Comparing the xcodebuild build and buck build steps, we saw a huge win since we were no longer cleaning our cache before every CI job. By switching to a tool with a trustworthy cache, we can share built artifacts between CI hosts too. Buck can also cache test results, so if we know a module has already been tested and is not affected by a certain code change, we can skip those tests.

Overall, our migration was a success. However, there were some costs with moving to a monorepo that we initially couldn’t avoid, such as decreased git performance and the reality that when master breaks, all code is affected.

For example, we started to see that perfectly valid commits that passed on CI would fail when rebased on top of the master branch. When we first launched the monorepo, we had to vigilantly revert failing commits at all hours of the day. On the worst days, 10% of commits would have to be reverted, which resulted in hours of lost developer time. To resolve this, we introduced a system, called Submit Queue, between the repository merge and push operations (which we call ‘land’ at Uber, since we use Arcanist). When an engineer attempts to land their commit, it gets enqueued on the Submit Queue. This system takes one commit at a time, rebases it against master, builds the code and runs the unit tests. If nothing breaks, it then gets merged into master. With Submit Queue in place, our master success rate jumped to 99%.

Conclusions

Moving to a new toolchain provided us with the foundation to experiment with new ideas to build our applications faster and better. Recently, our team has been working with Facebook’s Buck team to add full Swift support to Buck, as well as support for macOS and iOS Dynamic Frameworks. This has allowed us to utilize the benefits of Buck while we move our codebase into the Swift era.

Moving our code into one place and our apps to Buck were just a few steps on our path to modernizing how we do development. Alongside us, our Android team has also adopted Buck, and they too have seen the benefits of fast, reproducible builds. But we’re not stopping here; our team is committed to adding more features to Buck’s iOS support over the coming year, and we also plan to ramp up our Open Source efforts to contribute back to the wider mobile community.

Want to join our team? We’re hiring!

Photo Header Credit: “Mobile reedbuck calf running solo” by Conor Myhrvold, South Luangwa National Park, Zambia.