As part of Uber engineering’s wide efforts to reach profitability, recently our team was focused on reducing cost of compute capacity by improving efficiency. Some of the most impactful work was around GOGC optimization. In this blog we want to share our experience with a highly effective, low-risk, large-scale, semi-automated Go GC tuning mechanism.

Uber’s tech stack is composed of thousands of microservices, backed by a cloud-native, scheduler-based infrastructure. Most of these services are written in Go. Our team, Maps Production Engineering, has previously played an instrumental role in significantly improving the efficiency of multiple Java services by tuning GC. At the beginning of 2021, we explored the possibilities of having a similar impact on Go-based services. We ran several CPU profiles to assess the current state of affairs and we found that GC was the top CPU consumer for a vast majority of mission-critical services. Here is a representation of some CPU profiles where GC (identified by the runtime.scanobject method) is consuming a significant portion of allocated compute resources.

Emboldened by this finding, we commenced to tune GC for the relevant services. To our delight, Go’s GC implementation and the simplicity of tuning allowed us to automate the bulk of the detection and tuning mechanism. We detail our approach and its impact in the following sections.

Go runtime invokes a concurrent garbage collector at periodic intervals unless there is a triggering event before it. The triggering events are based on memory back pressure. Due to this, GC-impacted Go services benefit from more memory, since it reduces the times GC has to run. In addition, we realized that our host-level CPU to memory ratio is 1:5 (1 core : 5 GB RAM), while most Golang services were configured with a 1:1 to 1:2 ratio. Therefore, we were confident that we could leverage using more memory to reduce GC CPU impact. This is a service-agnostic mechanism that can yield a large impact when applied judiciously.

Delving deep into Go’s garbage collection is beyond the scope of this article, but here are the relevant bits for this work: Garbage collection in Go is concurrent and involves analyzing all objects to identify which ones are still reachable. We would call the reachable objects the “live dataset.” Go offers only one knob, GOGC, expressed in percentage of live dataset, to control garbage collection. The GOGC value acts as a multiplier for the dataset. The default value of GOGC is 100%, which means Go runtime will reserve the same amount of memory for new allocations as the live dataset. For instance:

hard_target = live_dataset + live_dataset * (GOGC / 100).

Then the pacer is in charge of predicting when it is the best time to trigger GC to avoid hitting the hard target (soft target).

We identified that a fixed GOGC value-based tuning is not suitable for services at Uber. Some of the challenges are:

The pain points previously presented are the reason for the conception of GOGCTuner. GOGCTuner is a library which simplifies the process of tuning garbage collection for service owners and adds a reliability layer on top of it.

GOGCTuner dynamically computes the correct GOGC value in accordance with the container’s memory limit (or the upper limit from the service owner) and sets it using Go’s runtime API. Following are the specifics of the GOGCTuner library’s features:

We found that we lacked some critical metrics which would give us more insights into garbage collection of each service.

Our initial approach was to have a ticker to run every second to monitor the heap metrics, and then adjust GOGC value accordingly. The disadvantage of this approach is that the overhead starts to become considerable, because in order to read heap metrics Go needs to do a STW (ReadMemStats) and it is somewhat inaccurate, because we can have more than one garbage collection per second.

Luckily we were able to find a good alternative. Go has finalizers (SetFinalizer), which are functions that run when the object is going to be garbage collected. They are mainly useful for cleaning memory in C code or some other resources. We were able to employ a self-referencing finalizer that resets itself on every GC invocation. This allows us to reduce any CPU overhead. For instance:

Calling runtime.SetFinalizer(f, finalizerHandler) inside of finalizerHandler is what allows the handler to run on every GC; it is basically not letting the reference die, since it is not a costly resource to keep alive (it is just a pointer).

After deploying GOGCTuner across a few dozen of our services, we dove deep on a few that showed significant, double-digit improvement in their CPU utilization. Accumulated cost savings from these services alone are around 70K cores. Following are 2 such examples:

The resulting CPU utilization reduction improves p99 latency (and associated SLA, user experience) tactically, and cost of capacity strategically (since services are scaled based on their utilization). 

Garbage collection is one of the most elusive and underestimated performance influencers of an application. Go’s robust GC mechanism and simplified tuning, our diverse, large-scale Go services footprint, and a robust internal platform (Go, compute, observability) collectively allowed us to make such a large-scale impact. We expect to continue improving how we tune GC as the problem space itself is evolving, due to changes in the tech and our competency.

To reiterate what we mentioned at the introduction: there is no one size fits all solution. We feel GC performance will remain variable in cloud-native setup due to the highly variable performance of both public clouds and containerized workloads that run within. Coupled with the fact that a vast majority of CNCF landscape projects that we use are written in Golang (Kubernetes, Prometheus, Jaeger, etc.), this means any large-scale deployment outside could also benefit from such effort.