As Uber’s architecture has grown to encompass thousands of interdependent microservices, we need to test our mission-critical components at max load in order to preserve reliability. Accurate load testing allows us to validate if a set of services are working at peak usage and optimal efficiency while retaining reliability.

Load testing those services within a short time frame comes with its unique set of challenges. Most of these load tests historically involved writing, running, and supervising tests manually. Moreover, the degree to which tests accurately represent production traffic patterns gradually decreases over time as traffic organically evolves, imposing a long-term maintenance burden. The scope of the load testing effort continuously increases as the number of services grows, incurring a hidden cost to adding new features.

With this in mind, we developed Ballast, an adaptive load test framework that leverages traffic capture using Berkeley Packet Filter (BPF) and replays the traffic using a PID Controller mechanism to adjust the number of requests per second (RPS) to each service. Ballast removes the toil of writing, running, and supervising load tests, improves load test coverage, and performs continuous load testing, providing insight into service capacity and improving deployment safety on an always-on basis.

In the following sections, we will describe the design of Ballast and how this powerful framework for load testing has freed us from the associated toil.

At a high level, Ballast consists of 6 major components: 

Ballast’s default load generator is called Shadower.

Shadower uses a coordinator/worker architecture. The coordinator is in charge of querying all the load tests in the scheduled state and finding a suitable worker to run them.

The coordinator is the public-facing component that uses leader election and it’s horizontally scaled. From any coordinator you can:

The worker is in charge of executing the load tests. It has the following features:

A command-line tool that provides encoding/decoding for all the available encodings we use at Uber (JSON, Thrift, etc). It has the following features:

These payloads are later uploaded to a persistent store for the worker to be able to read from them.

Example:

maps to →

The traffic capture component reads packets off the wire and assembles them into a valid request payload. It leverages the Berkeley packet filter (BPF) provided by package pcap to capture a specific service’s payload. The BPF is a technology used in certain computer operating systems for programs that need to analyze network traffic. 

It supports 3 protocols: HTTP 1.1, HTTP 2.0, and TChannel. TChannel is a networking framing protocol built at Uber for general RPC, supporting out-of-order responses with extremely high performance, allowing intermediaries to make forwarding decisions quickly.

This component can capture the packets for these protocols off the wire, assemble them into a valid service request, and be ready for the load generator reads.

This component is built on top of the Uber Metrics platform – M3 and provides the framework with the ability to retrieve the 4 golden signals (latency, traffic, errors, and saturation) for the load-tested service. During the load test, Ballast monitors the service’s health and availability by calling Golden Signals. 

After Ballast starts the load test, the Ballast PID controller continuously calculates an error value as the difference between the desired service load test state (goal state provided as r(t)) and a measurement of the service golden signals (y(t)) and applies a correction on the load generator target throughput based on proportional (P), integral (I), and derivative (D) terms. The following is how we define PID controller in Ballast:

Defining u(t) as the controller output, the final form of the PID algorithm is:

Where: 

K_p is the proportional gain, a tuning parameter,

K_i is the integral gain, a tuning parameter,

K_d is the derivative gain, a tuning parameter,

e(t) = r(t) – y(t) is the error (r(t) is the setpoint (goal state), and y(t) is the feedback value):

t is the time or instantaneous time (i.e., present),

τ is the variable of integration (takes on values from time 0 to the present t)

Using Ballast domain language:

r(t):  service golden signals SLO, including target CPU usage (e.g., 80%), request latency (e.g., 600ms), error rate (e.g., 0.1%) 

y(t): measured service golden signals in real-time

See an example of the Ballast load test controlled by the PID controller for our map search (P:4, I:0.2, D:1):

Shadower RPS ramped up quickly at the beginning but slowed down when its CPU usage was approaching the target CPU usage.

Ballast can run continuously without human intervention. The canary deployment is used to detect and prevent bugs before rolling it out globally. We enabled Ballast in the canary deployment for our services so that we always know our services’ capacity limit. With Ballast, it becomes simple math to prepare the proper capacity for our anticipated holiday peak: estimated holiday peak throughput divided by Ballast load test peak RPS per instance.

Ballast is used for improving canary deployment defect detection and reducing outages in production. We have enabled Ballast’s always-on feature to continuously run load tests in a service’s canary deployment. When a developer deploys new code in canary, the before/after metrics are immediately visible. For example, we know there’s been a performance degradation when:

Ballast is used for testing the behavior of a load shedder used by the Uber Eats backend. To ensure that only the lowest-priority requests are dropped under heavy load, Ballast can inject different combinations of requests to simulate various production traffic scenarios.

Ballast is used for replaying production high error rate traffic in staging to narrow down the failed requests. Developers can isolate the triggered code paths by analyzing the failed requests to efficiently debug the issues in production.

As we continue onboarding more services and use cases with Ballast, we are observing the emergence of an ecosystem of payload encoders, PID tuners, next generation of observability-based supervisors, chaos engineering scenarios, and more. Collectively these are analogous to operator architecture, with the Ballast controller running the reconciliation loop. Ballast lays the groundwork for the future of our resiliency testing, and we’re excited about its potential as our reliability platform. We hope this article is helpful to you!