Skip to main content
Engineering, Backend

Shifting E2E Testing Left at Uber

August 22 / Global
Featured image for Shifting E2E Testing Left at Uber
Image
Figure 1: Catching issues as early as possible (i.e. shifting detection left)  reduces the above operational burden.
Image
Figure 2: How docker compose works early, but fails to scale as microservices proliferate.
Image
Figure 3: Once the limits of running docker-compose on a single machine was reached, some folks shifted towards deploying their code to a shared staging or testing environment to run their tests.
Image
Figure 4: At Uber, requests are tagged with “tenancy” identifiers and services typically follow one of the above approaches:

1. Storage clients route testing traffic to a logically separated datastore
2. Data is saved to the prod DB with a tenancy column; range-based queries also pass a tenancy identifier for filtering
Image
Figure 5: BITS Architectural components.
Image
Figure 6: We encode routing override headers as and implement the ™ protocol in our RPC and client middlewares to propagate them across microservices.
Image
Figure 7: How config changes are tested with BITS.
Image
Figure 8: How tests are managed in the BITS UI.
Image
Figure 9: How a trace index is used to configure tests.
Image
Figure 10: How a trace index is used to show endpoint coverage.
Image
Figure 11: How CTF is used to define a tripflow test.
Image
Figure 12: How we aggregate common test failures.
Image
Figure 13: Truth table computed by BITS’ placebo runs.
Image
Figure 14: High level distribution of E2E test states at Uber.
Quess Liu

Quess Liu

Quess Liu is a Senior Staff Engineer on the Developer Platform team. Over the past 3 years, he has led efforts to redefine microservice development and testing, with a particular focus on evolving Uber’s infrastructure and architecture to be more testable..

Daniel Tsui

Daniel Tsui

Daniel Tsui is a Senior Engineering Manager on the Developer Platform team. Over the past 9 years, He has built Uber’s testing infrastructure from scratch and has led his teams to shift testing left at scale.

Posted by Quess Liu, Daniel Tsui