Revolutionizing Money Movements at Scale with Strong Data Consistency

Uber as a platform invites its users to leverage it, earn from it, and be delighted by it. Serving more than 18 million requests per day, in 10,000+ cities, has enabled people to move freely and to think broadly while earning a livelihood on it. As one of the underlying engines, Uber Money fulfills some of the most important aspects of people’s engagement in the Uber experience. A system like this should not only be robust, but should also be highly available with zero-tolerance to downtime, after our success mantra: “To collect and disburse on-time, accurately and in-compliance”.

While we expand to multiple lines of businesses, and strategize the next best, the engineers in Uber Money also thrive on building the next generation’s Payments Platform which extends Uber’s growth. In this blog, we introduce you to this platform and provide insights into our learnings. This includes migrating hundreds of millions customers between two asynchronous systems while maintaining data-consistency with a goal of zero impact on our users. 

Introduction and motivation

Gulfstream, Uber’s fifth generation collection and disbursement payment platform, is our latest. It is a single, integrated, SOX-compliant system built on the principles of double-entry accounting and  can  reconcile itself. We discuss some short-comings in the older model which we fixed in the new model in this article.

The legacy system had two internal systems. One provided collections from riders and eaters and the other provided disbursements to restaurants and partner-drivers. This had a lot of shortcomings such as no holistic view of end-to-end money movements. It also slowed the process to build more generic features such as Cash Trips, where Uber needed to collect commissions from its driver partners and so on. So we wanted to build a role-agnostic system which could collect and disburse money from any user. This enables faster onboarding for multiple lines of businesses. 

Advantages of the new system and architecture

Job/Order based system

Transaction-based systems are hard to extend for running balances and accounting for user entities. Tracking and enforcing the zero-sum principle is difficult. 

Our new architecture now uses Job/Order based systems. Each Job represents a ridesharing trip or eats/food delivery. There could be multiple orders that belong to the same Job due to adjustments, incentives, tips, etc. Each order contains multiple order entries and each order entry represents an amount of money moving in or out of a user’s account. Together it represents the money movement from a payer’s account and to a payee’s account. The sum of all the entries is zero (the system cannot create or destroy money). It flows from one account to another. The money movement, order-based system creates an analogy to the real world double-entry  accounting system.

This table illustrates a simple example of a ridesharing trip with a $20 total fare including a $2 service fee and a $18 trip fare. The sum of all the order entries is zero.

We decouple creating an order and processing it by leveraging message queueing system:

Highly available and active in all regions

 How do we achieve idempotency?

Data consistency between asynchronous platforms 

Due to the complexity of the legacy payment system and the scale of Uber’s user base as well as payment data, it took multiple years for us to migrate to the new payment platform. During the migration, we need to maintain both platforms and high data consistency across them. In order to achieve that, we persist every transactional change to the user’s payment accounts in entity change log so that our system serializes writebacks by the entity change log’s version number per user. We dual-write each transaction in the legacy system with a field including the version number. This way, the writeback is never out of order and the end result is always consistent even if we have multiple concurrent adjustments of the same job.

Migration and writebacks 

With our experience undertaking migration initiatives at Uber, we learned to: 

Dashboards and metrics

Before we could put the new system into production, we added various different metrics. This includes tracking counts, results, latency, and observability-based metrics for each flow. We set up various alerts for the production versus shadow traffic. This helped us  track business metrics on the system. In addition, we set up various dashboards to validate our service. We could also use these dashboards to understand how many successful business events we performed per active user and how many anomalies we detected between different systems. Our oncall engineer and rollout in-charge engineer tracked the dashboards on a daily basis. 

Smart rollout strategy

We designed the rollouts to migrate the system in multi-step fashion. We divided the rollouts broadly into: 

Sequential writebacks 

We update the payer and payee accounts when we process each order. Our services produce an EntityChangeLog to reflect the sequence of the account changes. Each changeLog entry has a version number and we increment the number per user. The service uses the version number to enforce the sequence of the writeback of orders.

The writeback service consumes EntityChangeLog events from the Messaging Queueing System. If it consumes an event that is not in sequence, our processing logic identifies the version mismatch and we retry the event a number of times. If it still fails, we publish a reconciliation event to another Messaging Queueing System topic. The service that consumes the reconciliation of the writeback service ingests the events. It checks the version number against what we recorded last in the legacy system. We fetch all the orders with versions in the gap from OrderStore and write them back to the legacy system, one after another. 

Validations and retries  

Our pre-rollout phase had various validation strategies setup in our system:

During the rollout-phase, we started expanding to more and more countries, which exponentially increased the load on all the services. We found an important feature missing between the legacy system and the new system. Specifically we had to build a service to consume reconciliations replay orders in the version gap we discussed above in Sequential writebacks.

Along with the validations, we also set up replay request APIs so that we could re-run the data to collect real-time analysis and debug issues faster. Since we built the system to be idempotency, this helped us recover from service-based failures faster and to record data quickly. 

Since this required manual interventions, we quickly built an ingester for retry queues. This enabled delayed replay mechanisms that could run basic validations before replaying orders. This helped to prevent loading our queues with bad data.

Lessons learnt 

Migrations are a reality for any company scaling quickly and trying to expand  the spectrum of products that they design. This complex project had multiple aspects, namely:

We believe what we learned can help engineers across the globe trying to deal with problems at  scale. And therefore we want to share some of the key concepts:

Acknowledgements

The project was a 2 year cross-team effort with more than 40 engineers, product managers and operations teams involved throughout the journey. Here, we are taking this opportunity to acknowledge all the engineers from the core Gulfstream team who invested heavily in the successful completion of the project. 

Conclusion and what’s next

We went through a journey of extensive architecture design, implementation and well-thought through rollouts and active monitoring. And we successfully launched the  platform with negligible  downtime in all countries. We onboarded new lines of businesses, namely Uber Freight, NEMO, seamlessly and efficiently.

As part of our future projects, the team is already thinking about truly turning the system into a platform.  This will reduce the engineering effort for newer use-case onboarding and migrate us away from payee-, payer-, and LOB- specific models to revolutionize the payments at Uber.

We hope these practices and architecture design prove useful for other engineers and teams performing migrations at scale.