At Uber Eats, we provide advertising tools for merchants to help them grow their businesses and boost sales. Our ads delivery system seeks to show the most relevant ads to customers in order to maximize returns for merchants. To improve the system, we need to explore various system configurations. For instance, we need to understand: what is the effect on customers when we increase the number of ad placements? How do customers react to different ad ranking algorithms? Will ads perform better if we modified our bidding algorithms?

We typically measure these changes using experiments. However, they come with challenges. While online experiments can yield precise results, they are costly in terms of time and resources, and pose a risk to hurting business performance. On the other hand, offline analysis using observational data is quicker, but biased.

To address the challenges of these traditional methods, we developed an Ads Simulator tool. This simulator replicates the outcomes of various counterfactual scenarios of the ads system and produces reliable results at a fraction of the cost and time of online experimentation. It is flexible, as it can be (simultaneously) applied to multiple changes such as ads placement design, auction mechanisms, and bidding strategies. Additionally, the simulator can emulate extreme cases that would be challenging to execute in the real world.

Furthermore, the simulator improves engineering efficiency. As the simulator shifts much of the exploratory phase offline, the engineering team can focus on implementing the most promising strategies.

To understand the design for an ads simulator, we first explore a stylized version of the current ads auction and bidding flow. This flow is made up of 6 essential steps that work together to drive successful ads campaigns (Figure 1):

Our Ads Simulator meticulously replicates the ads auction and bidding flow, yielding outcomes from potential ad strategies that mirror the dynamics of our actual ads marketplace. Unlike many simulators that assume auction and bid sequences operate independently, ours captures the real-world intricacies, especially the complexities driven by budget pacing. Moreover, instead of depending solely on synthetic data or a restricted advertiser set, our tool leverages the full array of advertisers in a given market, such as an entire city.

The architecture’s epicenter, depicted in Figure 2, features the Core Ads Flow Simulator. This robust engine manages recurring auctions and bids for every advertiser in the market. Accepting inputs from the Simulation Configuration and Auction Context, it processes interactions and produces comprehensive simulation data logs. Furthermore, our Ads Simulator offers an Analysis Package for a thorough assessment of simulation results. The following sections will detail each module. 

Ads Simulator can be tailored to probe diverse ad strategies: from bidding and pacing methods, to auction mechanisms and pricing models. Modifying or adding a few parameters suffices for most customizations, facilitating a straightforward configuration.

The Auction Context contains a list of auctions, along with bidders and user details. The Ads Simulator uses this list to replay auctions in order, simulate results, and adjust future auctions if necessary. This list can come from real auction logs or synthetic data. 

Real auction logs record all ads candidates at the request level, capturing their bids, contexts, and expected engagement rate. This allows us to see and evaluate ads that might have been filtered by later ranking stages. It addresses the common selection bias in big online ads or recommendation systems. Using this data enables more accurate responses from the counterfactual marketplace, considering complete insights rather than just previous outputs. Meanwhile, synthetic data gives the simulator the freedom to test various market conditions.

The core ads flow is designed to accurately replicate the interactions of the entire marketplace in the production environment. It consists of 3 essential components:

The Logging module captures simulation data, including auction outcomes, AdCampaign states, auction paths, marketplace metrics, etc. The Analysis Package provides a wide range of built-in functions and packages for metrics calculation, advanced visualization, and statistical analysis. Both modules are highly extensible, allowing developers to integrate customized logging mechanisms and analysis functions. Scientists and engineers can thus gather rich simulation data and derive valuable insights into ads strategies and marketplace dynamics.

The Ads Simulator offers valuable insights by accurately replicating the dynamics of the ads marketplace under new ads strategies. Its accuracy and reliability are demonstrated by the high degree of alignment between its simulation results and real-world experimentation results. Engineers and scientists at Uber should be confident to leverage the simulator to make informed decisions, optimize campaign performance, and discover new avenues to innovation.

In this section, we present two use cases that highlight the simulator’s accuracy and capabilities. Both use cases have been validated by online experimentation. They also demonstrate how the simulator provides actionable insights that drive business success.

A use case of the Ads Simulator is to simulate different reserve price designs. Traditionally, testing various reserve prices involves a time-consuming trial-and-error process with real public traffic, which may result in less ideal business outcomes. 

Figure 4 presents the simulation results for searching for the optimal reserve price design. The simulation illustrates the complexity of the task, highlighting that a traditional online experimentation search would be lengthy. The associated implementation cost can be significant with material impact on ads revenue and advertiser outcomes. 

Notably, the experimentation results for the chosen design align closely with the simulated results. These changes are so significant that they can rarely be extrapolated using offline analysis, relying on assumptions from historical data. By using this simulator, we prevent the high costs and unreliable results associated with traditional methods, enabling the team to carefully evaluate different designs and select the most effective strategy—all without risking real-world revenue or performance.

Another use case is to help the platform identify the optimal geographical regions for allocating campaign budgets. By determining these optimal regions, we can better guide advertisers on where they might get the most impact from their campaigns. It’s crucial for the marketplace to find the best match between customers and ads to provide a personalized customer experience. A focus on the right regions not only maximizes an advertiser’s campaign effectiveness but also ensures that customers are presented with more personalized ads tailored to their preferences and location. Consequently, when allocating or suggesting increased budgets, we need to ensure that we’re recommending the most effective regions to the advertisers. To do this, we need to understand the outcome of increasing budgets in different geographical regions. 

If we are to address this question through experimentation, we need to introduce random variations across various geographical regions. The standard options are impractical: modifying the actual budgets for all campaigns in each region would necessitate unscalable coordination with merchants; providing add-ons on top of the budget would be too expensive for the platform; and relying on budget wiggling experimentation is not a sustainable way of gathering results. 

On the contrary, extrapolating impact from observational data runs the risk of selection bias. Additionally, if increasing budgets in a geographical location leads to a change in market equilibrium, observational data would mask this effect. 

The Ads Simulator overcomes such limitations and generates reliable data for assessing the potential outcomes of budget scaling strategies. To validate the use of the simulator, we ran an online experiment that scales the budget for each geographical location along with a corresponding simulation that scales the budget by the same ratio. Figure 5 shows the resulting simulated Ads Spend effect and the experimental effect, with each point representing one geographic region. They appear to be well aligned.

We’ve detailed both the challenges and opportunities associated with more efficient and economical evaluation methods for Uber Eats Ads, leading to the introduction of the Ads Simulator. This tool has been seamlessly integrated into the development cycles of our scientists and engineers. It has proven instrumental in guiding and expediting product development, thereby driving our business growth.

Despite its success, there are areas for future improvements. For instance, we can extend the data collection framework by incorporating advertisers which are filtered out during the system’s early retrieval stages. This would broaden the simulator’s applicability to additional ad strategies, such as targeting. Ultimately, refining the user response simulator module—with improved calibration and sampling methods—will ensure that our simulations provide even more precise and reliable results.

Special thanks to Ludovica Rizzo and Solomon Barkley for their contribution on enhancing the simulator. Also thanks to Brian Lee and Bo Lin for providing data insights and results on our two successful use cases. The gratitude also goes to Ashish Gupta, Yuriy Bondaruk, and everyone on the Ads Delivery Science and Engineering Team for their insightful feedback and unwavering support. I am deeply grateful to James Lee, Erjie Ang, Guillaume De Roo, Jun Yan, and Ziwei Liu for their continuous support and meticulous review and editing. The success of the ideas in this blog post is a testament to the collective efforts of these remarkable individuals and teams.