Mediation Modeling at Uber: Understanding Why Product Changes Work (and Don’t Work)

At Uber Labs, our mission is to leverage insights and methodologies from behavioral science to help product and marketing teams improve the customer experience. Recently, we introduced mediation modeling, a statistical approach from academic research, to address user pain points.

Mediation modeling goes beyond simple cause and effect relationships in an attempt to understand what underlying mechanisms led to a result. Using this type of analysis, we can fine-tune product changes and develop new ones that focus on the underlying mechanisms behind successful features on the Uber platform.

At Uber, we have a strong culture of improvement, frequently conducting experiments to test whether one variable impacts another to ensure reliable, safe, and seamless user experiences.

Most of the time we have a hypothesis about why two variables are related. For example, suppose we believe that a promotion to give new riders trip discounts on their first few trips may improve new rider retention. While a standard analysis would tell us whether this promotion helped increase new rider retention, it would not tell us why. For example, do new riders return to the app after their first several trips specifically because of reduced trip fares? Or could it be that the promotion helped new riders familiarize themselves with the app, or something else entirely? If multiple mechanisms are present, which one plays a larger role and by how much? In a standard analysis, an underlying mechanism (i.e., why something happened) is often assumed to exist but not empirically tested with data.

While we may have some evidence that two variables are related, we may not have a clear understanding of why they are related, and when we do not understand why, we have to rely on trial and error. However, just like in academic research, knowing why is equally important for Uber because it helps us build better products for our users. For instance, in the example above, if we found that their familiarity with the app kept new riders on the platform, we should then prioritize product changes that encourage riders to use the app.

Mediation modeling opens the black box between a treatment and an outcome variable to reveal the underlying mechanisms, i.e. why something happened. Although widely used in academic research, this approach is under-utilized in business.^{1, 2, 3} When we have a causal assumption, instead of leaving it at that or relying on correlational evidence, mediation modeling lets us empirically test (vs. logically infer) the causal pathways between the two variables. More importantly, understanding these mechanisms enables us to develop better products quickly and more efficiently because it helps us pinpoint which features of these changes are responsible for making products successful.

So, what exactly can we do with mediation modeling to improve the user experience? We outline some sample hypothetical use cases in Figure 1, below:

First, we can use mediation modeling to test product assumptions. For example, we may believe that a new rider promotion could increase retention because of reduced trip fares (Figure 1a). With mediation modeling, we can empirically test this assumption. Our results of these test results can tell us whether our assumption is true, and if so, how much of the effect is because of reduced trip fares as opposed to other mechanisms (e.g., familiarity with the app).

Second, we can use mediation modeling to compare multiple mechanisms. In a hypothetical example, we may believe that a new design of Uber Eats menus can increase orders through more than one mechanism (Figure 1b). With mediation modeling, we are able to estimate how much of the treatment effect (i.e., increased orders) is accounted for by each of these mechanisms and which mechanism plays the largest role. The results help inform how we design our products and their future iterations.

Mediation modeling also allows us to connect intangible variables, such as consumer sentiment, with a specific feature to business metrics. As we know, customer feelings and satisfaction are critical to the success of a business. However, it is often difficult to quantify their business impact. Mediation modeling, however, enables us to test how these variables affect downstream business metrics (e.g., rider referrals, as depicted in Figure 1c).

Moreover, mediation modeling can be a creative way to break a long-term goal into smaller intermediate steps. For example, suppose our goal is to increase long-term rider satisfaction with Uber. How do we break this goal into smaller pieces that can be tied to our day-to-day work? If we have previously identified a key mediator behind rider satisfaction, we can then leverage this mediator as a short-term key performance indicator (KPI) (Figure 1d). If most of the effect of an intervention is mediated through a particular mechanism, then influencing that key mediator can be a necessary (although not sufficient) condition for the intervention to work.

Across various use cases, we identify upstream and downstream variables and test how they are connected with each other. Next, we take this explanation a step further and discuss the conceptual details behind mediation modeling.

To successfully execute this technique,  what exactly are the quantities we estimate and with what methodology? To answer these questions, we depict the simplest possible mediation model in Figure 2, below:

From this type of model, our goal is to estimate three key quantities:

In recent years, researchers have begun to understand mediation modeling in causal terms.^{4, 5, 6} This has allowed them to conceptualize mediation using formal frameworks developed for causal inference, such as the potential outcomes approach developed by Neyman, Rubin, and others.⁷

To better grasp how this framework functions, suppose we have an outcome Y and treatment assignment t, such that t is either 1 if an individual is in the treatment group or 0 if an individual is in the control group. Then, the outcome Y under treatment assignment t for individual i is Yi(t).

Now, we are often interested in estimating the difference in Y between the treatment and control, which for individual i is

Yi(1)−Yi(0)

However, most of the time we only observe one of these two outcomes because an individual is usually in just one experimental condition. So, for example if individual i is in the treatment condition (t=1), then outcome Yi(0) for that individual is just a potential outcome, i.e., an outcome that could have occurred but did not actually occur.

Since we usually cannot observe the treatment effects at an individual level, we estimate the group-level average treatment effect, which is defined as

E[Yi(1)−Yi(0)]

Here, the outcome Y under the treatment and the control assignment is estimated from the group-level quantities.

What does all this have to do with mediation modeling? Well, it turns out we can represent the three key mediation quantities using the above framework. Let M(t) correspond to the potential mediator value under treatment assignment t. Then, we can start defining the key mediation quantities as follows:

ATE=E[Yi(1,Mi(1))−Yi(0,Mi(0))]

In short, ATE is the difference between the potential outcomes under the treatment and the control assignment when the mediator changes as it actually does with the treatment assignment. This solves for the average effect of the treatment on the outcome.

As mentioned earlier, the goal of mediation analysis is to decompose the total treatment effect into two parts: the average direct effect (ADE) and the average causal mediated effect (ACME). In other words, ADE is the impact from the treatment on the outcome that does not go through the mediator. So, if you fix the value of the mediator while varying the value of the treatment status, then you will generate the direct effect. Using the potential outcomes notation, we have

ADE=E[Yi(1,Mi(t))−Yi(0,Mi(t))]

for t=0,1. So, the ADE can be understood as any additional contribution of the treatment assignment on the outcome once we prevent the mediator from changing with the treatment status.⁸ (This prevention does not happen literally. We use  statistical models to estimate what the outcome would have been if the mediator were fixed.)

Once we have established the ADE, it is clear that the ACME is simply its complement:

ACME=E[Yi(t,Mi(1))−Yi(t,Mi(0))]

for t=0,1. ACME corresponds to the difference in potential outcomes that would occur if we were to flip the mediator into the value it would take under the treatment status while holding the treatment status itself fixed.

One of the great things about these definitions is that they do not make any reference to a particular model. Consequently, researchers like Imai et. al.⁸ have developed algorithms that estimate the key mediation quantities using any valid model. This means, among other things, that we are free to estimate the relationships in the mediation graph using nonparametric and nonlinear models, a considerable advancement compared to the past.^{9, 10, 11, 12} For example, in a traditional approach such as Hayes’ PROCESS method, the mediators cannot be categorical variables and the outcome variables are restricted to only those that can be properly modeled with ordinary least squares or logistic regression¹³. The limitations on mediators and outcome variables imposed by the PROCESS method do not let us model discrete data, which makes it unsuitable for our work at Uber.

Finally, the potential outcomes framework proves helpful in laying out the identification assumptions for mediation effects. In the context of randomized experiments, the major assumption is that the mediator should be statistically independent of the potential outcomes of Y conditional on the observed treatment status and the values of the pretreatment covariates included in the model.^{14, 15} The reason why the mediation quantities are not consistently estimated without this assumption is that it would be otherwise possible for some third variable to confound the mediator-outcome relationship.

Even though the potential outcomes framework helps us to define this assumption clearly, it is  not possible to conclusively verify that the assumption holds. The best we can do is to (1) include any pretreatment covariates that theoretical considerations suggest could de-confound the relationship between the mediator and the outcome and (2) conduct sensitivity analyses to see how our estimates would change if our assumptions were not satisfied to different degrees.¹⁶ Luckily, as a data-driven technology company, we generally have a good set of pretreatment covariates that we can use in our models to mitigate the risk of confounding.

Now, let us take a look at a concrete example of how we applied mediation modeling at Uber through the lens of a recent analysis with our Customer Obsession team. We wanted to understand if adding a graph showing driver earnings led to fewer support tickets. To help users file less support tickets, we needed to understand why they were filing tickets in the first place.

In a previous experiment, the Customer Obsession team found that adding a graph displaying a driver-partner’s weekly earnings significantly increased their understanding of their earnings. Figure 3, below, shows the UI for the treatment and the control group:

In this analysis, earnings understanding was measured using a single-item survey that asked drivers how well they understood their earnings information on a five-point scale.

We wanted to know if earnings understanding was a significant behavioral mechanism behind earnings-related support tickets. In other words, we wanted to understand whether it was the case that the treatment improved earnings understanding, which in turn reduced support tickets. If that is the case, we would also like to estimate how much of the treatment effect was mediated through the path of earnings understanding (as opposed to other mechanisms).

To answer these questions, we used mediation modeling with the intervention as the independent variable, earnings understanding as the mediator, and earnings-related support tickets as the outcome variable (Figure 4). In addition, we included drivers’ pre-experiment support tickets and pre-experiment earnings experience (e.g., previous earnings, lifetime trips, and active days) as control variables. We finished the analysis by running a sensitivity analysis to determine how robust our results were against unmeasured mediator-outcome confounders.

The mediation modeling results showed that earnings understanding was indeed a significant mechanism behind earnings-related support tickets and it accounted for about 19 percent of the total treatment effect. Based on the results, we better understood how earnings understanding is an important path to reduce customer support tickets.

With these insights, we can better design products and communications that improve driver understanding of their earnings , thereby making the user experience more seamless and enjoyable. At the same time, we also recognized that there was a large opportunity for our team to further leverage models to uncover the mechanisms that mediate the remaining 81 percent of the effect. To accomplish this, we are partnering with the Customer Obsession team to test more of the remaining behavioral mechanisms. These results will inform the driver earnings product roadmap.

The future of mediation modeling is exciting: in recent years, academic researchers have started to investigate the prospect of conducting automatic mediator selection using high-dimensional data.^{17, 18} In these applications, we can specify potentially hundreds of mediators that are temporally between a treatment and an outcome. We then test whether there is a statistically reliable association between the treatment and each candidate mediator or between each candidate mediator and the outcome variable. With these approaches, we can potentially unpack the black box of the treatment-outcome relationship in unprecedented detail.

At Uber, we look forward to testing these data-driven algorithms and seeing how they compare with traditional theory-driven approaches through which researchers specifies the mediators before they conduct their research. We see mediation modeling as another interesting area where using machine learning methods to analyze experiments holds great potential.

If you are interested in leveraging applied behavioral science to power products and cities, learn about job opportunities with Uber Labs.

References: