Eats Safety Team On-Call Overview

Introduction

Our engineers have the responsibility of ensuring a consistent and positive experience for our riders, drivers, eaters, and delivery/restaurant partners.

Ensuring such an experience requires reliable systems: our apps have to work when anyone needs them. A major component of reliability is having engineers on call to deal with problems immediately as they arise. We set up our on-call engineers for success through training, tools, and processes.

In this article we will provide an overview of how we at the Eats Safety team ensure that our engineers are fully equipped to provide prompt, high-quality service—anywhere in the world, 24/7.

The Eats Safety Team

Created in late 2019, the Eats Safety team is based in the São Paulo Tech Center. We are responsible for building safety features and products for the UberEats marketplace.

The team is in a constant state of action: implementing and deploying systems, and combining different types of technology at massive scale. Mobile applications, cloud-based services, and web applications all need to interoperate effectively to achieve our desired business goals.

We have a lot of autonomy to build and deploy our own services and features. At the same time, we are responsible for ensuring that everything we built is performing as intended in production, and for that we have an on-call rotation.

On-Call Process

Roles

Being on call means being available for a period of time to respond to alerts with the appropriate level of urgency. In addition to mitigating incidents, on-call engineers also invest their time in calibrating alerts, updating runbooks, improving dashboards, automating operational processes, and providing support to other teams. On-call engineers often fall into 3 different roles, with the following responsibilities:

Primary: 

Secondary:

Nobody should be on-call alone. The secondary on-call is available to support the primary whenever needed.

Shadow:

The safest and most effective way to onboard new on-call engineers includes rigorous shadow training, which provides an “in the trenches” experience. We require that every new engineer on the team shadow another more experienced engineer before entering the rotation. A new on-call engineer will be reverse shadowed by an experienced engineer as well.

Alert Notification Policy

Alerts are triggered by monitoring platforms and depending how critical an alert is, the on-call engineers can be notified in different ways.

Critical Alerts:

Non-Critical Alerts:

If the primary on-call doesn’t acknowledge a critical alert within ten minutes, the alert will be escalated to the secondary in the current rotation.

Rotation

The on-call rotation is the schedule used to map who will be on call, and when. Every rotation should have at least a primary and a secondary. Some teams might also have a tertiary to cover the secondary, if necessary.

Defining an on-call rotation involves answering questions such as:

The answer can vary a lot according to the characteristics of each team, such as:

Example Setup

Eats Safety is a cross-functional team, composed of 4 mobile and 5 backend engineers.

Traditionally, different functional groups have their own rotation, even when they’re part of the same product team. In our case, that would mean having one on-call rotation for mobile and another for backend. However, assuming that we have a weekly rotation, it would cause a mobile pair to go on call every 2 weeks and we wanted:

Because of that, we decided to experiment with a merged mobile and backend on-call. It allowed us to reduce how often any given engineer goes on-call, and made it possible to draw from the expertise of both specialties at each shift, alternating between primary and secondary.

Here is what we learned from having a merged on-call rotation:

Before On-Call – The Preparation Checklist

The preparation checklist consists of steps an on-call engineer should take to be fully prepared for an upcoming on-call shift, such as:

After On-Call – The Handoff Meeting

The handoff meeting happens at the end of a shift. In this meeting, the whole context of the previous shift is passed to the engineers entering the new shift, including:

The reports based on alert annotation are a great source of insights for future alert improvements and runbook updates.

The participants in this meeting vary from one team to another. It can be the whole team, or just the engineers directly involved in the relevant shifts.

Monitoring

In order to know whether or not our services are performing as intended in production, they need to be monitored. Monitoring refers to our ability to quickly detect and understand problems, via system metrics or testing.

Monitoring can be grouped in 2 general categories:

Monitoring allows:

Good Alerts

Alerts are notifications sent to the on-call engineers that report a problem with a service/product. The type of notification will depend on how critical an alert is.

Here are our guidelines for what makes an alert good:

To guarantee the on-call load and quality, we track all alerts by annotating them, and then generating reports so we can understand what needs to be improved. 

Our Monitoring Tools

uMonitor is an alerting system and front end for bringing together all the different observability metrics, with the goal of enabling engineers to maintain visibility on their systems, detect issues as they arise, and quickly fix them.

Blackbox is designed to monitor the availability and accuracy of flows that are exposed externally to our customers, by calling the same APIs used for our mobile applications. You can read more in Observability at Scale: Building Uber’s Alerting Ecosystem.

On-Call Quality Metrics

We use an internal platform developed by the Production Engineering team, called On-Call Dashboard. It offers a seamless experience for our on-call engineers by compiling everything necessary to respond to an alert and recording a history of both alerts and the actions taken in response. This data gives us high level insight into alert trends which can be used to improve our software development and on-call response practices.

Annotating an alert comprises keeping track of the following information:

Metrics

We derive the following metrics from the alerts annotations, which help us understand the overall quality of the on-call shift:

Alerts: Alert count during the shift.

Annotations: The fraction of annotated alerts over total alerts. A high annotation rate indicates more complete information about the shift.

Signal-to-Noise Ratio Accurate: The fraction of alerts that represents a real problem over the total number of alerts. Noisy alerts should be calibrated in future on-call shifts.

Signal-to-Noise Ratio Actionable: The fraction of alerts that required an action to be taken by the on-call engineer or other teammate to resolve the alert.

Orphaned Alerts: Alerts that are still open by the end of the shift, but have not been annotated, indicating poor on-call engagement.

Poor Runbooks: Every alert should have an associated runbook. This metric represents the runbooks lack of accuracy based on a quality score assigned by on-call engineers on the shifts.

Disturbance: This indicates on-call action distribution along the shift. High disturbance score indicates a heavy shift with several incidents or noisy alerts.

Mean Time To Acknowledge Alert (MTTA): The average time it takes from when an alert is triggered to when it is acknowledged by the on-call engineer.

Mean Time To Resolution (MTTR): The average length of time needed to resolve alerts.

These metrics help engineers analyze and prioritize action items that should be addressed during the next on-call shift. Some of the insights we can take from these metrics are:

Training

As mentioned throughout the article, our on-call process involves tools, metrics, definition of roles, alert notification policies, etc. 

We conduct regular training sessions covering on-call and incident management as part of Engucation (Engineering + Education) for new engineers. Having a structured education process is important for ramping up new engineers, building confidence, and maintaining process standardization across teams.

During Engucation sessions, besides theory, new engineers also have hands-on experience that involves the first contact with many of the tools that we use on a daily basis.

Prior to joining an on-call rotation, all engineers should attend this training session, review their team’s runbooks, and complete one or more shadow rotations.

Final Considerations

The on-call process requires constant improvement. At Eats Safety we are continually evaluating and discussing what is working and what can be improved as part of our basic process during the shifts.

The standards presented here that we use at Eats Safety represent decades of collective experience and best practices from teams across the company. By teaching and following these processes, we hope to reduce chaos, maintain organization in the face of uncertainty, and improve outcomes for both engineers and customers.

We are hiring at the São Paulo Tech Center. Click here to see our positions.