Uber Open Source: Catching Up with Celina Ward, M3 Observability Engineer

When Celina Ward joined Uber in March 2018 to work on M3, an open source metrics platform created by Uber’s Observability team, she had no idea that she’d be co-headlining KubeCon just eight months later to talk about her experience. Now, just over a year since she started at the company, she and her co-presenter and Observability colleague, Matt Schallert, are tapped to discuss M3DB, M3’s open source storage database, at O’Reilly OSCON in July 2019.

Celina’s time at Uber has been a whirlwind, thanks in no small part to her open source work. Interested in engineering from an early age, she enjoys the impact and constant learning entailed by programming, particularly as it relates to mastering the ins and outs of Uber’s large-scale Observability infrastructure as part of Uber’s M3 team. Since M3 is open source, her team must assess project development from various perspectives, an additional layer of complexity—and opportunity—that Celina appreciates.

M3 was intended to be open source from early on, so Celina and her team considered the concerns of external stakeholders just as important as internal ones when it came to improving the software.

“Whatever end result you produce, you want other people to contribute to it,” she says of her team’s open source contributions. “You want them to feel like they’re a part of something important.”

On the eve of her appearance at the Uber Open Summit Sofia, we sat down with Celina to discuss her journey to open source, what it was like to present at Kubecon 2018, and what’s next for M3:

How did you first get interested in STEM and engineering?

I first took an interest in technology out of genuine interest. I wasn’t incredibly social as a kid, and all throughout elementary and high school, most of my friends lived in a different neighborhood. If I wanted to hang out with them, I’d have to get my parents to give me a ride and that wasn’t always possible. I could just browse the Web all day or I could do something productive with my time, so that’s how coding fell into my lap.

When I was around nine years old, I used to build websites themed to my personal interests at the time—in 4th grade, this meant coding up pages that highlighted my over-the-top enthusiasm for my favorite TV shows and animals. If I wanted music on one particular website, I would link to 30-second audio samples from pages selling music (usually AOL or some other well-known enterprise) and display the code in a textbox on my personal website so I could distribute them to my Neopets friends. I even made advertisements that displayed banners linking back to my website if users copied and pasted my specially formatted music code, so watching my website traffic grow was really exciting. Programming made the time go by super fast, particularly over the summer, and since I enjoyed it so much, I eventually decided to continue doing it!

How did you decide that software engineering was something you wanted to pursue as a career?

In college, I was writing a lot of essays and getting all of my core requirements down, and after awhile, I became pretty tired of it. I feel like I entered college with more of a freedom-seeking mindset that wanted to explore New York City, so when it finally came down to the wire of selecting a major, I asked myself, “What do I like to do that makes me feel energized and productive? What do I want to do more of?” I had to really think about what I enjoyed, valued, and was most meaningful to me.

Remembering how much gratification coding provided for me growing up, I decided to take my college’s Programming 101 course, and I found that I really enjoyed the homework assignments. After finishing each project, I felt a sense of accomplishment derived from a sense of independence and learning that other activities couldn’t provide.

The more I practiced at programming, the better I got. I loved that I could make incremental updates with software development to see visible progress. As with many skills, you don’t have to be a natural at coding; you can just keep practicing, and then, over time, become good at it.

Why did you decide to join Uber?

Before Uber, I worked at a large investment bank as an engineer. I quickly realized that the projects weren’t developing at the pace I was looking for; I had a lot of energy, so I wanted to be in an environment where my team and I could quickly iterate on new ideas and try out new technologies, even if it led to failure. I moved to a cybersecurity startup and that was super fun. I got to work with really well-known security veterans who hacked the first MacBook in 2007, and even one of the first thousand engineers from Google who was there when they acquired YouTube. Hearing all these stories of technology’s past was super interesting to me, and I was impressed how my colleagues accrued deep industry knowledge through trial and error. These profound accomplishments encouraged me to think about how my generation’s contributions could impact the field.

While I really enjoyed working with the team there, it became apparent that my scope of work was narrowing. Even though I wasn’t necessarily looking for another job, I had a friend working at Uber who suggested I apply. She introduced me to her team, and I really liked everyone I met. They were passionate, talented, and loved learning new things—but they also knew how to have fun and not take life too seriously. What really resonated with me was when they said “on this team, you’ll work on things that have never been done before.” They were right.

Besides the impressive people, Uber’s global scope and scale were compelling reasons for me to work here. If you go anywhere in the world and say you work for Uber, people know what that is. It seemed very intriguing to contribute to a company that was changing the way cities move. After I concluded that moving to Uber for M3 would be in my best interests, I forced myself to interview with other major tech companies too, but Uber really stood out because it wasn’t just another content platform; with cutting-edge technology, it was really changing the way humans think about transportation.

What is most rewarding about the type of work that you do at Uber?

On the surface, having impact at scale at every level of the organization is definitely gratifying. We’re actively changing the way we think about transportation, which is very cool, to say the least.

But also, in the software-specific level of our organization, M3, our metrics platform, and M3DB, our distributed time series database, are open source technologies that the industry hasn’t really seen before, and because of that, they’re extremely gratifying to work on. Time series databases are getting a lot of traction in the software community, and they’re kind of changing the way we think about storage and how we use observability data. Developing M3DB entails a lot of creativity at the software layer—you’re dealing with reliability, compression, and maintainability all at record-scale.

The Uber NYC office as a whole is dedicated to giving back to the STEM community. For instance, we sat on a Girls Who Code panel at Morgan Stanley. After the panel, we got to talk to the high school girls in attendance one-on-one about what they were working on and what it’s like to be a professional engineer. Engaging with the girls one-on-one was super rewarding because we could exchange stories, like how to succeed in the industry and how to make it work for our own personal interests.

What is most challenging about your work as an observability engineer at Uber?

When I first joined Uber’s Observability team, the ramp up to learning our architecture was incredibly steep. There are a couple of reasons for this: the diversity of our microservice-based architecture and our leverage of many open source tools. When the Uber codebase transitioned from a massive monolith to a collection of microservices, it introduced greater extensibility and maintainability, but also many more moving parts—which meant Uber generated a lot more observability data than I was used to. My first few weeks on the team were dedicated to learning about all these different services and how the microservices interact with each other. This architecture was the biggest system I’ve worked on to date.

Since much of our vendored software is open source, I needed to familiarize myself with other open source software platforms as well. Getting all of that knowledge down and then making sense, in just a few days, of why the team chose all of the different components was probably the most challenging part of my new role. Now that I’m more familiar with M3’s relationship to the rest of Uber’s technology stack, many of my job’s challenges involve communicating ideas effectively, agreeing on solutions, and scoping out time-bound tasks—basically just the normal engineering challenges.

How does Uber use M3?

Uber’s engineering organization heavily relies on metrics to effectively monitor the health of its entire stack. M3 is Uber’s distributed, in-house metrics platform. At the core of M3 is M3DB, its time series database. Time series databases store data with timestamps (metrics, in this case), and they are built specifically for measuring change over time. Engineers can instrument their services and applications with metrics and emit them to M3 for durable storage and querying.

High cardinality metrics are becoming increasingly valuable for drilldowns when monitoring components that fall under the same umbrella, but can fail independently of each other. For example, one can setup real-time alerting for their application by collecting p99 response times for all ride requests. With high cardinality, we can tag the metrics according to region, mobile operating system, or modality instead of aggregating all p99 response times into one time series. We also use M3 to track business metrics, so we can see how many drivers are online in a specific city or how many Uber Eats orders have been completed in a specified month.

We can also use M3 to do hardware capacity planning with low-level system metrics that are specific to hosts and their data centers. For instance, you can measure memory usage, container load average, disk read and write counts, and other aspects of hardware-level monitoring.

How do you balance requests from teams at Uber and the broader open source community when making updates to M3?

That’s an interesting question because I think, unlike many other projects, M3’s success is rooted so heavily in meeting the needs of both internal consumers and external ones. This is the first time I’ve worked on a project in the open source community that’s backed by an entire team and not just supported by an individual or two. Our team goals are aligned with making the product better both for our business and users in the broader open source community. A lot of times those requirements overlap.

We’ve found that open sourcing as a continuous strategy is the best way to consolidate all of those requests and come up with the best solution to make the system more robust. Making the strategy a priority is more valuable than just open sourcing some project without any sort of maintenance plan, hoping you’ll get free commits from people who just happen to come across your project. Creating value for other developers is key.

What do you enjoy most about being a member of the open source community?

Contributing to open source makes you feel good about the stuff you’re working on. When you open source your projects, you democratize the product. Everyone has a say in where it should go, and usually for good reason, because people tend to contribute to projects they use and that they’re passionate about. Since Uber targets so many people with our open source projects, it’s not always in our best interest to work in a silo. Also, the product gets a lot more use in several types of production environments than it would internally, so acknowledging all of those pain-points from different users and iterating on new ideas moves at a much faster velocity.

The M3 team really values listening to our customers. We have a chatroom where people can ask questions about using M3 in their own environments. We’re pretty active in helping them get the software up and running, listening to their pain points, and fixing things along the way.

You and your colleague Matt Schallert delivered one of four keynotes at KubeCon 2018 out of 1,600 submitted presentations. What was it like to present at one of the biggest open source conferences in the world?

I had attended KubeCon North America 2017 and was excited by what Kubernetes was doing and how versatile it is. I was using it for my personal projects, and the startup I was working at was also using it to deploy their product to customer environments.

Matt and I submitted our speaking proposal to KubeCon with the intention of delivering a regular talk on using local storage disks at scale (M3DB) on Kubernetes. We didn’t even think we’d get accepted, and then we found out we were chosen for a keynote. It was crazy!

At first, we were nervous, but the experience ended up being extremely positive. We got great feedback from leaders in the open source community about our approach to large-scale automated storage. They really thought our insights were useful. Through our presentation, we were able to give our feedback on what worked for us and what didn’t so that other members of the open source community can ultimately make their products better.

What advice would you give to engineers who are considering open sourcing their work?

Open source communities develop from thought-provoking conversations and stories about technology. The first question anyone should ask themselves before making this decision is, “why is my work interesting to me?” and if it’s not genuinely piquing your own interest, I don’t think anyone else will be interested in it enough to take time out of their day for it, either. At its core, the whole point of working on an open source project is to listen, empathize, and produce impactful work with other people. So, frame it from that perspective.

In terms of personal projects, you should consider why you’re open sourcing the project in the first place. Are you just looking to put something new on your portfolio or are you trying to give something back to the community and incorporate honest feedback? Whatever it is, be ready to talk about it with excitement and motivation. If you want to generally improve the software, open sourcing it might be the right call.

In terms of businesses choosing to open source their projects, there are several different considerations they need to make. Building enterprise software with the intent to open source it versus working on something for a few years and then deciding to open source it are two completely different problem sets. When you develop something with the mindset that the source code will be globally exposed from the start, you systematically isolate the open source code from the company’s proprietary code in such a way that it’s still easy to maintain, deploy, and share with the community. However, when you decide to open source something that initially established itself internally, there’s a lot of work to be done separating the parts that should be open sourced versus kept internal. From this, you still have to ensure that what’s released open source is viable, reliable, and makes sense to its potential users. With both strategies, however, you ultimately want other people to contribute to the project. You want them to feel like they’re a part of something important.

In either case, I think one of the most valuable things to consider is whether or not you are really going to empathize with your users to help them feel included in your project. And ask yourself how much time you’ll be able to communicate with contributors, because ignoring them is an easy way to tank your project.

Outside of your work at Uber, what drives you?

I like to keep hobbies that reap progressive results and lead to continual improvement. For instance, I like going to the gym, specifically for weightlifting, and cultivating plants. I also tend to have hobbies that I can either do alone or with friends because I don’t like to depend on other people too heavily or isolate myself, so I make it a point to balance that out.

I also love traveling and meeting new people. I used to be super shy but I’m getting better at putting myself outside of my comfort zone. This year I took up scuba diving, which I love because it has a technical side, but also lets me meet interesting people and learn more about marine animals. All of my hobbies force me to put myself out there, and, like programming, learn new things.

Interested in learning more about M3 and M3DB? Check out the projects for yourself!

Check out Celina’s upcoming speaking engagements at Uber Open Summit Sofia and O’Reilly OSCON 2019.

Want to work with Celina and other Observability engineers to develop large-scale metrics and storage solutions for Uber’s global operations? Apply for a role on our team!