Improving HDFS I/O Utilization for Efficiency

Scaling our data infrastructure with lower hardware costs while maintaining high performance and service reliability has been no easy feat. To accommodate the exponential growth in both Data Storage and Analytics Compute at Uber, the Data Infrastructure team massively overhauled its approach in scaling the Apache Hadoop^® Data File System (HDFS) by re-architecting the software layer in conjunction with a hardware redesign:

One may have seen the CAP theorem in the headlines of many articles and papers related to distributed systems, and it states: Consistency, Availability, and Partition tolerance: Pick Two!

The next generation of Data Infrastructure applies logic similar to the CAP theorem—that infrastructure can deliver only 2 of the 3 desired characteristics, namely: Cost efficiency, Availability, and Performance. 

Cost efficiency is one of Uber’s top priorities, and HDFS is an essential data service at Uber that must achieve 99.9+% availability. We cannot compromise either of these two characteristics. The question then becomes: do we have to sacrifice HDFS performance (especially IO performance) in exchange for cost efficiency and availability?

In the following blog, we try to analyze current HDFS disk IO utilization to evaluate whether we will hit the “IO wall” when multiple data services are running on our next generation, industry-leading, high-density hardware.

With that question in mind, we turned to usage metrics to analyze IO utilization for all 134,000 hard drives in the HDFS clusters. The data we got was astonishing: 

How are these busy disks distributed among all HDFS hosts: evenly distributed among a large number of the hosts, or concentrated in a small group of hosts? If the answer is the latter, then this may pose a significant issue for the upcoming high-density HDFS servers, which are running multiple services.

To answer this question, we selected the top 10% of the busiest disks (>13,000 disks) and checked how they were distributed among the ~5,600 HDFS hosts. Interestingly, the results showed that about 55% of the busiest drives comprised 10% of the HDFS hosts. 

The data showed that the busiest disks are indeed concentrated in a small group of hosts, rather than distributed among all hosts. This suggested that we should focus our efforts on these most IO-active hosts, since they have a much higher chance of becoming the IO bottleneck as we grow.

Our initial focus was the top 330 busiest hosts that we identified earlier. Further examination showed that these 330 hot Data Nodes resided in 4 out of the total ~20 HDFS clusters:

The data tells us one fact: cluster usage is a major factor of disk IO utilization. Cluster-level IO utilization, especially unbalanced IO traffic, is the top priority that the team should address. 

The Hadoop team took actions to address the issue immediately:

After actions were taken, we once again studied the top 10% of busiest HDFS nodes. We found that the small, busy clusters disappeared. However, the top 10% (or 558) most active hosts were all in the main HDFS cluster, which has over 3,000 Data Nodes. This raised another question: why were they all in one of the largest HDFS clusters? Further study showed that these 558 hosts have one thing in common: they were co-located with the newly added YARN services to fully utilize the hardware resources that were not being used by the HDFS service.

To understand the impact of co-located YARN services on the HDFS hosts, we examined the entire disk IO utilization again, and we compared all disk IO utilization based on the services running on the hosts. The differences were significant: disks taking both HDFS and YARN workloads had much higher IO utilization than those running HDFS only.

At the host level, the aggregated disk IO utilization was more significant: the co-located YARN service brought in much higher IO requests to the HDFS nodes at the per-host level.

Massive disk IO operations from YARN service were not only hurting the IO performance, but they also occupied disk space, generating a higher disk failure rate and adding additional cost to the daily operation of hosts. We recognized these challenges, and decided to add a dedicated SSD in our next generation HDFS servers to handle disk IO requests from YARN service. This will eliminate all the negative impacts introduced by YARN co-location with a fractional cost. Meanwhile, the Spark team has come up with a remote shuffle service that will cut down about 50% of local disk writes, and will roll this out in the next couple of months.

We believe that all excellent engineering wins come from understanding our applications well and shaping the engineering decisions around these use cases. If using your detective skills to determine how to optimize Big Data and make our Data Infrastructure more efficient appeals to you, consider applying for a role on our team!

Apache^®, Apache Hadoop^®, and Hadoop^® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.