WeRide Relies on Alluxio for its Cross-Region Hybrid Cloud Storage Gateway for Machine Learning and AI

Alluxio, the developer of open source data orchestration software for large-scale analytics and AI/ML workloads, announced that WeRide, China’s leading L4 autonomous driving company, is using Alluxio’s Data Orchestration software as a hybrid cloud storage gateway for applications on-premises to access public cloud storage like AWS S3. The new data architecture provides a localized cache per location to eliminate redundant requests to S3. In addition to removing the complexity of manual data synchronization, Alluxio directly serves data to engineers working with the same data in the same office, circumventing transfer costs associated with S3 and improving end-user work efficiency several fold.

To date, WeRide has accumulated more than four million kilometers of autonomous driving mileage and the rate of data collection will only increase as more testing vehicles are in service. In addition to data collected from test drives, applications such as simulation, SIL (Software in the loop) tests, and model benchmarking also produce terabytes of data daily. WeRide is a globally distributed company and data is generated and consumed in parallel by different teams across offices, with AWS S3 as the data lake.

Recommended AI News: Elevatus Surpasses One Million Video Assessments Completed Worldwide

Data Challenges at WeRide

Derek Tan, Executive Director of Infra & Simulation at WeRide, said, “When designing a new algorithm for our self-driving cars or fixing a bug in an existing one, our engineers need to test the algorithm against existing data. Given our data architecture, this caused bottlenecks such as slow iteration in development, high and unnecessary egress costs and error prone data synchronization.”

For example, prior to developing or debugging, developers need to download the latest data from the cloud to their local environment. This is often constrained by download speeds and network bandwidth. Each time data is downloaded from S3, there is a charge on the egress data transfer. Typically to debug one issue, the data transfer cost adds up to $5. This cost is further multiplied if multiple people are collaborating, even though they are downloading the same data. At WeRide, they built a custom data uploading process that copies data to the cloud and retains a local copy stored in NAS or HDFS. The local copy is necessary to give engineers faster access to data but this causes issues with data synchronization. Currently, WeRide maintains the local copies by running a cron job to clean up local data on a regular basis.

A New Architecture Using Alluxio

WeRide decided to explore existing technologies to fulfill their requirements for a solution that is a low or no-cost mature technology that is battle-tested for large scale data access, ready-to-use with easy integration and does not introduce new ETL jobs, and would allow them to scale by utilizing better hardware when their budget allows.

“With the above criteria in mind, Alluxio became a top choice to accelerate our data access,” said Tan. “In addition to being compatible with S3, it provides an easy access interface via its POSIX and HTTP endpoints. As an open source technology, we can incorporate it into our system without adding additional business costs.”

Recommended AI News: Cybersixgill Recognized as the Best Machine Learning Autonomous Solution by the 2021 Tech Ascension Awards

The Alluxio Deployment

In each office, WeRide deployed Alluxio as a small on-premise cluster, using S3 as the source of truth. Road test data is directly uploaded to the local Alluxio cluster, which can be immediately used by the engineers in the same office. Meanwhile, Alluxio automatically uploads the road test data to S3 in the background. As engineers in other offices want to use road test data, they can make a request via their local Alluxio cluster. The data will either be returned immediately if cached by Alluxio or fetched from S3 if not. To further reduce the fetch time of new data from S3, WeRide worked with the Alluxio team to implement a distributed load command which can open multiple simultaneous connections to download data. With Alluxio, application data fetched from the cloud is also cached locally, not previously possible if the data was not uploaded from the same office.

New Improvements with Alluxio

According to Tan, “We experienced many improvements using Alluxio including reducing the complexity of data synchronization by having a single interface to access data and removed the need to maintain a custom locally copy, having an out-of-the-box solution for in-office cache of the cloud data, fast access to data increased engineering productivity, and we reduced S3 data-out cost of downloading redundant data.”

Tan concluded, “WeRide aims at delivering L4 autonomous driving technology for the future. Data access is a critical part of developing smart mobility. Adopting Alluxio as a localized cache layer eliminates redundant requests to S3 while removing the complexity of data synchronization, reducing $5 per issue per engineer in data transfer. We look forward to further collaboration with Alluxio to achieve our data access goal economically.”

“Data orchestration for WeRide via Alluxio is now a critical component of connecting in-office machine learning applications with data in the cloud,” said Haoyuan Li, founder and CEO, Alluxio. “We are excited to further our collaboration with WeRide to add more features relating to data management policies to bring even more value to their engineers.”

Recommended AI News: IBM and Deloitte Launch New AI Offering to Unlock Business Insights in Hybrid Cloud Environments

[To share your insights with us, please write to sghosh@martechseries.com]