The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced Apache YuniKorn as a Top-Level Project (TLP).
Apache YuniKorn is a cloud-native, standalone Big Data and Machine Learning resource scheduler for batch jobs and long-running services on large scale distributed systems. The project was originally developed at Cloudera in March 2019, entered the Apache Incubator in January 2020, and graduated as a Top-Level Project in March 2022.
“The Apache YuniKorn community is striving together to solve the resource scheduling problems on the cloud,” said Weiwei Yang, Vice President of Apache YuniKorn. “It’s really great to see the Apache Way shine in the incubating process of YuniKorn. We are lucky to have such an open, collaborative, and diverse community, which is sympathetic and cares about everyone’s success. This motivates us to keep evolving and gets better every day.”
Recommended AI News: SAS Puts Humans at the Center With Responsible Innovation Initiative
Apache YuniKorn natively supports Big Data application workloads and mixed workloads, and provides a unified, cross-platform scheduling experience. Features include:
- Cloud native —runs on-premise and in a variety of public cloud environments; maximizes resource elasticity with better throughput.
- Hierarchical resource queues —efficiently manages cluster resources; provides the ability to control the resource consumption for each tenant.
- Application-aware scheduling —recognizes users, applications, and queues; schedules according to submission order, priority, resource usage, and more.
- Job ordering —built-in robust scheduling capabilities; supports fairness-based cross-queue preemption, hierarchies, pluggable node sorting policies, preemption, and more.
- Central management console —monitors performance across different tenants; one-stop-dashboard tracks resource utilization for managed nodes, clusters, applications and queues.
- Efficiency —reduces resource fragmentation and proactively triggers up-scaling; cloud elasticity lowers overall operational costs.
In addition, the Project has announced the release of Apache YuniKorn v1.0, the fifth update since entering the Apache Incubator. Improvements include:
- Decreased memory and cpu usage
- Extended metrics and diagnostics information
- New deployment model supporting future upgrades
- Technical preview of the plugin deployment mode
Optimized to run Apache Spark on Kubernetes (open source software container orchestration system), Apache YuniKorn’s performance makes it an optional replacement to the Kubernetes default scheduler. Apache YuniKorn excelled in benchmark tests with other schedulers in resource sharing, resource fairness, preemption, gang scheduling, and bin packing categories, with throughput exceeding 610 allocations per second across 2,000 nodes.
YuniKorn is in use at Alibaba, Apple, Cloudera, Lyft, Visa, and Zillow, among others.
Recommended AI News: Supermicro Accelerates AI Workloads, Cloud Gaming, Media Delivery with New Systems Supporting Intel’s Arctic Sound-M and Intel Habana Labs Gaudi2
“We’re thrilled to see this offering come to fruition. Apache YuniKorn powers Apache Spark workloads for Cloudera Data Engineering (CDE), a key Kubernetes-based service supporting the Cloudera Data Platform,” said Vinod Kumar Vavilapalli, Senior Director, Engineering at Cloudera and former PMC chair of Apache Hadoop. “As part of Cloudera’s Public and Private Cloud offerings, Apache YuniKorn adds tremendous flexibility and control when running large-scale analytics, enabling customers to better optimize the performance and value of their deployments.”
“Apache YuniKorn is an essential infra service for bringing Big Data/ML workloads onto the cloud,” said Chunde Ren, Engineering Manager at Alibaba Big Data Open-source team. “YuniKorn brings better scheduling capabilities, performance, elasticity, and usability for running workloads on Kubernetes, especially for Big Data and Machine Learning workloads, which benefits many users on the cloud. It’s a great pleasure for us to have participated in the YuniKorn community since its inception and to see it grow up to be a Top-Level Project.”
“Apache YuniKorn is becoming a popular choice for those who want to run Big Data workloads on Kubernetes, with more use cases developing,” added Yang. “We welcome all who are interested to join the YuniKorn community and work with us on solving these challenging problems.”
Recommended AI News: Copado Brings Affordable Automated Testing to Any Team Using the Salesforce Platform
[To share your insights with us, please write to sghosh@martechseries.com]
Comments are closed.