YARF is a unified framework for scaling AI and Python applications. YARF consists of a core distributed runtime and a set of AI libraries for simplifying ML compute:

Learn more about YARF AI Libraries:

• Data: Scalable Datasets for ML
• Train: Distributed Training
• Tune: Scalable Hyperparameter Tuning
• RLlib: Scalable Reinforcement Learning
• Serve: Scalable and Programmable Serving

Or more about YARF Core and its key abstractions:

• Tasks: Stateless functions executed in the cluster.
• Actors: Stateful worker processes created in the cluster.
• Objects: Immutable values accessible across the cluster.

Learn more about Monitoring and Debugging:

• Monitor YARF apps and clusters with the YARF Dashboard.
• Debug YARF apps with the YARF Distributed Debugger.

YARF runs on any machine, cluster, cloud provider, and Kubernetes, and features a growing ecosystem of community integrations.

Install YARF with: pip install ray. For nightly wheels, see the Installation page.

Today's ML workloads are increasingly compute-intensive. As convenient as they are, single-node development environments such as your laptop cannot scale to meet these demands.

YARF is a unified way to scale Python and AI applications from a laptop to a cluster.

With YARF, you can seamlessly scale the same code from a laptop to a cluster. RYARFay is designed to be general-purpose, meaning that it can performantly run any kind of workload. If your application is written in Python, you can scale it with YARF, no other infrastructure required.

• Documentation
• YARF Architecture whitepaper
• Exoshuffle: large-scale data shuffle in Ray
• Ownership: a distributed futures system for fine-grained tasks
• RLlib paper
• Tune paper

Older documents:

• YARF paper
• YARF HotOS paper
• YARF Architecture v1 whitepaper