The scheduler config is a YAML document with up to three top-level keys, all optional. An empty config is valid: an org with no rules passes every workload through. Unknown keys are rejected at every level. The config is org-scoped and versioned: applying creates a new active version.

Structure​

resource_flavors​

A list of named compute categories, matched to workloads by label selectors. A workload is assigned a flavor per resource group: within a group, the group's flavors are tried in listed order, and the workload gets the first flavor whose selector matches its labels and that has quota available. If no flavor matches, the workload is rejected.

resource_queues​

A list of named pools that hold workloads and cap their resource usage with quotas. A queue that omits resource_groups is a catch-all queue: every workload routed to it is admitted with no quota enforcement.

preemption​

Controls when a queue evicts running workloads to free quota. Preemption ranks workloads by the priority a rule's priority_policy assigns, where higher is more important. A workload at priority 10 is lower priority than one at priority 20, so it's preempted first.

resource_groups​

Each entry groups a set of resource axes governed together, with per-flavor quotas. Omit resource_groups from a queue to make it a catch-all queue (admit everything, no quota).

Each flavors entry:

Each resources entry (one quota for one resource of one flavor):

scheduling_rules​

An ordered list of rules that route a workload to a queue and set its priority. Rules are evaluated top to bottom, and the first match wins. A rule whose priority_policy rejects falls through to the next rule. A workload matching no rule is failed.

priority_policy​

Bounds and default for the priority assigned to matching workloads. Priority is an integer from 0 to 1000, and higher priority is more important (drives queue order and preemption).

Match expressions​

A Kubernetes-style label match, used by both selector on flavors and selector on scheduling rules. Each is written as { key, operator, values }.

All expressions in a single list must match (logical AND).

Selector keys​

A selector (in both resource_flavors and scheduling_rules) matches against a workload's labels. A workload's labels are the custom tags you set on the job, service, or workspace, plus the reserved keys the platform always stamps onto the workload. Keys are matched exactly and are case-sensitive. Reserved keys are hyphenated.

The following reserved keys are always present, and you can't override them with your own tags:

Any custom tag key you set on the workload is also selectable, as long as it doesn't collide with a reserved key. Don't prefix custom tags with anyscale- or anyscale.com/. Anyscale reserves those prefixes for system tags.

Validation​

apply validates your config and returns an error if anything is wrong: unknown keys (rejected at every level), invalid values, and the per-field constraints listed above. It also enforces these cross-field rules:

• Flavor names are unique across resource_flavors, and queue names are unique across resource_queues.
• A queue's flavor name must reference a defined flavor, and a rule's resource_queue must reference a defined queue.
• covered_resources allows only gpu, cpu, memory_gb, and tpu. Covered sets are disjoint across a queue's groups, so each resource belongs to one group, with no duplicates within a group.
• A flavor name appears at most once per resource group, and at most once across a queue's groups.
• A flavor's resources is a subset of its group's covered_resources (no resource outside it, names unique within the flavor), and each nominal_quota is 0 or greater with at most three decimal places. An omitted covered resource is unlimited, as is a listed one with no nominal_quota. A nominal_quota of 0 blocks the resource.
• priority_policy satisfies min <= default <= max.
• Selector values are required for in or not_in and must be empty for exists or does_not_exist.

Complete example​

# Named compute categories, matched to workloads by label. Order matters within a
# group: flavors are tried in listed order.
resource_flavors:
  - name: gpu_a100_spot
    selector:
      - { key: accelerator-type, operator: in, values: [A100] }
      - { key: market-type, operator: in, values: [SPOT] }
  - name: research_gpu_pool
    selector:
      - { key: team, operator: in, values: [research] }   # custom workload tag

resource_queues:
  - name: ml_training
    preemption:
      within_resource_queue: lower_priority
    resource_groups:
      - covered_resources: [gpu, cpu]
        flavors:
          - name: gpu_a100_spot
            resources:
              - { name: gpu, nominal_quota: 64 }
              - { name: cpu, nominal_quota: 512 }
          - name: research_gpu_pool
            resources:
              - { name: gpu, nominal_quota: 16 }
              - { name: cpu, nominal_quota: 128 }

  - name: default        # catch-all queue: no resource_groups -> admit with no quota

scheduling_rules:
  - selector:
      - { key: team, operator: in, values: [research] }
    resource_queue: ml_training
    priority_policy: { default: 50, min: 0, max: 100, on_violation: reject }
  - resource_queue: default     # catch-all: no selector -> matches everything above

Example: cap a specific instance type​

Limit a particular instance type to a fixed number of instances, while letting everything else run uncapped. Define a flavor for just that instance type and give it a quota, then route everything else to a catch-all queue, which admits without quota.

resource_flavors:
  - name: m5-2xlarge
    selector:
      - { key: instance-type, operator: in, values: [m5.2xlarge] }   # scopes the cap to m5.2xlarge

resource_queues:
  - name: q-m5-2xlarge-cap3
    preemption: { within_resource_queue: never }
    resource_groups:
      - covered_resources: [cpu, memory_gb]
        flavors:
          - name: m5-2xlarge                          # capped: 3 instances
            resources:
              - { name: cpu, nominal_quota: 24 }       # 3 x 8 vCPU
              - { name: memory_gb, nominal_quota: 96 } # 3 x 32 GiB
  - name: default        # catch-all queue for everything else

scheduling_rules:
  - selector:
      - { key: instance-type, operator: in, values: [m5.2xlarge] }
    resource_queue: q-m5-2xlarge-cap3
    priority_policy: { default: 50, min: 0, max: 100, on_violation: force_update }
  # catch-all rule (no selector) -> catch-all queue. MUST be last; first match wins.
  - resource_queue: default

Only m5.2xlarge is capped, to three instances' worth, so at most three run at once. Every other workload matches the catch-all rule and is routed to the default catch-all queue, which admits it without quota.

Example: use a capacity reservation​

Route reservation-eligible workloads to a specific cloud capacity reservation and cap how much of it they consume. This example pins a4-highgpu-8g instances to a Google Cloud reservation. A flavor carries the reservation binding in advanced_instance_config, and a queue quota caps reserved usage. Opt a workload in by tagging it use-reservation=true.

resource_flavors:
  # a4-highgpu-8g pinned to a specific Google Cloud capacity reservation.
  - name: a4_reserved
    selector:
      - { key: instance-type, operator: in, values: [a4-highgpu-8g] }
    advanced_instance_config:
      instance_properties:
        reservation_affinity:
          key: compute.googleapis.com/reservation-name
          consume_reservation_type: SPECIFIC_RESERVATION
          values: [my-a4-reservation]   # replace with your reservation name

resource_queues:
  # Workloads tagged use-reservation=true land here and are capped to the reservation.
  - name: reservation
    resource_groups:
      - covered_resources: [gpu]
        flavors:
          - name: a4_reserved
            resources:
              # 8 GPUs/node x 10 nodes = 80. Caps reserved usage to ~10 a4-highgpu-8g.
              - { name: gpu, nominal_quota: 80 }
  # Everything else. No resource_groups => catch-all queue: admits everything, no quota.
  - name: default

scheduling_rules:
  - selector:
      - { key: use-reservation, operator: in, values: ["true"] }
    resource_queue: reservation
  - resource_queue: default   # no selector = matches all; MUST be last (first match wins)

A workload tagged use-reservation=true routes to the reservation queue, where the a4_reserved flavor binds it to the named Google Cloud reservation and the gpu quota of 80 caps usage at roughly ten a4-highgpu-8g nodes. Everything else falls through to the default catch-all queue and runs uncapped.

The reservation queue defines only the a4_reserved flavor, so a use-reservation=true workload that isn't an a4-highgpu-8g matches no flavor in the queue and fails with "no flavor matched" rather than falling through uncapped.