Skip to content

SCHED-1648: Better support for Block topology - #2567

Merged
dstaroff merged 6 commits into
mainfrom
SCHED-1648/0
Jun 2, 2026
Merged

dstaroff merged 6 commits into
mainfrom
SCHED-1648/0

Conversation

@dstaroff

@dstaroff dstaroff commented Jun 1, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Existing worker-init init container dynamically registers worker topology as if the Slurm cluster is always has TopologyPlugin=topology/tree - it generates worker topology as this: default:<tier0>:<tier1>:<tier2>:<worker>. This doesn't work for block topology, as it expects workers to be part of only one block.

Solution

Provide separate dynamic topology generation based on the topology plugin:

  • For topology/tree - nothing changes
  • For topology/block - generate default:<tier0>:<worker>
  • Other topology plugins are not supported

Additional changes

Support for pattern names in topology.conf

We used to generate a comma-separated list of nodes, which may hit the CM size limit on really big clusters. Now we use patterns which are length-efficient and easy-readable. Sorted order and possible gaps are managed automatically.

Support for topology section of SlurmCluster CR to be set by Helm

It was impossible to set e.g. blockSize via Helm.

Release Notes

Feature: TopologyPlugin=topology/block is supported for dynamic worker topology setting.
Feature: topology.conf uses pattern names for nodes instead of comma-separated list.
Feature: topology section is possible to change via slurm-cluster Helm chart.

@dstaroff dstaroff self-assigned this Jun 1, 2026
@dstaroff dstaroff added bug Something isn't working enhancement New feature or request go Pull requests that update Go code feature fix helm Functional changes in Helm charts labels Jun 1, 2026
@dstaroff
dstaroff marked this pull request as ready for review June 2, 2026 09:29
Comment thread images/worker/worker_init.py Outdated
Comment thread images/worker/worker_init.py Outdated
Comment thread images/worker/worker_init.py Outdated
Comment thread internal/render/worker/container.go Outdated
Comment thread images/worker/worker_init.py Outdated
@dstaroff
dstaroff merged commit 5896221 into main Jun 2, 2026
40 of 41 checks passed
@dstaroff
dstaroff deleted the SCHED-1648/0 branch June 2, 2026 16:12
ali-sattari pushed a commit that referenced this pull request Jun 17, 2026
SCHED-1648: Better support for Block topology
andriishestakov pushed a commit to andriishestakov/soperator that referenced this pull request Jul 8, 2026
SCHED-1648: Better support for Block topology
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request feature fix go Pull requests that update Go code helm Functional changes in Helm charts

3 participants