Skip to content

SCHED-2442: Fix local NVMe disks in 4.1 - #2942

Merged
ali-sattari merged 1 commit into
soperator-release-4.1from
SCHED-2442/fix-nvme-start-stop
Sep 8, 2026
Merged

ali-sattari merged 1 commit into
soperator-release-4.1from
SCHED-2442/fix-nvme-start-stop

Conversation

@ali-sattari

@ali-sattari ali-sattari commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The NVMe health check scans all devices rather than validating the node-local mount used by workloads, and it cannot automatically return recovered nodes to service.

Solution

  • Resolve the RAID array from the configured mount point.
  • Run the check inside the jail where the node-local mount is visible.
  • Validate RAID state, NVMe membership, read/write access, and recent kernel errors.
  • Drain unhealthy nodes and automatically undrain them after recovery.

⚠️ if NVME_RAID_MOUNT_POINT env is not provided in command part of config for the script it will be skipped with no action.

Testing

  • CI and slurm-cluster Helm unit tests pass.

Release Notes

Fix: Improve local NVMe RAID health detection and automatically return recovered nodes to service.

@ali-sattari
ali-sattari changed the base branch from main to soperator-release-4.1 September 8, 2026 08:46
@ali-sattari ali-sattari added the bug Something isn't working label Sep 8, 2026
@ali-sattari
ali-sattari force-pushed the SCHED-2442/fix-nvme-start-stop branch from 06c1741 to 4df5572 Compare September 8, 2026 09:59
@ali-sattari
ali-sattari force-pushed the SCHED-2442/fix-nvme-start-stop branch from 4df5572 to e7e1a4b Compare September 8, 2026 10:49
@ali-sattari
ali-sattari marked this pull request as ready for review September 8, 2026 13:00
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-08T13:05:31.494766Z e7e1a4b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7e1a4b0ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread helm/slurm-cluster/slurm_scripts/nvme_raid_health.sh
@ali-sattari
ali-sattari merged commit 686907a into soperator-release-4.1 Sep 8, 2026
21 checks passed
@ali-sattari
ali-sattari deleted the SCHED-2442/fix-nvme-start-stop branch September 8, 2026 15:08
github-actions Bot added a commit that referenced this pull request Sep 8, 2026
…t-stop"

This reverts commit 686907a, reversing
changes made to 65404d5.
ali-sattari added a commit that referenced this pull request Sep 9, 2026
…t-stop"

This reverts commit 686907a, reversing
changes made to 65404d5.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

2 participants