[core] Avoid FabricManager stall on NVLink systems in GpuProfilingManager

[aschuh-hf]

Fixes #63243

On NVLink/NVSwitch nodes (e.g. A100 SXM4), bare nvidia-smi triggers a blocking RPC to the FabricManager daemon. If another process (e.g. dynolog) holds the NVML lock, this stalls 15–20 s—long enough to exceed the raylet's dashboard agent startup timeout and prevent ray.init() from succeeding.

Changes

• node_has_gpus(): replace bare nvidia-smi with nvidia-smi --query-gpu=name --format=csv,noheader, which queries NVML directly (~0.1 s) without touching the FabricManager.

# Before
subprocess.check_output(["nvidia-smi"], stderr=subprocess.DEVNULL)

# After
subprocess.check_output(
    ["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"],
    stderr=subprocess.DEVNULL,
)

• enabled property: move binary-presence checks before node_has_gpus() so the GPU detection call is skipped entirely on nodes without dynolog installed.

# Before: node_has_gpus() always called first
return self.node_has_gpus() and self._dynolog_bin is not None and self._dyno_bin is not None

# After: short-circuits before the GPU check when dynolog is absent
return self._dynolog_bin is not None and self._dyno_bin is not None and self.node_has_gpus()

• Tests: added test_node_has_gpus_uses_query_gpu_flag (asserts exact nvidia-smi flags) and test_enabled_does_not_call_node_has_gpus_when_dynolog_missing (asserts short-circuit behavior).

[gemini-code-assist]

Code Review

This pull request optimizes GPU detection in the GpuProfilingManager by reordering the enabled property to short-circuit the GPU check when required binaries are missing and updating the nvidia-smi command with specific query flags to avoid stalls. Review feedback suggests adding a timeout to the subprocess.check_output call for better robustness against system hangs. Additionally, it was noted that the class constructor might still trigger the GPU check unconditionally, which could interfere with the intended optimization and cause test failures.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 9df1a13. Configure here.}

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[aschuh-hf]

I believe this PR is ready and can be merged.

[edoakes]

why is it that these parameters skip the fabricmanager?

make sure to leave a comment here indicating why the "magic parameters" were chosen

[aschuh-hf]

Claude Code worked on debugging this issue on our on-premise GPU server. It measured runtime and looked at various system logs [including strace] to determine that these parameters by-pass the blocking communication with the fabric manager. I myself ran "time nvidia-smi" with and without these arguments, and the time difference was massive without any downsides.

[aschuh-hf]

The root cause analysis is documented in the linked issue #63243.

See also: https://share.google/aimode/cS0kazViPLaAK3fH9

[edoakes]

we probably still want this log line, or it might even make sense to reverse these (if there are no GPUs, then why even check for dynolog?)

is there a dependency between the dynolog_bin and node_has_gpus? or are you just trying to skip the nvidia-smi call?

[aschuh-hf]

The idea is that when the user has dynolog not installed, the GpuProfilingManager is anyway disabled, regardless if there is a GPU available. The check if the node has a GPU is what may cause problems. The check if dynolog is installed does not. Hence the switched order.

When a user does not need the GpuProfilingManager, they can simply use a container image without this dependency installed and thus completely bypass the init overhead of the GpuProfilingManager. When they did install and set up dynolog, only then we may assume they care about using the GpuProfilingManager.

Would you prefer that a separate warning alerts the user that dynolog_bin was not found, and that this is the primary reason why the GPU profiling is disabled?

[aschuh-hf]

Actually, the warning that dynolog is not available is already shown. So the user will always get a reason why GPU profiling is disabled:

1. dynolog must be installed and found
2. A CUDA device must be available

[edoakes]

Thanks for the contribution @aschuh-hf