Performance

Network Topology

TensorWave GPU clusters are designed around two complementary networking optimizations that directly affect multi-GPU workload performance.

Rail-optimized networking. Each GPU has a dedicated RDMA NIC, giving it an exclusive high-bandwidth path for collective communications. With 8 GPUs per node, there are 8 independent rails (rdma0-rdma7), one per GPU. This means RCCL ring and tree algorithms can saturate all available bandwidth simultaneously without GPUs competing for shared NIC resources.

Topology-aware scheduling. Nodes are organized into physical pods, each sharing a top-of-rack network fabric. Slurm's tree topology plugin uses a topology.conf that maps nodes to pods and pods to a spine, allowing the scheduler to preferentially allocate nodes within the same pod for a given job. For multi-node jobs, this reduces cross-switch hops and keeps the majority of collective traffic on the lower-latency intra-pod fabric.

RCCL All-Reduce Test

RCCL (ROCm Collective Communications Library) tests are pre-installed on compute nodes at /opt/rccl-tests/. The all_reduce_perf benchmark measures collective communication bandwidth across GPUs and is useful for validating interconnect performance and identifying nodes with degraded network throughput. A sample sbatch job driving all_reduce_perf is provided at /opt/tw/examples/libexec/rccl.sbatch.

Running the test

Create an output directory and submit the job:

sbatch /opt/tw/examples/libexec/rccl.sbatch

**rccl.sbatch:**

#!/bin/bash
#SBATCH --job-name=rccl_tests
#SBATCH --output=jid-%j.name-%x.log
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --gpus-per-node=8
#SBATCH --time=01:00:00
#SBATCH --nodes=2

set -euxo pipefail

# Use 2 InfiniBand queue pairs per connection between ranks
export NCCL_IB_QPS_PER_CONNECTION=2

# Double buffer size for NCCL communications
export NCCL_BUFFSIZE=8388608

# Prevent MPI from using InfiniBand
export UCX_NET_DEVICES=eno0

srun /opt/rccl-tests/all_reduce_perf -b 512M -e 8G -f 2 -g 1

To run on more nodes, override the --nodes value at submission time:

Script parameters

Environment variables

RCCL test arguments

Reading the output

A successful run completes without errors and shows increasing bus bandwidth as message size grows. Key fields in the output:

Results on an 4-node MI355X Cluster

• **algbw** — algorithm bandwidth: message size divided by time. Reflects how quickly one collective operation completes.

• **busbw** — bus bandwidth: algbw corrected for the number of ranks. Better reflects peak hardware utilization.

• **#wrong** — should be 0. Any non-zero value indicates a data correctness error.

• **Avg bus bandwidth** — average busbw across all message sizes. Useful as a single summary figure for comparison.

A healthy cluster shows busbw increasing steadily with message size and leveling off at a stable peak at larger sizes. Nodes with degraded interconnect will show lower busbw or fail to complete. Target busbw values will be dependent on your cluster architecture.

Training Benchmarks (MFU / TFLOPS)

Beyond raw interconnect bandwidth, it's often useful to measure end-to-end training performance — throughput (tokens/sec), compute utilization (TFLOPS), and Model FLOPs Utilization (MFU). For this, TensorWave recommends the Primus Auto Benchmark tool from AMD, which runs standardized model configs on the Megatron or TorchTitan backend and automatically generates a metrics summary table.

The tool runs inside the rocm/primus container on a single GPU node. It is interactive: after launching, it prompts you to choose a backend, auto-detects the GPU (MI300X / MI355X), and lets you select which model config(s) to run. Selected configs run sequentially, with logs and metrics written to the results/ directory.

Note: MFU is reported by the TorchTitan backend. The Megatron backend reports TPS, TFLOPS, memory utilization, and step time. Choose the backend based on the metric you need.

Running the benchmark

Because the tool runs interactively in Docker, first allocate a GPU node and open a shell on it. From a login pod, allocate a node and find its Slurm node name:

Then SSH to the allocated worker pod by its node name:

Pull the Primus container:

Launch the container:

Change into the auto-benchmark directory and launch the interactive tool:

Follow the on-screen prompts to select a backend, model config(s), and any parameter overrides. You will be asked for a HuggingFace token to pull model and tokenizer assets.

Reading the results

When all runs finish, the tool automatically generates a metrics table. Logs and the exact configs used are preserved for reproducibility under /workspace/Primus/tools/auto_benchmark/results/logs_<backend>/.

Preflight Smoke Test

preflight is a fast, self-contained health check meant to be dropped into the top of a job to efficiently validate hardware health before your real workload starts. It runs the pre-installed PyTorch environment at /opt/venv and exercises three things in under a minute:

1. matmul — every GPU runs a bf16 GEMM; validates the result is finite and measures throughput (TFLOPs).

2. intra-node RCCL — an all_reduce within each node's local GPUs; measures intra-node bus bandwidth.

3. inter-node RCCL — an all_reduce across the full allocation; measures cross-node bus bandwidth. Only runs when launched across more than one node (see below).

Each measurement must clear two bars: a per-SKU absolute floor and a peer-relative check (no GPU/link may sit more than a configurable tolerance below the fleet median), which catches a single throttled GPU or degraded link regardless of hardware SKU. On any failure preflight exits non-zero, so a set -e job aborts before wasting the allocation.

The launcher lives at /opt/tw/examples/bin/preflight and a ready-to-copy job is provided at /opt/tw/examples/libexec/preflight.sbatch.

Injecting it into your job

Add a single line near the top of your sbatch script, before your workload. Because the script uses set -e, a failed preflight stops the job before your workload runs:

You do not need to configure torchrun, master addresses, or GPU counts — the launcher detects the allocation from the SLURM environment and bootstraps torch.distributed.run for you.

Supported launch configurations

Key rules:

• Multi-node RCCL runs only under srun across >1 node. A bare preflight invocation (or a single-node allocation) stays single-node and never hangs waiting for peers that will not join.

• Launch with --ntasks-per-node=1. The launcher itself spawns one worker per GPU on each node; you only need one launcher task per node.

• Multi-node runs require full 8-GPU nodes. A partial node makes the inter-node comparison meaningless, so preflight hard-fails a multi-node run if any node has fewer than 8 GPUs. Partial-GPU checks (e.g. 2 or 4 GPUs) are supported single-node only.

Options and configuration

Any arguments after the launcher are forwarded to the check. Useful flags: