[ROCm][Quant][Perf] Minimax-M3: Enable fp8_per_channel for bf16 weights on mi300x

[hongxiayang]

Purpose

Improve the perf of Minimax-M3 bf16 model on MI300x (gfx942)/

The fp8 w8a8 MoE quant config dropped the SwiGLU-OAI alpha/beta that models
such as MiniMax-M3 pass to FusedMoE (swiglu_alpha=1.702, swiglu_beta=1.0).
Only swiglu_limit was forwarded, so the silu_and_mul_with_clamp kernel ran
with its default alpha=1.0/beta=0.0 and produced garbage (gsm8k 0.00) on both
the serialized (Fp8MoEMethod) and online (_Fp8OnlineMoEBase) fp8 MoE paths.

Plumb gemm1_alpha/gemm1_beta through the fp8 w8a8 MoE config chain:

fp8_w8a8_moe_quant_config: add params, forward to FusedMoEQuantConfig.make
make_fp8_moe_quant_config: forward them in the default/TRITON branch
Fp8MoEMethod.get_fused_moe_quant_config: read layer.swiglu_alpha/beta
_Fp8OnlineMoEBase.get_fused_moe_quant_config: same, for the online path
Also add "fp8_per_channel" to the ROCm supported_quantization allowlist. The
PTPC methods (Fp8PtpcOnline{Linear,MoE}Method) already exist and the ROCm
rowwise fp8 scaled-MM supports per-channel weight scales; the method was simply
not allowlisted on ROCm.

Verified on bf16 model with --quantization fp8_per_channel on MI300x.

For the 1k/1k config:

• halved weight bytes/token → ~2× weight-read bandwidth in decode (decode at batch is HBM-bandwidth-bound). This is the +28% conc64.
• 49% weight memory → +75% KV capacity → fit ~1.75× more concurrent requests (or ~1.75× longer context) before hitting the KV ceiling.

Use case beneficial:

• Long output-len, high concurrency (decode-dominated)
• Long total context × high concurrency

Aided with Claude.

Test Plan

serv

VLLM_USE_BREAKABLE_CUDAGRAPH=0 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=0 \
vllm serve <WEIGHTS> \
  --served-model-name mm3 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096 \
  --block-size 128 \
  --attention-backend TRITON_ATTN \
  --no-enable-prefix-caching \
  --quantization fp8_per_channel \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

Throughput bench (vllm bench serve, random 1k-in/1k-out)

vllm bench serve --backend vllm \
  --model mm3 --tokenizer <WEIGHTS> \
  --base-url http://localhost:8000 \
  --dataset-name random --random-input-len 1024 --random-output-len 1024 \
  --ignore-eos --num-prompts <N> --max-concurrency <C>   # C=1 (N=8), C=64 (N=128)

Accuracy (gsm8k via lm_eval, full 1319 samples)

# 20-shot (server --max-model-len 8192; ):
lm_eval --model local-completions \
  --model_args "model=mm3,base_url=http://localhost:8000/v1/completions,\
tokenized_requests=False,tokenizer_backend=None,num_concurrent=200,\
timeout=5000,max_length=8192" \
  --tasks gsm8k --num_fewshot 20 --limit 1319

Test Result

config	model GiB/GPU	GPU KV tok	conc1 tok/s (TPOT ms)	conc64 tok/s (TPOT ms)
bf16 (no quant)	99.9	1.55M	101.9 (9.72)	1476 (41.8)
bf16 + AITER linear	99.9	1.55M	100.8 (9.76)	1461 (42.4)
fp8_per_channel (PTPC)	50.6	2.72M	88.6 (11.04)	1801 (34.14)
fp8_per_channel + AITER linear	50.6	2.72M	94.0 (10.53)	1894 (32.42)

20-shot, full-1319 gsm8k	flex / strict	stderr
bf16 (no quant)	0.9424 / 0.9424	±0.0064
fp8_per_channel + AITER linear	0.9393 / 0.9393	±0.0066
gap	0.0031 (~0.3σ)	—

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[tjtanaa]

LGTM.

[tjtanaa]

@hongxiayang small NITS: we don't need the comment.

[tjtanaa]

@hongxiayang nits: we can remove these comments because the codebase already knew that this arguments are needed by minimaxm3

[tjtanaa]

@hongxiayang nits: we can remove these comments because the codebase already knew that this arguments are needed by minimaxm3

[tjtanaa]

LGTM

[mergify]

Hi @hongxiayang, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.