GLM-5.3-Flash on 8x B200: dual TP4/EP4 serving profile with long-context measurements

[stewtong]

I measured an SGLang serving profile for zai-org/GLM-5.3-Flash on one 8x NVIDIA B200 node. It uses two independent TP4/EP4 replicas, FP8 KV cache, TRT-LLM DSA backends, the FlashInfer TRT-LLM MoE runner, static NEXTN 5-1-6, and session-affine routing.

Update, September 27: on the pinned image, one server prefilling two 1M-token prompts together ran out of GPU memory in the sparse-attention indexer. #40854 fixes this upstream, and the repository now publishes an equivalent local patch. The results below predate the patch. The September run's findings are in this comment.

The pinned software stack, engine commands, proxy configuration, client setup, methods, and results are documented here: https://github.com/stewtong/b200-glm53

Validation observations:

• All 138 streaming Chat Completions requests returned HTTP 200 with the requested completion length. None timed out or returned HTTP 429.
• At concurrency 1, median end-to-end output rate including prefill was 245.3 tok/s at about 131K input tokens and 12.50 tok/s at about 1M input tokens.
• At about 524K input tokens and total concurrency 16 across both replicas, 48/48 requests completed without an observed out-of-memory error. Maximum capacity remains unmeasured.
• GSM8K scored 1,284/1,319 (97.35%) with the checkpoint's default max effort and 1,272/1,319 (96.44%) with reasoning_effort: low. The conditions used different output limits and are reported separately in the repository.
• In a repeated-prefix cache study with stable session affinity, median time to first output event was 3.10 seconds on turn 1 and 0.62 seconds on later turns. With a changed leading system line, the corresponding medians were 3.13 and 3.24 seconds.

The serving table and cache study used three repetitions per condition. Since September 3, the repository also publishes the benchmark harness and the request-level rows for both. The repository documents the tested boundaries and excludes unsupported aggregate throughput, decode TPOT, route-latency, and causal claims.

[stewtong]

Update, September 27: a new same-day run on the same 8x B200 setup found four things.

1. The pinned image can crash on concurrent 1M-token prompts. When one server prefilled two 1M-token prompts together, the sparse-attention indexer asked for a single 29.68 GiB buffer and ran out of GPU memory. The scheduler stopped generating, but the HTTP server kept answering, so the server still appeared healthy. [DSA] Chunk the kpool indexer MQA logits by query rows under a free-memory budget #40854 (merged September 26) fixes this upstream. The new results use an equivalent local patch, published in reproduce/patches/.
2. Two TP4/EP4 servers outperform one TP8/EP8 engine at long context. They delivered 1.80-2.11x its output rate at 128K input with 16 requests, 512K with 8 or 16, and 1M with 4 or 8. For single requests they were 1-4% faster.
3. FP8 KV cache showed no accuracy loss against BF16. On the full GSM8K test set, both scored 1,283 of 1,319 at the default effort. At low effort, FP8 scored 1,267 and BF16 1,272, a gap inside the run's uncertainty (95% interval -0.3 to +1.1 points). FP8 fits 1.80x as many tokens in the same memory.
4. A stopped server leaves truncated streams that still report HTTP 200. The streams ended without a finish reason or token counts; the 200 status had already been sent when each stream opened.

The run also covered 16K to 1M input at 1 to 16 concurrent requests: 744 of 744 requests succeeded, three repetitions per cell.

Every result ran with the local patch and a tool-call parser overlay. The TP8 run came after the main grid and used different prompts. The output rate is for one burst of requests, not sustained load. The original post said the raw traces were unpublished; the request-level data is now in the repository, and reproduce/derive-results.py recomputes these numbers. Full results and limits: BENCHMARKS.md.

[stewtong]

Revision note (September 27): the post now opens with the 1M out-of-memory finding from the comment above, and its last paragraph notes that the benchmark harness and request-level rows have been published since September 3. No measured values changed.