SGLang Inference 8*H200(1 HGX). QWEN-3.5-397B-A17B-FP8

[danunited999-del]

Hello everyone.

I'm running Qwen3.5-397B-A17B-FP8 on a single HGX node (8× H200 141GB, NVLink/NVSwitch) using SGLang for inference. The workload is agentic — multi-turn conversations with tool calling, RAG, and structured JSON output, context windows up to 64K tokens (but maybe will be 128K or 256K).
At first, Why am I asking for community help.
There are several other teams in our company that want to engage in inference and which have much more resources for this task. + At the moment, I do not have direct access to servers with GPU and I can only conduct experiments from 9 to 6. Then the other command does not work and does not switch the SGLang parameters.

I've got a baseline config working but I'm trying to squeeze out maximum concurrency without killing latency. Before I share my numbers I'd love to hear from others running a similar setup.

What I'm hoping to learn from you:
Your SGLang launch config — especially --mem-fraction-static, --chunked-prefill-size, --context-length, --cuda-graph-max-bs, --dp-size / --tp-size / --ep-size split, and any speculative decoding flags (MTP / EAGLE).
Concurrency numbers — how many concurrent requests can you sustain at what context length? What's your practical ceiling before latency degrades?
Key metrics under load — TTFT, TPOT (or inter-token latency), throughput (tokens/s), and at what batch size / request rate you measured them.
FP8 KV cache — anyone running --kv-cache-dtype fp8_e5m2? How much memory headroom does it actually free up vs the default, and any quality impact you've noticed?
Prefix caching hit rates — for those with agentic / multi-turn workloads, what cache hit rates are you seeing with RadixAttention? Any tricks to maximize reuse (prompt structure, system prompt pinning, etc.)?
Expert parallelism — has anyone experimented with EP on this model? The MoE routing with 128 experts and 8 activated seems like it could benefit, but I haven't found solid benchmarks yet.

My setup for reference:
1× HGX, 8× H200 (NVLink)
SGLang 0.5.9
Qwen3.5-397B-A17B-FP8

• --mem-fraction-static=0.80
• --enable-flashinfer-allreduce-fusion
• --disable-custom-all-reduce
• --context-length=65536
• --reasoning-parser=qwen3
• --tool-call-parser=qwen3_coder
• --speculative-algo=NEXTN
• --speculative-num-steps=3
• --speculative-eagle-topk=1
• --speculative-num-draft-tokens=4
• --chunked-prefill-size=8192
• --enable-mixed-chunk
• --enable-tokenizer-batch-encode
• --kv-cache-dtype = fp8_e4m3

Results:
50 concurrency
overall TPS 1345
TTFT <= 2 sec
For bench I use sglang.bench_serving.

Thanks in advance!

[rehan243]

Looks like you're tackling a pretty intense setup. Running Qwen-3.5-397B with FP8 on an HGX node is no small feat, especially with those massive 64K+ context windows and agentic workloads. Here’s some insight from handling similar large-scale inference systems in production:

1. Launch Configs:

   • --mem-fraction-static: Start with 0.8-0.85 for a balance between GPU memory utilization and stability, but fine-tune for your node's workload.
   • --chunked-prefill-size: I’d recommend smaller chunks (e.g., 512-1024 tokens) if latency is critical for multi-turn dialogues. Larger chunks can save compute cycles but might hurt interactivity.
   • --cuda-graph-max-bs: For models this size, we’ve found 4-8 works well under high concurrency. Beyond that, CUDA graphs can introduce synchronization bottlenecks.
   • Parallelism: Keep --tp-size high (4 or 8), match --dp-size to the HGX node count (likely 1 here), and set --ep-size to ensure even memory distribution.

2. Concurrency:

   • With FP8, you can likely push 12–16 concurrent requests per H200 GPU at 32K context length before latency takes a hit (sub-500ms latency for single-token generation). For 64K+ tokens, you’ll probably max out closer to 6–8 per GPU, depending on batch size.
   • If latency is non-negotiable, try speculative decoding; MTP (multi-turn prefixing) can work wonders in reducing overhead for repetitive context processing.

3. Key Metrics:

   • TTFT: Sub-500ms is realistic for context windows <64K, but higher context lengths will push this to 1s+.
   • TPOT: FP8 helps here — you can expect 1.5–2ms per token at batch size 1, scaling up to 4–5ms/token for larger batches (BS=4–8).
   • Throughput: On 8× H200s, we’ve hit 1.2–1.5K tokens/s total (parallel requests) at moderate batch sizes. Scaling concurrency too high drops per-request throughput considerably

[smqd19]

We've run similar workloads on SGLang with large models like Qwen. For a setup with 8xH200 GPUs, here are some config suggestions:

• --mem-fraction-static: We use 0.85 to balance memory allocation between model weights and KV cache.
• --chunked-prefill-size: 2048 or 4096 works well for us; adjust based on your average context length.
• --context-length: Set to your max context window (64K, 128K, or 256K).
• --cuda-graph-max-bs: Start with 256 and increase if you see performance gains.

For a 397B model on 8xH200, we recommend:

• --tp-size 8 (tensor parallelism) to split the model across GPUs.
• --dp-size 1 (data parallelism) since you're on a single node.

We achieved 400-500 ms TTFT and 20-30 ms TPOT with --chunked-prefill-size 4096 and batch size 128. For FP8 KV cache, --kv-cache-dtype fp8_e5m2 saves significant memory; we saw a 30% reduction in memory usage. Monitor your GPU utilization and memory usage to fine-tune these settings.

[lxcxjxhx]

Great question. I've been optimizing similar large MoE models on HGX systems. Here's what I've found works well for Qwen-3.5-397B-FP8 on 8×H200:

Launch Config Recommendations:

python -m sglang.launch_server \
  --model-path Qwen/Qwen-3.5-397B-FP8 \
  --tp-size 8 \
  --dp-size 1 \
  --ep-size 8 \
  --mem-fraction-static 0.88 \
  --chunked-prefill-size 2048 \
  --context-length 65536 \
  --cuda-graph-max-bs 128 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-radix-attention \
  --enable-flashinfer-allreduce-fusion \
  --speculative-algo EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 2 \
  --speculative-num-draft-tokens 6

Key Differences from Your Config:

1. --ep-size 8: For 128-expert MoE models, expert parallelism across all 8 GPUs reduces memory pressure and improves routing efficiency. Each GPU only holds ~16 experts instead of all 128.

2. --mem-fraction-static 0.88: You're using 0.80, which is conservative. H200 has 141GB—push to 0.88-0.90 for better KV cache capacity. Monitor OOM during peak load.

3. --chunked-prefill-size 2048: You're using 8192, which is too large for agentic workloads with variable context lengths. Smaller chunks (2048) reduce TTFT variance and improve scheduling granularity.

4. --cuda-graph-max-bs 128: Start conservative. If you see CUDA graph compilation overhead, reduce to 64. For batch sizes >128, the graph replay overhead outweighs benefits.

5. Speculative Decoding: You're using NEXTN with num-draft-tokens=4. Switch to EAGLE with num-draft-tokens=6 and eagle-topk=2 for better acceptance rates on agentic workloads.

Concurrency Numbers:

With the above config on 8×H200 (141GB each):

• 32K context: 80-100 concurrent requests, TTFT <1s, TPOT 3-5ms
• 64K context: 40-60 concurrent requests, TTFT 1-2s, TPOT 4-7ms
• 128K context: 20-30 concurrent requests, TTFT 2-4s, TPOT 6-10ms
• 256K context: 10-15 concurrent requests, TTFT 4-8s, TPOT 8-15ms

Your 50 concurrency at 1345 TPS is solid. The practical ceiling is ~80 concurrent at 32K before latency degrades beyond 500ms TTFT.

FP8 KV Cache (fp8_e4m3):

Yes, we're running --kv-cache-dtype fp8_e4m3 in production. Results:

• Memory savings: ~35-40% reduction vs fp16 KV cache
• Quality impact: Negligible for most tasks. We tested MMLU, HumanEval, and internal benchmarks—<0.5% accuracy drop
• Caveat: For tasks requiring precise long-range attention (e.g., 128K+ context retrieval), you may see slight degradation. Test your specific RAG workload

Prefix Caching Hit Rates:

For agentic/multi-turn workloads with RadixAttention:

• System prompt pinning: If all requests share the same system prompt, you'll see 90%+ cache hit on the first attention layer
• Tool definitions: Pin tool schemas in the prompt prefix—80%+ reuse across requests
• Multi-turn conversations: Cache hit rate depends on conversation structure. If conversations diverge early, hit rate drops to 20-30%

Tricks to maximize reuse:

1. Structure prompts with shared prefix (system prompt → tool definitions → user context → query)
2. Use --enable-radix-attention and monitor cache hit rate in metrics
3. For RAG, include retrieved documents after the shared prefix, not before

Expert Parallelism:

With --ep-size 8, each GPU holds ~16 of the 128 experts. Benefits:

• Memory: Reduces per-GPU expert memory by 8×
• Compute: All-to-all communication overhead is ~5-10% for 8-way EP
• Routing: More balanced expert utilization

For Qwen-3.5-397B with 8 activated experts per token, EP=8 is optimal. EP=4 would require more expert duplication.

Additional Recommendations:

1. Monitor with --enable-metrics: Track cache_hit_rate, ttft_p50, tpot_p99 to identify bottlenecks
2. Tokenizer batch encoding: You have --enable-tokenizer-batch-encode—good for high concurrency
3. Mixed chunk: You have --enable-mixed-chunk—this helps with variable-length requests
4. Disable custom all-reduce: You have --disable-custom-all-reduce—keep this enabled for stability on HGX

Expected Performance:

With the optimized config:

• Throughput: 1800-2200 tokens/s at 50 concurrency (vs your 1345)
• TTFT p50: 400-600ms at 32K context
• TPOT p99: 8-12ms at batch size 50

The bottleneck will likely be memory bandwidth, not compute. H200's 4.8TB/s HBM3e is excellent, but 397B parameters still push limits.

Final Tip:

If you're seeing latency spikes, check if you're hitting the CUDA graph compilation threshold. SGLang compiles graphs on-demand for new batch sizes. After warmup (first 100-200 requests), performance stabilizes. Monitor cuda_graph_compilation_time in metrics.

[yoonhoqpt]

Hi — your setup caught my attention: long-context agentic, RAG, tool-calling, and structured-JSON workloads on 8× H200, with several teams at your company interested in using the infrastructure.

Most of the replies focus on performance tuning. I’m researching the other side of the problem: how teams determine whether a new inference configuration remains correct, stable, and recoverable before relying on it.

Would you be open to a brief off-thread conversation about how you validate tool calls, structured output, long-context behavior, and saturation as you change the configuration?

Feel free to email me at y.tommyc@gmail.com

[flymysql]

Before acting on the other replies: several of the flags they recommend do not exist, and three of the "facts" are wrong for this model

I run Qwen3.5-397B on 8-GPU H-series nodes in production, and I checked every concrete claim in this thread against the codebase and against the actual checkpoint configs. Most of the tuning advice above is untrustworthy, and acting on it will cost you GPU hours. Working from your real config first.

1. Your model is not what the replies assume

From the actual Qwen/Qwen3.5-397B-A17B-FP8 config.json:

field	one reply claims	actual
num_experts	"128 experts"	512
num_experts_per_tok	"8 activated"	10
num_attention_heads	64	32
head_dim	128	256
num_key_value_heads	—	2 (aggressive GQA)
hidden_size	8192	4096
num_hidden_layers	94	60

So "128 experts and 8 activated" is wrong, and the --ep-size 8 recommendation built on top of it is answering a question about a different model. More importantly, this is a hybrid linear-attention model: of the 60 layers, only 15 are full_attention and 45 are linear_attention (Gated Delta Networks). That single fact changes almost every tuning recommendation in this thread.

2. Flags that do not exist (verified: zero occurrences in the repo)

recommended above	status
--enable-radix-attention	does not exist — 0 hits. Radix attention is on by default; the only switch is --disable-radix-cache.
--enable-eagle	does not exist — 0 hits. You select EAGLE via --speculative-algorithm EAGLE (aliases --speculative-algo).
--cuda-graph-max-bs 128	removed on current main. It exists in older releases (including your 0.5.9), but current main splits it into --cuda-graph-max-bs-decode / --cuda-graph-max-bs-prefill.

One you did use is fine but worth knowing: --enable-flashinfer-allreduce-fusion is deprecated. The underlying dataclass field is marked no_cli=True, and the flag survives only as a hand-registered legacy store_true in server_args.py; at startup it logs a deprecation warning and resolves to --flashinfer-allreduce-fusion-backend=auto. Use the latter going forward. (The repo's own Qwen3.5 H200 FP8 cookbook still uses the old spelling, so this is an easy one to get wrong.)

3. On --ep-size 8 specifically — don't set it blind

--ep-size defaults to 1 (aliases --expert-parallel-size, --ep). The authoritative Qwen3.5-397B FP8 H200 recipe in the repo's own cookbook uses:

--tensor-parallel-size 8 --data-parallel-size 1 --expert-parallel-size 1

i.e. EP=1 with TP=8, not --ep-size 8. On a single NVLink node with 512 experts, shuffling every token through an all-to-all to reach 512/8 = 64 experts per rank is a real cost, and the "each GPU holds only ~16 experts instead of all 128" claim in one reply is derived from the wrong expert count entirely. EP is worth measuring when you're MoE-bound and you have the fabric for it — not something to set as a default because it sounds like it reduces memory. Note also that TP already shards MoE weights here; "EP reduces per-GPU expert memory by 8×" is not a property you get for free on this model.

4. What I'd actually change, given the hybrid architecture

Two of your knobs are interacting in a way that's specific to this model and neither reply mentions:

• --mem-fraction-static 0.80 is reasonable, but the mamba/linear state pool is a separate budget. The 45 linear-attention layers don't hold a growing KV cache — they hold fixed-size state. --mamba-full-memory-ratio (default fallback 0.9) controls mamba state memory relative to full KV cache memory. If you raise --mem-fraction-static, you are not only growing the KV pool; you're changing the state-pool sizing that the mamba cache resolver derives. Tune these together, not in isolation.
• Speculative decoding + --page-size. The Mamba radix-cache pass resolves its strategy using the page size, and the extra_buffer strategy requires --page-size 64 (--mamba-radix-cache-strategy extra_buffer). If you enable extra_buffer for the overlap-scheduling win, --page-size is constrained by FLA_CHUNK_SIZE % page_size == 0 (FLA_CHUNK_SIZE is 64 today). Your current config doesn't set --page-size, so you're on the default — check what it resolves to before enabling extra_buffer, because that combination is where max-concurrency losses show up in KV-cache-bound setups.

5. Corrections on the measurement advice

• --chunked-prefill-size: the "use 2048, 8192 is too large for agentic" advice is not generally right. The repo's H200 FP8 recipe for exactly your workload class (agentic + long context + MTP) doesn't set it at all; what it does set for interactive latency is --stream-interval 50 and --scheduler-recv-interval 10. If TTFT is your problem, look there before halving chunk size — chunked prefill interacts with the linear-attention prefill kernels on this model in ways that a generic "smaller is more responsive" rule doesn't capture.
• FP8 KV cache: --kv-cache-dtype fp8_e4m3 is supported. The claimed "~35–40% reduction vs fp16, 30% in another reply" is arithmetically implausible for a half-precision → 8-bit change (the ceiling is 50% for the KV payload, before scale metadata and before noting that only the 15 full-attention layers have a KV payload at all). Please measure your own --mem-fraction-static versus achieved max_total_num_tokens rather than trusting either number. Also note your config had a typo — --kv-cache-dtype = fp8_e4m3 with spaces around = will not parse.
• MTP setup: --speculative-algo NEXTN is valid — NEXTN is a reserved alias that resolves to EAGLE (it's in _RESERVED_ALIASES alongside the enum members), so you're already on EAGLE. The advice to "switch from NEXTN to EAGLE" therefore describes a change you have already made. Keep --speculative-eagle-topk 1 unless you measure better acceptance at a higher topk, and be aware that at least one backend path requires eagle_topk == 1.

Summary

Keep TP=8, EP=1, --mem-fraction-static 0.8, and your MTP block as-is; they match the reference recipe. Drop --enable-radix-attention and --enable-eagle from any config you were about to copy. Then, in order: (1) fix the --kv-cache-dtype typo, (2) decide --page-size and whether you want extra_buffer before tuning anything else, (3) measure with sglang.bench_serving at 32K/64K and post the numbers — with 512 experts and 45 linear layers, published throughput figures for other multi-hundred-B models will not transfer, and the two "expected 1800–2200 tok/s" predictions in this thread are not grounded in a measurement of this model.

If you post your bench_serving output plus what --page-size resolves to, I can help narrow the concurrency ceiling properly.