Quantized KV Cache

Quantized KV cache reduces the memory footprint of key-value cache storage by using lower-precision data types (FP8 or FP4) instead of the default model precision in BF16. During autoregressive generation, LLMs cache previously computed key-value pairs to avoid redundant calculations. The KV cache typically consumes a significant portion of GPU memory, especially for long sequences. Quantized KV cache is a memory optimization technique that primarily benefits throughput by allowing more tokens to be cached, but may introduce minimal accuracy degradation depending on the quantization format used.

Performance Warning: When quantized KV cache must be dequantized before use in attention operations, performance can be extremely slow if dequantization is not fused with the attention kernel. Always verify that your chosen attention backend supports quantized KV cache. Backends without fused support may experience significant throughput degradation, potentially negating the memory benefits.Backend Support: Not all attention backends support quantized KV cache. Refer to Attention Backend for which backends support it.

Supported Formats

SGLang supports the following quantized KV cache formats:

FP8 Format

OCP (Open Compute Project) specifies two common 8-bit floating point formats:

E5M2 (5 exponent bits, 2 mantissa bits): Larger dynamic range (±57344.0), lower precision
E4M3 (4 exponent bits, 3 mantissa bits): Higher precision, smaller dynamic range (±240.0)

FP4 Format

FP4 quantization is currently experimental.

OCP (Open Compute Project) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format. SGLang exposes two experimental E2M1 KV-cache recipes:

nvfp4: NVIDIA FP4 with 16-value blocks, E4M3 block scales, and a per-tensor global scale.
fp4_mx_block16: An E2M1 block-size-16 compatibility recipe. It is distinct from the standard OCP MXFP4 block-size-32 format.

Usage

Enabling Quantized KV Cache

To enable quantized KV cache, use the --kv-cache-dtype argument when launching the server:

Command

# Enable FP8 E5M2 KV cache
python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-R1-0528 \
    --kv-cache-dtype fp8_e5m2 \

# Enable FP8 E4M3 KV cache
python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-R1-0528 \
    --kv-cache-dtype fp8_e4m3 \

# Enable NVFP4 FP4 E2M1 KV cache
python3 -m sglang.launch_server \
    --model-path nvidia/DeepSeek-R1-0528-NVFP4 \
    --kv-cache-dtype nvfp4 \

# Enable block-size-16 FP4 E2M1 KV cache
python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-R1-0528 \
    --kv-cache-dtype fp4_mx_block16 \

SM100 native NVFP4 recipes

On SM100, prefill can either consume packed NVFP4 directly or dequantize it into an FP8 E4M3 workspace. Select the online dequantization dtype with --prefill-kv-cache-dequant-dtype; SGLang chooses the corresponding attention implementation. For native NVFP4 prefill and decode:

Command

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3.5-35B-A3B-FP8 \
    --tp-size 1 \
    --kv-cache-dtype nvfp4 \
    --prefill-kv-cache-dequant-dtype nvfp4 \
    --page-size 16

For FP8 prefill backed by a temporary dequantization workspace, with native NVFP4 decode:

Command

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3.5-35B-A3B-FP8 \
    --tp-size 1 \
    --kv-cache-dtype nvfp4 \
    --prefill-kv-cache-dequant-dtype fp8_e4m3 \
    --page-size 16

nvfp4 means that prefill consumes the packed native NVFP4 cache directly without any additional dequantization. The default value, auto, selects this native NVFP4 mode on SM100 and FP8 E4M3 dequantization on SM120. Decode consumes native NVFP4 in both recipes. The native recipe avoids the FP8 workspace and its token-linear scale copy; the FP8 recipe retains both the linear scales used during prefill and the physical scale layout used during decode. KV data remains stored as packed FP4 either way.

Native NVFP4 prefill requires SM100, a page size divisible by 4, and an attention head dimension divisible by 64. TRT-LLM GenMHA uses FP8 query and output buffers internally; SGLang converts the result back to the model activation dtype. Top-k-1 EAGLE/EAGLE3/NEXTN and breadth-1 NGRAM speculative decoding are supported: an EAGLE-family draft worker uses trtllm_mha, and target verification consumes the physical NVFP4 cache directly in eager execution and CUDA Graphs. Set --speculative-ngram-max-bfs-breadth=1 for NGRAM. With the mixed FlashInfer-prefill/TRT-LLM-decode recipe, SGLang resolves --speculative-attention-mode to decode so verification does not depend on FlashInfer’s transient dequantization workspace. Other speculative algorithms, PD disaggregation, hierarchical KV cache, and LMCache are not currently supported by the SM100 native NVFP4 path. SM120 XQA continues to use its architecture-specific linear scale layout and BF16 query/output path.

Scaling Factors

FP8 quantization requires scaling factors to properly quantize and dequantize the KV cache.

Currently, only per-tensor (scalar) scaling factors are supported.

Scaling factors can be:

Loaded from checkpoints: Pre-quantized models (e.g., ModelOpt) may include k_scale and v_scale parameters that are automatically loaded
Provided via JSON: Supply scaling factors via --quantization-param-path.

The JSON file should follow this format:

Config

{
  "kv_cache": {
    "dtype": "float8_e4m3fn",
    "scaling_factor": {
      "0": {
        "0": 1.0,
        "1": 1.0
      }
    }
  }
}

Where the outer keys in scaling_factor are tensor parallel ranks and inner keys are layer indices.

If scaling factors are not provided and not found in the checkpoint, it will default to 1.0, which may cause accuracy issues.

FP4 (MXFP4): Unlike FP8, FP4 quantization handles scaling factors automatically on-the-fly during quantization and dequantization. No pre-quantized models or external scaling factor files are required—the block-based scaling factors are computed dynamically as needed.

Performance Considerations

Memory Savings

Quantized KV cache provides significant memory savings:

BF16 → FP4: Supports approximately 3.56× more tokens than BF16 (accounting for scaling factor overhead)

FP4 and FP8 quantization require additional memory for block-based scaling factors, which reduces the effective memory savings compared to the raw bit-width reduction. FP4 with block size 16 supports approximately 1.78× more tokens than FP8, and approximately 3.56× more tokens than BF16. The relative token capacity between FP8 and BF16 can be derived from these ratios.

This enables longer context lengths or more concurrent requests within the same memory budget. For native nvfp4, each logical scalar costs 0.5 bytes of packed FP4 data plus 1/16 byte of block-scale storage, compared with 2 bytes for BF16. The resulting theoretical KV-token capacity multiplier is 2 / 0.5625 = 3.56×. The mixed SM100 recipe owns a second scale layout and a shared one-layer FP8 prefill workspace, so its exact capacity depends on the number of full-attention layers. SGLang includes those auxiliary buffers in its KV-pool sizing calculation.

Accuracy Impact

FP8 Accuracy

FP8 E4M3 quantization typically introduces minimal accuracy degradation. The impact depends on model architecture, sequence length, and quantization format (generally, E4M3 has better accuracy than E5M2).

FP4 Accuracy

FP4 (MXFP4) quantization provides significant memory savings with varying accuracy impact depending on model size and dataset complexity. Preliminary accuracy test results from PR #10078 (MLA) and PR #12612 (MHA) show: Large Models (e.g., Qwen3-235B-A22B, DeepSeek-R1-0528) On large-scale models, FP4 maintains accuracy close to FP8/BF16, especially on simpler datasets:

Model	Dataset	KV16	KV8 (FP8 E4M3)	KV4 (FP4 E2M1)
Qwen3-235B-A22B	gsm8k	0.9168	0.9181	0.9186
Qwen3-235B-A22B	aime25	0.7733	0.7333	0.6000
Qwen3-235B-A22B	gpqa_diamond	0.7010	0.6899	0.6778
DeepSeek-R1-0528	gsm8k	0.9157	0.9154	0.9124
DeepSeek-R1-0528	aime25	0.5067	0.4934	0.4000
DeepSeek-R1-0528	gpqa_diamond	0.7707	0.7697	0.7273

Smaller Models (e.g., GPT-OSS-120B) On smaller models, FP4 shows more pronounced accuracy drops, particularly on challenging datasets:

Model	Dataset	KV16	KV8 (FP8 E4M3)	KV4 (FP4 E2M1)
GPT-OSS-120B	gsm8k	0.9161	0.9163	0.9152
GPT-OSS-120B	aime25	0.7533	0.7667	0.3533
GPT-OSS-120B	gpqa_diamond	0.5081	0.5434	0.3202

Key Observations:

Simple datasets (e.g., gsm8k): FP4 maintains accuracy close to FP8/BF16 across model sizes
Model size matters: Large models (200B+ parameters) generally tolerate FP4 quantization better than smaller models
Context length: Accuracy degradation may be more pronounced in long-context scenarios, as the accumulation of the quantization error may become significant.

Evaluate FP4 accuracy on your specific model and workload. Large models on simpler tasks typically show minimal degradation, while smaller models or complex reasoning tasks may require FP8 or BF16 for acceptable accuracy.

Best Practices

Use pre-quantized models: Prefer models quantized offline with scaling factors included in the checkpoint.
Choose the right format: Use fp8_e4m3 for better accuracy (recommended), fp8_e5m2 for larger dynamic range, or nvfp4 / fp4_mx_block16 for maximum memory savings (experimental)
Check backend compatibility: Verify that your chosen attention backend supports quantized KV cache

Basic Usage

Advanced Features

Supported Models

Developer Guide

References

Supported Formats

FP8 Format

FP4 Format

Usage

Enabling Quantized KV Cache

SM100 native NVFP4 recipes

Scaling Factors

Performance Considerations

Memory Savings

Accuracy Impact

FP8 Accuracy

FP4 Accuracy

Best Practices

​Supported Formats

​FP8 Format

​FP4 Format

​Usage

​Enabling Quantized KV Cache

​SM100 native NVFP4 recipes

​Scaling Factors

​Performance Considerations

​Memory Savings

​Accuracy Impact

​FP8 Accuracy

​FP4 Accuracy

​Best Practices

Supported Formats

FP8 Format

FP4 Format

Usage

Enabling Quantized KV Cache

SM100 native NVFP4 recipes

Scaling Factors

Performance Considerations

Memory Savings

Accuracy Impact

FP8 Accuracy

FP4 Accuracy

Best Practices