Skip to content

[bugfix]Indexer init skip and MTP TopK share for iteration - #45895

Merged
youkaichao merged 6 commits into
vllm-project:mainfrom
JaredforReal:indexer_init_skip
Jun 19, 2026
Merged

youkaichao merged 6 commits into
vllm-project:mainfrom
JaredforReal:indexer_init_skip

Conversation

@JaredforReal

@JaredforReal JaredforReal commented Jun 17, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

fix GLM-5.2 BF16 init Indexer in skip_topk layer
fix GLM-5.2 MTP Port-Norm cycle

Test Plan

# setup with
VLLM_DEEP_GEMM_WARMUP=skip  vllm serve zai-org/GLM-5.2-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --max-num-seqs 32 \
  --cudagraph-capture-sizes 1 2 4 8 16 32 \
  --kv-cache-dtype fp8_e4m3

Test Result

MTP Mean Acceptance Length raise form 3~ to 4~, while Mean Acceptance Rate raise to 60%
IFBench remain 74.62


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: JaredforReal <w13431838023@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adds support for GLM-DSA MTP “post-final-norm hidden recycling” in speculative decoding and refactors sparse MLA attention to consume a shared top‑k indices buffer explicitly (enabling “skip” layers without an indexer).

Changes:

  • Add mtp_recycle_post_norm plumbing across config override, draft model (MTP) forward output shape, and proposer tuple-handling logic.
  • Pass topk_indices_buffer explicitly through MLA layers/attention into sparse MLA backends, decoupling them from indexer.
  • Refine DeepSeek v3.2 IndexCache “skip topk” logic to always build indexers for MTP/nextn layers while optionally skipping some backbone layers.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
vllm/v1/spec_decode/llm_base_proposer.py Detects GLM-DSA recycling and adjusts whether MTP returns tuples.
vllm/v1/attention/backends/mla/xpu_mla_sparse.py Switches sparse MLA backend to take a shared topk_indices_buffer directly.
vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py Same as above for ROCm aiter sparse MLA backend.
vllm/v1/attention/backends/mla/flashmla_sparse.py Same buffer wiring change for FlashMLA sparse backend.
vllm/v1/attention/backends/mla/flashinfer_mla_sparse.py Same buffer wiring change, removing the “indexer required” assertion.
vllm/model_executor/models/deepseek_v2.py Updates IndexCache skip/topk pattern logic and MTP-layer exceptions.
vllm/model_executor/models/deepseek_mtp.py Adds optional tuple return for GLM-DSA recycling behavior.
vllm/model_executor/layers/mla.py Plumbs topk_indices_buffer into the MLA attention layer.
vllm/model_executor/layers/attention/mla_attention.py For sparse MLA, passes topk_indices_buffer down to backend impl args.
vllm/config/speculative.py Sets mtp_recycle_post_norm based on original GLM model_type.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +1004 to +1007
_index_topk_freq = getattr(config, "index_topk_freq", 1)
_index_topk_pattern = getattr(config, "index_topk_pattern", None)
_index_skip_topk_offset = getattr(config, "index_skip_topk_offset", 2)
layer_id = extract_layer_index(prefix)
Comment on lines +1010 to +1012
_skip_topk = (
max(layer_id - _index_skip_topk_offset + 1, 0) % _index_topk_freq != 0
)
Comment on lines +441 to +444
# Sparse MLA reads top-k indices from a shared buffer. Pass it
# explicitly so backbone "skip" layers (indexer=None) still find it.
if use_sparse:
extra_impl_args["topk_indices_buffer"] = topk_indices_buffer

assert indexer is not None, "Indexer required for sparse MLA"
self.topk_indices_buffer: torch.Tensor | None = indexer.topk_indices_buffer
self.topk_indices_buffer: torch.Tensor | None = topk_indices_buffer
kv_sharing_target_layer_name: str | None,
# MLA Specific Arguments
topk_indice_buffer: torch.Tensor | None = None,
topk_indices_buffer: torch.Tensor | None = None,
@mergify mergify Bot added deepseek Related to DeepSeek models nvidia rocm Related to AMD ROCm intel-gpu Related to Intel GPU labels Jun 17, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Jun 17, 2026
@mergify mergify Bot added v1 bug Something isn't working labels Jun 17, 2026
Signed-off-by: JaredforReal <w13431838023@gmail.com>
realliujiaxu pushed a commit to vllm-project/vllm-ascend that referenced this pull request Jul 2, 2026
### What this PR does / why we need it?
This PR adds support for GLM-5.2 on Ascend. This pull request updates
`model_returns_tuple` in `llm_base_proposer.py` to support
DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP
recycles the post-final-norm hidden state, its forward pass returns a
tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families
return a single tensor.

vllm's PR: vllm-project/vllm#45895

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@dc68bd8

Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
MengqingCao pushed a commit to vllm-project/vllm-ascend that referenced this pull request Jul 6, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
wangyichao1999 pushed a commit to wangyichao1999/vllm-ascend that referenced this pull request Jul 9, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Spicy-Stick pushed a commit to Spicy-Stick/vllm-ascend that referenced this pull request Jul 10, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

---

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

---

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

---

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
---
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Signed-off-by: Spicy-Stick <873805887@qq.com>
xqchen7 pushed a commit to nv-action/vllm-benchmarks that referenced this pull request Jul 15, 2026
### What this PR does / why we need it?
This PR adds support for GLM-5.2 on Ascend. This pull request updates
`model_returns_tuple` in `llm_base_proposer.py` to support
DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP
recycles the post-final-norm hidden state, its forward pass returns a
tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families
return a single tensor.

vllm's PR: vllm-project/vllm#45895

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@dc68bd8

Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
Signed-off-by: xqchen7 <chenxueqing7@huawei.com>
xqchen7 pushed a commit to nv-action/vllm-benchmarks that referenced this pull request Jul 15, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

---

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

---

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

---

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

---

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
---
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Signed-off-by: xqchen7 <chenxueqing7@huawei.com>
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
Liamup777 pushed a commit to Liamup777/vllm-ascend that referenced this pull request Jul 20, 2026
### What this PR does / why we need it?
This PR adds support for GLM-5.2 on Ascend. This pull request updates
`model_returns_tuple` in `llm_base_proposer.py` to support
DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP
recycles the post-final-norm hidden state, its forward pass returns a
tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families
return a single tensor.

vllm's PR: vllm-project/vllm#45895

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@dc68bd8

Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
Alex-stack-hub pushed a commit to 0moyi0-2024/vllm-ascend_tp that referenced this pull request Jul 27, 2026
### What this PR does / why we need it?
This PR adds support for GLM-5.2 on Ascend. This pull request updates
`model_returns_tuple` in `llm_base_proposer.py` to support
DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP
recycles the post-final-norm hidden state, its forward pass returns a
tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families
return a single tensor.

vllm's PR: vllm-project/vllm#45895

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@dc68bd8

Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
Alex-stack-hub pushed a commit to 0moyi0-2024/vllm-ascend_tp that referenced this pull request Jul 27, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
eble-amd pushed a commit to ROCm/vllm that referenced this pull request Aug 14, 2026
MmMmaru pushed a commit to jiaqi-lee/vllm-ascend that referenced this pull request Aug 19, 2026
### What this PR does / why we need it?
This PR adds support for GLM-5.2 on Ascend. This pull request updates
`model_returns_tuple` in `llm_base_proposer.py` to support
DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP
recycles the post-final-norm hidden state, its forward pass returns a
tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families
return a single tensor.

vllm's PR: vllm-project/vllm#45895

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@dc68bd8

Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
MmMmaru pushed a commit to jiaqi-lee/vllm-ascend that referenced this pull request Aug 19, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
shiqiangA pushed a commit to shiqiangA/vllm-ascend that referenced this pull request Aug 20, 2026
### What this PR does / why we need it?
This PR adds support for GLM-5.2 on Ascend. This pull request updates
`model_returns_tuple` in `llm_base_proposer.py` to support
DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP
recycles the post-final-norm hidden state, its forward pass returns a
tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families
return a single tensor.

vllm's PR: vllm-project/vllm#45895

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@dc68bd8

Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
shiqiangA pushed a commit to shiqiangA/vllm-ascend that referenced this pull request Aug 20, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Leetrytry pushed a commit to Leetrytry/vllm-ascend that referenced this pull request Sep 11, 2026
### What this PR does / why we need it?
This PR adds support for GLM-5.2 on Ascend. This pull request updates
`model_returns_tuple` in `llm_base_proposer.py` to support
DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP
recycles the post-final-norm hidden state, its forward pass returns a
tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families
return a single tensor.

vllm's PR: vllm-project/vllm#45895

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@dc68bd8

Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
Leetrytry pushed a commit to Leetrytry/vllm-ascend that referenced this pull request Sep 11, 2026
### What this PR does / why we need it?
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
- Conditionally implement start_weight_update() and
finish_weight_update() as no-op methods for non-0.23.0 releases.
- Keep the NPU IPC weight transfer engine compatible with the updated
WeightTransferEngine interface.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

---

#### vllm_ascend/patch/platform/patch_torch_accelerator.py
- Redirect torch.accelerator.get_memory_info() to
torch.npu.mem_get_info() on non-0.23.0.
- Avoid crashes caused by the unsupported C10 DeviceAllocator path when
constructing MemorySnapshot.
- Align with the existing NPU-specific memory API patches.
- Upstream source: commit 747b068 (v0.24.0+
MemorySnapshot(device=device) path).

--- 

#### vllm_ascend/patch/worker/patch_qwen3_dflash.py
- Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional
mask embedding download failures.
- Preserve the expected "mask embedding not present" behavior when the
file is unavailable.
- Upstream source: vllm#46104
(vllm-project/vllm#46104).

--- 

#### vllm_ascend/worker/v2/model_runner.py
#### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py
- Forward is_padding and prompt_lens when constructing AscendInputBatch.
- Match the updated upstream InputBatch interface and avoid
initialization failures on newer releases.
- Upstream source: vllm#40654
(vllm-project/vllm#40654).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Add the reduce_results argument to DeepseekV2MLAAttention.
- Forward the parameter to RowParallelLinear to stay compatible with the
updated upstream attention initialization.
- Keep the implementation compatible across all supported vLLM versions.
- Upstream source: vllm#45895
(vllm-project/vllm#45895).

--- 

#### vllm_ascend/distributed/device_communicators/npu_communicator.py
- Register a no-op all2all_manager for NPUCommunicator.
- Bypass the upstream MoE fault-tolerance check (which queries
all2all_manager when data_parallel_size > 1 and is_moe) while preserving
the existing MC2 communication path.
- Keep compatibility with the updated distributed initialization.
- Related upstream changes:
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/ops/fused_moe/fused_moe.py
- Share routed expert parameters through direct nn.Parameter aliasing
instead of creating wrapper parameters.
- Ensure both legacy and routed_experts parameter paths reference the
same underlying weights.
- Apply the aliasing strategy to all routed-expert MoE models on newer
vLLM releases.
- Related upstream changes:
- vllm#40996 (vllm-project/vllm#40996)
- vllm#46892 (vllm-project/vllm#46892)

--- 

#### vllm_ascend/worker/worker.py
#### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py
#### vllm_ascend/distributed/weight_transfer/hccl_engine.py
#### vllm_ascend/patch/platform/patch_weight_transfer_engine.py
- Adapt WeightTransferEngineFactory.create_engine() and
WeightTransferEngine.__init__() to support both legacy and current
upstream signatures.
- Keep weight transfer compatible across v0.23.0, v0.24.0, and newer
upstream releases.
- Upstream source: vllm#44353
(vllm-project/vllm#44353).

--- 

#### vllm_ascend/patch/worker/patch_deepseek_v2.py
- Remove the upstream model-level all-gather path for DeepSeek-V2 on
non-0.23.0.
- Keep the implementation compatible with the Ascend MC2 dispatch flow.
- Avoid tensor shape mismatches and residual concatenation failures
introduced by the upstream refactor.
- Related upstream changes:
- vllm#41184 (vllm-project/vllm#41184)
--- 
#### vllm_ascend/ops/fused_moe/fused_moe.py
- Remove the unnecessary .contiguous() call after weight transposition
on non-0.23.0.
- Reduce transient NPU peak memory during MoE weight loading.
- Prevent OOM caused by duplicate temporary tensor allocations.
- Related upstream changes:
- vllm#44589 (vllm-project/vllm#44589)

- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@ee0da84
---------
Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working deepseek Related to DeepSeek models intel-gpu Related to Intel GPU nvidia ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm speculative-decoding v1

5 participants