[bugfix]Indexer init skip and MTP TopK share for iteration - #45895
Merged
Merged
Conversation
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: JaredforReal <w13431838023@gmail.com>
JaredforReal
requested review from
AndreasKaratzas,
LucasWilkinson,
MatthewBonanni,
ProExpertProg,
WoosukKwon,
benchislett,
houseroad,
luccafong,
mgoin,
pavanimajety,
robertgshaw2-redhat,
tjtanaa,
tlrmchlsmth,
yewentao256 and
youkaichao
as code owners
June 17, 2026 04:52
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR adds support for GLM-DSA MTP “post-final-norm hidden recycling” in speculative decoding and refactors sparse MLA attention to consume a shared top‑k indices buffer explicitly (enabling “skip” layers without an indexer).
Changes:
- Add
mtp_recycle_post_normplumbing across config override, draft model (MTP) forward output shape, and proposer tuple-handling logic. - Pass
topk_indices_bufferexplicitly through MLA layers/attention into sparse MLA backends, decoupling them fromindexer. - Refine DeepSeek v3.2 IndexCache “skip topk” logic to always build indexers for MTP/nextn layers while optionally skipping some backbone layers.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| vllm/v1/spec_decode/llm_base_proposer.py | Detects GLM-DSA recycling and adjusts whether MTP returns tuples. |
| vllm/v1/attention/backends/mla/xpu_mla_sparse.py | Switches sparse MLA backend to take a shared topk_indices_buffer directly. |
| vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py | Same as above for ROCm aiter sparse MLA backend. |
| vllm/v1/attention/backends/mla/flashmla_sparse.py | Same buffer wiring change for FlashMLA sparse backend. |
| vllm/v1/attention/backends/mla/flashinfer_mla_sparse.py | Same buffer wiring change, removing the “indexer required” assertion. |
| vllm/model_executor/models/deepseek_v2.py | Updates IndexCache skip/topk pattern logic and MTP-layer exceptions. |
| vllm/model_executor/models/deepseek_mtp.py | Adds optional tuple return for GLM-DSA recycling behavior. |
| vllm/model_executor/layers/mla.py | Plumbs topk_indices_buffer into the MLA attention layer. |
| vllm/model_executor/layers/attention/mla_attention.py | For sparse MLA, passes topk_indices_buffer down to backend impl args. |
| vllm/config/speculative.py | Sets mtp_recycle_post_norm based on original GLM model_type. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1004
to
+1007
| _index_topk_freq = getattr(config, "index_topk_freq", 1) | ||
| _index_topk_pattern = getattr(config, "index_topk_pattern", None) | ||
| _index_skip_topk_offset = getattr(config, "index_skip_topk_offset", 2) | ||
| layer_id = extract_layer_index(prefix) |
Comment on lines
+1010
to
+1012
| _skip_topk = ( | ||
| max(layer_id - _index_skip_topk_offset + 1, 0) % _index_topk_freq != 0 | ||
| ) |
Comment on lines
+441
to
+444
| # Sparse MLA reads top-k indices from a shared buffer. Pass it | ||
| # explicitly so backbone "skip" layers (indexer=None) still find it. | ||
| if use_sparse: | ||
| extra_impl_args["topk_indices_buffer"] = topk_indices_buffer |
|
|
||
| assert indexer is not None, "Indexer required for sparse MLA" | ||
| self.topk_indices_buffer: torch.Tensor | None = indexer.topk_indices_buffer | ||
| self.topk_indices_buffer: torch.Tensor | None = topk_indices_buffer |
| kv_sharing_target_layer_name: str | None, | ||
| # MLA Specific Arguments | ||
| topk_indice_buffer: torch.Tensor | None = None, | ||
| topk_indices_buffer: torch.Tensor | None = None, |
Signed-off-by: JaredforReal <w13431838023@gmail.com>
1 task done
realliujiaxu
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 2, 2026
### What this PR does / why we need it? This PR adds support for GLM-5.2 on Ascend. This pull request updates `model_returns_tuple` in `llm_base_proposer.py` to support DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP recycles the post-final-norm hidden state, its forward pass returns a tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families return a single tensor. vllm's PR: vllm-project/vllm#45895 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
4 tasks
MengqingCao
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 6, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
wangyichao1999
pushed a commit
to wangyichao1999/vllm-ascend
that referenced
this pull request
Jul 9, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Spicy-Stick
pushed a commit
to Spicy-Stick/vllm-ascend
that referenced
this pull request
Jul 10, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com> Signed-off-by: Spicy-Stick <873805887@qq.com>
xqchen7
pushed a commit
to nv-action/vllm-benchmarks
that referenced
this pull request
Jul 15, 2026
### What this PR does / why we need it? This PR adds support for GLM-5.2 on Ascend. This pull request updates `model_returns_tuple` in `llm_base_proposer.py` to support DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP recycles the post-final-norm hidden state, its forward pass returns a tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families return a single tensor. vllm's PR: vllm-project/vllm#45895 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com> Signed-off-by: xqchen7 <chenxueqing7@huawei.com>
xqchen7
pushed a commit
to nv-action/vllm-benchmarks
that referenced
this pull request
Jul 15, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com> Signed-off-by: xqchen7 <chenxueqing7@huawei.com>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
…ect#45895) Signed-off-by: JaredforReal <w13431838023@gmail.com>
Liamup777
pushed a commit
to Liamup777/vllm-ascend
that referenced
this pull request
Jul 20, 2026
### What this PR does / why we need it? This PR adds support for GLM-5.2 on Ascend. This pull request updates `model_returns_tuple` in `llm_base_proposer.py` to support DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP recycles the post-final-norm hidden state, its forward pass returns a tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families return a single tensor. vllm's PR: vllm-project/vllm#45895 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
6 of 12 tasks
Alex-stack-hub
pushed a commit
to 0moyi0-2024/vllm-ascend_tp
that referenced
this pull request
Jul 27, 2026
### What this PR does / why we need it? This PR adds support for GLM-5.2 on Ascend. This pull request updates `model_returns_tuple` in `llm_base_proposer.py` to support DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP recycles the post-final-norm hidden state, its forward pass returns a tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families return a single tensor. vllm's PR: vllm-project/vllm#45895 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
Alex-stack-hub
pushed a commit
to 0moyi0-2024/vllm-ascend_tp
that referenced
this pull request
Jul 27, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
1 task
1 of 13 tasks
21 tasks
eble-amd
pushed a commit
to ROCm/vllm
that referenced
this pull request
Aug 14, 2026
…ect#45895) Signed-off-by: JaredforReal <w13431838023@gmail.com>
MmMmaru
pushed a commit
to jiaqi-lee/vllm-ascend
that referenced
this pull request
Aug 19, 2026
### What this PR does / why we need it? This PR adds support for GLM-5.2 on Ascend. This pull request updates `model_returns_tuple` in `llm_base_proposer.py` to support DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP recycles the post-final-norm hidden state, its forward pass returns a tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families return a single tensor. vllm's PR: vllm-project/vllm#45895 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
MmMmaru
pushed a commit
to jiaqi-lee/vllm-ascend
that referenced
this pull request
Aug 19, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
shiqiangA
pushed a commit
to shiqiangA/vllm-ascend
that referenced
this pull request
Aug 20, 2026
### What this PR does / why we need it? This PR adds support for GLM-5.2 on Ascend. This pull request updates `model_returns_tuple` in `llm_base_proposer.py` to support DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP recycles the post-final-norm hidden state, its forward pass returns a tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families return a single tensor. vllm's PR: vllm-project/vllm#45895 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
shiqiangA
pushed a commit
to shiqiangA/vllm-ascend
that referenced
this pull request
Aug 20, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
Leetrytry
pushed a commit
to Leetrytry/vllm-ascend
that referenced
this pull request
Sep 11, 2026
### What this PR does / why we need it? This PR adds support for GLM-5.2 on Ascend. This pull request updates `model_returns_tuple` in `llm_base_proposer.py` to support DeepSeek-family MTP models (`DeepSeekMTPModel`). Since DeepSeek MTP recycles the post-final-norm hidden state, its forward pass returns a tuple `(logit_hidden, recycle_hidden)`, whereas other MTP families return a single tensor. vllm's PR: vllm-project/vllm#45895 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: jiajinzhu2 <jiajinzhu@huawei.com>
Leetrytry
pushed a commit
to Leetrytry/vllm-ascend
that referenced
this pull request
Sep 11, 2026
### What this PR does / why we need it? #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py - Conditionally implement start_weight_update() and finish_weight_update() as no-op methods for non-0.23.0 releases. - Keep the NPU IPC weight transfer engine compatible with the updated WeightTransferEngine interface. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/platform/patch_torch_accelerator.py - Redirect torch.accelerator.get_memory_info() to torch.npu.mem_get_info() on non-0.23.0. - Avoid crashes caused by the unsupported C10 DeviceAllocator path when constructing MemorySnapshot. - Align with the existing NPU-specific memory API patches. - Upstream source: commit 747b068 (v0.24.0+ MemorySnapshot(device=device) path). --- #### vllm_ascend/patch/worker/patch_qwen3_dflash.py - Wrap DFlashQwen3ForCausalLM._read_mask_embedding() to ignore optional mask embedding download failures. - Preserve the expected "mask embedding not present" behavior when the file is unavailable. - Upstream source: vllm#46104 (vllm-project/vllm#46104). --- #### vllm_ascend/worker/v2/model_runner.py #### vllm_ascend/patch/worker/patch_v2/patch_input_batch.py - Forward is_padding and prompt_lens when constructing AscendInputBatch. - Match the updated upstream InputBatch interface and avoid initialization failures on newer releases. - Upstream source: vllm#40654 (vllm-project/vllm#40654). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Add the reduce_results argument to DeepseekV2MLAAttention. - Forward the parameter to RowParallelLinear to stay compatible with the updated upstream attention initialization. - Keep the implementation compatible across all supported vLLM versions. - Upstream source: vllm#45895 (vllm-project/vllm#45895). --- #### vllm_ascend/distributed/device_communicators/npu_communicator.py - Register a no-op all2all_manager for NPUCommunicator. - Bypass the upstream MoE fault-tolerance check (which queries all2all_manager when data_parallel_size > 1 and is_moe) while preserving the existing MC2 communication path. - Keep compatibility with the updated distributed initialization. - Related upstream changes: - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Share routed expert parameters through direct nn.Parameter aliasing instead of creating wrapper parameters. - Ensure both legacy and routed_experts parameter paths reference the same underlying weights. - Apply the aliasing strategy to all routed-expert MoE models on newer vLLM releases. - Related upstream changes: - vllm#40996 (vllm-project/vllm#40996) - vllm#46892 (vllm-project/vllm#46892) --- #### vllm_ascend/worker/worker.py #### vllm_ascend/distributed/weight_transfer/npu_ipc_engine.py #### vllm_ascend/distributed/weight_transfer/hccl_engine.py #### vllm_ascend/patch/platform/patch_weight_transfer_engine.py - Adapt WeightTransferEngineFactory.create_engine() and WeightTransferEngine.__init__() to support both legacy and current upstream signatures. - Keep weight transfer compatible across v0.23.0, v0.24.0, and newer upstream releases. - Upstream source: vllm#44353 (vllm-project/vllm#44353). --- #### vllm_ascend/patch/worker/patch_deepseek_v2.py - Remove the upstream model-level all-gather path for DeepSeek-V2 on non-0.23.0. - Keep the implementation compatible with the Ascend MC2 dispatch flow. - Avoid tensor shape mismatches and residual concatenation failures introduced by the upstream refactor. - Related upstream changes: - vllm#41184 (vllm-project/vllm#41184) --- #### vllm_ascend/ops/fused_moe/fused_moe.py - Remove the unnecessary .contiguous() call after weight transposition on non-0.23.0. - Reduce transient NPU peak memory during MoE weight loading. - Prevent OOM caused by duplicate temporary tensor allocations. - Related upstream changes: - vllm#44589 (vllm-project/vllm#44589) - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@ee0da84 --------- Signed-off-by: hfadzxy <starmoon_zhang@163.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
fix GLM-5.2 BF16 init Indexer in skip_topk layer
fix GLM-5.2 MTP Port-Norm cycle
Test Plan
Test Result
MTP Mean Acceptance Length raise form 3~ to 4~, while Mean Acceptance Rate raise to 60%
IFBench remain 74.62
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.