Skip to content

[Compressed Tensors] Add XPU wNa16 support - #29484

Merged
jikunshang merged 16 commits into
vllm-project:mainfrom
yiliu30:woq-xpu
Dec 5, 2025
Merged

jikunshang merged 16 commits into
vllm-project:mainfrom
yiliu30:woq-xpu

Conversation

@yiliu30

@yiliu30 yiliu30 commented Nov 26, 2025 •

Copy link
Copy Markdown
Contributor

Purpose

cd vllm
python examples/offline_inference/basic/generate.py  \
    --model Intel/Qwen3-8B-W4A16-G128-AutoRound-LLMC-TEST-ONLY \
    --gpu_memory_utilization 0.75 \
    --enforce-eager 

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for wNa16 compressed tensors on XPU by adding a new IPEXwNa16LinearKernel. The changes are mostly self-contained in a new file and registration of the new kernel. However, I've found a critical issue in the implementation of the new kernel within vllm/model_executor/layers/quantization/kernels/mixed_precision/ipex.py. The input feature size for the underlying IPEX linear layer is calculated incorrectly, which will likely lead to runtime failures or incorrect computations. I have also pointed out a confusing and redundant variable assignment that should be cleaned up. Please see the detailed comments for suggestions on how to fix these issues.

Comment thread vllm/model_executor/layers/quantization/kernels/mixed_precision/ipex.py Outdated
Comment thread vllm/model_executor/layers/quantization/kernels/mixed_precision/ipex.py Outdated
@hshen14

hshen14 commented Nov 26, 2025

Copy link
Copy Markdown

@robertgshaw2-redhat @mgoin Please help review. The PR is to support the quantized models with compressed tensor format (e.g., quantized by LLM-C/AutoRound) for Intel GPUs.

Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
@xuechendi

Copy link
Copy Markdown
Contributor

@mgoin @robertgshaw2-redhat , may you help to take a look?

@mergify mergify Bot added the ci/build label Dec 5, 2025
@jikunshang jikunshang added the ready ONLY add when PR is ready to merge/full CI is needed label Dec 5, 2025

@jikunshang jikunshang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for xpu part.

@jikunshang
jikunshang merged commit 0d8a7d8 into vllm-project:main Dec 5, 2025
53 of 55 checks passed
@mgoin

mgoin commented Dec 9, 2025

Copy link
Copy Markdown
Member

Great work! Sorry for missing this

mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
my-other-github-account pushed a commit to my-other-github-account/vllm that referenced this pull request May 15, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
0826joyce pushed a commit to 0826joyce/vllm-serving-optimization that referenced this pull request May 19, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build ready ONLY add when PR is ready to merge/full CI is needed

5 participants