[Migration] Migrate GGUF quantization support to plugin by Isotr0py · Pull Request #39612 · vllm-project/vllm

Isotr0py · 2026-04-12T06:34:14Z

Purpose

After this PR, GGUF support will be migrated to https://github.com/vllm-project/vllm-gguf-plugin, you can still use GGUF models normally after plugin installation!

uv pip install vllm-gguf-plugin

Related issues: [RFC]: Migrate bitsandbytes and GGUF quantization support to OOT plugin #39583 and [RFC]: Decouple vllm-omni GGUF quantization from upstream vllm#39583 vllm-omni#2700
~~This is a draft PR used to assist with GGUF plugin development.~~

Test Plan

uv pip install "vllm-gguf-plugin>=0.0.2"
pytest -s -v tests/plugins_tests/gguf/

Test Result

All tests pass.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify · 2026-04-12T06:34:51Z

Documentation preview: https://vllm--39612.org.readthedocs.build/en/39612/

gemini-code-assist

Code Review

This pull request removes hardcoded GGUF support from the core vLLM codebase and replaces it with a more extensible ModelFormatHandler architecture. The changes involve deleting GGUF-specific CUDA kernels, documentation, and tests, while refactoring model loaders and layers (Linear, MoE, Embedding) to use generic quantization configuration hooks. I have no feedback to provide.

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify · 2026-04-13T21:23:52Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify · 2026-05-07T14:28:09Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Isotr0py · 2026-06-12T16:04:46Z

@mgoin The GGUF plugin test pass now: https://buildkite.com/vllm/ci/builds/71683#019ebaa4-74dd-4462-9b0b-24cff7c5bbf3, perhaps it's safe to merge this PR to catch up the v0.23 release now?

mgoin

LGTM, great work getting this all together. I still do think it would be good to keep it installed by default to avoid breaking existing flows, but usage is not that high

…ddings After vllm-project#39612, ParallelLMHead.tie_weights delegates to quant_method.tie_weights and QuantizeMethodBase.tie_weights raised NotImplementedError. Only UnquantizedEmbeddingMethod implemented it, so a model with tied word embeddings quantized via ModelOpt (NVFP4/FP8) -- or whose lm_head is excluded and gets UnquantizedLinearMethod -- crashed at load with NotImplementedError (e.g. the Gemma family). Default the base to sharing the weight tensor, matching the behavior ParallelLMHead.tie_weights had directly before vllm-project#39612; methods needing special handling (e.g. GGUF) still override. Fixes vllm-project#45543. Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com> (cherry picked from commit 9752054)

…#39612) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

…ddings After vllm-project#39612, ParallelLMHead.tie_weights delegates to quant_method.tie_weights and QuantizeMethodBase.tie_weights raised NotImplementedError. Only UnquantizedEmbeddingMethod implemented it, so a model with tied word embeddings quantized via ModelOpt (NVFP4/FP8) -- or whose lm_head is excluded and gets UnquantizedLinearMethod -- crashed at load with NotImplementedError (e.g. the Gemma family). Default the base to sharing the weight tensor, matching the behavior ParallelLMHead.tie_weights had directly before vllm-project#39612; methods needing special handling (e.g. GGUF) still override. Fixes vllm-project#45543. Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com>

…ddings After vllm-project#39612, ParallelLMHead.tie_weights delegates to quant_method.tie_weights and QuantizeMethodBase.tie_weights raised NotImplementedError. Only UnquantizedEmbeddingMethod implemented it, so a model with tied word embeddings quantized via ModelOpt (NVFP4/FP8) -- or whose lm_head is excluded and gets UnquantizedLinearMethod -- crashed at load with NotImplementedError (e.g. the Gemma family). Default the base to sharing the weight tensor, matching the behavior ParallelLMHead.tie_weights had directly before vllm-project#39612; methods needing special handling (e.g. GGUF) still override. Fixes vllm-project#45543. Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com> (cherry picked from commit 9752054)

…ddings After vllm-project#39612, ParallelLMHead.tie_weights delegates to quant_method.tie_weights and QuantizeMethodBase.tie_weights raised NotImplementedError. Only UnquantizedEmbeddingMethod implemented it, so a model with tied word embeddings quantized via ModelOpt (NVFP4/FP8) -- or whose lm_head is excluded and gets UnquantizedLinearMethod -- crashed at load with NotImplementedError (e.g. the Gemma family). Default the base to sharing the weight tensor, matching the behavior ParallelLMHead.tie_weights had directly before vllm-project#39612; methods needing special handling (e.g. GGUF) still override. Fixes vllm-project#45543. Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com>

…ddings After vllm-project#39612, ParallelLMHead.tie_weights delegates to quant_method.tie_weights and QuantizeMethodBase.tie_weights raised NotImplementedError. Only UnquantizedEmbeddingMethod implemented it, so a model with tied word embeddings quantized via ModelOpt (NVFP4/FP8) -- or whose lm_head is excluded and gets UnquantizedLinearMethod -- crashed at load with NotImplementedError (e.g. the Gemma family). Default the base to sharing the weight tensor, matching the behavior ParallelLMHead.tie_weights had directly before vllm-project#39612; methods needing special handling (e.g. GGUF) still override. Fixes vllm-project#45543. Signed-off-by: Mike G <180722391+mikekg@users.noreply.github.com> (cherry picked from commit 9752054)

draft gguf remove

937eae3

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot added documentation Improvements or additions to documentation ci/build llama Related to Llama models multi-modality Related to multi-modality (#4194) rocm Related to AMD ROCm v1 labels Apr 12, 2026

github-project-automation Bot added this to AMD Apr 12, 2026

github-project-automation Bot moved this to Todo in AMD Apr 12, 2026

gemini-code-assist Bot reviewed Apr 12, 2026

View reviewed changes

Isotr0py added 4 commits April 12, 2026 14:44

remove gguf materialization

642e1ec

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean

1b53ba7

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

remove gguf materialization

6332347

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean

a270d85

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

This was referenced Apr 12, 2026

Initialize plugin implementation vllm-project/vllm-gguf-plugin#1

Merged

[RFC]: Migrate bitsandbytes and GGUF quantization support to OOT plugin #39583

Open

mergify Bot added the needs-rebase label Apr 13, 2026

sync main

f4e25f6

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot removed the needs-rebase label May 5, 2026

workaround tie words embedding

4d940cb

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot added the needs-rebase label May 7, 2026

Isotr0py added 6 commits May 8, 2026 23:25

remove siglip maybe_swap_ffn_param

83fe14d

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

pass quant_config

df565fa

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean

ca76814

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean openpangu

4b1e2d0

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean dead code

070ce48

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

clean MoE weight loader

de99b2b

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Isotr0py requested a review from khluu as a code owner June 11, 2026 13:07

Isotr0py added 5 commits June 11, 2026 22:01

add soft failed gguf plugin tests

9942842

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

fix relative import

e21919f

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

use olmoe Q6_K

e62257b

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

only download one gemma3 model

0e2cd2c

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Merge remote-tracking branch 'upstream/main' into remove-gguf

11ec61e

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

mergify Bot removed the needs-rebase label Jun 11, 2026

Isotr0py added 4 commits June 11, 2026 23:06

add tp test

c4e048a

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

increase timeout minutes

84fe076

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

ooops

71e7843

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Merge branch 'main' into remove-gguf

4750949

mgoin approved these changes Jun 12, 2026

View reviewed changes

vllm-bot merged commit 6635279 into vllm-project:main Jun 12, 2026
189 of 192 checks passed

github-project-automation Bot moved this from Todo to Done in AMD Jun 12, 2026

Oxygen56 mentioned this pull request Jun 13, 2026

Bugfix: Accept file-type-only quant types (IQ2_M, IQ3_XS, MXFP4_MOE) in remote GGUF model IDs #44235

Closed

This was referenced Jun 13, 2026

[Bug]: Tied word embeddings (tie_word_embeddings) crash with NotImplementedError for ModelOpt-quantized models after #39612 #45543

Closed

[Bugfix] Default tie_weights to sharing the weight (fix tied quantized embeddings, e.g. ModelOpt Gemma4) #45544

Merged

Saddss pushed a commit to Saddss/vllm that referenced this pull request Jun 14, 2026

[Migration] Migrate GGUF quantization support to plugin (vllm-project…

351f22a

…#39612) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

hmellor mentioned this pull request Jun 30, 2026

[Feature] Improve GGUF loading from HuggingFace user experience #25340

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Migration] Migrate GGUF quantization support to plugin#39612

[Migration] Migrate GGUF quantization support to plugin#39612
vllm-bot merged 46 commits into
vllm-project:mainfrom
Isotr0py:remove-gguf

Isotr0py commented Apr 12, 2026 •

edited

Loading

mergify Bot commented Apr 12, 2026

gemini-code-assist Bot left a comment

mergify Bot commented Apr 13, 2026

mergify Bot commented May 7, 2026

Isotr0py commented Jun 12, 2026

mgoin left a comment •

edited

Loading

Uh oh!

Labels

4 participants

Uh oh!

Uh oh!

Conversation

Isotr0py commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

mergify Bot commented Apr 12, 2026

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

mergify Bot commented Apr 13, 2026

mergify Bot commented May 7, 2026

Isotr0py commented Jun 12, 2026

mgoin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Labels

4 participants

Isotr0py commented Apr 12, 2026 •

edited

Loading

mgoin left a comment •

edited

Loading