Skip to content

[Multimodal] Add Qwen2-VL/Qwen2.5-VL processor-mapped video loader#45555

Merged
Isotr0py merged 7 commits into
vllm-project:mainfrom
WindChimeRan:qwen2-vl-video-loader
Jun 21, 2026
Merged

[Multimodal] Add Qwen2-VL/Qwen2.5-VL processor-mapped video loader#45555
Isotr0py merged 7 commits into
vllm-project:mainfrom
WindChimeRan:qwen2-vl-video-loader

Conversation

@WindChimeRan

@WindChimeRan WindChimeRan commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

Qwen2-VL and Qwen2.5-VL currently fall back to the generic opencv video loader (uniform num_frames=32) because they ship no video_processor_type in their preprocessor config.

This adds Qwen2VLVideoBackend, so the loader is auto-selected (#44126). Same pattern as the Qwen3-VL loader (#44412). This will be the baseline groundtruth of #45203

fps takes precedence over num_frames because vLLM's VideoMediaIO always injects num_frames=32, which would otherwise make fps sampling unreachable; default fps=2.

Result

Qwen2.5-VL-7B-Instruct, NExTQA (100) + MVBench (90), offline llm.chat, max_tokens=1, 1×A100, identical engine config (max_pixels=128·28²), back-to-back:

loader NExTQA MVBench overall req/s
opencv num_frames=32 (main default) 0.800 0.611 0.711 2.16
opencv fps=2, cap 768 (same-budget control) 0.840 0.700 0.774 1.45
qwen2_vl fps=2 (this PR) 0.840 0.700 0.774 1.57

Correct sampling gains accuracy (+8.9 pt on MVBench's motion/temporal tasks) at a throughput cost (fps=2 samples more frames than a fixed 32).

Test Plan / Result

Extends test_video_processor_from_model_repo with both repos (exercises the model_type resolver fallback) and adds test_qwen2vl_loader_matches_hf_sample_frames, which asserts the loader's indices equal Qwen2VLVideoProcessor.sample_frames` across an fps/duration grid (including an out-of-range-tail case that exercises the clamp).

pytest tests/multimodal/test_video.py -k qwen2vl -v   # 4 passed
pytest tests/multimodal/test_video.py -q              # passed, no regressions

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
@mergify mergify Bot added multi-modality Related to multi-modality (#4194) qwen Related to Qwen models labels Jun 14, 2026
@WindChimeRan WindChimeRan force-pushed the qwen2-vl-video-loader branch 4 times, most recently from fc7815d to be3f75d Compare June 14, 2026 05:25
@WindChimeRan WindChimeRan marked this pull request as ready for review June 14, 2026 05:37
@WindChimeRan

WindChimeRan commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

@Isotr0py followup of #45203

@Isotr0py Isotr0py self-assigned this Jun 15, 2026
Comment thread vllm/transformers_utils/processor.py Outdated
@mergify

mergify Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @WindChimeRan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 16, 2026
@WindChimeRan WindChimeRan force-pushed the qwen2-vl-video-loader branch from be3f75d to 75e8fd2 Compare June 16, 2026 06:48
@WindChimeRan WindChimeRan requested a review from Isotr0py June 16, 2026 06:48
@mergify mergify Bot removed the needs-rebase label Jun 16, 2026
Comment thread tests/multimodal/test_video.py Outdated
Signed-off-by: Ranran <hzz5361@psu.edu>
@WindChimeRan WindChimeRan requested a review from Isotr0py June 17, 2026 04:25
@Isotr0py Isotr0py enabled auto-merge (squash) June 17, 2026 10:32
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 17, 2026
…ed 32-frame default, so the ~30s test clip needs a larger context.

Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>
auto-merge was automatically disabled June 21, 2026 08:16

Head branch was pushed to by a user without write access

@WindChimeRan WindChimeRan requested a review from noooop as a code owner June 21, 2026 08:16
@WindChimeRan

Copy link
Copy Markdown
Contributor Author

fixed CI tests.

@Isotr0py

Copy link
Copy Markdown
Member

Seems https://buildkite.com/vllm/ci/builds/73265#019ee942-329b-4f80-bfe2-37f276a6d480 failure also related.

[2026-06-21T09:16:35Z] FAILED entrypoints/pooling/classify/test_online_vision.py::test_chat_video_url_request[muziyongshixin/Qwen2.5-VL-7B-for-VideoCls] - AssertionError: assert 8993 == 4807
[2026-06-21T09:16:35Z]  +  where 8993 = UsageInfo(prompt_tokens=8993, total_tokens=8993, completion_tokens=0, prompt_tokens_details=None).prompt_tokens
[2026-06-21T09:16:35Z]  +    where UsageInfo(prompt_tokens=8993, total_tokens=8993, completion_tokens=0, prompt_tokens_details=None) = ClassificationResponse(id='classify-a68fd2ee63ce0dfd', object='list', created=1782032047, model='muziyongshixin/Qwen2.5-VL-7B-for-VideoCls', data=[ClassificationData(index=0, label='LABEL_1', probs=[3.55368274540524e-06, 0.9999964237213135], num_classes=2)], usage=UsageInfo(prompt_tokens=8993, total_tokens=8993, completion_tokens=0, prompt_tokens_details=None)).usage
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py Isotr0py enabled auto-merge (squash) June 21, 2026 17:04
@Isotr0py Isotr0py merged commit 635c383 into vllm-project:main Jun 21, 2026
66 checks passed
tunglinwood pushed a commit to tunglinwood/vllm that referenced this pull request Jun 22, 2026
…llm-project#45555)

Signed-off-by: Ranran <hzz5361@psu.edu>
Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
nkzhenhua pushed a commit to nkzhenhua/vllm that referenced this pull request Jun 24, 2026
…llm-project#45555)

Signed-off-by: Ranran <hzz5361@psu.edu>
Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
qli88 pushed a commit to qli88/vllm that referenced this pull request Jun 26, 2026
…llm-project#45555)

Signed-off-by: Ranran <hzz5361@psu.edu>
Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Qiang Li <qiang.li2@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

multi-modality Related to multi-modality (#4194) qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed

2 participants