[Frontend] Replace legacy Gemma4 parsers with engine-based implementation

[bbrowning]

Purpose

Migrate Gemma4 from separate hand-coded reasoning and tool parsers to the unified ParserEngine framework introduced for Qwen3. A single state machine in vllm/parser/gemma4.py now handles both channel-based reasoning extraction and custom tool call parsing with declarative configuration. The streaming engine parser adapters are registered under the existing gemma4 name so no user-facing configuration changes are needed.

This cleans up a multitude of issues with Gemma4 models (including DiffusionGemma) when used with spec decoding, stream_interval > 1, and general handling of reasoning / tool call boundaries, especially in streaming scenarios. The biggest improvement will be felt by users in scenarios like tool call heavy streaming code assistants (Claude Code, Codex CLI, OpenCode, Pi, etc) using spec decoding (such as --speculative-config '{"model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4}' and similar shown in model cards and recipes.vllm.ai). This also makes DiffusionGemma quite usable, as it natively emits multiple tokens at once - similar to if the regular models emitted 256 speculative tokens from a parsing standpoint.

Note that this is a draft until #45413 merges, as it builds on top of it. All commits from that PR are included here for now just so this is a working branch if pulled in isolation. Once 45413 lands, this will show far few code changes because the core engine will already be in.

Test Plan

There are some minor changes to the existing Gemma4 reasoning and tool call parser tests - standardizing on None instead of "" when there is no reasoning or content present (like our other parsers do), adjusting type coercion to the new two-layered approach where we first parse everything to string Python dicts then later coerce consistently across all engine-based parsers, and minor assertion changes where the new parser handles some cases that the previous one could not, such as partial tool calls.

There's also comprehensive new tests specific to the Gemma4 engine parser to test all kinds of real-world scenarios I encountered while testing this on live servers.

Unit Tests

pytest \
  tests/tool_use/test_gemma4_responses_adjust_request.py \
  tests/parser/engine/test_gemma4_streaming_reasoning.py \
  tests/parser/engine/test_delegating_replay.py \
  tests/parser/engine/test_replay.py \
  tests/reasoning/test_gemma4_reasoning_parser.py \
  tests/tool_parsers/test_gemma4_tool_parser.py

Test Result

Unit Tests

515 passed, 19 warnings

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @bbrowning.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[schoennenbeck]

Looking forward to this release. The combination of tool calling, reasoning and MTP for Gemma4 is currently incredibly broken and stopping me from actually using MTP which does give a very nice performance boost. Thanks for implementing this.

[bbrowning]

@schoennenbeck Have you been able to try the updated Gemma4 parser from this branch at all? I'm running it through a lot of testing myself, but other feedback is always valuable 😄

[bbrowning]

Pulling this out of draft, as the streaming parser engine core has merged and I've rebased this on top and tested it with an assortment of coding assistants with large stream_interval and MTP settings just to make sure things look good.

Note that this touches more files and has more changes than other future parsers will, as it handles another set of edge cases in the streaming parser engine (reasoning -> tool transition under a very specific circumstance) as well as an ability to notice and recover from part or all of the reasoning start token patterns missing in Gemma4 as long as we can notice the miss before we start streaming content back as reasoning. There was an existing Gemma4 test that tests for this, and I wanted to be able to handle that without loosening that test assertions. And, in the real world, sometimes I see the model generate thought instead of <|channel>thought in longer multi-turn scenarios which this now handles via that mechanism.

[sfeng33]

I've had agent run through all the open issues/prs for gemma4, and these are the issues identified, do you think if any of these is worth fixing in this pr?

Gap 1: Dict key <|"|> delimiter stripping (HIGH)

Issues: #44715 — 4+ duplicate PRs (#44811, #44877, #44717, #44716)

File: vllm/parser/gemma4.py, _parse_gemma4_args() lines 128-133

Bug: Key parsing scans for : separator but doesn't strip <|"|> from keys. When model emits <|"|>key_name<|"|>:value, the key becomes <|"|>key_name<|"|> instead of key_name.

Fix: Before scanning for :, check if position starts with STRING_DELIM. If so, extract key by scanning for closing delimiter (same approach as value parsing). Falls back to bare key scan for unquoted keys.

Gap 2: is_reasoning_end() returns False when thinking never started (MEDIUM-HIGH)

Issue: #39130

File: vllm/parser/gemma4.py, Gemma4Parser.is_reasoning_end() line 500

Bug: When enable_thinking=false, generated tokens contain no reasoning markers. The backward scan finds nothing and falls through to self._reasoning_ended (False). This blocks xgrammar structured output from activating.

Fix: Change the fallthrough from return self._reasoning_ended to return True. Rationale: if no reasoning/tool/turn markers exist in the generated tokens, thinking was never started, so it's "ended" for structured output purposes. This is safe because:

• If reasoning is active (<|channel> found), scan returns False correctly
• If reasoning ended (<channel|> found), scan returns True correctly
• If tool call is active (<|tool_call> found), scan returns True correctly
• If no markers at all → thinking never started → return True

Also update the base ParserEngine.is_reasoning_end() at line 511 to match: change return False to return True.

[bbrowning]

I pulled in a fix for #44715 since that bug is also present here, given that we just copied over how the Gemma4 args get parsed from the old way. I took inspiration for the fix from one of the mentioned PRs and inspiration for the tests from another, so added co-authorship to that last commit for both of those.

Will take a look at #39130 to see if it's already fixed here or if I need to add something.

[bbrowning]

Gap 2: is_reasoning_end() returns False when thinking never started (MEDIUM-HIGH)

Issue: #39130

This one is a bit trickier. We can do better here when enable_thinking=False and signal our default notion of reasoning ended or not based on that. That won't handle 100% of possible cases, mainly because when thinking is enabled the Gemma 4 chat templates don't add the start thinking token (they let the model decide to think or not) so even when thinking is enabled we don't reliably know if it's going to think (and thus whether the thinking has ended). But, we could get a lot closer to right out of the box for the majority of cases by something like this I think, but needs more testing and I'd prefer to do this as a follow-up or review what others do here.

diff --git a/vllm/parser/gemma4.py b/vllm/parser/gemma4.py
index d8bdc2eca..cb12b35c6 100644
--- a/vllm/parser/gemma4.py
+++ b/vllm/parser/gemma4.py
@@ -423,6 +423,8 @@ class Gemma4Parser(ParserEngine):
         tools: list[Tool] | None = None,
         **kwargs,
     ) -> None:
+        chat_kwargs = kwargs.get("chat_template_kwargs", {}) or {}
+        self._thinking_enabled = chat_kwargs.get("enable_thinking", False)
         super().__init__(
             tokenizer,
             tools,
@@ -494,9 +496,9 @@ class Gemma4Parser(ParserEngine):
             if tool_call_id is not None and tid == tool_call_id:
                 return True
             if new_turn_id is not None and tid == new_turn_id:
-                return False
+                return not self._thinking_enabled
             if tool_response_id is not None and tid == tool_response_id:
-                return False
+                return not self._thinking_enabled
             if end_id is not None and tid == end_id:
                 return True
         return self._reasoning_ended

That's not fully tested, and I'd want to test something like that with live versions of the smaller E2B/E4B as well as the larger 26B/31B models with various combinations of enable_thinking to ensure it works for most cases.

[sfeng33]

Thanks for the work!

[schoennenbeck]

@schoennenbeck Have you been able to try the updated Gemma4 parser from this branch at all? I'm running it through a lot of testing myself, but other feedback is always valuable 😄

Haven't had the chance yet but will do so as soon as I have the time.

[m4r1k]

Thanks for #45588. I verified it fixes the streaming auto tool-call corruption under MTP from #41967.
While testing on HEAD I hit a separate regression in the required / named tool-choice paths that I don't see reported yet.

Environment

• vLLM 0.23.1rc1.dev48+g8bf374955 (HEAD 8bf374955, built with VLLM_USE_PRECOMPILED=1)
• google/gemma-4-31B-it (fp8), MTP draft google/gemma-4-31B-it-assistant (num_speculative_tokens=8)
• --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4
• 1x NVIDIA RTX PRO 6000
• temperature=0

What works

• tool_choice="auto" is correct, streaming and non-streaming (this is [Bug]: Gemma4 + MTP speculative decoding drops first tool-call arguments in streaming multi-tool auto-tool-choice #41967, and it's fixed: 5/5 two-tool streaming repros reconstruct both calls, MTP active ~90% acceptance).

Regression (required and named)

With tool_choice="required", the model emits the forced FunctionDefinition JSON, but it comes back as content and tool_calls is empty:

curl -s $BASE/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{
  "model":"google/gemma-4-31B-it","tool_choice":"required",
  "messages":[{"role":"user","content":"What is the weather in Milano and Roma?"}],
  "tools":[{"type":"function","function":{"name":"get_weather","description":"Weather for a city",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}'

{"choices":[{"finish_reason":"tool_calls","message":{
  "content":"[\n  {\"name\":\"get_weather\",\"parameters\":{\"city\":\"Milano\"}},\n  {\"name\":\"get_weather\",\"parameters\":{\"city\":\"Roma\"}}\n]",
  "tool_calls":[]}}]}

Same for named choice (tool_calls=[], args JSON in content), and for both stream=true and stream=false. Expected: tool_calls populated, content empty.

For Gemma4 the engine adapter sets supports_required_and_named = False. In vllm/parser/abstract_parser.py, required/named with that flag false is routed into the is_auto_tool_choice fallback, which calls extract_tool_calls(), the native Gemma4 parser that only understands <|tool_call>call:.... But the request still gets the forced FunctionDefinition JSON structured output, and the branch that would parse that JSON into tool calls is gated on supports_required_and_named being true, so it is skipped.
TL;DR JSON is generated, the native parser cannot read it, and it leaks to content.

The pre-#45588 legacy Gemma4ToolParser avoided this: its adjust_request skipped the JSON structured-output for required/named so the model emitted the native <|tool_call> format, which the native parser then extracted. The new Gemma4Parser.adjust_request at vllm/parser/gemma4.py no longer does this, so required/named regressed. On eb28452b1 (the pre-#45588 fix) the same request returned proper tool_calls.

Fix

I opened a PR with the fix: #45795. It parses the forced FunctionDefinition JSON in the not-supported required/named fallback in vllm/parser/abstract_parser.py, with a fallback to native extraction when the content is not that JSON (so native-emitting parsers like mistral are unaffected). It adds required/named regression tests in tests/parser/test_parse.py for a parser with supports_required_and_named=False that fail before the change and pass after.

Check	Before	After
tool_choice="required" + MTP	tool_calls=[], JSON in content	get_weather(Milano) + get_weather(Roma)
named tool choice + MTP	tool_calls=[]	get_weather(Torino)
#41967 streaming auto (x5)	5/5	5/5
tests/parser/test_parse.py	14 passed	16 passed (2 new)
the 2 new regression tests	2 failed	2 passed (they catch the bug)
gemma4 parser unit suite	1353 passed	1353 passed

@schoennenbeck @bbrowning PTAL 🙏

[tolgayan]

Thanks for landing this. This looks like it fixes the blocking Gemma4 26b-a4b-it tool-calling issue in vLLM + multi-turn agent flows.

Since v0.23.0 was cut just before this merged, would you consider cherry-picking it into a v0.23.x patch release? This would unblock reliable Gemma4 tool calling for many developers.

[yichieh-lu]

+1 to @tolgayan - I also think a v0.23.x patch release with this fix would be a great help for many developers. Thanks for all the hard work!

[berkayersoyy]

+1 to @tolgayan 🆙

[m4r1k]

My two PRs #45795 and #45832 fixing tool usage got in yesterday, they are worth cherry-picking into a v0.23.x patch release as well