[Bugfix] [Parser] Fix empty tool block silently dropping subsequent content

[bbrowning]

Purpose

I regularly run reviews of vLLM code through a variety coding assistants point to self-hosted models. One of my sessions (Codex CLI running w/ Qwen 3.6-27B) picked up a theoretical bug where an empty tool block (ie no parsed function, just open and close tool markers with empty or nonsense content in the middle) would cause all subsequent content to get dropped. So, I had a patch sitting on the side to get around to that.

Today, in another session (Claude Code running w/ Nemotron 3 Super), I hit that theoretical bug as it was reviewing some GLM 4.7 parser changes. The model was tracing some of the parser code paths, and when it started emitting content like <tool_call>func_name<arg_key>key</arg_key><arg_value>value</arg_value></tool_call> it got stuck and didn't emit any more output. That particular output sequence caused the model to output its exact expected <tool_call> token ID and </tool_call> token ID, causing the parser to see start/end tool call but without any meaningful content in the middle. So, inadvertently, I tripped this exact bug.

Now, the parser still parses this as a tool call (because it is the right special tokens, and we're streaming so we cannot just buffer the entirety of the tool call to make an educated guess after we see all of it). But, when the tool call leads to empty content, we keep emitting the rest of the content following the tool call. Previously everything after that bad tool call was dropped from the response.

This PR adds the missing TOOL_PREAMBLE → TOOL_END transitions to Qwen3 and Gemma4 parsers so empty tool blocks (open marker immediately followed by close marker) recover to CONTENT state instead of staying stuck in TOOL_PREAMBLE and silently dropping all subsequent output. It also fixes deferred content release in the engine to emit content when tool events produce no actual deltas. MiniMax M2 already had this transition but gains the new empty-tool-block test scenario.

Test Plan

A test for this scenario was added to trace_builder.py which means we test this behavior for all parsers using the new engine across multiple token per delta sizes. And, all the other unit tests from the impacted parsers were run.

Unit tests

pytest -v \
  tests/parser/engine/ \
  tests/tool_parsers/test_qwen3coder_tool_parser.py \
  tests/tool_parsers/test_gemma4_tool_parser.py \
  tests/tool_parsers/test_minimax_m2_tool_parser.py \
  tests/reasoning/test_qwen3_reasoning_parser.py \
  tests/reasoning/test_gemma4_reasoning_parser.py \
  tests/reasoning/test_minimax_m2_reasoning_parser.py \
  tests/reasoning/test_nemotron_v3_reasoning_parser.py

Test Result

Unit tests

2333 passed, 2 warnings

[sfeng33]

LGTM, thanks for the work!