[Rust Frontend] Add Python bridge for Rust tool parsers

[BugenZhao]

Purpose

This PR adds a generic PyO3 bridge that exposes Rust tool parsers to Python through vllm._rust_tool_parser, along with a Python RustToolParser adapter for the existing vLLM ToolParser interface.

This is not yet wired into any production parser path. The intent is to provide a future extension point where Python adapters can delegate parser state and grammar to Rust while keeping protocol adaptation in Python.

This updates the existing PR for this Rust tool parser bridge work rather than opening a duplicate PR. AI assistance was used.

Tests

Verified locally:

• VLLM_RS_TARGET_PATH=/tmp/vllm-rust-artifacts/vllm-rs VLLM_RUST_EXTENSION_TARGET_DIR=/tmp/vllm-rust-artifacts bash build_rust.sh - passed; produced vllm-rs and _rust_tool_parser.abi3.so.
• Temporary import smoke for vllm._rust_tool_parser using the built _rust_tool_parser.abi3.so - passed.
• cargo test --manifest-path rust/Cargo.toml -p vllm-tool-parser-py - passed.
• .venv/bin/pytest -q tests/tool_parsers/test_rust_tool_parser.py - passed.
• git diff --check - passed.

Verified in Buildkite build 70210 before the DCO-only commit-message rewrite:

• docker-build-image - passed; log confirmed _rust_tool_parser.abi3.so was added to the wheel.
• docker-smoking-non-root-smoke-tests - passed.
• async-engine-inputs-utils-worker-config-cpu reached tests/tool_parsers/test_rust_tool_parser.py; all four Rust tool parser Python tests passed before tokenizers_ started.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

[Harry-Chen]

Please compile with python limited API just like other native extensions.

You can try adding something like:

https://github.com/vllm-project/router/blob/e667ebba0fa7f9f1132ffc6ab8e3386b6a06e1ab/pyproject.toml#L47-L48

I'm not sure whether this works well with vllm's complex packaging scripts.

[mergify]

Hi @BugenZhao, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[depthfirst-app]

⚪ Severity: LOW

After the Rust parse_into raises an exception (line 310-316), execution falls through to process the potentially partially-mutated parser_output. Unlike _parse_complete (which correctly returns None inside its except block), here the partial/corrupt parser output is forwarded as a valid tool call delta. This can cause incomplete tool calls (e.g., truncated arguments) to be streamed to downstream tool executors.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Add return None at the end of the except block (after the logger.exception call) to prevent fall-through to processing the potentially partially-mutated parser_output. This matches the pattern used in _parse_complete (line 226) which correctly returns None inside its except block.

⚠️ Experimental Feature: This code suggestion is automatically generated. Please review carefully.

Suggested change

	except Exception as error:
	self._error = error
	logger.exception(
	"Error parsing %s streaming tool call output.",
	self.rust_parser_name,
	)
	delta_message = self._delta_message_from_parser_output(parser_output)
	except Exception as error:
	self._error = error
	logger.exception(
	"Error parsing %s streaming tool call output.",
	self.rust_parser_name,
	)
	return None
	delta_message = self._delta_message_from_parser_output(parser_output)