Skip to content

Batch path nests tools inside generationConfig, producing an invalid batch request #532

Description

@shoemoney

Environment: langextract at HEAD 9853e44, Vertex AI backend, batch mode.

Bug: GeminiLanguageModel with tools configured produces an invalid batch request. The batch branch in langextract/providers/gemini.py pops system_instruction and safety_settings out of the config to place them at the request top level, but never pops tools (allowlisted in _API_CONFIG_KEYS at gemini.py:112 and documented in the constructor as forwarded to the API). gemini_batch._build_request then camelizes the leftover key into generationConfig, producing:

{"generationConfig": {"temperature": 0.0, "tools": [{"google_search": {}}], "candidateCount": 1}}

In the GenerateContentRequest schema, tools is a top-level request field like systemInstruction and safetySettings, not a GenerationConfig field, so Vertex batch validation rejects the request. With ignore_item_errors the outputs silently pad to empty strings instead. The identical model config works on the realtime path because generate_content(config=call_config) accepts tools at that level.

Repro (no API call needed):

from langextract.providers import gemini_batch
gen_config = {"temperature": 0.0, "tools": [{"google_search": {}}], "candidate_count": 1}
req = gemini_batch._build_request(prompt="p", gen_config=gen_config, ...)
# req["generationConfig"] contains "tools"; the schema wants it at req top level

Trigger: GeminiLanguageModel(vertexai=True, project=..., location=..., tools=[{"google_search": {}}], batch={"threshold": 1}) then extract/infer over >= threshold prompts.

Fix shape: pop tools in the batch branch next to the system_instruction/safety_settings pops and pass it through infer_batch into _build_request, which sets request["tools"] at the top level, mirroring the existing systemInstruction/safetySettings handling. Every other _API_CONFIG_KEYS member left in the dict is a legitimate generationConfig field; tools is the single misplaced one.

I have this fix implemented with a regression test and can open a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions