Skip to content

[Serve][4/4] Add build_openai_applications for multi-model direct streaming - #66520

Draft
celinaky wants to merge 1 commit into
celina/router-app-3-llm-router-controlfrom
celina/router-app-4-openai-applications
Draft

celinaky wants to merge 1 commit into
celina/router-app-3-llm-router-controlfrom
celina/router-app-4-openai-applications

Conversation

@celinaky

@celinaky celinaky commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Why this is needed

This PR adds a public (alpha) builder that supports multi-model direct streaming via a multiple model applications supervised by a router application and a control application.

Summary

ray.serve.llm.build_openai_applications(llm_serving_args, *, name="llm", route_prefix="/") -> List[RunTarget]

For N configured models, the builder returns N + 2 application targets: one per model application, a router application, and a control application.
Each model application uses the existing single-model direct-streaming topology.

Example usage


from ray import serve
from ray.serve.llm import LLMConfig, build_openai_applications

apps = build_openai_applications({"llm_configs": [qwen_config, llama_config]})
serve.run_many(apps)

POST /v1/chat/completions {"model": "qwen"}   → router → streamed from a qwen LLMServer replica
GET  /v1/models                               → router → control application
POST /models/qwen/v1/chat/completions         → qwen application directly (skips the router)

Test plan

  • pytest .../routers/test_builder_openai_applications.py: builder unit tests
  • RAY_SERVE_ENABLE_HA_PROECT_STREAMING=1 pytest.../test_openai_applications_direct_streaming.py
  • pre-commit run on changed files
@celinaky
celinaky added this pull request to stack #66518 September 27, 2026 19:58
@celinaky celinaky changed the title [Serve][4/4] Sep 29, 2026
@celinaky celinaky added the go add ONLY when ready to merge, run all tests label Sep 29, 2026
Signed-off-by: Celina Yang <celina.yang@anyscale.com>
@celinaky
celinaky force-pushed the celina/router-app-4-openai-applications branch from 47a300c to 640521d Compare October 1, 2026 06:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests

1 participant