[Serve][LLM] Don't fetch LLMConfig from replicas at ingress startup

[kouroshHakha]

Bug

With min_replicas: 0, deploying an OpenAI ingress force-starts every LLMServer at deploy time, defeating scale-to-zero. The ingress also can't pass check_health until every model has successfully started at least once, so one broken model blocks the whole service from becoming healthy.

Root cause

OpenAiIngress.__init__ schedules _setup_handle_and_config_maps, which awaits handle.llm_config.remote() for each deployment. With 0 replicas the call queues at the router, bumps ongoing_requests to 1, and the autoscaler scales 0 → 1. The fetched LLMConfig is byte-identical to what build_openai_app already holds in memory.

Fix

Pass static metadata into the ingress directly. OpenAiIngress.__init__ now takes:

• llm_deployments: Dict[str, DeploymentHandle]
• model_cards: Dict[str, ModelCard]
• lora_paths: Optional[Dict[str, str]] = None

build_openai_app, build_dev_openai_app, build_pd_openai_app, and build_dp_openai_app build these dicts up front. get_lora_model_metadata takes base_path: str instead of LLMConfig. No remote llm_config() call at startup; check_health is a no-op.

Test plan

• pytest python/ray/llm/tests/serve/cpu/deployments/routers/test_router.py
• pytest python/ray/llm/tests/serve/cpu/deployments/routers/test_dev_ingress.py
• pytest python/ray/llm/tests/serve/cpu/deployments/llm/multiplex/test_lora_deployment_base_client.py
• Manually deploy multi-model app with min_replicas: 0, send no traffic, verify no replicas start and service reports healthy.

[gemini-code-assist]

Code Review

This pull request refactors the OpenAiIngress initialization to accept llm_deployments, model_cards, and lora_paths as dictionaries keyed by model ID, replacing the previous list-based approach. This change removes the asynchronous remote fetching of model configurations during setup, simplifying the ingress lifecycle and improving initialization reliability. Feedback suggests consolidating the duplicated logic for preparing these arguments in build_openai_app and build_dev_openai_app into a shared helper function to improve maintainability.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit 652cc31. Configure here.}

[cursor]

Property access may trigger router init from wrong thread

Low Severity

_get_request_router accesses router._asyncio_router.request_router (a property with lazy initialization) instead of the previous router._asyncio_router._request_router (a direct attribute read). The property's lazy init creates a RequestRouter and calls self._request_router_initialized.set() on an asyncio.Event that was created on a different thread's event loop. Calling set() from the broadcast thread is not thread-safe per asyncio's contract and could race with the event loop thread calling update_deployment_targets.

 

^{Reviewed by Cursor Bugbot for commit 652cc31. Configure here.}

[eicherseiji]

Is this change meant to be included?

[kouroshHakha]

I have to check what claude did, it was not able to pass the ci/cd tests and now it's passing. will report back :D

[kouroshHakha]

yeah, I don't know why this has been triggered specifically during this pr, but it's basically safe guarding a race condition. without these test_dev_ingress would fail.

[eicherseiji]

lgtm