[serve][llm] Replace LLM ingress router replica selection with choose_replica

[jeffreywang88]

Description

Removes the custom round-robin pick from LLMRouter and delegates replica selection to DeploymentHandle.choose_replica on the underlying LLMServer deployment. The pick policy is now whatever the user configured via llm_config.deployment_config.request_router_config; the builder fills in RoundRobinRouter as the default when nothing was set.

API

• Declarative (YAML)

  applications:
  - import_path: ray.serve.llm:build_openai_app
    name: llm_app
    route_prefix: "/"
    args:
      llm_configs:
        - model_loading_config:
            model_id: qwen3-4b
            model_source: Qwen/Qwen3-4B
          deployment_config:
            autoscaling_config: { min_replicas: 8, max_replicas: 8 }
            request_router_config: 
              request_router_class: ray.serve.llm.request_router.PrefixCacheAffinityRouter
              request_router_kwargs:
                match_rate_threshold: 0.1

• Imperative (Python)

  llm_config = LLMConfig(
      model_loading_config={"model_id": "qwen3-4b", "model_source": "Qwen/Qwen3-4B"},
      deployment_config={
          "autoscaling_config": {"min_replicas": 8, "max_replicas": 8},
          "request_router_config": RequestRouterConfig(
              request_router_class=PrefixCacheAffinityRouter,
          ),
      },
  )

Benchmark

No regression from master's implementation on L4, but there is regression identified on H100. Need to apply the changes in this commit to avoid regression, which @jeffreywang-anyscale will do in a follow-up PR.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request refactors the LLMRouter to delegate replica selection to DeploymentHandle.choose_replica, replacing the manual round-robin logic and internal replica caching. The LLMServer deployment is now configured to use the RoundRobinRouter via RequestRouterConfig. Additionally, the temporary _get_request_router method in DeploymentHandle has been removed, and tests have been updated to reflect the new asynchronous routing logic and configuration. I have no feedback to provide.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 36ee8f6. Configure here.}