[serve][3/N] Introduce experimental ConsistentHashRouter for session-sticky routing

[jeffreywang88]

Summary

Adds ConsistentHashRouter, an experimental subclass of RequestRouter that maps session_id → replica via a consistent-hash ring for sticky-session request routing. Client must send session_id along with its request header to benefit from session-stickiness.

Preceding PRs:

• Plumbing: [serve][1/N] Plumb session_id through request metadata and proxy layers #62905
• mmh3 dependency introduction: [serve][2/N] Add mmh3 for consistent hashing #63096

Changes

• Build a ring with V=100 virtual nodes per replica. When the assigned replica rejects the request, walk clockwise for up to K=2 fallback replicas.
• Route session-less requests through the same ring using internal_request_id. Do not fall back to power-of-two-choices.
• Rebuild the ring only when the replica set changes.

Implementation gotchas

• choose_replicas returns [[primary], [fallback_1], [fallback_2]] instead of one multi-element rank; otherwise the framework's _select_from_candidate_replicas would pick the lowest-queue-length replica, defeating stickiness.
• Override _fulfill_pending_requests: ConsistentHashRouter cannot safely use RequestRouter's FIFO-style task-shedding behavior. Once a routing task pops a request from _pending_requests_to_route, that task owns the request metadata needed to compute the consistent-hash replica. If the base loop exits early because there are “too many” routing tasks, the popped request can remain unfulfilled but no longer be available for another task to route. Pow-2 can recover from that with FIFO fallback. Consistent hashing cannot, because assigning a replica chosen for one request/session to a different pending request would break stickiness. Therefore, the override enforces: if a task pops a request, it must keep trying until that exact request is fulfilled.

Opt-in API

@serve.deployment(
    request_router_config=RequestRouterConfig(
        request_router_class=(
            "ray.serve.experimental.consistent_hash_router:ConsistentHashRouter"
        ),
        request_router_kwargs={"num_virtual_nodes": 100, "num_fallback_replicas": 2},
    ),
)
class SessionAwareDeployment: ...

Benchmarks

Performance comparison w/ PowerOfTwoChoicesRouter

No overhead over power-of-two-choices router.

Correctness

During a scaling event, the session affinity rate drops by M/N+1 because M/N+1 sessions are re-assigned to different replicas.

Effectiveness -- LLM session affinity

Replica assignment distribution

With a higher number of virtual nodes, the request -> replica assignment is more uniform.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request introduces a ConsistentHashRouter to Ray Serve, providing session stickiness via a consistent-hash ring with virtual nodes. The implementation supports fallback replicas during backpressure and maintains affinity during scaling. The PR also includes extensive unit and integration tests, refactors test utilities, and adds the mmh3 dependency. Review feedback suggests optimizing the ring unzipping logic and the ranked replica lookup loop for better performance and idiomatic Python usage.

[jeffreywang88]

@cursor review

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d3e3ff9. Configure here.}

[cursor]

Routing task probes cancelled request indefinitely when under target

Medium Severity

The break condition requires both pending_request.future.done() AND len(self._routing_tasks) > self.target_num_routing_tasks. When a request is externally cancelled but routing tasks are at or below target, the routing task continues probing replicas indefinitely (with backoff sleeps) for a request nobody is waiting for. Since the non-FIFO _fulfill_next_pending_request cannot reassign a found replica to another request, any probed replica with capacity is simply wasted. Under sustained load with cancellations and saturated replicas, this can keep a routing task slot occupied for extended periods, blocking real pending requests from being routed.

 

^{Reviewed by Cursor Bugbot for commit d3e3ff9. Configure here.}