[serve][1/3] Add replica-side slot reservation primitive by jeffreywang88 · Pull Request #63252 · ray-project/ray

jeffreywang88 · 2026-05-09T03:52:58Z

Description

The choose_replica / dispatch flow needs the router to hold a slot on the replica before dispatching, so that when dispatch finally sends the request it can use with_rejection=False (slot already reserved on the actor). This PR adds the underlying primitive: an actor-side reserve_slot / release_slot pair backed by the existing semaphore, plus the wire format and ReplicaSelection wrapper. There are no callers yet — the router integration is the next PR.

Lifecycle

This PR adds the primitives a future caller (implemented in #63254 and #63255) will use as a four-step lifecycle:

Reserve: Grab a permit on the replica's max_ongoing_requests semaphore before the request is built; get back a token and ground-truth (accepted, num_ongoing_requests).
Hand off: Wrap the token in a ReplicaSelection (address + node + AZ) the caller can pass around.
Consume on dispatch: The token rides on RequestMetadata; the replica skips re-acquiring the semaphore on the way in.
Release on abort: Early return / exception / cancellation returns the token and frees the permit.

Primitives

Replica.reserve_slot / release_slot (exposed through ReplicaActor): the actor-side primitive, leveraging the existing _start_request semaphore so reservations count against the same capacity bound and show up in get_num_ongoing_requests().
- Cross-language (Java) replicas raise RuntimeError("Slot reservation not supported for Java.") since there's no actor-side semaphore on the Java replica.
RunningReplica.reserve_slot / release_slot: async wrappers over the actor RPC that return ReplicaQueueLengthInfo for cache updates.
ReplicaSelection: wraps the token plus replica metadata; enforces single dispatch via _mark_dispatched.
ReplicaUnavailableError: raised when a selection is invalidated before dispatch.

FAQ

1. What's still missing in this PR?

Gap	Fixed in
Nothing calls `reserve_slot` yet — the primitive exists but is unwired.	#63254
`RunningReplica.reserve_slot` doesn't yet retry on `ActorDiedError` / `ActorUnavailableError`.	#63254 caller in `AsyncioRouter.choose_replica`
`Router` abstract base, `SingletonThreadRouter`, `CurrentLoopRouter`, and `DeploymentHandle` have no `choose_replica` / `dispatch` methods yet.	#63255

2. Why do we need both token and replica-side semaphore?

Reservation creates a gap: reserve_slot and dispatch are two separate router → actor RPCs, and the design needs to handle both capacity and identity across that gap.

The semaphore is the capacity gate. It's anonymous, shared with every other entry path (handle_request, handle_request_with_rejection), so all paths agree on what "at capacity" means.

The token is the reservation's identity. The semaphore alone can't tell you:

Whether an incoming dispatch should acquire a fresh slot or consume one already held — _start_request branches on the token's presence.
Whether a request that asked to skip semaphore acquisition is legitimately consuming a reservation, vs. forging past the capacity gate.

Related issues

RFC: #59792
Original PR: #60865
Next PR: #63254

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

gemini-code-assist

Code Review

This pull request introduces a slot reservation mechanism for Ray Serve replicas to manage capacity during the choose-and-dispatch process. Key changes include adding reserve_slot and release_slot methods to the replica implementation, updating RequestMetadata to track reservation tokens, and introducing a ReplicaSelection class to manage the reservation lifecycle. Additionally, a new ReplicaUnavailableError is added. Feedback was provided regarding a potential blocking call in the reserve_slot method that could occur if a race condition happens between the capacity check and semaphore acquisition.

gemini-code-assist · 2026-05-09T03:58:24Z

+        if not self._can_accept_request(request_metadata):
+            return False, self.get_num_ongoing_requests()
+
+        await self._semaphore.acquire()


The reserve_slot method checks _can_accept_request before awaiting the semaphore. While this provides an early exit, await self._semaphore.acquire() can still block if a race condition occurs between the check and the acquisition. If the intention is for the router to never block on a specific replica during selection, consider using a non-blocking acquisition or a timeout.

Co-Authored-By: machichima <nary12321@gmail.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

…thInfo wrapper) Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 · 2026-05-09T06:00:22Z

+
+    # Internal fields (not part of public API)
+    _replica: RunningReplica
+    _deployment_id: Optional[DeploymentID]


will be used in #63255

jeffreywang88 · 2026-05-09T06:00:40Z

+    _request_metadata: RequestMetadata
+    _method_name: str


will be used in #63254

jeffreywang88 · 2026-05-09T21:25:52Z

+        return self._metrics_manager.get_num_ongoing_requests() + len(
+            self._reserved_slots
+        )


Note: Requests that have actually entered _start_request + capacity already held by reserve_slot() but not yet dispatched.

jeffreywang88 · 2026-05-09T21:34:34Z

+                    "Request tried to consume an unknown reserved slot "
+                    f"{reserved_slot_token}."
+                )
+            self._reserved_slots.remove(reserved_slot_token)


Note: semaphore has already been acquired at reserve_slot()

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 · 2026-05-09T21:52:23Z

+
+
+@PublicAPI(stability="alpha")
+class ReplicaUnavailableError(RayServeException):


will be used in #63254

kouroshHakha

Clean, well-scoped foundation. The semaphore accounting in _start_request is correct — the refactor maintains the invariant that exactly one acquire maps to exactly one release across both the reserved-slot and classic paths. The drain fix (counting _reserved_slots in get_num_ongoing_requests) is the right call; without it a replica could exit between reserve_slot and dispatch. Tests are solid and the test structure (isolated FakeServeReplicaForSlotReservation, no Ray runtime) is exactly right for this layer.

Three items below, two of which I'd want addressed before merge.

Note

This review was co-written with AI assistance (Claude Code).

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e036f17. Configure here.}

cursor · 2026-05-10T04:54:16Z

+        """Returns the replica address in host:port format."""
+        if self.port:
+            return f"{self.node_ip}:{self.port}"
+        return self.node_ip


Falsy port check excludes valid port zero

Low Severity

The address property uses if self.port: which is falsy for both None and 0. Since port is typed as Optional[int], the intended check is likely if self.port is not None: to distinguish "no port configured" (None) from a valid port number. While port 0 is uncommon in production, this is a new public-facing API where the semantics of the truthiness check may surprise future callers.

^{Reviewed by Cursor Bugbot for commit e036f17. Configure here.}

kouroshHakha · 2026-05-10T06:02:17Z

LGTM

…#63252) Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com> Co-authored-by: machichima <nary12321@gmail.com>

jeffreywang88 added the go add ONLY when ready to merge, run all tests label May 9, 2026

jeffreywang88 force-pushed the decouple-routing-primitives-1 branch from 738e50b to 18d7ec5 Compare May 9, 2026 03:58

gemini-code-assist Bot reviewed May 9, 2026

View reviewed changes

[Serve] add replica-side slot reservation primitive

b22b7d9

Co-Authored-By: machichima <nary12321@gmail.com> Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 force-pushed the decouple-routing-primitives-1 branch from 18d7ec5 to b22b7d9 Compare May 9, 2026 04:00

jeffreywang88 mentioned this pull request May 9, 2026

[serve][2/3] Add choose_replica/dispatch to AsyncioRouter #63254

Merged

jeffreywang88 added 3 commits May 9, 2026 05:33

Draining should respect reserved slots

2364565

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

RunningReplica.release_slot returns int (drop unused ReplicaQueueLeng…

54ff466

…thInfo wrapper) Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

Add more unit tests

4250fe2

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 marked this pull request as ready for review May 9, 2026 05:58

jeffreywang88 requested a review from a team as a code owner May 9, 2026 05:58

jeffreywang88 commented May 9, 2026

View reviewed changes

cursor Bot reviewed May 9, 2026

View reviewed changes

Comment thread python/ray/serve/_private/replica.py

ray-gardener Bot added the serve Ray Serve Related Issue label May 9, 2026

jeffreywang88 changed the title ~~[Serve] Add replica-side slot reservation primitive~~ May 9, 2026

jeffreywang88 commented May 9, 2026

View reviewed changes

Clean up tests

2e99d4b

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

jeffreywang88 commented May 9, 2026

View reviewed changes

jeffreywang88 requested a review from kouroshHakha May 9, 2026 21:58

kouroshHakha reviewed May 10, 2026

View reviewed changes

Comment thread python/ray/serve/_private/replica.py

Comment thread python/ray/serve/_private/request_router/replica_wrapper.py

Comment thread python/ray/serve/exceptions.py

Comment thread python/ray/serve/tests/unit/test_router.py

Address CR feedback

e036f17

Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>

cursor Bot reviewed May 10, 2026

View reviewed changes

kouroshHakha enabled auto-merge (squash) May 10, 2026 06:02

kouroshHakha approved these changes May 10, 2026

View reviewed changes

kouroshHakha merged commit 4cc69e0 into master May 10, 2026
7 checks passed

kouroshHakha deleted the decouple-routing-primitives-1 branch May 10, 2026 06:30

chillCode404 mentioned this pull request May 10, 2026

[core] Split raylet cython file into multiple files (timeseries_utils.pxi) #63256

Open

5 tasks

This was referenced May 12, 2026

[Serve] decouple routing primitives #60865

Closed

[serve] Add metrics for decoupled routing primitives (Phase 2) (#62163) #62356

Open

Lucas61000 pushed a commit to Lucas61000/ray that referenced this pull request May 15, 2026

[serve][1/3] Add replica-side slot reservation primitive (ray-project…

16f3e9e

…#63252) Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com> Co-authored-by: machichima <nary12321@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[serve][1/3] Add replica-side slot reservation primitive#63252

[serve][1/3] Add replica-side slot reservation primitive#63252
kouroshHakha merged 6 commits into
masterfrom
decouple-routing-primitives-1

jeffreywang88 commented May 9, 2026 •

edited

Loading

gemini-code-assist Bot left a comment

gemini-code-assist Bot May 9, 2026

jeffreywang88 May 9, 2026

jeffreywang88 May 9, 2026

Uh oh!

jeffreywang88 May 9, 2026

jeffreywang88 May 9, 2026 •

edited

Loading

jeffreywang88 May 9, 2026

kouroshHakha left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

cursor Bot May 10, 2026

kouroshHakha commented May 10, 2026

Uh oh!

Labels

2 participants



		@PublicAPI(stability="alpha")
		class ReplicaUnavailableError(RayServeException):

Uh oh!

Conversation

jeffreywang88 commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Lifecycle

Primitives

FAQ

Related issues

Additional information

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

gemini-code-assist Bot May 9, 2026

Choose a reason for hiding this comment

jeffreywang88 May 9, 2026

Choose a reason for hiding this comment

jeffreywang88 May 9, 2026

Choose a reason for hiding this comment

Uh oh!

jeffreywang88 May 9, 2026

Choose a reason for hiding this comment

jeffreywang88 May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

jeffreywang88 May 9, 2026

Choose a reason for hiding this comment

kouroshHakha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

cursor Bot May 10, 2026

Choose a reason for hiding this comment

Falsy port check excludes valid port zero

kouroshHakha commented May 10, 2026

Uh oh!

Labels

2 participants

jeffreywang88 commented May 9, 2026 •

edited

Loading

jeffreywang88 May 9, 2026 •

edited

Loading