[serve] Push-based replica health: replica self-checks and heartbeats

[johntaylor-cell]

Stacked on #66243, which added the controller-side receive path.

The controller health-checks every replica by firing check_health.remote() on a
timer. At 8k replicas the submit alone is ~465 us and GIL-bound, and the sweep
accounts for essentially all of deployment-state update time.

Replicas now run their own check twice per health-check period and heartbeat the
result, so the pull probe fires only when a pushed result goes stale.

Behind RAY_SERVE_ENABLE_PUSH_HEALTH, default off. The flag exists so the rest
of the stack can be reviewed without changing behavior; the last PR deletes it.

The in-flight guard is bounded. A push overdue past
RAY_SERVE_METRICS_PUSH_STUCK_S (20 s) is abandoned rather than waited on. An
unbounded guard lets one stuck receiver silence a replica permanently — measured
during development, one node's replicas dropped from ~420 to ~1.5 reports/s and
stayed there. The pre-existing metric push had the identical unguarded pattern, so
the same bound now covers it too.

Probe timeouts are not charged to a replica while the controller is behind. The
registry compares arrivals against the rate the fleet should be publishing; below
half of it, a timed-out probe scores as no information rather than a failure. It
deliberately declines to decide whose fault the silence is, because the only thing
it gates is whether to count a strike, and declining is safe either way. A returned
failure still counts, and a crashed actor still reports ACTOR_CRASHED. With the
feature off the expected rate stays 0, so the gate is inert.

Testing. 24 new unit tests. Verified by mutation: each of ten independent
removals — the lag gate, the timeout flag, the lagging ratio, the rate window, the
expected-rate cadence, the stuck bound, once-per-period failure counting, the
threshold latch, the unhealthy bypass, and repeat-unhealthy suppression — is caught
by at least one test. test_deployment_state.py 294 passed and
test_metrics_utils.py 108 passed, with test_replica_backpressure,
test_replica_quiesce, test_user_callable_wrapper, test_controller,
test_application_state and test_deployment_rank_manager unchanged.

[gemini-code-assist]

Code Review

This pull request introduces a replica-pushed self-health mechanism to Ray Serve, allowing replicas to periodically run their own health checks and push the results to the controller, thereby reducing active pull-probing overhead. It also adds a mechanism to detect controller ingest lag and avoid incorrectly marking replicas as unhealthy due to probe timeouts when the controller is falling behind. Feedback on the changes highlights a potential ZeroDivisionError in expected_push_rate if the metrics interval or health check period is configured to zero, which should be guarded against.

[cursor]

Stale Bugbot comment from a previous run.

[gemini-code-assist]

Code Review

This pull request introduces a push-based health check mechanism for Ray Serve replicas, allowing replicas to periodically run their own health checks and push the results to the controller. This mechanism includes a fallback to pull probes when pushed results go stale, and an ingest lag gate to suppress probe timeouts when the controller is lagging. Feedback on these changes highlights a potential memory leak in ReplicaHealthPushRegistry where dead replica entries are never pruned if the total count remains below _PRUNE_THRESHOLD (65,536), and suggests triggering pruning periodically regardless of the size threshold.

[gemini-code-assist]

In clusters with high replica churn but a total lifetime replica count below _PRUNE_THRESHOLD (65,536), dead replica entries will never be pruned and will accumulate in self._state indefinitely, causing a slow memory leak. Since pruning is extremely cheap for smaller dictionary sizes, we should also trigger a prune periodically (e.g., every _PRUNE_MAX_AGE_S seconds) regardless of the size threshold, while still respecting the minimum interval to avoid thrashing.

Suggested change

	if (
	len(self._state) > self._PRUNE_THRESHOLD
	and now - self._last_prune_time > self._PRUNE_MIN_INTERVAL_S
	):
	if (
	len(self._state) > self._PRUNE_THRESHOLD
	or now - self._last_prune_time > self._PRUNE_MAX_AGE_S
	) and now - self._last_prune_time > self._PRUNE_MIN_INTERVAL_S:

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 91d4d34. Configure here.}

[cursor]

Stale failure watermark skips new strikes

Medium Severity

_last_mirrored_push_failures is not cleared when a replica recovers. A later unhealthy push that restarts at count 1 matches the stale watermark, so the increment is cancelled and the strike is dropped. Intermittent fail-then-recover cycles can therefore never reach the unhealthy threshold while fresh heartbeats keep deferring pull probes.

Additional Locations (1)

• python/ray/serve/_private/deployment_state.py#L2051-L2061

 

^{Reviewed by Cursor Bugbot for commit 91d4d34. Configure here.}