[serve,data][llm] LLM telemetry bugfixes

[eicherseiji]

Why are these changes needed?

An audit of the LLM usage telemetry found the per-model statistics are structurally wrong, independent of any particular deployment. The detached telemetry actor appends one entry to its model list per replica engine-start (and again on every restart / autoscale / redeploy) and is never reset, so each comma-joined tag:

• grows without bound over a long-lived cluster's lifetime (risking silent truncation at the usage-tag value size limit), and
• duplicates each model by its replica/restart count, inflating every per-model list and count.

Two values were also wrong by construction: num_replicas was hardcoded to 1 for non-autoscaling deployments (and num_replicas="auto" was dropped entirely, since the stored config keeps the literal string), and the JSON_MODE tags were built from a hardcoded use_json_mode=True for a dimension that has no deployment-time config, making them duplicates of the MODELS / NUM_REPLICAS tags. These are observable failure modes, not hypothetical.

Serve (observability/usage_telemetry/usage.py)

• Dedup by model_id (last-write-wins dict) instead of an append-only list. Fixes the double-counting across replicas/restarts and the unbounded memory growth on the head-node actor; tags now carry one entry per distinct model. model_id is identity only and is never recorded as a value.
• Report the configured replica count: fixed num_replicas instead of hardcoded 1, and resolve num_replicas="auto" to its autoscaling config (reported as autoscaling) rather than dropping it.
• Drop the JSON_MODE tags (proto 602/603 reserved); they carried no signal.
• Telemetry can no longer break engine start: _multiple_apps never raises, and per-model reporting is wrapped so a failure is logged, not propagated.
• Atomic agent creation via get_if_exists; removed the shadowed retry args.

Batch (batch/observability/usage_telemetry/usage.py + processors)

• Same dedup fix (key by telemetry identity) plus a _reset hook.
• Guarded, atomic agent creation and a non-raising push path so telemetry cannot break processor construction.
• HTTP processor now passes batch_size; vLLM processor now sources data_parallel_size (both previously reported 0).
• Fixed the LLM_BATCH_CONCURRENCY proto comment.

Tests

Updated existing expectations and added regression tests: replica/restart dedup, fixed num_replicas, and num_replicas="auto" reported as autoscaling.

Note

This reserves two released usage tags (602/603) and stops recording them. Per the proto header, data-collection changes need sign-off from @pcmoritz / @thomasdesr.

Checks

• Added regression tests.
• Signed off with DCO.

[gemini-code-assist]

Code Review

This pull request refactors the LLM batch and serve telemetry agents to deduplicate reported models and replicas by keying tracking collections on model/telemetry identity instead of accumulating them in lists. It also improves robustness by wrapping telemetry reporting in try-except blocks to prevent telemetry failures from breaking main execution paths, and updates actor creation to be atomic using get_if_exists=True. The feedback suggests further improving robustness by wrapping the remote actor creation in TelemetryAgent in a try-except block to avoid breaking processor construction, and defensively handling non-integer values (such as 'auto' or None) for num_replicas in Ray Serve to prevent Pydantic validation errors.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[cursor]

Proto file modified requires RPC fault-tolerance review

Low Severity

⚠️ This PR modifies one or more .proto files.
Please review the RPC fault-tolerance & idempotency standards guide here:
https://github.com/ray-project/ray/tree/master/doc/source/ray-core/internals/rpc-fault-tolerance.rst

 

^{Triggered by project rule: Bugbot Rules}

^{Reviewed by Cursor Bugbot for commit c507d8d. Configure here.}

[thomasdesr]

Maybe overly-cute, but what if we captured the id(model) when you're storing this instead of the actual model value? That way we're just capturing the memory address of the string instead of its original value? It wouldn't be comparable across anything, but would maybe still be semantically meaningful?

[eicherseiji]

Like it! I think the string will be recreated with each remote call/deserialization though, right? Used a hash which I think will be equivalent, but lmk if I missed sth

[thomasdesr]

Hash'll work!

A note for the future though: hashes are guessable even when they aren't reversible, and they're comparable across unrelated domains (same input, same hash, everywhere).

Breaking those two properties is over-engineering here, since these values literally don't go anywhere. But if we ever need to, I'd fold two things into a keyed hash (HMAC, or BLAKE3's derive-key mode):

1. A domain separator: a static key that semi-uniquely identifies this particular use, e.g. ray_telemetry_model_id_masking. That puts the values in their own namespace so they "will never collide with anything else at random."
2. A source of randomness somewhere (per-process, per-pipeline, per-object) that prevents direct comparison even within the domain.

[jeffreywang88]

are we okay with not knowing the exact model from our telemetry? do you know what we use our telemetry for?

[eicherseiji]

The model_id is high cardinality (user specified) and may contain identifying information, so it's just used as a local dedupe key. Instead we log model arch since it's lower cardinality while retaining some signal about the models used.

Could be interesting to log some more information about the model structure though (e.g. total/active params)

[thomasdesr]

Stamping for telemetry review, defer rest to others :D

[jeffreywang88]

Just leaving nits.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{Reviewed by Cursor Bugbot for commit 9484d36. Configure here.}