Skip to content

[Data] Move resource budget Prometheus gauges to ExecutionCallback#62209

Merged
bveeramani merged 6 commits into
ray-project:masterfrom
limarkdcunha:task/moving-data-gauges-out-of-streaming-executor
Apr 2, 2026
Merged

[Data] Move resource budget Prometheus gauges to ExecutionCallback#62209
bveeramani merged 6 commits into
ray-project:masterfrom
limarkdcunha:task/moving-data-gauges-out-of-streaming-executor

Conversation

@limarkdcunha

Copy link
Copy Markdown
Contributor

Description

This PR moves out responsibility of tracking Prometheus metrics for resource budgets into a new class ResourceAllocatorPrometheusCallback.
This is in accordance with single responsibility principle.

Closes #60269

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
@limarkdcunha limarkdcunha requested a review from a team as a code owner March 30, 2026 19:00

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the resource allocation metrics logic by moving it from the StreamingExecutor into a dedicated ResourceAllocatorPrometheusCallback. The new callback handles the initialization and updating of Prometheus gauges for CPU, GPU, memory, and object store memory budgets, as well as the maximum bytes to read. The StreamingExecutor now includes this callback by default and extends its callback list when additional ones are provided during execution. I have no feedback to provide.

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Mar 30, 2026
Comment thread python/ray/data/_internal/execution/streaming_executor.py Outdated
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Comment thread python/ray/data/context.py Outdated
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
@bveeramani bveeramani self-assigned this Apr 1, 2026

@bveeramani bveeramani left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM

Comment thread python/ray/data/_internal/execution/streaming_executor.py Outdated
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
@bveeramani bveeramani enabled auto-merge (squash) April 1, 2026 17:48
@github-actions github-actions Bot added the go add ONLY when ready to merge, run all tests label Apr 1, 2026
@github-actions github-actions Bot disabled auto-merge April 1, 2026 21:05
@bveeramani bveeramani merged commit d3b06f8 into ray-project:master Apr 2, 2026
6 checks passed
mancfactor pushed a commit to mancfactor/ray that referenced this pull request Apr 2, 2026
…ay-project#62209)

### Description

This PR moves out responsibility of tracking Prometheus metrics for
resource budgets into a new class ResourceAllocatorPrometheusCallback.
This is in accordance with single responsibility principle.

Closes ray-project#60269

---------

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Frank Mancina <fmancina@haproxy.com>
Lucas61000 pushed a commit to Lucas61000/ray that referenced this pull request May 15, 2026
…ay-project#62209)

### Description

This PR moves out responsibility of tracking Prometheus metrics for
resource budgets into a new class ResourceAllocatorPrometheusCallback.
This is in accordance with single responsibility principle.

Closes ray-project#60269

---------

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues go add ONLY when ready to merge, run all tests

2 participants