[Data] Fixed scheduling metrics computation#62031
Conversation
![gemini-code-assist[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the op_runtime_metrics.py file to improve the accuracy of task scheduling and output backpressure metrics. Key changes include correcting the calculation of average_task_scheduling_time_s to use num_tasks_have_outputs instead of num_tasks_finished, and refactoring the on_task_output_generated method to correctly track the first output generation and associated timing. Minor updates include a clarification in a metric description and a formatting adjustment. There is no feedback to provide.
| return self.task_scheduling_time_s / self.num_tasks_finished | ||
| # NOTE: For correct calculation, we must use `num_tasks_have_outputs`, since | ||
| # scheduling time is incremented upon receiving of the first output | ||
| return self.task_scheduling_time_s / self.num_tasks_have_outputs |
There was a problem hiding this comment.
Just want to ensure we're measuring the above.
- Should we not divide by
num_tasks_submittedinstead ofnum_tasks_have_outputs?
There was a problem hiding this comment.
Check how task_scheduling_time_s is incremented -- we increment it with the first output, hence denominator
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
alexeykudinkin
force-pushed
the
ak/op-mcs-fix
branch
from
4b69514 to
1c8a0ae
Compare
ray-gardener
Bot
added
data
## Description Previously `average_task_scheduling_time_s` was computed incorrectly: - Cumulative scheduling duration was incremented whenever task would get first output - But denominator would only increase only when the task would finish Meaning that for the time while the task is running after first output is produced, but before it finishes the metric will be inflated, incorrectly. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
3 participants
Description
Previously
average_task_scheduling_time_swas computed incorrectly:Meaning that for the time while the task is running after first output is produced, but before it finishes the metric will be inflated, incorrectly.
Related issues
Additional information