[Core] Compute per component memory usage in MiB

[Kunchd]

Description

We have observed that the idle worker memory usage as reported in the per component memory usage metric grows over time. Previous investigation has concluded that it's possible for idle workers to "leak" memory due to imported libraries caching memory regions between tasks, resulting in a large idle worker footprint between runs. We have also previously uncovered that memory leaks in popular libraries can also result in idle worker memory growth: apache/arrow#39808.

However a recent investigation into the trivial workload below with no library usage still showed idle worker memory growth over time.

import ray
from tqdm import tqdm
import time

ray.init()

@ray.remote
def produce():
    a = b"0" * int(0.5 * 1024**3)
    time.sleep(5)
    return a

@ray.remote
def consume(bytes):
    time.sleep(5)
    return "done"

refs = []
for _ in tqdm(range(16)):
    ref = produce.remote()
    refs.append(ref)
    consume_refs = []
    for i in range(4):
        consume_refs.append(consume.remote(ref))
    print(ray.get(consume_refs))

Per component metric showing idle worker memory growth:

This growth is attributed to how "unique set size" is calculated in the query for per component memory usage metric. Previously, we computed the unique set size via the following query:

(sum(ray_component_rss_mb{instance=~".+",SessionName=~".+",ray_io_cluster=~".*",ClusterId=~"ID"} * 1024 * 1024) by (Component)) - (sum(ray_component_mem_shared_bytes{instance=~".+",SessionName=~".+",ray_io_cluster=~".*",ClusterId=~"ID"}) by (Component))

The problematic part is the * 1024 * 1024 which attempts to convert the rss back into bytes. However, the RSS was actually recorded in megabytes instead mebibytes, resulting in over calculation. As the RSS for the worker grows, this calculation grows increasingly incorrect as shown by the graph below where we demonstrate the difference between computing pre-existing query (were we incorrectly convert megabytes into bytes) and the actual unique set size.

This PR addresses this mis-conversion by establishing bytes as the unit for all per component memory metrics. The updated graph of idle worker memory usage in the per component memory usage panel is shown below:

Related issues

Additional information

[gemini-code-assist]

Code Review

This pull request updates the system metrics ray_component_rss_mb and ray_component_uss_mb to output in bytes instead of megabytes, renaming them to ray_component_rss_bytes and ray_component_uss_bytes respectively. The changes span documentation, dashboard panels, reporter agents, tests, and release scripts. Feedback on the changes highlights two issues: first, the Gauge definitions in reporter_agent.py mistakenly specify "MiB" as the unit metadata instead of "bytes"; second, in mem_check.py, the variable uss_mb_for_agent_component and its downstream assertions still assume megabytes, which will cause assertion failures now that the metric returns bytes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Yicheng-Lu-llll]

https://github.com/search?q=repo%3Aray-project%2Fray+1.0e6&type=code

Seems some of the paths still use/ 1.0e6

[edoakes]

naming is a bit inconsistent here -- should we call it mem_uss_bytes or drop the mem_ from the shared one?

[edoakes]

As I mentioned offline, I am concerned about blanket breaking compatibility on the metric naming. Can we leave the _mb ones in place but update the dashboards to use _bytes? We can also note that these ones are deprecated in our monitoring docs.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 1c410b8. Configure here.}

[Kunchd]

@edoakes Could you take another look when you get the chance? Thanks!

[edoakes]

🫡