feat: Initial implementation of power metrics in aiperf.

[FrankD412]

This PR adds some basic power metrics leveraging GPU telemetry to compute the amount of power used during the benchmark.

Summary by CodeRabbit

• New Features

  • GPU efficiency metrics: total power, total energy, and output-tokens-per-joule.
  • On-demand GPU telemetry collection entrypoint.
  • Baseline GPU metric capture during telemetry setup.

• Tests

  • Updated unit tests to exercise the public collection entrypoint and support new telemetry behaviors.

• Chores

  • Expanded test logging mocks to include informational logs.

---

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@8a9d8aa680fea00293f8856c280c212c2c70cc80

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@8a9d8aa680fea00293f8856c280c212c2c70cc80

---

Last updated for commit: 8a9d8aa • Browse code

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds GPU efficiency metric computation and an explicit collector entrypoint: a new accumulator method computes total GPU power, energy, and output-tokens-per-joule; collector collection method made public for on-demand baseline captures; records and manager invoke these flows.

Changes

Cohort / File(s)	Summary
Efficiency computation src/aiperf/gpu_telemetry/accumulator.py	Added imports for EnergyMetricUnit/PowerMetricUnit and new GPUTelemetryAccumulator.compute_efficiency_metrics(metric_results, time_filter) which aggregates per-GPU power and energy across DCGM endpoints and conditionally emits total_gpu_power, total_gpu_energy, and output_tokens_per_joule.
Protocols src/aiperf/gpu_telemetry/protocols.py	Added async def collect_and_process_metrics(self) -> None to GPUTelemetryCollectorProtocol and def compute_efficiency_metrics(self, metric_results: list[MetricResult], time_filter: TimeRangeFilter) -> list[MetricResult] to GPUTelemetryAccumulatorProtocol; added TimeRangeFilter import under TYPE_CHECKING.
Records integration src/aiperf/records/records_manager.py	When GPU accumulator exists, builds a TimeRangeFilter from phase stats and calls compute_efficiency_metrics(...), extending records_results with returned efficiency metrics before publishing.
Manager baseline capture src/aiperf/gpu_telemetry/manager.py	After creating a PyNVML collector, performs a baseline capture via collector.initialize() and collector.collect_and_process_metrics(), logging success at info and warning on failure; DCGM endpoint baseline logs changed from debug→info.
Collector API rename src/aiperf/gpu_telemetry/pynvml_collector.py	Renamed private _collect_and_process_metrics to public collect_and_process_metrics and updated its invocation in the background loop.
Tests updated tests/unit/gpu_telemetry/test_pynvml_collector.py, tests/unit/gpu_telemetry/test_telemetry_manager.py	Tests updated to call the public collect_and_process_metrics() entrypoint and to mock manager.info in telemetry manager tests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I hopped through logs at break of dawn,

watts and joules I proudly fawn,
tokens counted, energy measured,
metrics stitched and neatly treasured,
a tiny rabbit cheers the run 🌱⚡

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately reflects the main change: adding initial power metrics functionality to aiperf via GPU telemetry integration.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

❌ Patch coverage is 89.07563% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/aiperf/gpu_telemetry/accumulator.py	85.71%	7 Missing and 2 partials ⚠️
src/aiperf/gpu_telemetry/pynvml_collector.py	33.33%	2 Missing ⚠️
src/aiperf/records/records_manager.py	83.33%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[FrankD412]

This PR addresses: https://linear.app/nvidia/issue/AIP-838/implement-an-energytoken-benchmark-estimation

[FrankD412]

After some local testing, some of the metrics show up but not all. I need to figure out how to run a metric calculation for energy/token and have it show up.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/superpowers/plans/2026-04-13-monitor.md`:
- Around line 13-28: Replace the ASCII table under "File Map" with a mermaid
diagram block (```mermaid ... ```) that groups the created source files, the
modified source file, and the created test files as subgraphs; include nodes for
monitor/__init__.py (Package marker), monitor/models.py (GpuSnapshot model),
monitor/gpu.py (query_gpus), monitor/api.py (create_app), monitor/poller.py
(CsvPoller), cli_commands/monitor.py (CLI subcommand), the modified cli.py
(Register monitor command), and the three test files
tests/unit/monitor/__init__.py, test_gpu.py, test_api.py, and test_poller.py,
following the suggested structure so the diagram replaces the table exactly.

In `@src/aiperf/gpu_telemetry/manager.py`:
- Around line 212-215: The broad exception handler in manager.py (the "except
Exception as e" that calls self.warning(f"GPU Telemetry: Failed to capture
baseline from pynvml: {e}")) is missing the BLE001 suppression comment; add "#
noqa: BLE001" to that except line to match other handlers and include the same
short explanatory comment used at lines 240/334/428, so the except Exception as
e line becomes explicitly marked as intentionally broad while keeping the
existing self.warning call intact.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: cd0f4da4-9c3d-4a42-8ace-c06012576f1b

📥 Commits

Reviewing files that changed from the base of the PR and between 946bb59 and deaf3a6.

📒 Files selected for processing (9)

• docs/superpowers/plans/2026-04-13-monitor.md
• docs/superpowers/specs/2026-04-13-monitor-design.md
• src/aiperf/gpu_telemetry/accumulator.py
• src/aiperf/gpu_telemetry/manager.py
• src/aiperf/gpu_telemetry/protocols.py
• src/aiperf/gpu_telemetry/pynvml_collector.py
• src/aiperf/records/records_manager.py
• tests/unit/gpu_telemetry/test_pynvml_collector.py
• tests/unit/gpu_telemetry/test_telemetry_manager.py

✅ Files skipped from review due to trivial changes (1)

• docs/superpowers/specs/2026-04-13-monitor-design.md

🚧 Files skipped from review as they are similar to previous changes (3)

• src/aiperf/records/records_manager.py
• src/aiperf/gpu_telemetry/protocols.py
• src/aiperf/gpu_telemetry/accumulator.py

[FrankD412]

Ok, I was concerned when aiperf was reporting different numbers than the reference implementation in trtllm-bench, but I ended up just wrapping both calls in a simple script to measure. I expected it to over-estimate slightly in both cases, but it looks like the math works out. I'm really surprised that the trtllm-bench case is quite different under a concurrency of 1024 with a ISL/OSL of 1k/1k. Here's what Llama-3.1-7B on 8xH200 looks like with a simple config:

CSV Export: /home/scratch.fdinatale_coreai/code/aiperf/h200/artifacts/meta-llama_Llama-3.1-70B-Instruct-openai-completions-concurrency1024-request_rateinf/profile_export_aiperf.csv
JSON Export: /home/scratch.fdinatale_coreai/code/aiperf/h200/artifacts/meta-llama_Llama-3.1-70B-Instruct-openai-completions-concurrency1024-request_rateinf/profile_export_aiperf.json
Log File: /home/scratch.fdinatale_coreai/code/aiperf/h200/artifacts/meta-llama_Llama-3.1-70B-Instruct-openai-completions-concurrency1024-request_rateinf/logs/aiperf.log


GPU 0 (NVIDIA H200): 143653.77 J
GPU 1 (NVIDIA H200): 147784.43 J
GPU 2 (NVIDIA H200): 149026.62 J
GPU 3 (NVIDIA H200): 148156.23 J
GPU 4 (NVIDIA H200): 145397.66 J
GPU 5 (NVIDIA H200): 148267.18 J
GPU 6 (NVIDIA H200): 153568.17 J
GPU 7 (NVIDIA H200): 149799.60 J
Total energy: 1185653.65 J

Energy reported by aiperf

  "osl_mismatch_count": {
    "unit": "requests",
    "avg": 1421.0
  },
  "total_token_throughput": {
    "unit": "tokens/sec",
    "avg": 27921.504814287888
  },
  "total_gpu_power": {
    "unit": "W",
    "avg": 4724.019916949152
  },
  "total_gpu_energy": {
    "unit": "J",
    "avg": 1067107.4119999986
  },
  "output_token_throughput_per_watt": {
    "unit": "tps/W",
    "avg": 3.027279746105967

trtllm-bench

-- Energy Metrics --------------------------------------

Total Energy (J):                                 1647268.6290
Output Tokens per Second per Watt (tps/W):         1.8649
Average GPU Power (W):                            666.9243

-- Request Latency Breakdown (ms) -----------------------

Validation output:

GPU 0 (NVIDIA H200): 214223.76 J
GPU 1 (NVIDIA H200): 217174.15 J
GPU 2 (NVIDIA H200): 218900.36 J
GPU 3 (NVIDIA H200): 218245.88 J
GPU 4 (NVIDIA H200): 215508.04 J
GPU 5 (NVIDIA H200): 217891.44 J
GPU 6 (NVIDIA H200): 222311.65 J
GPU 7 (NVIDIA H200): 219760.09 J
Total energy: 1744015.37 J

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/aiperf/gpu_telemetry/accumulator.py`:
- Around line 337-368: The current per-GPU metric extraction only catches
NoMetricValue in the blocks calling gpu_data.get_metric_result for
"gpu_power_usage" and "energy_consumption" (power_result / energy_result),
allowing any other exception to bubble up and abort processing; update those
try/except blocks to also catch generic Exception as e (after NoMetricValue) and
log the unexpected error (include the gpu_uuid and exception text) via
self.debug or self.error so the loop continues to the next GPU and overall
efficiency metric aggregation isn't aborted.
- Around line 370-372: The early return when power_gpu_count == 0 in the
accumulator (around power_gpu_count check) is dropping metrics that don't
require instantaneous power (e.g., total_gpu_energy and
output_tokens_per_joule); change the logic in the method containing that check
(look for power_gpu_count, total_gpu_energy, output_tokens_per_joule) so you no
longer return [] when power data is missing—instead compute and include metrics
that can be derived from available energy data and only skip or set to None
those metrics that specifically require gpu_power_usage/instantaneous power;
ensure the final returned list still contains non-power metrics and that
power-dependent fields are omitted or null when power_gpu_count == 0.
- Around line 325-329: The code incorrectly reads
total_output_tokens_result.avg; since TotalOutputTokensMetric inherits
DerivedSumMetric and the derived scalar is stored as a sum, change the
assignment that computes total_output_tokens to read
total_output_tokens_result.sum instead of .avg (locate the
total_output_tokens_result variable and the total_output_tokens assignment near
the TotalOutputTokensMetric / DerivedSumMetric usage) so the value semantically
matches the DerivedSumMetric behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a0bffc0c-ed37-496c-ad35-c82557ac1743

📥 Commits

Reviewing files that changed from the base of the PR and between deaf3a6 and 6676436.

📒 Files selected for processing (1)

• src/aiperf/gpu_telemetry/accumulator.py

[devin-ai-integration]

Devin Review found 1 potential issue.

View 3 additional findings in Devin Review.

[debermudez]

Four issues found during review.

[FrankD412]

Four issues found during review.

Alright -- fixed the issues.

[debermudez]

Code Review — PR #803 (power-efficiency metrics)

Reviewed against b4f1f58c (current PR head). Two passes combined:

1. Branch code review — 10 findings (3 High / 6 Medium / 1 Low).
2. LLM-ergonomics review — 8 findings (2 High / 5 Medium / 1 Low).

Most findings anchor to specific lines and are posted inline. Two cross-cutting items don't anchor cleanly:

High — Public metrics added without docs updates (ergonomics Axis 6)

This PR adds three new public metrics (total_gpu_power, total_gpu_energy, output_tokens_per_joule), a new unit string (tokens/J), and a new "externally-injected derived metric" pattern, but no docs are updated. CLAUDE.md mandates: "DOCUMENTATION IS REQUIRED, NOT OPTIONAL ... Metrics definitions or formulas → docs/metrics-reference.md". Suggested:

• Add a ## GPU Power Efficiency Metrics section to docs/metrics-reference.md with the three tags, units, and formulas (Σ avg(gpu_power_usage), Σ delta(energy_consumption), total_output_tokens / total_gpu_energy).
• Add an ### Externally-Injected Derived Metric entry to docs/dev/patterns.md citing power_efficiency_metrics.py as the reference file.
• If the pattern warrants it, sync across the four-file rule (AGENTS.md / CLAUDE.md / .github/copilot-instructions.md / .cursor/rules/python.mdc) and run make check-agent-files-sync.

Low — tools/ debug runner duplication

tools/run_trtllm_bench_with_power.py, tools/run_trtllm_serve_with_power.py, and tools/run_trtllm_serve_with_power_dcgm.py each define their own http_get / http_post / wait_for_server / write_json helpers. Extracting tools/_power_runner_common.py would prevent helper drift (though tools/ is out of strict ergonomics scope).

Runtime reproduction (deferred — should run on a GPU host pre-merge)

# F-04: degenerate phase_stats
aiperf profile -m <model> --no-warmup --request-count 0 --gpu-telemetry pynvml \
  --artifact_dir /tmp/aiperf-f04 --ui simple

# F-08: GPU flags absent when telemetry disabled
aiperf profile -m <model> --request-count 50 --no-gpu-telemetry \
  --artifact_dir /tmp/aiperf-f08 --ui simple
# expect: profile_export_aiperf.json does NOT contain total_gpu_power / total_gpu_energy / output_tokens_per_joule

# F-02: cross-phase energy leak (requires multi-phase plan)
aiperf profile ... --plan two-phase.yaml ...
# expect: phase 2's total_gpu_energy reflects only phase 2 delta

What's good

• The asymmetric gauge/counter time-window comment at accumulator.py:407-408 is exactly the "why, not what" comment style CLAUDE.md prescribes, and test_energy_filter_widens_end_ns_while_power_filter_stays_bounded (test_accumulator.py:549) locks the invariant in.
• _capture_collector_baseline (manager.py:266-292) cleanly relocates collector.initialize() to the baseline-capture path so the energy counter has a baseline before profile start.
• Lambda-wrapped debug logs (accumulator.py:325-338, 364-380) match the "lambda for expensive logs" rule and correctly use uuid=gpu_uuid to dodge the loop-variable closure bug.
• Mechanical floor is clean: check-ergonomics (257 baselined, 0 new) and ruff-baselined (305 baselined, 0 new) both pass.

Full review artifacts (local-only): artifacts/code-review.md and artifacts/code-review-2026-05-27/llm-ergonomics-pr-803-feat-power-efficiency-metrics.md.

[FrankD412]

Confirmed that the energy_per_user statistic is showing up and things appear to be behaving. Updating and setting to auto-merge once tests complete.

[debermudez]

Nice work man.