fix(ci): run validators/ unit tests in make test, exclude from coverage gate - #1918
Conversation
…ge gate Makefile excluded /validators from 'go list ./...' in make test, so regression suites added in NVIDIA#1745/NVIDIA#1748 (and the older nvidia-smi, inference-perf, nccl, conformance tests) never ran in CI. - Drop -e /validators from the make test package filter. - Exclude validators/ from the test-coverage threshold calc (its package coverage is 41-92%, pulling aggregate from 80.0% to 75.7% and breaking the 80% gate) while still running its tests. Fixes NVIDIA#1752 Signed-off-by: Mohit Yadav <ymohit799057@gmail.com>
📝 WalkthroughWalkthrough
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Makefile`:
- Around line 230-231: Update the coverage output in the test-coverage target so
it uses the same validator-excluding filter as the enforced coverage metric,
keeping the displayed total consistent with the reported percentage.
- Line 235: Update the coverage gate around the coverage variable to align with
the documented project-wide threshold: either remove the /validators/ exclusion
from the aggregate calculation, or add and enforce a separate validator coverage
policy and update the contributor testing documentation accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: e3e0e1da-8358-4521-85c5-1944dfa606f0
📒 Files selected for processing (1)
Makefile
njhensley
left a comment
njhensley
left a comment
There was a problem hiding this comment.
📋 Multi-persona review — PR #1918
▎ Method: 3 persona passes (Correctness/Shell-Make, CI-DX/Operability, Test-Strategy/Domain), each finding independently confirmed or refuted by a senior meta-reviewer against the resolved code. Measurements reproduced independently. Line links pinned to head 95040365.
▎ Tier legend: 🔴 Blocker · 🟠 Major · 🟡 Minor · 🔵 Nitpick · ✅ Confirmed non-issue
Overall assessment — Request changes
The diagnosis is right and valuable: -e /validators was hiding 54 test files across 7 packages, and that exclusion was never deliberate — it was collateral in the 60-file refactor b741e99e (#290), with no rationale in the commit message and 8 validator test files already present at that commit. The revived tests are in good shape: all 7 packages pass under -race, hermetically, adding only ~60–100s to a 15-minute job budget.
The problem is that the mitigation is in the wrong layer. The coverage exclusion was added to make test-coverage, but CI never runs that target — grep -rn 'test-coverage' .github/ returns zero hits. CI runs make test and then re-computes the threshold from the raw coverage.out via .github/actions/go-coverage, which hard-fails below 80. So as written this PR ships the coverage regression without the mitigation and turns the merge-gate Test job red. Details inline on Makefile L235.
🔵 Nitpick — PR title typo covergae will land in main's history
The commit subject at 95040365 is spelled correctly, but the repo squash-merges (verified: recent main subjects all end in (#NNNN), GitHub's PR-title-derived format), so the PR title becomes the permanent commit subject. Suggest retitling to fix(ci): run validators/ unit tests in make test, exclude from coverage gate (68 chars, within the 70-char limit).
✅ Confirmed non-issues (checked and cleared)
- The revived tests are healthy. All 7 packages
okunder-race -short; re-run withKUBECONFIG=/dev/null HOME=/tmp/nohomestill allok. External interactions arehttptest.NewServerand fake clientsets only — no cluster, envtest, network, or GPU dependency.make testis the correct home for them. - No timeout risk.
TEST_TIMEOUT=10mis a per-test-binary limit, not aggregate; the slowest new package isvalidators/conformanceat ~30s, against a 15-minute job budget. validators/chainsawis not swallowed by the surviving-e /tests/chainsaw/filter — its import path lacks the required/tests/segment. The filter is correctly specific.go tool cover -func=/dev/stdinworks correctly — byte-identical output to reading a file, reads fine from a non-seekable pipe, andmode: atomicsurvives the grep. Verified on darwin/arm64 go1.26.5. (Linux-runner behavior unverified — no Docker in the review sandbox — but/dev/stdin→/proc/self/fd/0is standard onubuntu-latest.)- Make escaping is correct —
$$3,$$(...), the single-quoted grep pattern,.PHONY, and the@prefix are all fine. - Refuted during meta-review: a reviewer claimed #1752's "include validator packages in coverage reporting" criterion was unmet. Not true — validators are in
coverage.outnow and in every downstream reporting consumer. Only the threshold gate excludes them. - Refuted during meta-review: a reviewer claimed the first
go-coverage-reportdelta comment would be skewed by baseline-vs-PR asymmetry. Not for this PR — the delta step is gated on changed.gofiles, and this PR changes none, so no delta comment fires at all.
Summary
| Tier | Count | Items |
|---|---|---|
| 🔴 Blocker | 1 | Coverage exclusion sits in a target CI never runs → merge gate fails at 75.8% vs 80 |
| 🟠 Major | 0 | — |
| 🟡 Minor | 5 | Unbounded exemption; unanchored grep; no anti-regression guard; two divergent coverage numbers; pre-existing fail-open |
| 🔵 Nitpick | 2 | Undocumented carve-out + newly-false "project-wide"; PR title typo |
Recommendation: The blocker must be fixed before merge — move the exclusion to where CI reads it, then confirm the merge-gate Test job is actually green (it has not run yet on this PR). The unanchored-grep fix is a one-token change worth folding in. The remaining minors and both nitpicks are fine to defer in writing, but the unbounded exemption should get a tracking issue before the carve-out becomes permanent by default.
Thanks for digging out a four-month-old silent test gap — the underlying find here is a good one.
| test-coverage: test ## Runs tests and enforces coverage threshold (from .settings.yaml quality.coverage_threshold) | ||
| @coverage=$$(go tool cover -func=coverage.out | grep total | awk '{print $$3}' | sed 's/%//'); \ | ||
| @coverage=$$(grep -v '/validators/' coverage.out | go tool cover -func=/dev/stdin | grep total | awk '{print $$3}' | sed 's/%//'); \ | ||
| echo "Coverage: $$coverage% (threshold: $(COVERAGE_THRESHOLD)%)"; \ |
There was a problem hiding this comment.
🟡 Minor — Pre-existing fail-open in the coverage gate (informational — your change improves one case)
test-coverage has no set -o pipefail. On a corrupt profile $coverage ends up empty, bc emits a parse error, [ -eq 1 ] errors with 'unary operator expected', the if body is skipped, and the recipe prints "Coverage check passed" and exits 0. Reproduced in a scratch Makefile on both the pre- and post-change recipes, so this is pre-existing and not introduced here. Your change incidentally improves one case: a missing coverage.out now fails closed (empty stdin → total: 0.0% → gate errors) where it previously failed open.
Blast radius: Local make qualify / make test-coverage only — CI is unaffected because .github/actions/go-coverage uses set -euo pipefail and explicitly checks for an empty $COVERAGE.
Fix: Not yours to fix in this PR. Worth a drive-by follow-up: if [ -z "$$coverage" ]; then echo "ERROR: failed to compute coverage"; exit 1; fi after the assignment, mirroring what the CI action already does.
| GOFLAGS="-mod=vendor" go test -short -count=1 -race -timeout=$(TEST_TIMEOUT) -covermode=atomic -coverprofile=coverage.out $$(go list ./... | grep -v -e /tests/chainsaw/ -e /validators) || exit 1; \ | ||
| GOFLAGS="-mod=vendor" go test -short -count=1 -race -timeout=$(TEST_TIMEOUT) -covermode=atomic -coverprofile=coverage.out $$(go list ./... | grep -v -e /tests/chainsaw/) || exit 1; \ | ||
| echo "Test coverage:"; \ | ||
| go tool cover -func=coverage.out | tail -1 |
There was a problem hiding this comment.
🟡 Minor — Two divergent coverage numbers for the same commit; decide deliberately which one is published
This line still prints the unfiltered total, so a single make test-coverage run emits Test coverage: 75.8% followed two lines later by Coverage: 80.2% (threshold: 80%) — a 4.4-point contradiction with no explanation, which reads like a computation bug. The same raw coverage.out also feeds the shields.io badge gist (75.8% flips the color bucket from brightgreen (>=80) to green (>=70)), the coverage-pr/coverage-baseline artifacts, and the go-coverage-report delta comment.
Blast radius: Contributor confusion locally, plus a ~4.4pt drop in the publicly reported coverage badge on main that reflects no real regression.
Fix: Filter once at profile-production time (see the blocker) so this line, the gate, the badge, and the delta report all agree. If the filter must stay in test-coverage, at minimum label the two numbers explicitly, e.g. Total (incl. validators, ungated): … vs Gated (excl. validators): …. The point is to make it a deliberate choice rather than an accident.
| go tool cover -func=coverage.out | tail -1 | ||
|
|
||
| .PHONY: test-coverage | ||
| test-coverage: test ## Runs tests and enforces coverage threshold (from .settings.yaml quality.coverage_threshold) |
There was a problem hiding this comment.
🔵 Nitpick — Carve-out is undocumented, and docs/contributor/tests.md newly says something false
No Makefile comment explains the exclusion — which is exactly how the original -e /validators survived unexamined for four months. Separately, the word "project-wide" in docs/contributor/tests.md:94 (and the mirrored AGENTS.md/.claude/CLAUDE.md:585) becomes factually wrong once validators are carved out.
Blast radius: Maintainability — a future contributor either "fixes" the filter (re-breaking the gate) or widens it without realizing it is a deliberate carve-out.
Fix: Add a comment naming the reason, the tracking issue, and the removal condition. Fix "project-wide" in docs/contributor/tests.md:94 and AGENTS.md/CLAUDE.md:585 (those two must change together — AGENTS.md is a CI-enforced mirror). Note the 75%/70%-vs-actual-80 drift in those same lines plus DEVELOPMENT.md:458 is entirely pre-existing — flagging it, not asking you to fix it here.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/validators_tests_included_test.sh`:
- Around line 1-5: Add the repository-standard license header to the shell
script before its existing comments, using the project’s license tooling or
established header format; preserve the shebang as the first line and leave the
test guard content unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 613d88a6-5dc8-4983-83d8-5f39ce3b9ab5
📒 Files selected for processing (5)
.claude/CLAUDE.mdAGENTS.mdMakefiledocs/contributor/tests.mdtools/validators_tests_included_test.sh
| #!/usr/bin/env bash | ||
| # Guards against the /validators exclusion regression fixed in #1752: | ||
| # asserts the package set make test runs still includes the packages | ||
| # carrying the #1745/#1748 regression suites. | ||
| set -euo pipefail |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add the repository license header.
This new .sh source file contains only the shebang and comments. Add the standard header using the repository’s license tooling before merging.
As per coding guidelines, **/*.{go,yaml,yml,sh} files require license headers verified by repository tooling.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tools/validators_tests_included_test.sh` around lines 1 - 5, Add the
repository-standard license header to the shell script before its existing
comments, using the project’s license tooling or established header format;
preserve the shebang as the first line and leave the test guard content
unchanged.
Source: Coding guidelines
njhensley
left a comment
njhensley
left a comment
There was a problem hiding this comment.
📋 Re-review — PR #1918
▎ Method: follow-up pass on the deltas since the prior multi-persona review, verified directly against head a2edf95b.
▎ Tier legend: 🔴 Blocker · 🟠 Major · 🟡 Minor · 🔵 Nitpick · ✅ Confirmed non-issue
Overall assessment — Request changes
Thanks for the fast turnaround — the substance of the prior review is addressed well. The original blocker is genuinely fixed: the filter moved into the test target so CI now gates on the validators-excluded coverage.out, and the docs/comment cleanups all landed. One item still blocks merge, and it's one I need to partly walk back: the anti-regression guard was my suggestion (prior finding F7, which I'd tiered Minor / reasonable to defer), and as I scoped it, it can't actually do its job. Recommend deleting it — details inline on tools/validators_tests_included_test.sh.
✅ Prior 🔴 Blocker — RESOLVED (verified)
Makefile now writes the unfiltered profile to coverage.full.out, then grep -v '^github.com/NVIDIA/aicr/validators/' coverage.full.out > coverage.out. Since CI's go-coverage action reads ./coverage.out, it gates on the filtered ~80% number — the correct layer. I confirmed the anchored pattern preserves the mode: atomic header. This also closes prior F2 (make test, the badge, and the artifacts now all read the same filtered coverage.out), F4 (anchored ^github.com/NVIDIA/aicr/validators/), and F6 (docs no longer claim "project-wide").
✅ Confirmed non-issues / resolved since last pass
- Coverage-layer fix verified correct (see above) — the headline concern is fully addressed.
- Docs:
.claude/CLAUDE.md,AGENTS.md, anddocs/contributor/tests.mdall updated to note the validators exclusion; the false "project-wide" wording is gone. - (Superseded) the prior "missing trailing newline" nit on the guard file is moot if the file is deleted.
Summary
| Tier | Count | Items |
|---|---|---|
| 🔴 Blocker | 1 | Inclusion guard breaks CI and doesn't test its named regression → delete (our own F7, reframed) |
| 🟠 Major | 0 | — |
| 🟡 Minor | 1 | #<ISSUE> placeholder unfilled |
| 🔵 Nitpick | 1 | coverage.full.out not cleaned/ignored |
Recommendation: The coverage fix is right and done. The only thing blocking merge is the guard I asked for — simplest path is to delete tools/validators_tests_included_test.sh and either fill in or drop the #<ISSUE> line, at which point this is an easy approve. (The PR-title typo covergae → coverage is also still worth fixing before squash-merge.)
|
|
||
| for pkg in \ | ||
| github.com/NVIDIA/aicr/validators/deployment \ | ||
| github.com/NVIDIA/aicr/validators/performance \ |
There was a problem hiding this comment.
🔴 Blocker — The inclusion guard is broken, and (my scoping) can't catch what it's named for — recommend removing it
Context: this script exists because of my prior finding F7. I'd flagged the missing #1752 guard as Minor and "reasonable to defer," and proposed a tools/*_test.sh asserting the package set contains the two validator packages. That suggestion was under-specified, and this is a faithful implementation of it — so treat this as a correction of my own advice as much as of the code.
Two problems, both pointing at removal:
- It fails CI as written. Line 12 ends with a trailing
\, which line-continues intodo, so bash readsfor pkg in deployment performance doand hitsifwheredowas expected:
$ bash -n tools/validators_tests_included_test.sh
tools/…:14: syntax error near unexpected token `if' (rc=2)
test-shell auto-runs every tools/*_test.sh under set -e and is a prerequisite of make test, so this aborts the Test job before any Go test runs — the same class of red gate this PR set out to fix.
- Even fixed, it guards nothing. It hardcodes its own copy of the filter (
go list ./... | grep -v -e /tests/chainsaw/) rather than exercising the Makefile'stestrecipe. So the regression it is named for — someone re-adding-e /validatorsto the Makefile — would leave this test green. It only fails if a validator package is deleted/renamed outright, which other checks already surface loudly. It asserts a constant it defined two lines earlier.
Blast radius: CI: the syntax error aborts test-shell → make test → the merge-gate Test job, on this PR and every push, until fixed. Separately, as a guard it provides false assurance against the exact silent-exclusion regression it advertises.
Fix: Delete the file. The #1752 "add a guard" criterion isn't load-bearing now that validators run and appear in the coverage profile, and a guard that gives false assurance is worse than none. If you'd rather keep the checkbox closed, the only version that earns its place asserts the artifact the recipe actually produces:
# coverage.full.out is the UNfiltered profile `make test` writes
for pkg in validators/deployment validators/performance; do
grep -q "^github.com/NVIDIA/aicr/$pkg/" coverage.full.out \
|| { echo "FAIL: $pkg not in coverage profile — did make test stop running it?"; exit 1; }
doneThat catches a Makefile filter re-adding -e /validators (the packages vanish from coverage.full.out); the current script can't. Either way, run make test-shell before pushing.
| # validators/ tests run as part of `make test` but are excluded from the | ||
| # coverage.out this target emits: per-package coverage there runs 41-92% | ||
| # (see #1752), which would pull the project-wide gate from ~80% to ~75.8%. | ||
| # Tracked for a separate, lower validators coverage floor in #<ISSUE>. |
There was a problem hiding this comment.
🟡 Minor — #<ISSUE> is a literal unfilled placeholder
# Tracked for a separate, lower validators coverage floor in #<ISSUE>. This is the deferred ratchet from prior finding F3, but the reference is a dangling placeholder — a literal #<ISSUE> would land in main.
Blast radius: A permanent placeholder in the tree; readers can't find the tracking issue because there isn't one.
Fix: Either file the tracking issue and drop in the real number, or remove the sentence. (This is also where a real coverage assertion on validators would eventually live, which is the honest reason the guard above can only check presence today — validators sit below the 80% bar and are carved out, so there's no enforced number to assert yet.)
| KUBEBUILDER_ASSETS=$$(setup-envtest use -p path 2>/dev/null || echo "") \ | ||
| AICR_CRITERIA_STRICT=1 \ | ||
| GOFLAGS="-mod=vendor" go test -short -count=1 -race -timeout=$(TEST_TIMEOUT) -covermode=atomic -coverprofile=coverage.out $$(go list ./... | grep -v -e /tests/chainsaw/ -e /validators) || exit 1; \ | ||
| GOFLAGS="-mod=vendor" go test -short -count=1 -race -timeout=$(TEST_TIMEOUT) -covermode=atomic -coverprofile=coverage.full.out $$(go list ./... | grep -v -e /tests/chainsaw/) || exit 1; \ |
There was a problem hiding this comment.
🔵 Nitpick — New coverage.full.out isn't cleaned or ignored
make clean (line 650) removes ./coverage.out but not the newly-introduced coverage.full.out, and .gitignore covers neither.
Blast radius: A run leaves an untracked artifact in the working tree.
Fix: Add coverage.full.out to the clean target's rm list (and optionally to .gitignore).
Signed-off-by: Mohit Yadav <ymohit799057@gmail.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
docs/contributor/tests.md (1)
94-96: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winCoverage floor stated as 75% contradicts the 80% cited elsewhere in this same file.
This line reads "Coverage floor: 75%", but lines 170, 481, and 532 of this same document all state the floor is 80%. Per the PR's own rationale, 75.7% is the aggregate without the validators carve-out — the actual
.settings.yaml quality.coverage_thresholdgate remains 80%; only the validators packages are excluded from the computed number. Stating the floor itself as 75% is misleading and inconsistent with the rest of the document.📝 Proposed fix
-**Coverage floor: 75%** (from `.settings.yaml` -`quality.coverage_threshold`; excludes `validators/`, see `#1752`). `make test-coverage` enforces it. +**Coverage floor: 80%** (from `.settings.yaml` +`quality.coverage_threshold`). `validators/` packages are excluded from the +computed coverage, see `#1752`. `make test-coverage` enforces it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/contributor/tests.md` around lines 94 - 96, Update the “Coverage floor” statement in the contributor testing documentation to say 80%, matching the quality.coverage_threshold gate and the other references in the file. Retain the validators exclusion, enforcement command, and per-package decrease guidance unchanged.Makefile (1)
234-235: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRemove
coverage.full.outinmake cleantoo.*.outalready ignores it, but the clean target still leaves the intermediate file behind.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Makefile` around lines 234 - 235, Update the Makefile clean target to remove coverage.full.out alongside the existing coverage artifacts. Preserve the current test coverage generation and filtering commands unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/contributor/tests.md`:
- Around line 94-96: Update the “Coverage floor” statement in the contributor
testing documentation to say 80%, matching the quality.coverage_threshold gate
and the other references in the file. Retain the validators exclusion,
enforcement command, and per-package decrease guidance unchanged.
In `@Makefile`:
- Around line 234-235: Update the Makefile clean target to remove
coverage.full.out alongside the existing coverage artifacts. Preserve the
current test coverage generation and filtering commands unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: c45a8369-e49a-4d1b-b66a-0ed0a0226b7f
📒 Files selected for processing (2)
Makefiledocs/contributor/tests.md
…ge.full.out - Remove tools/validators_tests_included_test.sh: it syntax-errored (trailing backslash swallowed 'do'), aborting test-shell before any test ran, and even fixed it only re-asserted its own hardcoded filter rather than exercising the Makefile's actual coverage output. - Resolve the #<ISSUE> placeholder by dropping the sentence. - Add coverage.full.out to make clean. - Sync docs/contributor/tests.md, AGENTS.md, .claude/CLAUDE.md from 75% to the actual 80% threshold in .settings.yaml (tests.md had drifted internally inconsistent after NVIDIA#1937 partially fixed this). Addresses njhensley re-review on NVIDIA#1918. Signed-off-by: Mohit Yadav <ymohit799057@gmail.com>
|
gh pr comment 1918 --body "@coderabbitai review" |
|
✅ Action performedReview finished.
|
njhensley
changed the title
…alth checks Closes NVIDIA#1246. - kueue: resourceflavors/clusterqueues/localqueues.kueue.x-k8s.io (verified via in-tree manifests) - kubeflow-trainer: trainjobs.trainer.kubeflow.org (verified via tests/chainsaw/ai-conformance/kind-training-kubeflow/assert-crds.yaml) - network-operator: nicclusterpolicies.mellanox.com (verified via components/network-operator/manifests/nic-cluster-policy-aks.yaml) - k8s-nim-operator: nimservices/nimcaches/nimpipelines.apps.nvidia.com (verified via docs/conformance/cncf/v1.35/nim-eks/evidence/robust-operator.md) - gatekeeper: constrainttemplates.templates.gatekeeper.sh + configs.config.gatekeeper.sh (needs live-cluster confirmation against chart 3.22.2, flagged inline) - slinky-slurm-operator: negative finding documented - crds.enabled: false, CRDs owned entirely by sibling slinky-slurm-operator-crds chart Signed-off-by: Mohit Yadav <ymohit799057@gmail.com> docs: restore slinky-slurm-operator health check, add comment-only negative finding Signed-off-by: Mohit Yadav <ymohit799057@gmail.com> fix: address review comments on NVIDIA#1246 CRD backfill fix: sync kubeflow-trainer CRD docs with asserted trainingruntimes (coderabbitai) address review: soften gatekeeper verification wording, fix volatile line refs, rename CRD step for consistency chore: pausing scans chore: skills cleanup chore(deps): Update python Docker tag to v3.14 (NVIDIA#1906) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com> revert(validators): pin aiperf-bench base back to python:3.13-slim (NVIDIA#1909) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(rekor-monitor): correlate identity via release-workflow run history (NVIDIA#1903) Signed-off-by: Brian Lockwood <lockwobr@gmail.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com> fix(validators): multi-stage aiperf-bench build; bump aiperf to 0.11.0 (NVIDIA#1912) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> feat(skills): download UAT debug bundles for failure triage (NVIDIA#1913) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> chore: deps: bump renovatebot/github-action from 46.1.20 to 46.1.21 (NVIDIA#1923) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> feat(skills): add aicr-triage skill for project board triage (NVIDIA#1911) chore(deps): regenerate stale THIRD_PARTY_NOTICES.md (NVIDIA#1926) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> feat(server): non-interactive bundle attestation for /v1/bundle (NVIDIA#1891) chore(deps): Update dependency awscli to v1.45.52 (NVIDIA#1905) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> ci(stale): drop phantom priority/critical exempt label (NVIDIA#1925) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore(deps): Update dependency sigstore/cosign to v3.1.2 (NVIDIA#1924) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> Co-authored-by: Brian Lockwood <lockwobr@gmail.com> feat(skills): add aicr-cross-review skill for multi-agent PR review (NVIDIA#1915) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore: deps: bump github.com/prometheus/client_golang from 1.24.0 to 1.24.1 (NVIDIA#1922) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> Co-authored-by: Brian Lockwood <lockwobr@gmail.com> fix: resolve goconst and errorlint warnings across OCI, coverage, validators (NVIDIA#1916) Signed-off-by: Andrew White <andrewh@cdw.com> chore(deps): Update testing-tools (NVIDIA#1934) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> fix(skills): harden UAT debug-bundle triage against malformed input (NVIDIA#1914) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> chore: deps: bump actions/stale from 10.4.0 to 11.0.0 (NVIDIA#1931) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> fix(ci): sync coverage docs to 80% and fail closed on bad threshold (NVIDIA#1937) docs: correct issue/PR lifecycle table, fix reminder dedupe (NVIDIA#1921) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> chore: deps: bump actions/checkout from 7.0.0 to 7.0.1 (NVIDIA#1930) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> Co-authored-by: Brian Lockwood <lockwobr@gmail.com> fix(recipe,client): reject duplicate ComponentRef names, omit disable… (NVIDIA#1917) Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> fix(validators): pass aiperf model via --model flag (NVIDIA#1939) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(skills): recover cross-review lanes, harden consensus/fallback (NVIDIA#1932) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> feat(rekor-monitor): resumable identity scan to catch up large backlogs (NVIDIA#1929) Signed-off-by: Brian Lockwood <lockwobr@gmail.com> fix(bundler): create argocd-helm static/ only when populated (NVIDIA#1944) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore: deps: update hashicorp/azurerm requirement from ~> 4.0 to ~> 5.0 in /infra/uat-azure-account (NVIDIA#1945) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix(ci): run validators/ unit tests in make test, exclude from coverage gate (NVIDIA#1918) Signed-off-by: Mohit Yadav <ymohit799057@gmail.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> feat(corroborate): improve evidence dashboard UX (NVIDIA#1935) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(api): accept versionless legacy bundle recipes (NVIDIA#1943) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> test(bundler): cover argocd-helm OCP bundling and pin its error contract (NVIDIA#1950) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore: remove stray MOFED debug artifacts from repo root (NVIDIA#1955) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(deps): bump network-operator default to v26.4.1 (NVIDIA#1938) Signed-off-by: Atif Mahmood <atif1996@users.noreply.github.com> Co-authored-by: Atif Mahmood <atif1996@users.noreply.github.com> chore: deps: bump renovatebot/github-action from 46.1.21 to 46.2.0 (NVIDIA#1964) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): Update ministackorg/ministack Docker tag to v1.4.7 (NVIDIA#1965) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> feat(recipe): implement the ADR-015 profile core (NVIDIA#1933) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> fix(bom): extract nested operator images (NVIDIA#1960) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> address yuanchen8911 review: bump network-operator to 26.4.1 with nicnodepolicies exclusion, close gatekeeper TODO with verification
…alth checks Closes NVIDIA#1246. - kueue: resourceflavors/clusterqueues/localqueues.kueue.x-k8s.io (verified via in-tree manifests) - kubeflow-trainer: trainjobs.trainer.kubeflow.org (verified via tests/chainsaw/ai-conformance/kind-training-kubeflow/assert-crds.yaml) - network-operator: nicclusterpolicies.mellanox.com (verified via components/network-operator/manifests/nic-cluster-policy-aks.yaml) - k8s-nim-operator: nimservices/nimcaches/nimpipelines.apps.nvidia.com (verified via docs/conformance/cncf/v1.35/nim-eks/evidence/robust-operator.md) - gatekeeper: constrainttemplates.templates.gatekeeper.sh + configs.config.gatekeeper.sh (needs live-cluster confirmation against chart 3.22.2, flagged inline) - slinky-slurm-operator: negative finding documented - crds.enabled: false, CRDs owned entirely by sibling slinky-slurm-operator-crds chart Signed-off-by: Mohit Yadav <ymohit799057@gmail.com> docs: restore slinky-slurm-operator health check, add comment-only negative finding Signed-off-by: Mohit Yadav <ymohit799057@gmail.com> fix: address review comments on NVIDIA#1246 CRD backfill fix: sync kubeflow-trainer CRD docs with asserted trainingruntimes (coderabbitai) address review: soften gatekeeper verification wording, fix volatile line refs, rename CRD step for consistency chore: pausing scans chore: skills cleanup chore(deps): Update python Docker tag to v3.14 (NVIDIA#1906) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com> revert(validators): pin aiperf-bench base back to python:3.13-slim (NVIDIA#1909) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(rekor-monitor): correlate identity via release-workflow run history (NVIDIA#1903) Signed-off-by: Brian Lockwood <lockwobr@gmail.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com> fix(validators): multi-stage aiperf-bench build; bump aiperf to 0.11.0 (NVIDIA#1912) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> feat(skills): download UAT debug bundles for failure triage (NVIDIA#1913) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> chore: deps: bump renovatebot/github-action from 46.1.20 to 46.1.21 (NVIDIA#1923) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> feat(skills): add aicr-triage skill for project board triage (NVIDIA#1911) chore(deps): regenerate stale THIRD_PARTY_NOTICES.md (NVIDIA#1926) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> feat(server): non-interactive bundle attestation for /v1/bundle (NVIDIA#1891) chore(deps): Update dependency awscli to v1.45.52 (NVIDIA#1905) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> ci(stale): drop phantom priority/critical exempt label (NVIDIA#1925) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore(deps): Update dependency sigstore/cosign to v3.1.2 (NVIDIA#1924) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> Co-authored-by: Brian Lockwood <lockwobr@gmail.com> feat(skills): add aicr-cross-review skill for multi-agent PR review (NVIDIA#1915) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore: deps: bump github.com/prometheus/client_golang from 1.24.0 to 1.24.1 (NVIDIA#1922) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> Co-authored-by: Brian Lockwood <lockwobr@gmail.com> fix: resolve goconst and errorlint warnings across OCI, coverage, validators (NVIDIA#1916) Signed-off-by: Andrew White <andrewh@cdw.com> chore(deps): Update testing-tools (NVIDIA#1934) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> fix(skills): harden UAT debug-bundle triage against malformed input (NVIDIA#1914) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> chore: deps: bump actions/stale from 10.4.0 to 11.0.0 (NVIDIA#1931) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> fix(ci): sync coverage docs to 80% and fail closed on bad threshold (NVIDIA#1937) docs: correct issue/PR lifecycle table, fix reminder dedupe (NVIDIA#1921) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> chore: deps: bump actions/checkout from 7.0.0 to 7.0.1 (NVIDIA#1930) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> Co-authored-by: Brian Lockwood <lockwobr@gmail.com> fix(recipe,client): reject duplicate ComponentRef names, omit disable… (NVIDIA#1917) Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> fix(validators): pass aiperf model via --model flag (NVIDIA#1939) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(skills): recover cross-review lanes, harden consensus/fallback (NVIDIA#1932) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> feat(rekor-monitor): resumable identity scan to catch up large backlogs (NVIDIA#1929) Signed-off-by: Brian Lockwood <lockwobr@gmail.com> fix(bundler): create argocd-helm static/ only when populated (NVIDIA#1944) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore: deps: update hashicorp/azurerm requirement from ~> 4.0 to ~> 5.0 in /infra/uat-azure-account (NVIDIA#1945) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix(ci): run validators/ unit tests in make test, exclude from coverage gate (NVIDIA#1918) Signed-off-by: Mohit Yadav <ymohit799057@gmail.com> Co-authored-by: Nathan Hensley <229213852+njhensley@users.noreply.github.com> feat(corroborate): improve evidence dashboard UX (NVIDIA#1935) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(api): accept versionless legacy bundle recipes (NVIDIA#1943) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> test(bundler): cover argocd-helm OCP bundling and pin its error contract (NVIDIA#1950) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> chore: remove stray MOFED debug artifacts from repo root (NVIDIA#1955) Signed-off-by: Nathan Hensley <nhensley@nvidia.com> fix(deps): bump network-operator default to v26.4.1 (NVIDIA#1938) Signed-off-by: Atif Mahmood <atif1996@users.noreply.github.com> Co-authored-by: Atif Mahmood <atif1996@users.noreply.github.com> chore: deps: bump renovatebot/github-action from 46.1.21 to 46.2.0 (NVIDIA#1964) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): Update ministackorg/ministack Docker tag to v1.4.7 (NVIDIA#1965) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Lalit Adithya <ladithyav@nvidia.com> feat(recipe): implement the ADR-015 profile core (NVIDIA#1933) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> fix(bom): extract nested operator images (NVIDIA#1960) Signed-off-by: Yuan Chen <yuanchen97@gmail.com> address yuanchen8911 review: bump network-operator to 26.4.1 with nicnodepolicies exclusion, close gatekeeper TODO with verification
2 participants
Makefile excluded /validators from 'go list ./...' in make test, so regression suites added in #1745/#1748 (and the older nvidia-smi, inference-perf, nccl, conformance tests) never ran in CI.
Fixes #1752