As our last v0 release before v1 (scheduled on October 5th), this release focuses on component upgrade safety, validation hardening, and bundler and deployer reliability, landing alongside new platform and accelerator coverage and the last phase of ADR-022 described apiVersion migration that v1.0.0 will complete.

Highlights

Component Upgrade Safety - v1 commits AICR to safe component upgrades, and this release adds additional capabilities to that mechanism. ADR-021 defined a transition record, a per-component, per-version-boundary statement of safe, manual, or blocked, with the steps and the evidence that backs it, and a fail-closed loader that reports unknown rather than guessing. The new aicr upgrade-check command reads those records and reports the verdict for the boundary you are about to cross. Records also live beside the component they describe, so the obligation renews on each pin bump rather than decaying.

Validation Hardening - Every check that v1 will stand behind has to fail for the right reason. The DRA support check now gains an MNNVL IMEX channel subtest. The NCCL benchmark is derived from the shipped TCPXO runtime rather than a separately pinned image, with AICR_NCCL_RUNTIME_IMAGE available where a site needs its own. The GB200 NET preflight is now driver-version-aware, unsupported Hyperdisk types are rejected on a4x nodes, AKS cluster autoscaling is validated, NodeWright is read by discovery, and Job deadlines get headroom over the check budget so a slow check fails as a check rather than a timeout.

Bundler and Deployer Reliability — The bundle layout is a frozen v1 surface, so what it emits has to be right on every deployer, not just the common one. bundle-info.yaml now records the deployer and layout that produced the bundle. recipe.yaml is written for every deployer, not only Helm; ownsCRDs is honored on the helm and helmfile deployers; helm connection flags are translated for the CRD step, with the flag value redacted when the step rejects it; and dry-run no longer previews a stale cached chart archive.

Recipes and Coverage

• k0s joins the supported services, with a validated H200 training leaf, a setup guide, and Preview coordinate coverage
• GKE gains a GB200 (a4X) recipe with NVLS NCCL validation, and a torch-distributed-tcpxo runtime with the network mapping TCPXO requires
• Five new training-kubeflow leaves, each with its evidence link kept intact
• GB300 gets node tuning enabled and bumped NodeWright packages, plus its own NCCL thresholds
• Kueue moves to 0.19.3 and kai-scheduler to v0.16.9

Other Improvements

• Five opt-in NVSentinel mixins widen what a cluster reports about itself: audit logging and tracing, a Kubernetes Object Monitor, a preflight mixin, Node Problem Detector conditions consumed as NVSentinel signal, and NIC and fabric fault detection on Mellanox platforms.
• SLSA provenance is attested per platform manifest; recipe digests are content-only via PredicateTypeV3; evidence fails closed on a mutable validator image tag; and overlay recipe digests are canonicalized so the same recipe hashes the same way
• New signed evidence for GB300 EKS training and inference, VR200 training-kubeflow, and the H100 AKS training-kubeflow and inference-dynamo coordinates
• A recipe can pin deployment identity so registry defaults can move underneath it; a profile value can tighten a composed constraint; and a direct -r overlay whose mixins were not applied is rejected rather than silently under-resolved

Thanks to @ArangoGutierrez, @atif1996, @ayuskauskas, @chris320211, @coffeepac, @lockwobr, @mikecook, @mohityadav8, @njhensley, @ramessesii2, @rorajani, @srao-nv, @sylvesterkaczmarek, @varmesh, @vineeth-bandi, @yuanchen8911, and @mchmarny.

Changelog

New Features

• 693a7f8: feat(api): switch artifact emitters to ADR-022 target apiVersions (#2703) (@mchmarny)
• c7d0825: feat(bundler): derive the DRA eviction label with dra-node-labeler (#2813) (@atif1996)
• 85d1280: feat(bundler): record deployer and layout in bundle-info.yaml (#2816) (@mchmarny)
• 6e880a2: feat(ci): attest SLSA provenance per platform manifest (#2729) (@lockwobr)
• c113877: feat(ci): notify Slack when main goes red (@mchmarny)
• 3462ada: feat(ci): report registry chart drift to Slack weekly (#2779) (@mchmarny)
• bc29f9a: feat(cli): aicr upgrade-check reports ADR-021 upgrade verdicts (#2760) (@lockwobr)
• 79b89df: feat(evidence): publish GB300 EKS training and inference evidence at v0.21.1 (#2812) (@yuanchen8911)
• a4e952f: feat(gb300): enable node tuning and bump nodewright packages (#2772) (@ayuskauskas)
• 9fc4f86: feat(recipe)!: ship NodeWright CRs, move default namespace (#2861) (@lockwobr)
• 1f27c0d: feat(recipe): pin deployment identity so registry defaults can move (#2839) (@lockwobr)
• 4e3e8f0: feat(recipes)!: ship torch-distributed-tcpxo GKE runtime with required network mapping (#2705) (@srao-nv)
• b73f08d: feat(recipes): add GKE GB200 (A4X) recipe with NVLS NCCL validation (#2338) (@mikecook)
• 758b552: feat(recipes): add five training-kubeflow leaves, keep evidence links (#2717) (@yuanchen8911)
• e665888: feat(recipes): add k0s service overlays with validated H200 training leaf (#2656) (@ramessesii2)
• 3d9dac5: feat(recipes): add opt-in nvsentinel-preflight mixin (#2778) (@varmesh)
• 3351de8: feat(recipes): bump kueue chart to 0.19.3 (#2598) (@ArangoGutierrez)
• c9a5230: feat(recipes): consume Node Problem Detector conditions in NVSentinel (#2804) (@varmesh)
• f8ca115: feat(recipes): detect NIC and fabric faults on Mellanox platforms (#2823) (@varmesh)
• 4f97dbc: feat(upgrade): add ADR-021 transition record schema and loader (#2704) (@ayuskauskas)
• 640a114: feat(upgrade): nodewright v0.18.0 record; relax rule 2 to safe only (#2815) (@ayuskauskas)
• f451468: feat(validator): derive NCCL benchmark from the shipped TCPXO runtime (#2751) (@yuanchen8911)
• 9f5ca25: feat(validator): dra-support MNNVL IMEX channel subtest (v1 milestone) (#2770) (@yuanchen8911)
• 764932b: feat(validators): add inference-model-cache-storage-class recipe cons… (#2669) (@mikecook)
• ddefb8f: feat(validators): support AICR_NCCL_RUNTIME_IMAGE override for NCCL checks (#2386) (@mohityadav8)
• de89f27: feat: GB300 nccl thresholds (#2782) (@coffeepac)
• f2c3fa4: feat: add opt-in NVSentinel Kubernetes Object Monitor mixin (#2763) (@varmesh)
• 031b010: feat: add opt-in NVSentinel audit logging and tracing mixin (#2712) (@varmesh)

Bug Fixes

• 87bcb45: fix(build): pin distroless static v4.1.3 so images ship /app (#2771) (@lockwobr)
• 4aa8337: fix(bundler): honor ownsCRDs on the helm and helmfile deployers (#2725) (@lockwobr)
• ebddbb2: fix(bundler): redact the flag value when the CRD step rejects it (#2856) (@mchmarny)
• b510a0f: fix(bundler): stop dry-run previewing a stale cached chart archive (#2855) (@lockwobr)
• 6f1e622: fix(bundler): translate helm connection flags for the CRD step (#2849) (@mchmarny)
• cdfb814: fix(bundler): write recipe.yaml for every deployer, not just helm (#2759) (@lockwobr)
• b99c8e4: fix(ci): gate gcp-h100 training UAT cell to v0.22.0+ (@mchmarny)
• ec7fef4: fix(ci): ignore CSIDriver schema-lag field in KWOK's Argo CD instance (#2592) (@mikecook)
• eb405a0: fix(ci): install ORAS directly instead of via setup-oras (#2657) (@mchmarny)
• 5849fb4: fix(ci): release-gate fixes for cache, oasdiff, checksums, docs gate (#2673) (@mchmarny)
• ccebdce: fix(ci): remove go install from the qualification gate (#2713) (@mchmarny)
• 8b24cff: fix(ci): stop fork runs writing caches from lint, e2e and cli-e2e (#2680) (@mchmarny)
• 8efa372: fix(ci): stop merge gate duplicating and cancelling main push runs (@mchmarny)
• 5bb01eb: fix(ci): validate YAML code blocks in documentation (#2750) (@chris320211)
• 0ab0c2e: fix(errors): distinguish context cancellation from deadline expiry (#2852) (@lockwobr)
• 81555ac: fix(evidence): fail closed on mutable validator image tags (#2884) (@mchmarny)
• 2e7f69f: fix(evidence): make recipe digest content-only via PredicateTypeV3 (#2789) (@mikecook)
• 3a346c7: fix(issues): quote "No" dropdown option in feature request issue form (#2707) (@vineeth-bandi)
• 9912fa4: fix(recipe): let a profile value tighten...

Changelog

Bug Fixes

• 804b41d: fix(ci): install ORAS directly instead of via setup-oras (#2657) (@mchmarny)

Other Tasks

• 41bd4bb: chore(ci): drop the demo deploy job from the release pipeline (@mchmarny)

This release focuses on API surface hardening, GB300 and Vera Rubin support, major GPU Operator and Dynamo version bumps, and Kubernetes 1.37 with broad validation and bundler hardening.

Highlights

New Hardware and Platform Support - GB300 lands on both the cloud and bare-metal (NCP) paths: a generic gb300-generic-ubuntu-training recipe for self-managed Kubernetes (#2568), EKS training and inference recipes (#2382) with signed evidence (#2410), and a Slurm variant (#2544). Vera Rubin (VR200) arrives as preview overlays on RKE2 (#2520), with published evidence (#2573).

Component Upgrades - GPU Operator moves to v26.7.0 and the DRA driver to 0.5.0, with a workaround for the ComputeDomain CRD conflict v26.7.0 introduces and the driver held at 580 (#2439). dynamo-platform moves to 1.4.2: the request plane defaults to TCP and KV events to ZMQ, so bundled NATS is no longer deployed - re-enabling it is opt-in for standing clusters. The same bump takes Grove to v0.1.0-alpha.12 and picks up a NIXL loader-path fix (#1983).

Recipes and Coverage

• GKE self-installed driver pools now carry the driver in the bundle: gpuStack=bundle-installer replaces driver-installer, pinning the version in the recipe so upgrades roll with the bundle (#2360)
• OKE gains a gpuStack profile (#2355), with RDMA fabric wiring for L40S RoCE and GB200 InfiniBand landing alongside it (#2356)
• An l40-any accelerator overlay and a declared RTX PRO 6000 Server Edition driver floor
• Per-value readiness constraints for configuration profiles

Validation Hardening - Every NCCL benchmark run gets its own namespace, and cleanup fails the run when that namespace will not terminate. The validator rejects an incompatible StorageClass before creating the model-cache PVC, tolerates tainted nodes with a wider Trainer readiness timeout, retries transient reads while polling gang-scheduling pods, and resolves the GPU-node universe label per service. DaemonSet checks are guarded against stale rollouts, and concurrent snapshot runs no longer collide.

API Surface Hardening - The four surfaces AICR freezes for v1 - REST, Go SDK, CLI, and artifact schemas - are now each held to a committed baseline. The profile-aware /v2 REST family folds into a single /v1 (#2112) - a pre-adoption restructure with no consumers to notify. JSON Schemas for the v1 artifacts are published and gated, the per-deployer bundle layout is frozen, and artifacts read both their alpha apiVersion and the maturity-appropriate target from ADR-022. pkg/client/v1 is now the only path the CLI and server take into business logic, enforced by an architecture test. Backing all of it: gates that replay documented REST examples and CLI invocations against real handlers and assert the spec and server agree on routes.

Bundler and Deployer Reliability - A readiness gate landed for the Helm network-operator, DRA eviction is wired for GPU driver upgrades behind an opt-in node label, and OCP gates the GPU Operator on network readiness. Bundles missing a required node selector now fail instead of shipping, generated wrappers are stamped with both the AICR and payload versions, and Argo CD is forced to replace the readiness-gate Job on upgrade.

Other Improvements - Kubernetes moves to 1.37: recipe floors are raised to clear the DRA chart's kubeVersion, agentgateway moves to v1.5.0 for the tightened CRD cost budget, client libraries are on v0.37.0, and Kind and KWOK lanes run 1.37 with Argo CD server-side diff. Toolchain moves to Go 1.27.1.
CycloneDX SBOMs and OpenVEX documents are published per platform, and vendor/ is gone in favor of Go proxy resolution verified reproducible on every push to main (ADR-023), and ADR-025 proposes an NVIDIA Cluster Readiness Engine component.

Thanks to @atif1996, @ayuskauskas, @ezhang3333, @framsouza, @giuliocalzo, @haarchri, @Kevin-Hawkins, @lalitadithya, @lockwobr, @MDhamani, @mikecook, @mohityadav8, @njhensley, @ntheanh201, @rorajani, @srao-nv, @varmesh, @xdu31, @yankay, @yuanchen8911, and @mchmarny.

Changelog

New Features

• dc3818e: feat(agent): isolate concurrent snapshot runs with per-run resources (#2357) (@ayuskauskas)
• d380113: feat(api)!: collapse the REST families into a single /v1 (#2464) (@mchmarny)
• 54f31b4: feat(api): deprecation channel, CLI surface gate, ADR-022 emit guards (#2436) (@mchmarny)
• 3f70c95: feat(api): gate the REST contract against a committed baseline (#2468) (@mchmarny)
• 482f670: feat(api): publish and gate JSON Schemas for the v1 artifacts (#2470) (@mchmarny)
• 05b75b7: feat(bundler): add readiness gate for the Helm network-operator (#2337) (@framsouza)
• d85b4ba: feat(bundler): freeze the per-deployer bundle layout (#2475) (@mchmarny)
• cefd89d: feat(bundler): wire DRA eviction for GPU driver upgrades (#2401) (@MDhamani)
• 3e8bef2: feat(recipe): per-value readiness constraints for configuration profiles (#2347) (@atif1996)
• 33f0860: feat(recipes)!: GKE bundle-installer replaces driver-installer (#2360) (@atif1996)
• ffe0cc2: feat(recipes): OKE RDMA fabric wiring (L40S RoCE + GB200 IB) (#2356) (@atif1996)
• e08293b: feat(recipes): add GB300 EKS Ubuntu training Slurm recipe (#2544) (@varmesh)
• de6514b: feat(recipes): add GB300 EKS training and inference overlays (#2382) (@yuanchen8911)
• bb0b059: feat(recipes): add VR200 (Vera Rubin) RKE2 preview overlays (#2520) (@yuanchen8911)
• bf36a91: feat(recipes): add l40-any accelerator overlay (#2366) (@ntheanh201)
• 432f597: feat(recipes): declare RTX PRO 6000 Server Edition driver floor (#2446) (@yuanchen8911)
• 83ce737: feat(recipes): generic bare-metal GB300 training recipe (#2568) (@atif1996)
• 0cf961f: feat(recipes): gpuStack profile for the OKE family (#2355) (@atif1996)
• a874184: feat(recipes): register nvcre Helm component (#2524) (@rorajani)
• 4e8a1c9: feat(sdk): complete the facade surface for snapshot criteria and mirror inventory (#2478) (@mchmarny)
• aa337f3: feat(sdk): derive CollectSnapshot AgentConfig from spec.snapshot (#2538) (@mchmarny)
• 245b693: feat(sdk): derive bundle and validate options from AICRConfig (#2521) (@mchmarny)
• 949f672: feat(sdk): derive evidence attestation options from AICRConfig (#2542) (@mchmarny)
• 84983e8: feat(sdk): migrate pkg/cli off pkg/config onto the aicr.Config facade (#2548) (@mchmarny)
• 31ee1a3: feat: accept target artifact versions defined by ADR-022 (#2404) (@mchmarny)

Bug Fixes

• a9a1be6: fix(api)!: accept HEAD and make GET and POST echo the same criteria (#2472) (@mchmarny)
• e312609: fix(api): stage ADR-022 default-track target in recipe response enums (#2419) (@mchmarny)
• d961042: fix(bundler): force ArgoCD replace on readiness-gate Job upgrade (#2408) (@Kevin-Hawkins)
• f71bd8b: fix(bundler): make the DRA eviction node label opt-in (#2471) (@yuanchen8911)
• 7469fdb: fix(bundler): stamp AICR and payload versions into generated wrappers (#2571) (@mchmarny)
• 3bc6a10: fix(bundler): stop Argo CD from flipping computedomains CRD ownership (#2547) (@yuanchen8911)
• 315463d: fix(bundler): warn when GPU nodes lack the DRA kubelet-plugin label (#2457) (@yuanchen8911)
• a455a93: fix(chainsaw): guard DaemonSet health checks against stale rollouts (#2453) (@mikecook)
• bea09a3: fix(ci): enable Argo CD server-side diff for k8s 1.37 KWOK lanes (#2603) (@rorajani)
• d4bf52e: fix(ci): extend api-diff package load timeout (#2399) (@yuanchen8911)
• 46b5764: fix(ci): install pinned E2E tools instead of runner defaults (@mchmarny)
• dab6f81: fix(ci): preload KWOK registry and Gitea images into the Kind node (#2480) (@mchmarny)
• ca8556c: fix(ci): pull the KWOK images once per run instead of once per job (#2497) (@mchmarny)
• e83c15d: fix...

This release focuses on SDK completeness, digest-pinned OCI recipes, runtime AI inventory, and validation and bundler reliability.

Highlights

Complete SDK Contract - Users can now verify and sign artifacts through the pkg/client/v1 facade. Compiled examples ship with the package, the CLI and server signing paths use the facade, and API compatibility remains gated.

Digest-Pinned OCI Recipe Sources - SDK consumers can load digest-pinned recipe catalogs from OCI registries. Each OCI-backed client uses a private workspace cleaned by Client.Close().

Runtime AI Inventory (ADR-019) - AICR adds optional k8s-aibom v1.3.0 to generate CycloneDX ML-BOMs for selected namespaces, with stock GKE adoption and CRD storage-version health checks.

Validation Hardening - GKE recipes now fail during deployment when required GPU NIC networks are missing, GPU host-driver floors are enforced, and build suffixes are compared correctly. Kubeflow Trainer lifecycle follows the recipe, partial installs roll back, and controller waits use the discovered name. Deadline-killed validators report the actual failure, while evidence verdicts require cleanup, avoid over-claiming, and clarify operator SKIP and Trainer namespace semantics.

Bundler & Deployer Reliability - Flux upgrades can replace component-owned CRDs, empty sources/ directories are omitted, and transient Helm index failures retry automatically. NVSentinel misconfigurations and unsupported --dynamic use with local-chart argocd-helm components now fail closed. Mixed vendored components receive a tracked -post release, and bundle vendoring is hardened against SSRF and resource exhaustion.

Recipes & Components

• NVSentinel v1.20.0, configured for AKS, GKE-COS, OKE, and Kind
• topograph moved to v1.0.0 across recipes; GAIE v1.5.0 CRDs vendored for agentgateway
• Slinky/Slurm gained configurable Enroot settings with epilog/prolog examples
• Nodewright tuning resources changed to whole-number equivalents

Other Improvements - nodes is now metadata-only for overlay selection while --nodes still satisfies CLI specificity checks. NIM gains a pinned, credential-free CNCF AI conformance example, and the toolchain moves to Go 1.27.

Thanks to @ayuskauskas, @bmcadams1, @Dhirenderchoudhary, @framsouza, @kaynetu, @Kevin-Hawkins, @lalitadithya, @lockwobr, @njhensley, @pdmack, @ravisoundar, @rorajani, @srao-nv, @TerryHowe, @tjrasche, @varmesh, @yuanchen8911, and @mchmarny.

Changelog

New Features

• def13e0: feat(demos): credential-free NIM for CNCF AI conformance evidence (#2244) (@yuanchen8911)
• 916dae3: feat(recipe): add generation-time runtime inventory selection (#2317) (@mchmarny)
• af9099f: feat(recipes): add qualified k8s-aibom component (#2259) (@mchmarny)
• 7ba3ac9: feat(recipes): adopt k8s-aibom in the h100-gke-cos-inference recipe (#2328) (@mchmarny)
• 8af10c1: feat(recipes): assert k8s-aibom CRD storage version in the health check (#2305) (@mchmarny)
• 42d35a6: feat(recipes): requalify and re-pin k8s-aibom to v1.3.0 (#2309) (@mchmarny)
• 9dc8b6e: feat(sdk): add Client.LoadSnapshot (#2229) (@mchmarny)
• 710b2b9: feat(sdk): bind AICRConfig to the facade for verify and recipe (#2243) (@mchmarny)
• 9fc11bd: feat(sdk): expose snapshot criteria relaxation as a resolve option (#2247) (@mchmarny)
• 06d2cbb: feat(sdk): expose snapshot diff facade (#2307) (@tjrasche)
• 39d5867: feat(sdk): expose verification and signing on pkg/client/v1 (#2218) (@mchmarny)
• c69db7e: feat(sdk): load digest-pinned OCI recipe sources (#2212) (@tjrasche)
• 5e55a8e: feat(uat): add argocd deployer variant on aws-h100 training (#2250) (@framsouza)
• 78b97cd: feat(validation): detect missing GKE GPU NIC networks at deployment phase (#2248) (@yuanchen8911)

Bug Fixes

• 131aeb0: fix(api): normalize legacy recipe kind on bundle ingest (#2162) (@yuanchen8911)
• 26eef38: fix(bom): support digest field in structured image descriptors (#2263) (@TerryHowe)
• b349a59: fix(bundler): create flux sources/ only when populated (#2161) (@yuanchen8911)
• 534b540: fix(bundler): fail closed on silent NVSentinel misconfigurations (#2183) (@yuanchen8911)
• 3d76752: fix(bundler): let CRD-owning components replace their CRDs on Flux upgrade (#2312) (@mchmarny)
• 67f5d1b: fix(bundler): reject --dynamic on local-chart argocd-helm components (#2200) (@rorajani)
• 1c7aa96: fix(bundler): retry transient Helm index fetch failures (#2268) (@yuanchen8911)
• f1a5213: fix(bundler): tracked -post release for vendored mixed components (#2061) (@rorajani)
• 4cae5aa: fix(ci): code-block-aware MDX sanitization in Fern workflows (#2319) (@pdmack)
• 63c79ec: fix(ci): count in-flight release runs in rekor monitor allowlist (#2154) (@lockwobr)
• b8a6ead: fix(ci): dispatch TestGrid from evidence ingest (#2330) (@srao-nv)
• 875006c: fix(ci): pin UAT GPU worker AMIs to Ubuntu 24.04 (guard 26.04 drift) (#2191) (@njhensley)
• c17850f: fix(ci): re-pin golangci-lint install.sh checksum for v2.13.1 (@mchmarny)
• 27af274: fix(ci): watch the license gate's own inputs in the deps path filter (#2166) (@yuanchen8911)
• 427c2ec: fix(constraints): compare GKE build suffixes in version constraints (#2252) (@Kevin-Hawkins)
• 0dc556e: fix(coverage): resolve UAT wiring from the reservation registry (#2138) (@Dhirenderchoudhary)
• 68be22f: fix(deps): bump golang.org/x/mod to v0.40.0 for GO-2026-6179/6180 (#2205) (@mchmarny)
• 21b538d: fix(evidence): clarify operator SKIP, pin Trainer namespace split (#2269) (@yuanchen8911)
• 1e0ea13: fix(evidence): gate verdicts on cleanup and stop over-claiming (#2222) (@yuanchen8911)
• 6f5eb1e: fix(lint): resolve gofmt and SA4023 failures from golangci-lint v2.13.1 (@mchmarny)
• 508074c: fix(mirror): fail closed when component values cannot be hydrated (#2285) (@mchmarny)
• 908a2ca: fix(notices): anchor go-licenses test guards and cover the fail-closed path (#2163) (@yuanchen8911)
• b2e5908: fix(notices): resolve GOROOT explicitly for go-licenses (#2159) (@mchmarny)
• 619cf93: fix(recipe): fold the OS guard into one joint-coverage rule (#2322) (@lockwobr)
• 7227511: fix(recipes)!: configure NVSentinel for AKS, GKE-COS, OKE and Kind (#2249) (@yuanchen8911)
• 76a8024: fix(sdk): address OCI source follow-up feedback (#2313) (@tjrasche)
• effd3b5: fix(sdk): harden transparent alias API diff (#2157) (@tjrasche)
• 142c792: fix(server): harden bundle vendor path against SSRF and resource exhaustion (#2170) (@framsouza)
• 484c9e0: fix(skills): cross-review sandbox probe and companion resolution (#2172) (@yuanchen8911)
• 1ec0796: fix(skills): reap leaked cross-review refs/cr/* and temp diffs (#2195) (@yuanchen8911)
• db8dcb9: fix(snapshotter): default live agent tolerations (#2320) (@yuanchen8911)
• 78e9bf3: fix(uat): pin helm-diff plugin with verified checksum (#2267) (@lockwobr)
• 31970aa: fix(validator): decide Trainer lifecycle from recipe, not cluster (#2321) (@yuanchen8911)
• 9dfb678: fix(validator): enforce GPU host-driver version floors (#2257) (@rorajani)
• ea6aaee: fix(validator): report deadline-killed validator as failed, not "pod not found" (#2187) (@yuanchen8911)
• 8c27768: fix(validators): roll back partial Kubeflow Trainer installs (#2171) (@mchmarny)
• 6bf2bd5: fix(validators): wait on the discovered Trainer controller name (#2173) (@yuanchen8911)
• aa6d5c5: fix(vendor): correct k8s-launch-kit vendor path casing (#2158) (@rorajani)
• 4130a48: fix: enable Go build cache for CodeQL and freshness gates (#2272) (@bmcadams1)
• 90a7f41: fix: overlap Kind cluster creation with host Go builds in E2E (#2289) (@bmcadams1)
• 3ba53ce: fix: pin the E2E Kind node image to .settings.yaml (#2284) (@bmcadams1)
• 16e0773: fix: suffix UAT artifact names with run_attempt (#2196) (@bmcadams1)

Other Tasks

• 49ad450: chore(deps): Update dependency go to v1.26.6 (#2208) (@github-actions[bot])
• b82c438: c...

This release focuses on GPU stack profiles, broader recipe and platform coverage, signed release provenance, and validation hardening.

Highlights

GPU Stack Profiles (ADR-015) - As AICR covers more use-cases, one criteria combination increasingly maps to more than one valid infrastructure configuration. The only prior way to express the second was a bundle-time --set override that validation could not see, so aicr validate checked the stock configuration against a cluster running a different one. Recipes now declare a named gpuStack profile naming who owns a layer of the stack, the cloud service or the GPU Operator. One value is selected at generation time via aicr recipe --profile gpuStack=<value> or the declared default, hydrated into recipe.yaml, and its owned paths are locked against diverging overrides at bundle and mirror time. Selection is explicit intent, never inferred. So for example, the AKS pool reading supplied by aicr snapshot --aks-gpu-pools qualifies a selection and fails generation closed on mismatch. The mechanism landed in v0.18 with no adopter; AKS and the full GKE family have since converted. It is service-agnostic, so OKE addons and OCP operators extend it without new schema.

Supply Chain & Provenance - Release metadata and the aiperf-bench third-party source are published as signed OCI referrers alongside release artifacts, the latest release is re-verified daily, and Rekor identity monitoring gained a resumable scan that catches up large backlogs. aicr verify can be driven from AICRConfig (spec.verify), and POST /v1/bundle supports non-interactive attestation. A registry-inventory egress gate bounds which registries the supply chain may reach.

Validation Hardening - A pass now has to be earned. The validator fails closed on unmatched declared checks and on capability checks whose declared dependencies are missing, rolls back cluster-admin RBAC when prep fails, and re-establishes the Job watch after an apiserver idle-stream closure instead of reporting a false failure. Cordoned GPU and RDMA nodes are disclosed rather than silently skipped, gang-scheduling and webhook conformance tests are now behavioral, and six operator-owning health checks assert CRD Established=True. The evidence dashboard at validation.aicr.run gained multiple UX improvements, and validator outcomes carry redaction-safe CTRF extra data.

SDK Contract - The exported surface of pkg/client/v1 is pinned and gated by a compatibility check, alias and semver contracts are reconciled, and the selector API is context-aware with snapshot Jobs routed through the facade.

Recipes & Evidence

• AKS H100 Ubuntu training and inference (Dynamo) recipe evidence
• AKS training and inference Kubernetes floors aligned to the DRA-GA >= 1.34 rationale
• Slurm/Slinky gains accounting ownership modes and shared RWX storage for home and data
• Network Operator default bumped to v26.4.1; EKS GB200 enrolled in the nightly UAT
• OpenShift Support - OCP-compatible components for cert-manager, prometheus-adapter, nvidia-dra-driver-gpu, and k8s-nim-operator, argocd-helm bundling now covers the OCP path with a pinned error contract, and recipes enabling both gpu-operator and gpu-operator-ocp are rejected at resolution

Other Improvements

• aicr snapshot fails closed on unusable --snapshot input; the collector fails loud when Kubernetes collection is canceled mid-propagation and emits lossless label/taint readings
• GET /v1/query requires a selector, versionless legacy bundle recipes are accepted, and duplicate ComponentRef names are rejected
• Constraint measurement paths are validated at recipe load time; aicr validate --fail-on-error scoping and its exit-code and CTRF-suite semantics are documented
• The BOM extracts nested operator images; the aiperf-bench runtime moved to NVIDIA distroless Python with pip removed
• Readiness checks run in-process via pkg/chainsaw (promoted from validators/chainsaw), and the external chainsaw binary is gone from the validator images

Thanks to @andrewwhitecdw, @atif1996, @framsouza, @gat786, @kaynetu, @krtadev, @lockwobr, @mohityadav8, @njhensley, @rorajani, @rsd-darshan, @tjrasche, @varmesh, @yuanchen8911, and @mchmarny.

Changelog

New Features

• 76fb50c: feat(ci): add UAT orphan janitor (dry-run-first, all clouds) (#2068) (@njhensley)
• 9b180ac: feat(ci): add registry-inventory supply-chain egress gate (#2043) (@njhensley)
• 0716a37: feat(ci): enroll EKS GB200 into the nightly UAT batch (#1979) (@njhensley)
• 9a1064d: feat(cli): support aicr verify via AICRConfig (spec.verify) (#2046) (@lockwobr)
• 81e43cf: feat(corroborate): improve evidence dashboard UX (#1935) (@njhensley)
• 24183d9: feat(notices): add Python dependency licenses to third-party notices (#2004) (@lockwobr)
• c482c2e: feat(recipe): adopt the ADR-015 gpuStack profile on AKS (#1967) (@yuanchen8911)
• 01932ea: feat(recipe): convert GKE family to the gpuStack profile (ADR-015 PR 3) (#2044) (@yuanchen8911)
• 9a2002e: feat(recipe): implement the ADR-015 profile core (#1933) (@yuanchen8911)
• 018dd55: feat(recipes): backfill CRD Established=True for 6 operator-owning health checks (#1919) (@mohityadav8)
• 16a550c: feat(server): non-interactive bundle attestation for /v1/bundle (#1891) (@lockwobr)
• 1808312: feat(skills): add aicr-cross-review skill for multi-agent PR review (#1915) (@yuanchen8911)
• ff0301b: feat(skills): add aicr-triage skill for project board triage (#1911) (@yuanchen8911)
• 7e0f4a1: feat(skills): download UAT debug bundles for failure triage (#1913) (@njhensley)
• 580f516: feat(slinky): add shared RWX storage support for home and data (#2079) (@kaynetu)
• 0752ea1: feat(slurm): add accounting ownership modes (#1970) (@kaynetu)
• c0615e6: feat(uat): slot-aware daytime cluster-name convention (ADR-017) (#2081) (@njhensley)
• f2b54e5: feat(validator): carry redaction-safe CTRF extra outcome data (#1973) (@njhensley)

Bug Fixes

• 470397f: fix(api): accept versionless legacy bundle recipes (#1943) (@yuanchen8911)
• b8d410f: fix(api): drop unemitted metadata.created from spec (#2034) (@varmesh)
• f822560: fix(api): require selector on v1 query GET (#2013) (@tjrasche)
• 93328af: fix(bom): extract nested operator images (#1960) (@yuanchen8911)
• fd26a47: fix(bundler): create argocd-helm static/ only when populated (#1944) (@yuanchen8911)
• c7fa85a: fix(ci): break orphaned TF state lock before UAT teardown (#2049) (@njhensley)
• cb97658: fix(ci): complete UAT teardown on cancel, resize UAT budgets (#2080) (@njhensley)
• 00cb051: fix(ci): retry GKE UAT bringup on transient node-pool timeout (#2066) (@njhensley)
• 2fdcb78: fix(ci): run validators/ unit tests in make test, exclude from coverage gate (#1918) (@mohityadav8)
• d1b7800: fix(ci): size the GKE UAT timeout for the bringup retry (#2078) (@lockwobr)
• 052a18e: fix(ci): sync coverage docs to 80% and fail closed on bad threshold (#1937) (@lockwobr)
• 0f611ce: fix(ci): tear down daytime-up UAT cluster on failed/cancelled run (#2067) (@njhensley)
• 5e2304c: fix(collector): emit lossless label/taint readings (#2093) (@varmesh)
• a268a00: fix(collector): fail loud when K8s collection is canceled mid-propagation (#2036) (@mchmarny)
• 30ec323: fix(deps): bump network-operator default to v26.4.1 (#1938) (@atif1996)
• f2400a4: fix(docs): fail-closed MDX check for bare '<' + MDX-safe content (#2127) (@njhensley)
• d96d977: fix(evidence): bound on-disk bundle reads with FileReadTimeout ctx (#2071) (@njhensley)
• 19a4e3a: fix(evidence): bound residual on-disk reads on publish/verify paths (#2100) (@mchmarny)
• 6ed2486: fix(evidence): correct profile recipe name emphasis (#2099) (@njhensley)
• 5bfc5d9: fix(evidence): make gang and webhook conformance tests behavioral (#2097) (@yuanchen8911)
• e8acb9c: fix(gate): close residual fail-closed and execution-parity gaps (#2052) (@mchmarny)
• 73cd9ec: fix(kwok): fail closed on unmapped profiles; skip in batch (#2033) (@framsouza)
• 6cb3ab9: fix(kwok): harden profile-select tree-fault handling and test-harness coverage (#2110) (@framsouza)
• 9fafddd: fix(kwok): strict criteria reader in CI classify; document fail-closed contract (#2109) (@framsouza)
• a153567: fix(pod): guard Job watch resume against backoff/cancel race (#2012) (@njhensley)...

This release focuses on expanded Slurm-on-Kubernetes support, air-gapped signing and verification, parallel bundle deployment, and first-class GPU driver ownership.

Highlights

Slurm on Kubernetes

• AKS H100 Slinky/Slurm recipe with Enroot/Pyxis support and Gres/GPU configuration
• New topograph component with the slinky engine for topology-aware scheduling
• Snapshot support for Slinky Slurm and the MariaDB operator
• Functional Kueue with default quota CRs

Recipes & Coverage

• Kubeflow training overlay for RTX PRO 6000 on EKS, plus RTX PRO 6000 inference (Dynamo) evidence
• VR200 training and inference (Dynamo) recipe evidence on RKE2
• AKS H100 driver-only and managed-driver recipe evidence
• Recipe resolution now enforces stated-criteria coverage, failing fast when a recipe cannot satisfy its declared criteria

Air-Gapped Supply Chain - Bundles can now be signed and verified entirely offline: aicr bundle accepts --tlog-upload=false to skip transparency-log upload, and verification gains --insecure-ignore-tlog for disconnected environments. KMS-backed signing also adds HashiCorp Vault support. Verification also got stricter - bundles now verify as fully self-contained, air-gapped artifacts, recipe.yaml is covered by bundle checksums, and AICR now runs its own Rekor v2 identity monitoring for the release signer.

Parallel Deployment - The deployer now deploys independent components in parallel, following the dependency graph so unrelated components no longer wait on each other.

GPU Driver Ownership - Driver management is now an explicit contract: recipes auto-detect a pre-installed GPU driver from the snapshot, AKS defaults to the Azure-managed GPU driver profile, and a bundle-time coherence check catches conflicting driver-ownership configuration before deploy.

Validation & Evidence - The validator gains a configurable inference-perf router mode, recipe-supplied NCCL benchmark runtime via --data, a uniform-RDMA-fabric readiness gate, GPU readiness gated on runtime-required taint clearance, and fail-fast on degraded GPU nodes. The evidence dashboard at validation.aicr.run gains a cross-version combined view with deep links from recipe health.

Other Improvements

• SDK supports custom kubeconfig paths for validation jobs, and aicr validate --kubeconfig now selects the target cluster end-to-end
• aicr diff compares structured snapshot fields
• GPU Operator upgraded to v26.3.3 with driver 580.173.02; nvidia-tuned bumped to 0.3.2
• AICR container images are now based on nvcr.io/nvidia/distroless/static

Thanks to @ArangoGutierrez, @atif1996, @ayuskauskas, @kaynetu, @lockwobr, @mohityadav8, @njhensley, @tjrasche, @xdu31, @yuanchen8911, and @mchmarny.

Changelog

New Features

• d89d2c6: feat(bundle): KMS-backed signing via HashiCorp Vault (hashivault://) (#1729) (@lockwobr)
• 6f9050b: feat(bundle): air-gapped signing with --tlog-upload=false (#1797) (@lockwobr)
• 26afcca: feat(bundler): bundle-time GPU driver-ownership coherence check (#1819) (@yuanchen8911)
• 28d3c51: feat(ci): collect UAT cluster debug bundle on failure (#1764) (@njhensley)
• c5d4ede: feat(ci): consolidate KWOK tier workflows into single reusable runner (#1790) (@mohityadav8)
• 2f6367b: feat(ci): onboard GB200 AWS UAT reservation (#1774) (@njhensley)
• 65f598d: feat(ci): own Rekor v2 identity monitoring via tools/rekor-monitor (#1754) (@lockwobr)
• d1f62de: feat(config): add KMS signingKey to AICRConfig bundle attestation (#1796) (@lockwobr)
• e06d23f: feat(corroborate): add cross-version combined dashboard view (#1771) (@njhensley)
• 76cdb46: feat(deployer): deploy independent components in parallel (#1746) (@lockwobr)
• 6de53db: feat(recipe): auto-detect pre-installed GPU driver from snapshot (#1697) (@xdu31)
• ce7ede7: feat(recipes): add kubeflow training overlay for rtx-pro-6000 on EKS (#1857) (@mohityadav8)
• 13a49f5: feat(recipes): default AKS to Azure-managed GPU driver profile (#1756) (@yuanchen8911)
• 3b3e3a3: feat(recipes): functional kueue with default quota CRs (#1686) (@ArangoGutierrez)
• 709b620: feat(sdk): add support for custom kubeconfig paths for validation jobs to SDK (#1735) (@tjrasche)
• 2fdec48: feat(skills): add aicr-uat-report UAT health report skill (@mchmarny)
• 70eea6c: feat(uat): capture GPU driver state on shortfall in debug bundle (#1867) (@njhensley)
• 477285d: feat(uat): enroll nvkind (kind-h100) in the nightly batch (#1871) (@njhensley)
• de5cea3: feat(uat): nvkind H100 real-hardware evidence lane (DC5) (#1843) (@njhensley)
• ae4c014: feat(validator): configurable inference-perf router mode (#1745) (@yuanchen8911)
• fd4f7a1: feat(validator): recipe-supplied NCCL benchmark runtime via --data (#1805) (@njhensley)
• 6548f17: feat(verify): offline / air-gapped verification (--insecure-ignore-tlog) (#1798) (@lockwobr)

Bug Fixes

• 3574f80: fix(bundler): cover recipe.yaml with bundle checksums (#1750) (@yuanchen8911)
• 2e0bc38: fix(ci): defer release aliases until security gates pass (#1763) (@mchmarny)
• 16b7358: fix(ci): guard UAT image cleanup on numeric version id (#1765) (@njhensley)
• 132dadf: fix(ci): secure GCP UAT actuator execution (#1759) (@mchmarny)
• e105808: fix(cleanup): backstop check namespaces; fence out-of-band installs (#1773) (@lockwobr)
• 958e1de: fix(cli): validate --kubeconfig selects the cluster end-to-end (#1868) (@mchmarny)
• 0d0c1ed: fix(collector): address Slinky snapshot review (@mchmarny)
• 93ef7d5: fix(corroborate): partial phase rollup + show untested categories (#1845) (@njhensley)
• ee10a60: fix(diff): compare structured snapshot fields (#1760) (@mchmarny)
• 7f5f25c: fix(nfd): raise topology-updater memory limit for high-core GPU nodes (#1825) (@njhensley)
• 3fa4614: fix(oci): restore layer title annotation for file-store unpack (#1794) (@njhensley)
• 544491c: fix(recipe): enforce stated-criteria coverage on recipe resolution (#1784) (@mchmarny)
• 3aa5283: fix(recipe): require Kubernetes 1.34 for EKS GB200 training (#1747) (@yuanchen8911)
• 849b959: fix(test): scrub inherited VAULT_* env in vault KMS e2e runner (#1895) (@lockwobr)
• 57019a1: fix(uat): harness correctness bugs + budget-aware nightly time-box (#1847) (@njhensley)
• a897f34: fix(uat): nvkind os-agnostic recipe + dashboard coordinate (#1851) (@njhensley)
• 33f06c5: fix(uat): refresh the EKS STS session mid-gate on AWS (#1872) (@njhensley)
• 431bdc8: fix(validate): never emit evidence in --no-cluster dry-run mode (#1810) (@njhensley)
• 9724b49: fix(validator): fail fast on degraded GPU node; dedicated model-cache timeout (#1866) (@njhensley)
• 49c3cc4: fix(validator): gate GPU readiness on runtime-required taint clearance (#1811) (@njhensley)
• 53a43e9: fix(validator): gate deployment readiness on uniform RDMA fabric (#1865) (@njhensley)
• 1179103: fix(validator): make nodewright deployment health check value-aware (#1849) (@njhensley)
• 2b1be85: fix: enforce closed-world bundle verification (#1758) (@mchmarny)

Other Tasks

• 3cfc476: Accept renamed nvidia-smi banner fields (#1748) (@yuanchen8911)
• 1439f2f: Enable scheduled execution for Rekor Monitor (@mchmarny)
• 86272d0: Fix OCP readiness gate OLM RBAC (#1749) (@yuanchen8911)
• e0efbb5: Revert "feat(slinky-slurm): AKS H100 recipe, Enroot/Pyxis, Gres/GPU (#1744)" (#1788) (@njhensley)
• 08121ad: Update rekor-monitor.yaml (@mchmarny)
• 46b9e01: chore(deps): Update Helm release gpu-operator to v26 (#1804) (@github-actions[bot])
• c1b9653: chore(deps): Update docker.io/library/busybox Docker tag to v1.38.0 (#1880) (@github-actions[bot])
• d989e0c: chore(deps): Update ministackorg/ministack Docker tag to v1.4.3 (#1856) (@github-actions[bot])
• e63b7a3: chore(deps): Update openbao/openbao Docker tag to v2.5.5 (#1786) (@github-actions[bot])
• e3fc8fb: chore(deps): Update supply-chain (#1816) (@github-actions[bot])
• 7d8ac35: chore(deps): Update testing-tools (#1741) (@github-actions[bot])
• b8c0e63...

This release continued our focus on scaling AICR validation across new services, intents, and new accelerators (GB300, metal3, GB200), as well as a new recipe-driven GPU allocation policy.

Highlights

New Accelerators & Services

• GB300 accelerator and metal3 service support
• GB200 Slurm recipe with IMEX compute-domain and channel conformance
• slinky/slurm-operator upgraded to v1.2.0 with configurable operator and webhook placement
• Akamai Cloud / Linode LKE provider-ID mapping

Expanding UAT Validation - The automated User Acceptance Testing (UAT) pipeline added Azure AKS to the AICR fleet with OIDC federation, a phase runner, and per-intent configs. A new reservation broker registry now gates GPU-pool acquisition, ensuring that scheduling lands GPU nodes only when needed. You can view evidence of every AICR UAT validation with a responsive corroborate dashboard UI and version-aware consensus at validation.aicr.run.

GPU Allocation Policy - GPU allocation became a first-class, recipe-driven contract. Recipes now default to the device-plugin allocation strategy in production, the validator enforces the recipe-configured allocation policy, and new capability inspection hardens allocation against misconfiguration.

NCCL & Fabric Validation - NCCL validation matured substantially: a GKE NCCL preflight with launcher-failure mitigations, AKS H100 all-reduce performance validation, recipe-driven benchmark applicability, cross-node fabric resource homogeneity checks, and rotation-proof bandwidth capture via pod termination messages. NCCL launcher diagnostics now surface to stdout for faster root-causing.

OS Tuning - AKS moved off the ib-node-config DaemonSet to nodewright tuning, a tuning-status table is now auto-generated, and nvidia-setup (v0.4.0) and nvidia-tuned (v0.3.1) were bumped.

Bundler

• Multi-tenant Argo deploy options
• argocd-helm children-only render via includeRootApp
• repoURL default baked into values.yaml at bundle push time

Supply Chain & Evidence - Keyless and KMS signing now default to Rekor v2, with Rekor v2 identity monitoring for the release signer and restored SLSA Build L3 image provenance in the reusable workflow. The verifier now fails closed on invalid attestations under a unified allowlist. Evidence collection gained mode-aware CNCF DRA-support and secure-access phases, propagates CNCF section failures, and renders divergent recipe version pins as BOM variants.

Other Improvements

• NFD bumped to v0.19.0, nvidia-dra-driver-gpu to v0.4.1 GA, dynamo-platform/runtime to v1.2.1

Thanks to @ArangoGutierrez, @atif1996, @ayuskauskas, @eljohnson92, @hogeheer499-commits, @kaynetu, @lockwobr, @mohityadav8, @njhensley, @srao-nv, @tjrasche, @yuanchen8911, and @mchmarny.

Changelog

New Features

• ddea27d: feat(bom): render divergent recipe version pins as BOM variants (#1634) (@yuanchen8911)
• 038ae9c: feat(bundle): multi-tenant Argo deploy options (#1656) (@mchmarny)
• eb8684f: feat(bundler): argocd-helm children-only render via includeRootApp (#1724) (@mchmarny)
• fe55152: feat(bundler): bake repoURL default into values.yaml at bundle push t… (#1562) (@mohityadav8)
• 4fe9dc8: feat(ci): Rekor v2 identity monitoring for the release signer (#1727) (@lockwobr)
• a363ce4: feat(ci): add UAT reservation registry and broker helper (#1274) (#1559) (@njhensley)
• bdd91fc: feat(ci): add testgrid-publish workflow (TG5) (#1477) (@srao-nv)
• 79de60e: feat(ci): daytime human-access deployment scheduler (#1281) (#1587) (@njhensley)
• 83d9825: feat(ci): nightly UAT version matrix + versioned install (#1274) (#1579) (@njhensley)
• 55fd14f: feat(ci): per-intent + daytime-lifecycle UAT cluster acquisition (#1586) (@njhensley)
• e4fdf16: feat(ci): route UAT through the reservation broker (#1274) (#1569) (@njhensley)
• 1eb9140: feat(ci): surface superseded UAT runs (#1274) (#1583) (@njhensley)
• 1a0c96f: feat(corroborate): responsive dashboard UI + version-aware consensus (#1571) (@njhensley)
• 8f198eb: feat(evidence): mode-aware CNCF dra-support and secure-access collection (#1694) (@yuanchen8911)
• a0f0f2d: feat(infra): grant classic-ELB sweep permissions to UAT AWS role (#1617) (#1622) (@njhensley)
• a98c845: feat(recipe): add gb300 accelerator and metal3 service support (#1608) (@atif1996)
• d1aaafc: feat(recipes): bump dynamo-platform and runtime to 1.2.1 (#1581) (@yuanchen8911)
• 1805af0: feat(recipes): device-plugin GPU allocation production default (#1671) (@yuanchen8911)
• 1329b7f: feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning (#1698) (@ayuskauskas)
• d331244: feat(tuning): auto-generate nodewright tuning-status table (#1594) (@ayuskauskas)
• 4a762d2: feat(uat): Azure AKS UAT phase runner + intent configs (2/2) (#1722) (@mchmarny)
• 7082dd7: feat(uat): Azure AKS UAT — OIDC federation, dispatch, pipeline (1/2) (#1709) (@mchmarny)
• f37bba2: feat(uat): enroll azure-h100 in the nightly batch with [training] (#1726) (@mchmarny)
• 70bc0b3: feat(uat): gate AWS GPU pool with skyhook runtime-required taint (#1614) (@mchmarny)
• 23e97a7: feat(validator): AKS H100 NCCL all-reduce performance validation (#1695) (@yuanchen8911)
• 4608e13: feat(validator): GKE NCCL preflight + launcher failure mitigations (#1631) (@njhensley)
• 8fa9893: feat(validator): GPU allocation capability inspection and hardening (#1620) (@yuanchen8911)
• 3af23fa: feat(validator): recipe-driven NCCL benchmark applicability (#1719) (@njhensley)
• 526893f: feat: enforce recipe-configured GPU allocation policy (#1664) (@yuanchen8911)

Bug Fixes

• e8450d5: fix(bundler): emit disableValidation on CRD-dependent -post wrappers (#1687) (@mchmarny)
• a4be19a: fix(bundler): honor bundlers query param; driver-managed cleanup (#1606) (@mchmarny)
• ca6799f: fix(ci): build+push aiperf-bench image in UAT so inference-perf runs (#1638) (@njhensley)
• 7e41661: fix(ci): guarantee helm-diff keyring cleanup on all paths (#1578) (@njhensley)
• 01a2f7a: fix(ci): make helm v4.2.2 pin effective, verify helm-diff provenance (#1575) (@njhensley)
• a90193b: fix(ci): point aws-h100 UAT reservation at cr-0e16ad417f9a5bf69 (#1659) (@njhensley)
• 82c752e: fix(ci): reap orphaned LB ELB/SG before UAT VPC teardown (#1617) (#1618) (@njhensley)
• 6205cf0: fix(ci): restore SLSA Build L3 image provenance (reusable workflow) (#1558) (@mchmarny)
• dc12250: fix(ci): target aws-h100 UAT nodes at cr-0e16ad417f9a5bf69 (#1660) (@njhensley)
• f465b92: fix(ci): trim UAT nightly matrix, quiet run-watch, run 2 GPU nodes (#1592) (@njhensley)
• 713f263: fix(evidence): propagate CNCF section failures (#1730) (@hogeheer499-commits)
• 079e173: fix(kwok): deadline-derived sync-gate budgets + 18m tier timeout (#1708) (@ArangoGutierrez)
• 7c7a12b: fix(kwok): mirror registry and gitea images off Docker Hub (#1707) (@ArangoGutierrez)
• a0befce: fix(lke): add provider ID mapping for Akamai Cloud / Linode LKE (#1560) (@eljohnson92)
• 94b05c5: fix(recipe): align H100 NCCL bandwidth gate across services (#1565) (@yuanchen8911)
• 1ee6eeb: fix(recipe): reject ComponentRefs with incoherent type/field combinations (#1585) (@yuanchen8911)
• fbe0e21: fix(recipe): reject enabled Helm refs lacking a deployable, versioned chart primary (#1621) (@yuanchen8911)
• b178e7a: fix(recipe): reject unapplied ComponentRef.Patches; fix coherence godoc (#1589) (@yuanchen8911)
• 13d47cf: fix(recipes): align AKS training with GPU Operator-managed default (#1680) (@yuanchen8911)
• f07962a: fix(recipes): guard overlay version pins against registry defaults (#1572) (@mchmarny)
• 2358d65: fix(recipes): pin toolkit containerd config source to file on AKS (#1689) (@mchmarny)
• 22651e2: fix(recipes): raise ebs-csi node sidecar memory limits for p5 nodes (#1693) (@njhensley)
• ef12565: fix(recipes): raise nvidia-setup-full resources for EFA DKMS build (#1738) (@ayuskauskas)
• 7708b26: fix(uat): bump AKS actuator to v0.1.7 (60m GPU-pool timeouts) (@mchmarny)
• 05b9b54: fix(uat): drop gVNIC network from GKE GPU pool (TCPXO RxDM 7/8) (#1662) (@njhensle...

This release focuses on portable recipe validation evidence, the API domain migration to aicr.run, network topology discovery, and broader conformance validation.

Highlights

Evidence & Trust - Evidence handling matured into an end-to-end signing and verification workflow. aicr evidence sign signs an already-pushed bundle, push gains a --no-sign mode, and pointers are auto-signed on push under a per-source pointer contract with a signer allowlist. A new fork-based signing workflow lets external contributors publish signed evidence, the evidence gate now partitions protected vs. other sources with a component-scoped cascade, and aicr verify --trust-root validates against a private Sigstore trust root. A Rekor identity + consistency monitor watches the transparency log, evidence bundles minimize sensitive content, and the updated ingest path automates bundle verification.

Evidence Dashboard & Corroboration - New corroborate consensus model rolls per-source results into a unified verdict (with PARTIAL handling for incomplete phase coverage) and generates a dashboard published to GitHub Pages at validation.aicr.run. This is initial work using only 1st party data to validate that flow. More to come here in next release.

aicr.run API Domain Migration - The artifact API domain migrated to aicr.run, bumping the contract to v1alpha2/v1beta2 (see ADR-013). aicr validate now warns on aicr version skew across inputs and enforces artifact apiVersion compatibility so mismatched recipes and snapshots fail closed.

Network Topology Discovery - A new NetworkTopology measurement type and collector integrate the l8k library based on NVIDIA k8s-launch-kit project to capture cluster network fabric, enabling cross-repo consumption via Subtype.Items.

New Validators & Recipes

• L40S accelerator support on OKE (Oracle Kubernetes Engine), with a new ol (Oracle Linux) OS value; OKE overlays now require explicit OS selection (no os: any fallback)
• slinky-slurm conformance validator with a Critical User Journey (CUJ) demo
• RoCE NET variant for nccl-all-reduce-bw
• Prometheus/Alertmanager StatefulSet readiness in the kube-prometheus-stack check
• OpenShift Container Platform (OCP) added as a supported --service value

Bundler

• Structured deployer output with color and step headers
• Private-by-default agentgateway inference-gateway exposure
• Dynamic toleration injection for pre-built OCI bundles

Other Improvements

• inference-perf frontend routing switched to least-loaded
• New kwok argocd-git/gitea lanes support in test flow
• Deployment ordering now honors enabled: false

Thanks to @almaslennikov, @atif1996, @haarchri, @kaynetu, @lockwobr, @mohityadav8, @njhensley, @rsd-darshan, @srao-nv, @yuanchen8911, and @mchmarny.

Changelog

New Features

• ddc4dde: feat(bundler): private-by-default agentgateway inference-gateway exposure (#1450) (@yuanchen8911)
• e9aba0d: feat(bundler): structured deploy.sh output with color and step headers (#1393) (@mohityadav8)
• f7545fd: feat(bundler): support dynamic toleration injection for pre-built OCI bundles (#1471) (@mohityadav8)
• 70b743c: feat(ci): GP5 GitHub Pages publish for evidence dashboard (#1504) (@njhensley)
• b3b4eee: feat(ci): enforce single-owner model for /assign and self-only /unassign (@mchmarny)
• 9e6374b: feat(ci): self-serve issue assignment via /assign and /unassign comments (@mchmarny)
• f144ddd: feat(corroborate): consensus model and dashboard generator (#1483) (@njhensley)
• e878161: feat(evidence): --no-sign push and pending/structured-cause verify (#1445) (@mchmarny)
• 14e347a: feat(evidence): GP2 ingest — verify bundles to source-keyed GCS (#1484) (@njhensley)
• a758de2: feat(evidence): aicr evidence sign — sign an existing pushed bundle (#1446) (@mchmarny)
• db3ad73: feat(evidence): auto-sign evidence pointers on push (#1454) (@mchmarny)
• 480b306: feat(evidence): fork-based signing workflow + publishing docs (#1451) (@mchmarny)
• 03b14d9: feat(evidence): minimize sensitive content in evidence bundles (#1414) (@njhensley)
• a2e0690: feat(evidence): per-source pointer contract + signer allowlist (#1401) (#1485) (@njhensley)
• 9be382b: feat(evidence): regenerate signed demo evidence under aicr.run (#1509) (@yuanchen8911)
• da21e22: feat(measurement): NetworkTopology type + Subtype.Items for cross-repo l8k integration (#1417) (@almaslennikov)
• dc23abf: feat(network): NetworkTopology collector + l8k library integration (#1474) (@almaslennikov)
• c7f4373: feat(recipe): add L40S accelerator support (#1510) (@atif1996)
• a1b6e20: feat(recipe): shared coordinate mapping + taxonomy spec (TG6) (#1409) (@mchmarny)
• caecdd0: feat(recipes): add Prometheus/Alertmanager StatefulSet readiness to kube-prometheus-stack check (#1497) (@mohityadav8)
• ffe7522: feat(testgrid): add testgrid-publish CLI tool (TG2) (#1447) (@srao-nv)
• 9a741e6: feat(validate): enforce artifact apiVersion compatibility (#1387) (@mchmarny)
• 68eee88: feat(validate): warn on aicr version skew across inputs (#1386) (@mchmarny)
• d57f7ac: feat(validation): switch inference-perf frontend routing to least-loaded (#1399) (@yuanchen8911)
• 381ba2a: feat(validators): RoCE NET variant for nccl-all-reduce-bw (#1428) (@yuanchen8911)
• f0ddcd0: feat(verify): private Sigstore trust root (aicr verify --trust-root) (#1449) (@lockwobr)
• 9be847e: feat: migrate API domain to aicr.run, bump v1alpha2/v1beta2 (#1499) (@mchmarny)

Bug Fixes

• 732a8dd: fix(attestation): retry ambient OIDC token acquisition (#1389) (@mchmarny)
• 24edd51: fix(ci): correct docs-only detection in merge gate (#1381) (@mchmarny)
• eebd348: fix(ci): link digest-pinned evidence ref in UAT summaries (#1498) (@njhensley)
• eea2a81: fix(cli): validate/recipe UX papercuts (#1383 items 1-7) (#1391) (@mchmarny)
• fc5e814: fix(corroborate): PARTIAL rollup for incomplete phase coverage (#1503) (@njhensley)
• ab022b2: fix(corroborate): escape boot() error + fail-closed meta schema gate (#1500) (@njhensley)
• fd8ba27: fix(evidence): key-constrain all redaction allowlist subtypes (#1419) (@njhensley)
• e19674c: fix(evidence): make unsigned-commit CI-sign flow coherent (#1530) (#1538) (@njhensley)
• 7839464: fix(evidence): relocate idempotency, strict digest, package docs (#1543) (@njhensley)
• 9cd3fb7: fix(recipe): fix deployment ordering to honor enabled:false (#1465) (@yuanchen8911)
• 7f7ae67: fix(test): argocd-sync gate: use all-semantics, handle health-gated apps (#1397) (@rsd-darshan)
• a0f2d09: fix(uat): bind Prometheus PVC to cluster default StorageClass (#1455) (@njhensley)
• 026804e: fix(uat): gate install on the deployment validation phase (#1439) (@njhensley)
• ebbd322: fix(uat): gate on Prometheus readiness; retry conformance metric APIs (#1452) (@njhensley)
• 6663eb5: fix(uat): gate validate on cluster convergence (readiness wait + failFast) (#1426) (@njhensley)
• c4f0872: fix(uat): run all validate phases to gate on GPU readiness (#1416) (@njhensley)
• 58721c5: fix(uat): wait for nodewright tuning before validate readiness gate (#1429) (@njhensley)
• 6ac2cfc: fix(validator): drop single-shot ClusterPolicy check from expected-resources (#1495) (@njhensley)
• 410e31a: fix(validator): pod-autoscaling passes on external-metric HPA path (DRA clusters) (#1408) (@yuanchen8911)
• 4b24b7d: fix(validator): wait for KAI deployments ready in gang-scheduling check (#1514) (@mchmarny)
• a74fe77: fix: harden timeout/false-PASS handling, evidence integrity, CI gaps (#1511) (@mchmarny)

Other Tasks

• bf294ed: Add OpenShift Container Platform (OCP) as a service type (#1380) (@kaponco)
• 3307169: Delete .github/workflows/devtrace.yaml (@mchmarny)
• e927f39: chore (validator): pin Trainer self-install JobSet image to promoted registry (#1440) (@yuanchen8911)
• b391a17: chore(ci): add kaynetu to copy-pr-bot trustees (#1412) (@kaynetu)
• 9c2d362: chore(deps): Update dependency awscli to ...

This release focuses on recipe health scoring, improved deployment validation, improved snapshot/discovery, and extending software supply chain capabilities for enterprise users.

Highlights

Recipe Structural Health - New pkg/health engine computes per-recipe health signals (chart_pinned, constraints_wellformed, declared_coverage) and rolls them up into a recipe-health matrix. aicr recipe list surfaces structural-health columns (with a --no-health opt-out), a tools/health generator and weekly recipe-health-refresh workflow keep the matrix current, and a lint guard now requires healthCheck.assertFile.

Improved Deployment Validation - The chainsaw deployment-phase runner is now an in-process executor rather than a shelled-out binary. aicr validate runs all phases by default with a --fail-fast opt-in, fails closed on evaluator errors, and is nil-safe across health checks.

Snapshot/Discovery - The collector now discovers GPU SKUs without nvidia-smi, removing the CUDA base image dependency and matching SKUs on token boundaries instead of substrings.

Closed Supply Chain - Signing and verification now work end-to-end in air-gapped and enterprise environments. aicr bundle supports KMS-backed signing (--signing-key) and private Sigstore deployments (--fulcio-url, --rekor-url); aicr verify --key validates bundles against a KMS or public key; and aicr evidence publish signs recipe evidence off-network. The recipe catalog itself now ships signed provenance for the V1 closed supply chain, and keyless signing warns before publishing identity to the public transparency log.

New Recipes & Overlays

• A100 training Kubeflow overlay chains for EKS, AKS, GKE COS, and OKE
• GB300 concrete EKS service-bound overlays
• OKE GB200 and AKS H100 Dynamo performance checks

CLI & Bundling

• aicr recipe list subcommand for catalog enumeration
• Gatekeeper added as an optional component

Inference Performance & Validation

• Inference-performance validation enhanced and tuned; gated on all worker services Ready
• nccl-all-reduce-bw gates wired for EKS + H200; GKE NCCL node selector made dynamic
• Bounded absent-resource retries in deployment-phase health checks

Thanks to @atif1996, @cdesiniotis, @dims, @haarchri, @JaydipGabani, @lalitadithya, @lockwobr, @njhensley, @pdmack, @pedjak, @rsd-darshan, @sttts, @xdu31, @yuanchen8911, and @mchmarny.

Changelog

New Features

• cfb0cb0: feat(agentgateway): scope inference-gateway LB to allowed source ranges (#1138) (@yuanchen8911)
• 3b8b1f3: feat(bundle): KMS-backed signing via --signing-key (#407) (#1205) (@lockwobr)
• 5316d3c: feat(bundle): private Sigstore via --fulcio-url and --rekor-url (#1158) (@lockwobr)
• b847f96: feat(bundler): retry sign.Bundle on transient Sigstore failures (#1251) (@mchmarny)
• 3c1f525: feat(bundler): warn on open agentgateway inference-gateway exposure (#1163) (@yuanchen8911)
• cd30fe5: feat(ci): add weekly recipe-health-refresh workflow (#1320) (@njhensley)
• 5f8647d: feat(cli): add --no-health opt-out to recipe list (#1314) (@njhensley)
• f0490fb: feat(cli): add --set-json/--set-file for list and object bundle overrides (#1162) (@yuanchen8911)
• b401339: feat(cli): add structural-health columns to recipe list (#1302) (@njhensley)
• a47a53f: feat(cli): warn before keyless signing publishes identity to public log (#1300) (@njhensley)
• 97934ad: feat(collector): driver-free GPU SKU discovery; remove nvidia-smi + CUDA base (#1352) (@mchmarny)
• eb8728d: feat(coverage): generated CUJ/CLI coverage matrix (RQ3) (#1316) (@mchmarny)
• 1674ea4: feat(evidence): add aicr evidence publish for off-network signing (#1140) (@njhensley)
• e1b0160: feat(health): add tools/health generator and recipe-health matrix (#1304) (@njhensley)
• 8d0da78: feat(health): chart_pinned signal + declared_coverage descriptor (#1293) (@mchmarny)
• fa92b43: feat(health): constraints_wellformed signal (parse-only, hermetic) (#1301) (@njhensley)
• 10b08f1: feat(health): pkg/health core Compute loop, resolves signal, rollup (#1291) (@mchmarny)
• 3e2f823: feat(recipe): add AKS H100 Dynamo perf check (#1232) (@yuanchen8911)
• 8f8bc56: feat(recipe): add OKE GB200 perf check (#1233) (@yuanchen8911)
• ec95e20: feat(recipe): add aicr recipe list subcommand for catalog enumeration (#1208) (@rsd-darshan)
• 57fbed0: feat(recipe): hydrate healthCheck.assertFile + suppression sentinel (#1231) (@mchmarny)
• ae6819c: feat(recipe): lint guard requiring healthCheck.assertFile + allowlist (#1244) (@mchmarny)
• 0bf2267: feat(recipe): signed catalog provenance for V1 closed supply chain (#1216) (@mchmarny)
• 463d6a1: feat(recipes): add A100 AKS training Kubeflow overlay chain (#1295) (@yuanchen8911)
• d8d3070: feat(recipes): add A100 EKS training Kubeflow overlay chain (#1305) (@yuanchen8911)
• fd64dd7: feat(recipes): add A100 GKE COS training Kubeflow overlay chain (#1306) (@yuanchen8911)
• 6eb85ac: feat(recipes): add A100 OKE training Kubeflow overlay chain (#1294) (@yuanchen8911)
• 4b817ce: feat(recipes): add concrete GB300 EKS service-bound overlays (#1319) (@yuanchen8911)
• cad0142: feat(recipes): backfill chainsaw health checks for 5 missing components (#1243) (@mchmarny)
• 81daab3: feat(recipes): deepen 21 chainsaw health checks; close epic #660 (#1245) (@mchmarny)
• 7bb7059: feat(recipes): migrate nvidia-dra-driver-gpu to registry.k8s.io v0.4.0 (#1285) (@mchmarny)
• e848e1f: feat(tests): KMS bundle-signing e2e against MiniStack over TLS (#1298) (@lockwobr)
• 81c3fb0: feat(tests): private-Sigstore bundle-signing e2e via Helm scaffold (#1321) (@lockwobr)
• 25a6cdd: feat(validator): ship chainsaw binary; activate deployment-phase runner (#1235) (@mchmarny)
• 4d5ac90: feat(validators): enhance inference performance validation (#1133) (@yuanchen8911)
• f6cb3cd: feat(validators): replace chainsaw binary with in-process executor (#1252) (@mchmarny)
• e3460fc: feat(verify): KMS/public-key bundle verification (aicr verify --key) (#1238) (@lockwobr)

Bug Fixes

• e3aa6b4: fix(bundler): disable kataSandboxDevicePlugin in gpu-operator values (#1343) (@atif1996)
• 5f51fe8: fix(ci): actually tear down AWS UAT cluster (destroy → apply) (#1213) (@njhensley)
• 484c61a: fix(ci): always upload recipe-evidence report so comment gate works (#1292) (@njhensley)
• eddc075: fix(ci): build patched nvkind with --config-source=file (#1237) (#1258) (@mchmarny)
• 1ef5bdb: fix(ci): resolve fork PRs for recipe-evidence sticky comment (#1297) (@njhensley)
• 7068779: fix(ci): shard Tier 3 KWOK matrix to stay under 256-config cap (#1173) (@njhensley)
• ff8a756: fix(ci): stamp publish with resolved tag instead of releases/latest API (#1136) (@pdmack)
• 85daf65: fix(ci): suppress chainsaw CVEs + apply VEX on release scan (#1366) (@mchmarny)
• 4e8f778: fix(ci): unblock build-attested workflow on missing HOMEBREW_DEPLOY_KEY (#1296) (@lockwobr)
• 4e86524: fix(docs): catch bare tags in MDX safety check (#1170) (@pedjak)
• c606d17: fix(docs): cover contributor docs in MDX check; catch autolinks (#1151) (@mchmarny)
• 915ed66: fix(docs): escape bare < for Fern MDX + harden MDX checker (#1367) (@mchmarny)
• 4b2864a: fix(docs): keep --set-json code span on single line for MDX check (@mchmarny)
• de2d9dd: fix(docs): use MDX comments in recipe-health.md so Fern publish parses (#1365) (@mchmarny)
• 833a2e5: fix(evidence): emit recipe-evidence pointer.yaml at 2-space indent (#1165) (@yuanchen8911)
• af563b2: fix(evidence): pull by digest and auto-tag pushes instead of :v1 (#1168) (@njhensley)
• 410f5a3: fix(fingerprint): match GPU SKUs on token boundaries not substrings (#1350) (@mchmarny)
• 6e95906: fix(health): exempt manifest-only Helm components from chart_pinned (#1303) (@njhensley)
• e32375c: fix(recipes): pin nvidia-dra-driver-gpu to 0.4.1-rc.1 for strict-YAML fix (#1341) (@yuanchen8911)
• 032b707: fix(scan): match VEX PURL to grype's image PURL + surface CVE IDs (@mchmarny)
• 909629b...

This release focuses on further expansion to AICR recipe matrix (H200, B200, RTX PRO 6000, BCM, Slinky Slurm, Run:ai), new Go client for embedding AICR, and significant engine hardening that removes process-global state from the recipe and improves validator pipelines.

Highlights

In-Process Go Library — New pkg/aicr package exposes an aicr.Client facade for in-process consumers, allowing products and SDKs to drive recipe resolution, bundling, and validation without forking a CLI. The CLI and HTTP server already implement thin adapters over this facade.

aicr mirror — New top-level command for mirroring container images referenced by a recipe to an alternate registry, completing the air-gapped story that began with Helm vendoring in v0.13.0. The command reuses the recipe-bound DataProvider so its manifest reads are identical to what bundle and validate see.

Engine Hardening — The recipe and validator pipelines are now free of process-global state. Builder owns an isolated DataProvider, the criteria registry is per-provider (no more singleton), and the DataProvider interface is context-aware. Embedders can safely run multiple Builder instances concurrently against different sources.

New Recipes & Overlays

• H200 promoted to a first-class accelerator type with EKS overlays
• GKE B200 service-bound overlays
• RTX PRO 6000 Blackwell (B40) overlays for EKS
• BCM service type added with overlays and nodewright reapply-on-reboot
• NVIDIA Run:ai platform support
• Slinky Slurm gains a cluster chart with EKS, Kind, and GKE COS H100 leaves
• H100 and generic tuning extended to H200 and RTX PRO 6000 EKS recipes

Validation & Performance

• Strict performance floors scoped to accelerator-bound recipes
• Deployment-phase floor delivered at per-accelerator wildcards
• Per-field union merge for validation phase checks

Other Improvements

• Bundle command supports app name for configurable Argo CD parent
• argocd-helm mixed-component bundles use native OCI source shape
• Recipe-set scheduling paths now correctly override CLI defaults
• Kubeconfig-aware serializer for ConfigMap output

Thanks to @ayuskauskas, @faganihajizada, @fallintoplace, @gat786, @haarchri, @hkii, @lockwobr, @njhensley, @pdmack, @resker, @xdu31, @yuanchen8911, and @mchmarny.

Changelog

New Features

• ae6c948: feat(aicr): top-level Go library facade for in-process consumers (#1072) (@hkii)
• 5c63c10: feat(aicr): wrap facade alias types as facade-owned structs (#1111) (@mchmarny)
• 971024e: feat(bundler): configurable parent Application name (--app-name) (#1036) (@mchmarny)
• 7adc586: feat(bundler): derive DRA chart-version annotation from resolved recipe (#973) (#1033) (@yuanchen8911)
• 9a31cc0: feat(bundler): drop generated undeploy.sh; delegate to helm uninstall (#1095) (@mchmarny)
• 6098765: feat(bundler): route registry/manifest reads through recipe-bound provider (#1016) (@mchmarny)
• ed6b480: feat(ci): allow publish-fern-docs to target a specific or latest tag (@mchmarny)
• 9e1841f: feat(cli): add evidence digest subcommand for recipe canonical hash (#1055) (@njhensley)
• 0e10705: feat(cli): add release-notes drafting skill for Claude Code (@mchmarny)
• 940fb15: feat(fern): adopt global-theme nvidia, remove per-repo theme assets (#995) (@pdmack)
• 095eb8d: feat(mirror): add mirror command (#967) (@haarchri)
• a1fc3ab: feat(mirror): thread recipe-bound DataProvider through manifest reads (#1123) (@mchmarny)
• 735c233: feat(recipe): add bcm service type with overlays (#1060) (@mchmarny)
• 94149a9: feat(recipe): context-aware DataProvider interface (#1121) (@mchmarny)
• c210874: feat(recipe): deliver deployment-phase floor at per-accelerator wildcards (#1001) (@yuanchen8911)
• c148e96: feat(recipe): extensible criteria values via catalog-driven runtime registry (#998) (#999) (@mchmarny)
• 4e6cd9c: feat(recipe): per-Builder DataProvider isolation; deprecate process globals (#1015) (@mchmarny)
• 76647b7: feat(recipe): per-field union merge for validation phase checks (#1103) (@mchmarny)
• cb098fa: feat(recipe): push owner-token guard down to pkg/recipe.RecipeResult (#1113) (@mchmarny)
• ba787a5: feat(recipe): register h200 as first-class accelerator type (#1091) (@yuanchen8911)
• f385af2: feat(recipes): add RTX PRO 6000 Blackwell (B40) overlays for EKS (#1046) (@yuanchen8911)
• bbf8176: feat(recipes): add concrete GKE B200 service-bound overlays (#1053) (@yuanchen8911)
• 8b49397: feat(recipes): add inference-perf to gb200-eks-ubuntu-inference-dynamo (#977) (@yuanchen8911)
• 81c0789: feat(recipes): add nodewright h100 tuning to H200 EKS recipes (#1102) (@yuanchen8911)
• 178bbe4: feat(recipes): add nodewright to bcm with reapply-on-reboot (#1105) (@ayuskauskas)
• 4daf2af: feat(recipes): apply nodewright generic tuning to rtx-pro-6000 EKS (#1101) (@yuanchen8911)
• cce25f0: feat(recipes): pin nodewright-customizations packages by digest (#1037) (@ayuskauskas)
• 8435e1f: feat(validation): scope strict perf floor to accelerator-bound recipes (#1009) (@yuanchen8911)
• e565ce0: feat(validator): co-locate ai-service-metrics with Prometheus (#1066) (@njhensley)
• e30f18a: feat(validator): parameterize image override env vars (#1028) (@haarchri)
• 686d329: feat(validators): warm up inference-perf benchmark and make it tunable (#1096) (@yuanchen8911)
• 2dfa5e3: feat: add NVIDIA Run:ai as platform-runai (#955) (@resker)

Bug Fixes

• 506507b: fix(bundler): argocd-helm parent App uses native-OCI source shape (#1051) (@yuanchen8911)
• b05fc2e: fix(bundler): argocd-helm repoURL UX — surface key, emit install hint (#1081) (@mchmarny)
• 04e4418: fix(bundler): correct helmfile bundle README for stratified layout (#959) (@lockwobr)
• 51eb4b5: fix(bundler): derive argocd-helm chart name from OCI artifact path (#1032) (@mchmarny)
• d638d00: fix(bundler): fix bugs - assemble full OCI artifact in path-based child apps; quote chart name (#1034) (#1035) (@yuanchen8911)
• ef16e9e: fix(bundler): pin OCI sync for argocd-helm mixed-component -pre/-post children (#1039) (@mchmarny)
• ceec36e: fix(bundler): quote argocd app-of-apps metadata.name (#1011) (#1040) (@yuanchen8911)
• 3eab313: fix(bundler): recipe-set scheduling paths override CLI defaults (#982) (#1082) (@mchmarny)
• a578244: fix(bundler): replace placeholder GitRepository with ArtifactGenerator for Flux OCI (#1017) (@haarchri)
• e83728f: fix(bundler): scope helmfile disableValidation to primary release (#1125) (@mchmarny)
• 91a4552: fix(bundler): tolerate trailing slash on repoURL; quote Chart.yaml version (#1038) (@yuanchen8911)
• eec6349: fix(ci): broaden Renovate postUpgradeTasks allowlist (@mchmarny)
• 31c912a: fix(ci): drop privileged grants from /ok-to-test fork path (#1067) (@mchmarny)
• 37b3127: fix(ci): move fern registry PR step after publish (@mchmarny)
• 620c6f6: fix(ci): open PR instead of pushing fern version registry to main (@mchmarny)
• e423b8f: fix(ci): restore fetch-tags in publish-fern-docs checkout (@mchmarny)
• 734d3a9: fix(ci): retry release-asset curls to absorb transient 502s (@mchmarny)
• 423bc08: fix(ci): skip version registration for latest release and sort by semver (#979) (@pdmack)
• 1c82645: fix(ci): unblock packaging dry-run and /ok-to-test startup failures (#1128) (@mchmarny)
• 3c2aa56: fix(ci): use yq instead of sed for version stamp in publish workflow (#943) (@pdmack)
• 2cd479f: fix(deps): refresh helmfile checksums for v1.5.2 (@mchmarny)
• 45f3f1a: fix(kwok): drop recipe suffix from argocd-helm-oci in-cluster repoURL (#1047) (@yuanchen8911)
• b0bf12d: fix(kwok): make argocd OCI repoURL per-lane (follow-up to #1047) (#1048) (@yuanchen8911)
• 5f72700: fix(kwok): require Succeeded operationState in argocd sync gate (#1062) (@yuanchen8911)
• 216c76c: fix(kwok): single-snapshot verify_pods + settle retry (#1090) (#1092) (@yuanchen8911)
• 047123f: fix(performance): add GKE inference performance validation + cold-s...