feat(evidence): publish GB300 EKS training and inference evidence at v0.21.1

[yuanchen8911]

Summary

Publishes signed recipe-evidence for both v1 Supported GB300 EKS coordinates, produced at the released v0.21.1 binary:

Coordinate	Bundle digest	Rekor
gb300-eks-ubuntu-training-kubeflow	sha256:320d48c1…cc2e0	114273867
gb300-eks-ubuntu-inference-dynamo	sha256:f8988045…2619a	114273668

The existing evidence for these coordinates records aicrVersion: dev in its signed predicate, which is what #2806 reports: the board shows evidence, but not against a shipped recipe version, so AICR's generated health table renders both rows as pending.

Motivation / Context

Refs #2806

ROADMAP §2 declares both coordinates Supported with "deployment, conformance, and performance evidence published". These bundles record aicrVersion: 0.21.1, supplying that at a released version.

Type of Change

• New feature (non-breaking change that adds functionality)

Component(s) Affected

• Recipe engine / data (recipes/)

Implementation Notes

Run performed on EKS GB300 (p6e-gb300r.36xlarge, 2 GPU nodes / 8 GPUs, Ubuntu 24.04, K8s v1.35.6), deployed from a clean teardown: full tools/cleanup, GPU-node reboot (driver-managed cluster), then training bundle, then inference layered incrementally.

Signed via the fork CI workflow with GitHub Actions ambient OIDC — no personal identity in the transparency log. Source slug 5bf9e82f… is already allowlisted, so no allowlist.yaml change is needed.

Testing

Both recipes validated with aicr validate --emit-attestation, all phases green:

gb300-eks-ubuntu-training-kubeflow — deployment 4/4, conformance 8/8, performance 2/2

• nccl-all-reduce-bw-net: 44.98 GB/s (threshold 40, floor 36)
• nccl-all-reduce-bw-nvls: 843.51 GB/s (threshold 700, floor 630)

gb300-eks-ubuntu-inference-dynamo — deployment 4/4, conformance 11/11, performance 1/1

• inference-perf: 86,105.78 tok/s (constraint ≥ 60000), TTFT p99 183.51 ms (constraint ≤ 2000)

aicr evidence verify returns exit 0 for both pointers.

Risk Assessment

Low — adds two evidence pointer files, no code or recipe changes.

One thing reviewers should weigh rather than discover later:

• The inference run required a manual workaround for kai-scheduler Helm upgrade silently breaks all gang scheduling (stale ServiceAccount token) #2636. Layering the inference bundle onto the training deployment upgrades kai-scheduler, which leaves kai-scheduler-default holding a stale ServiceAccount token; gang-scheduling failed and Dynamo workers would not schedule. kubectl rollout restart deployment/kai-scheduler-default cleared it (401s 18→0), after which conformance passed 11/11. kai-scheduler Helm upgrade silently breaks all gang scheduling (stale ServiceAccount token) #2636 is closed but its proposed kai-scheduler-post mitigation was never implemented, so any two-intent incremental deploy still hits this.

Checklist

• Commits signed (-S) and signed off (-s)
• aicr evidence verify exit 0 on both pointers
• No allowlist change required (existing allowlisted source slug)

[github-actions]

Recipe evidence check

Protected recipes

Recipes with committed evidence (recipes/evidence/<slug>/<source>/<digest>.yaml) that this PR affects: 2

Recipe	Source	Pointer	Verify	Digest match
gb300-eks-ubuntu-inference-dynamo	5bf9e82f0e90a11528ac85f4bcb866c8	sha256-b6f03b62702a258a1d5049a4a56eaa1685af63de5dbb1dcb7491e2bbce5a7e3a	✅ passed	⚠️ stale (52e5b9bc9ada… vs current 0696c5c33d88…)
gb300-eks-ubuntu-inference-dynamo	5bf9e82f0e90a11528ac85f4bcb866c8	sha256-f89880455101b90b092ccb549dac7bce1112525055245189b94dc9343262619a	✅ passed	⚠️ stale (394770514dfa… vs current 0696c5c33d88…)
gb300-eks-ubuntu-training-kubeflow	5bf9e82f0e90a11528ac85f4bcb866c8	sha256-320d48c10adeaded2a304dd4e0db12b7586b4dc9180e12a440caf8ae575cc2e0	✅ passed	⚠️ stale (e5a5ebcddc9f… vs current 2c04825e44de…)
gb300-eks-ubuntu-training-kubeflow	5bf9e82f0e90a11528ac85f4bcb866c8	sha256-c19d7932a51fc76366eb095a95c57fdaaa13d5b5cd48b77635dc1d58ec8ed886	✅ passed	⚠️ stale (de43585aa39f… vs current 2c04825e44de…)

How to refresh evidence

Run on a cluster matching the recipe's criteria:

aicr snapshot -o snapshot.yaml
# Profiled families (AKS/GKE gpuStack): hydrate the recipe with the
# pointer's recorded 'profile:' selection first — validating the raw
# overlay resolves only the declaration default, and 'aicr validate'
# has no --profile flag. AKS additionally needs the pool projection
# (GKE uses the plain snapshot above):
#   az aks nodepool list -g <rg> --cluster-name <cluster> -o json > pools.json
#   aicr snapshot --aks-gpu-pools pools.json -o snapshot.yaml
#   aicr recipe -s snapshot.yaml --intent <intent> [--platform <platform>] \
#     --profile <name>=<value> -o recipe.yaml
# State the target leaf's intent/platform explicitly (the snapshot
# fingerprint supplies service/accelerator/OS but intent and platform
# default to 'any') and pass -r recipe.yaml below instead of the raw
# overlay.
aicr validate \
  -r recipes/overlays/<slug>.yaml \
  -s snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<your-fork>/aicr-evidence
# Copy to the per-source path printed in the emit 'copyTo' hint:
#   recipes/evidence/<slug>/<source>/<bundle-digest>.yaml

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 9fb30896-d0e2-4ca8-9f58-6d8e239e2c0f

📥 Commits

Reviewing files that changed from the base of the PR and between f594793 and f509629.

📒 Files selected for processing (2)

• recipes/evidence/gb300-eks-ubuntu-inference-dynamo/5bf9e82f0e90a11528ac85f4bcb866c8/sha256-f89880455101b90b092ccb549dac7bce1112525055245189b94dc9343262619a.yaml
• recipes/evidence/gb300-eks-ubuntu-training-kubeflow/5bf9e82f0e90a11528ac85f4bcb866c8/sha256-320d48c10adeaded2a304dd4e0db12b7586b4dc9180e12a440caf8ae575cc2e0.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Walkthrough

Walkthrough

Added schema-versioned attestation metadata for the gb300-eks-ubuntu-inference-dynamo and gb300-eks-ubuntu-training-kubeflow recipes. Each record includes its evidence bundle digest and OCI location, timestamp, predicate type, GitHub Actions signer, token issuer, and Rekor log index.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to f5096

The added evidence records conform to the repository’s validation contract, with no confirmed merge-blocking risk.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the publication of GB300 EKS training and inference evidence for version v0.21.1.
Description check	✅ Passed	The description directly explains the two signed evidence bundles, validation results, affected recipes, and the deployment workaround.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[yuanchen8911]

Rebased onto current main after #2814: f509629 → e07851e. The evidence-only commit rebased without conflicts, and make qualify passes.