[Epic]: k8s-aibom - stock adoption in one GKE recipe, proven end to end #2271

Goal

Adopt k8s-aibom into exactly one stock GKE recipe and prove the full AICR workflow end to end on a real GKE cluster: recipe generation selects the component, the bundle deploys it, the health check passes, and the controller emits a correct ML-BOM for a real GPU inference workload.

This is the follow-on phase to #2234, which admitted k8s-aibom v1.2.0 to the component registry as an optional, registry-only component under ADR-019 (Accepted 2026-08-19).

Registry-only adoption proved the component is safe to offer. It deliberately proved nothing about what happens when a stock recipe ships it by default, because no stock recipe references it and the Kind integration test is a simulation of a managed cluster rather than one.

Decisions taken with upstream, 2026-08-19

Agreed with the k8s-aibom maintainers:

1. Upstream ships a v1beta1 API. Schema-identical to v1alpha1 (field-frozen since v1.0.0), conversion stays None with no webhook, storage flips to v1beta1, and v1alpha1 remains served through 1.x. Upstream design note in progress.
2. AICR's "non-alpha storage API" requirement means the storage flip, not object migration. The gate is spec.versions[?storage].name == 'v1beta1' on the served CRD. Migrating existing stored objects and cleaning status.storedVersions are not required at graduation, because conversion None over an identical schema makes existing objects readable immediately. Those become the gate for a later v1alpha1 removal release, which upstream will announce separately with a documented migration step.
3. This phase pins v1.2.0 now and requalifies after graduation. Waiting would stall version-independent work on a date we do not control.
4. The Kubernetes range question is closed. Upstream is replacing the fixed 1.27-1.35 statement with a policy (stable APIs only, no known ceiling, tested floor 1.27) backed by a version-matrix CI job. AICR's Kind 1.36.1 pass becomes part of that evidence.

Stability boundary: no preview label

Discussion with upstream used the phrase "explicit preview label." AICR will not implement one. Everything in recipes/registry.yaml is assumed stable; that is a core premise of the project, and a preview-labeled registry entry would contradict it.

AICR already has the mechanism this phase needs, and it is the registry-only versus stock-recipe boundary that #2234 established:

• Registry presence asserts the component is qualified and stable enough to offer. k8s-aibom is already here.
• Stock recipe presence asserts AICR ships it by default. That is the assertion which must wait for v1beta1.

So the sequencing is: everything except the overlay merge proceeds now against v1.2.0, and the overlay change lands only once the graduation release is qualified. The end-to-end GKE demonstration runs before then against a custom recipe or an unmerged branch, which proves the workflow without asserting stock stability.

This is recorded in ADR-019's amendment (2026-08-19), section B. No separate ADR-020 is being written: ADR-019's Follow-Up Decisions permits "a separate ADR or explicit amendment," and the amendment carries only the decisions that change a published contract. This epic carries the plan.

Why this is the prescribed next step

ADR-019's Follow-Up Decisions section already specifies this phase. It requires a separate ADR or explicit amendment defining six things:

Related, but not a dependency: the CRD upgrade path

#2264 is decoupled from this epic. It is an independent P1 on its own merit: the Flux CRD-upgrade gap affects every CRD-shipping component AICR delivers, not just this one, and upstream is not gating their graduation release on it either.

It stays relevant as context rather than as a gate. The graduation release is upstream's first CRD-changing release, and on Flux, Helm, and Helmfile an existing CRD is never updated on upgrade (only Argo CD does). So whatever state #2264 is in when #2282 runs is what the cross-boundary upgrade evidence will record: with the fix, evidence of a working upgrade path; without it, documented evidence of the gap on three of five deployers. Both are legitimate outcomes for that issue, and neither blocks it.

Note also that the stranded-CRD case is not silent, contrary to an earlier reading recorded here. Upstream's controller requests v1beta1 via informers, a stranded CRD does not serve it, caches never sync, and readiness gates on cache sync as of v1.1.0, so the pod holds NotReady. AICR's existing Deployment health-check assertion already fails on that. See the correction on #2281.

What "end to end on a real cluster" must prove

On a live GKE cluster, not Kind:

1. aicr recipe for the target criteria resolves with k8s-aibom selected, recorded in the recipe, and opt-out expressed as a recipe-level contract.
2. aicr bundle renders the component through every supported deployer, and the deployed bundle installs cleanly on GKE.
3. aicr validate passes, including the k8s-aibom health check, against the managed control plane.
4. A real GPU inference workload in an opted-in namespace produces a correct CycloneDX ML-BOM.
5. Controller and API-server cost measured on GKE, compared against ADR-019's Kind-measured envelope (50m/128Mi requests, 1 CPU/256Mi limits).
6. Upgrade, rollback, and uninstall evidence captured across the v1.2.0 to graduation-release boundary, not within either version. Since that boundary is upstream's first CRD change, running it there makes the re-pin the most valuable evidence in the phase instead of duplicated effort.

Any stock recipe not in scope must still resolve to the same bytes it does today. The golden parity tests added in #2259 (pkg/bundler/testdata/stock_render_golden.yaml, pkg/recipe/testdata/catalog_parity_golden.yaml) will change for the target recipe only; that diff is the proof, and it should be reviewed line by line rather than regenerated.

Target recipe

Confirmed: h100-gke-cos-inference (overlay).

• Inference produces the most legible ML-BOM content, since the point of an AI BOM is inventorying models and AI workloads rather than generic pods. A training recipe demonstrates the same mechanics with a weaker story.
• H100 on GKE COS is a mainstream, widely exercised combination, so the demo is representative rather than exotic.
• It is a leaf overlay, so adoption does not touch gke-cos-inference, gke-cos, or any shared base, keeping the blast radius to one recipe as intended.

Fallback if a real model-serving workload proves impractical to stand up on GKE: h100-gke-cos-training, same mechanics with a weaker AIBOM narrative. Switching would require amending the ADR-019 amendment section C, not a silent substitution.

Sequence

1. Selection and opt-out semantics decided. The one requirement the ADR-019 amendment left open. ComponentRef.IsEnabled() already reads a recipe-recorded enabled override, so the question is narrower than it first looked: does a user generating from a stock recipe decline the component by authoring a custom overlay (existing mechanism), or by a generation-time flag that aicr recipe records into the emitted recipe (new CLI contract)? Resolve by amending the ADR section, then proceed.
2. Storage-version assertion added to the health check, so a stranded CRD fails validation instead of passing quietly.
3. Selection semantics implemented in the recipe layer, with opt-out recorded at generation time.
4. End-to-end GKE demonstration against v1.2.0, via a custom recipe or unmerged branch.
5. Graduation release requalification and re-pin, capturing the cross-boundary upgrade evidence.
6. Overlay change adding the component to the one target recipe, with golden-parity diffs reviewed. Lands only after step 6.
7. Docs: component catalog, recipe docs, and the upstream range policy pointer.

Out of scope

• Any recipe other than the single named target. No shared base, no mixin, no second service.
• A preview, maturity, or stability field in the component registry. See the stability boundary section above.
• A new AICR evidence predicate for AIBOM output, or any change to ADR-007. ADR-019 keeps this a separate decision, and folding it in here would couple a deployment question to an evidence-format question.
• Consuming the upstream signature verifier.
• Provisioning sinks, buckets, webhook receivers, identities, credentials, or retention policies.
• Automatic progression from this recipe to broader adoption. That is a later decision informed by what this phase measures.

Open questions

• Does "end to end" here include surfacing the AIBOM as AICR evidence, or is producing a correct ML-BOM on the cluster sufficient for this phase? Scoped out above; say so if that is wrong.
• Readiness gates, not dates, govern this phase. k8s-aibom: assert CRD storage version in the health check #2281 gates k8s-aibom: requalify and re-pin against the v1beta1 graduation release #2282; k8s-aibom: requalify and re-pin against the v1beta1 graduation release #2282 gates the overlay merge. fix(bundler): Flux HelmRelease never updates existing CRDs on upgrade #2264 is decoupled and proceeds on its own track. Upstream timing is best-effort context only.