fix(recipes): raise ebs-csi node sidecar memory limits for p5 nodes

[njhensley]

Summary

Raise the aws-ebs-csi-driver node DaemonSet sidecar memory limits (livenessProbe, nodeDriverRegistrar) from the chart-default 32Mi to 128Mi, so they don't OOM on high-core GPU instances (p5.48xlarge, 192 vCPU).

Motivation / Context

On p5.48xlarge UAT clusters, expected-resources (deployment-phase validation) intermittently-to-consistently fails on the aws-ebs-csi-driver health check:

[chainsaw] aws-ebs-csi-driver: health check failed:
Pod kube-system/ebs-csi-node-... matches forbidden shape
(phase=Running, waiting=CrashLoopBackOff, ...)

Root cause (from node dmesg during bringup):

livenessprobe invoked oom-killer ... constraint=CONSTRAINT_MEMCG
Memory cgroup out of memory: Killed process (livenessprobe)
total-vm:1368464kB, anon-rss:7680kB, file-rss:24032kB   → RSS ≈ 31.7 MB vs 32 MiB limit

The chart's 32Mi limit on the node sidecars is too tight on high-core nodes. The Go binary's resident set alone is ~24 MiB (file-rss); per-P runtime overhead (Go sizes runtime structures by GOMAXPROCS, which defaults to the host core count — 192 on p5) pushes the working set to ~31.7 MiB during cold start under bringup load, cgroup-OOM-killing the livenessProbe container. That removes the /healthz endpoint the ebs-plugin livenessProbe depends on → kubelet restarts ebs-plugin → CrashLoopBackOff → the aws-ebs-csi-driver health check trips → expected-resources fails → UAT fails on p5. The smaller m7i.xlarge system nodes never hit this (fewer cores → smaller runtime footprint), which is why it is p5-specific.

Fix: give both node sidecars headroom above the observed ~31.7 MiB peak (128Mi), independent of instance core count.

Fixes: N/A
Related: N/A

Type of Change

• Bug fix (non-breaking change that fixes an issue)

Component(s) Affected

• Recipe engine / data (pkg/recipe) — component values (recipes/components/aws-ebs-csi-driver/values.yaml)

Implementation Notes

• Adds a sidecars: block setting nodeDriverRegistrar and livenessProbe limits to 128Mi (requests unchanged at 40Mi). These sidecar keys apply to both node and controller instances in the chart.
• GOMAXPROCS pinning was considered but deliberately not used as the fix: forensics show it is a minor contributor (~5 MiB of the working set), and raising the limit is the deterministic, core-count-independent fix. It can be added later as defense-in-depth if desired.
• No image, chart version, or pin changes — the BOM is unaffected.

Testing

# values-only change (no Go); validated by:
yq '.sidecars' recipes/components/aws-ebs-csi-driver/values.yaml   # parses, limits applied

• Diagnosed on a live p5 UAT cluster: confirmed the cgroup-OOM chain end-to-end via node dmesg, per-container exit codes (livenessprobe exit 137/OOMKilled at 32Mi; ebs-plugin exit 2 from liveness kill), and the failing aws-ebs-csi-driver chainsaw assert.
• Honest caveat on validation: the OOM only manifests under real bringup load (cold cache + 192-core CPU/memory contention). It could not be reproduced on the same node 3h post-bringup once quiesced, so this fix is justified deductively (a cgroup-OOM at a measured 31.7 MiB peak is eliminated by a 128 MiB limit) rather than by a live A/B. Full validation will come from the next p5 UAT bringup with this change in place.
• No Go changes, so the mandatory Go lint gate does not apply; CI (make qualify) covers recipe/render checks.

Risk Assessment

• Low — Isolated change (a single component's sidecar memory limits), trivially revertible, cannot regress smaller nodes (they used far less than 32Mi already).

Rollout notes: Takes effect on the next deploy/UAT bringup. Backwards compatible; no migration.

Checklist

• Tests pass locally (make test with -race) — N/A (no Go changes)
• Linter passes (make lint) — N/A (values-only)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality — N/A (component values)
• I updated docs if user-facing behavior changed — N/A (internal deploy tuning)
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 31017f9e-dee1-4d13-b835-690e8cff55fb

📥 Commits

Reviewing files that changed from the base of the PR and between e8450d5 and 517c371.

📒 Files selected for processing (1)

• recipes/components/aws-ebs-csi-driver/values.yaml

---

📝 Walkthrough

Walkthrough

This change modifies the AWS EBS CSI driver Helm values configuration to add explicit resource limits for two sidecar containers, nodeDriverRegistrar and livenessProbe. Each sidecar receives a 128Mi memory limit along with associated CPU and memory request values, accompanied by comments explaining the rationale related to chart defaults on high-core instances and potential liveness failures from tight limits.

Estimated code review effort: 1 (Trivial) | ~3 minutes

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: increasing aws-ebs-csi node sidecar memory limits for p5 nodes.
Description check	✅ Passed	The description is directly related and accurately explains the memory-limit change and the p5-specific failure it fixes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Recipe evidence check

Protected recipes

Recipes with committed evidence (recipes/evidence/<slug>/<source>/<digest>.yaml) that this PR affects: 1

Recipe	Source	Pointer	Verify	Digest match
gb200-eks-ubuntu-training	7c4c0edc8c765a95a0f3afdb3bbb8e91	sha256-93fac974407a873d5b6a52a72bafcaa18b019190545a23d03031680d6aabd2bc	❌ invalid — registry-forbidden (HTTP 401): registry not accessible (make the fork's aicr-evidence package public, or provide registry credentials)	⚠️ skipped (no signed digest)

Other affected recipes without evidence yet: 23

These recipes are affected by this PR but carry no committed evidence pointer, so there is
nothing to verify. This is expected — evidence is hardware-gated and added over time.

• a100-eks-training
• a100-eks-ubuntu-training-kubeflow
• a100-eks-ubuntu-training
• gb200-eks-inference
• gb200-eks-training
• gb200-eks-ubuntu-inference-dynamo
• gb200-eks-ubuntu-inference
• gb200-eks-ubuntu-training-kubeflow
• gb200-eks-ubuntu-training-slurm
• h100-eks-inference
• h100-eks-training
• h100-eks-ubuntu-inference-dynamo
• h100-eks-ubuntu-inference-nim
• h100-eks-ubuntu-inference
• h100-eks-ubuntu-training-kubeflow
• h100-eks-ubuntu-training-slurm
• h100-eks-ubuntu-training
• h200-eks-inference
• h200-eks-training
• rtx-pro-6000-eks-inference
• rtx-pro-6000-eks-ubuntu-inference-dynamo
• rtx-pro-6000-eks-ubuntu-inference-nim
• rtx-pro-6000-eks-ubuntu-inference

How to refresh evidence

Run on a cluster matching the recipe's criteria:

aicr snapshot -o snapshot.yaml
aicr validate \
  -r recipes/overlays/<slug>.yaml \
  -s snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<your-fork>/aicr-evidence
# Copy to the per-source path printed in the emit 'copyTo' hint:
#   recipes/evidence/<slug>/<source>/<bundle-digest>.yaml

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.