fix(ci): size the GKE UAT timeout for the bringup retry - #2078
Merged
Merged
Conversation
lockwobr
added
the
theme/ci-dx
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe GCP UAT job timeout increases from 280 to 320 minutes. The budget documentation now includes the Bringup Infra retry, actuator timeout, and 60-second backoff. It documents an approximately 299-minute worst-case runtime and notes that future actuator timeout changes require a budget recalculation. Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/uat-gcp.yaml:
- Around line 86-93: Correct the timeout-budget comments around the UAT
workflow: state that a 320-minute cap leaves approximately 21 minutes above the
estimated 299-minute retry path, not 12 minutes. Revise the rollback guidance to
explain that a future reduction requires recalculating the provisioning and
teardown budgets, especially if `#2065` increases one provisioning attempt from 30
to 60 minutes; removing retries alone does not justify returning to 280.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 445823ea-3984-44a4-b116-d8f7522d1e5e
📒 Files selected for processing (1)
.github/workflows/uat-gcp.yaml
Contributor
Coverage Report ✅
Coverage BadgeNo Go source files changed in this PR. |
#2066 added a second bringup attempt to uat-gcp without raising the job timeout, which was budgeted for one. The step carries no step-level timeout, and the failure it retries is the actuator's own hardcoded 30m node-pool timeout — so the first attempt burns the full 30m before failing, plus a 60s backoff. That is ~31m on top of the ~268m the comment budgets, i.e. ~299m against a 280m cap. The consequence is not a slow run. As the same comment notes, a job-level timeout cancels pending always() steps, so tripping the cap skips Destroy Cluster and leaks the GPU node and its VPC network group — precisely what #2066 was written to prevent. It only bites when the first attempt burns its full timeout AND the retry succeeds AND the run is otherwise near budget, but that is the exact path a retry is supposed to rescue. Raise the cap to 320, leaving ~21m over the worst case, and record the arithmetic so the next change to either number has the reasoning in front of it. The comment also warns against reducing the cap on the strength of #2065: raising the actuator's node-pool timeout 30m->60m makes a single attempt cost up to ~30m more, so the base budget rises to ~298 even if the retry never fires. That change trades a frequent retry for a longer first attempt; it does not free budget. uat-aws and uat-azure are unaffected: their bringup runs a single apply, and their retry loops are short API-call retries the existing headroom already absorbs. Signed-off-by: Brian Lockwood <lockwobr@gmail.com>
lockwobr
force-pushed
the
fix/uat-gcp-retry-timeout-budget
branch
from
August 5, 2026 21:51
70e012e to
ef9f7d3
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
lockwobr
enabled auto-merge (squash)
August 5, 2026 21:51
mchmarny
approved these changes
Aug 5, 2026
11 of 25 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Raises
uat-gcp's job timeout from 280 to 320 minutes. #2066 added a second bringup attempt without adjusting the budget, which was sized for one.Motivation / Context
The
Bringup Infrastep has no step-level timeout, and the failure it retries is the actuator's own hardcoded 30m node-pool timeout — quoted from the comment #2066 added:So the first attempt burns the full 30 minutes before failing, then sleeps 60s, then retries. That is ~31m on top of the ~268m the existing comment budgets:
~299m against a 280m cap.
The consequence is not a slow run. From the same comment:
Tripping the cap cancels the
always()Destroy Cluster step and leaks the GPU node and its VPC network group — exactly the outcome #2066 exists to prevent. Its own rationale says a stranded cluster "leaks the GPU node and NETWORKS quota."It only bites when the first attempt burns its full timeout and the retry succeeds and the run is otherwise near budget. Narrow, but that is precisely the path a retry is meant to rescue.
Found while reviewing #2073, where the file was pulled into the diff by mistake.
Fixes: N/A
Related: #2066, #2065
Type of Change
Component(s) Affected
cmd/aicr,pkg/cli)cmd/aicrd,pkg/server)pkg/recipe)pkg/bundler,pkg/component/*)pkg/collector,pkg/snapshotter)pkg/validator)pkg/errors,pkg/k8s)docs/,examples/)Implementation Notes
320 preserves the original ~12m of teardown headroom over the ~299m worst case. The comment now records the arithmetic, so the next change to either number has the reasoning in front of it rather than having to re-derive it from the retry loop.
This is a stopgap. The durable fix is the upstream node-pool timeout bump 30m -> 60m tracked in #2065 (mirroring the AKS actuator,
mchmarny/cluster#31); once that lands the retry becomes rare and the cap can drop back toward 280.uat-aws and uat-azure were checked and are unaffected. Their
Bringup Infraruns a singleapplywith no retry; their retry loops are short API-call retries (aws eks list-clusters, AKS auth propagation, az throttling) costing seconds to a few minutes, which the existing headroom already absorbs. Only GCP retries a 30-minute provisioning attempt.Testing
Comment-and-constant change only; there is no behaviour to unit test. The 3
SC2016findings actionlint reports on this file are pre-existing (identical count onmain) and untouched here.The change is only observable on a run where bringup actually retries, which is the scenario #2065 tracks eliminating.
Risk Assessment
Rollout notes: Raising a timeout cap cannot make a passing run fail. The only effect is that a run which would have been cancelled at 280m now has room to finish and, importantly, to tear its cluster down. Worst case a genuinely hung run occupies a runner ~40m longer before being cancelled.
Checklist
make testwith-race)make lint)git commit -S)