fix: status check lists all events

[ericzzzzzzz]

Fixes: #8972
Related: #8624

Description

• change the way how skaffold should detect when a deployment request was rejected by admission webhook

• The original implementation repeatedly lists all events to determine if request was denied by admission webhook, there are some problems with this approach

  • The check is pretty loose, and may produce false positive reported in Unexpected "Error creating: Unauthorized" #8972
  • It lists all events from around to around, when failCreate event found, the checking cannot be moved forward even tolerate-failure-until-deadline is used, even worse, the failCreate event may not come from the current skaffold session, it seems that by default the retention for events is 1 hr.
  • there is no other success indicator can change the state so tolerate-failure-until-deadline probably won't make sense, even if we assume events are reliable and in chronological order, which probably not true
  • Events should only be informational, generally, it's not a good idea to rely on event message to determine business logic

• The replacement hack still sort of relies on content from a message, this is bad, but it seems that there are no other reliable way to do this kind of check at the moment, we cannot check the status of pods from a deployment when admission webhook rejects request as those pods haven't been created yet.

• This implementation may be broken when message from k8s server changes, but shouldn't produce other issues

Test Plan

• Manually tested against GKE cluster w/ binauthz enabled following the guide here https://cloud.google.com/architecture/binary-auth-with-cloud-build-and-gke, example/nodejs is a good candidate to test this

[codecov]

Codecov Report

Merging #9015 (f9c678a) into main (290280e) will decrease coverage by 6.85%.
Report is 995 commits behind head on main.
The diff coverage is 49.80%.

@@            Coverage Diff             @@
##             main    #9015      +/-   ##
==========================================
- Coverage   70.48%   63.64%   -6.85%     
==========================================
  Files         515      624     +109     
  Lines       23150    31915    +8765     
==========================================
+ Hits        16317    20311    +3994     
- Misses       5776    10078    +4302     
- Partials     1057     1526     +469     

Files Changed	Coverage Δ
cmd/skaffold/app/cmd/completion.go	13.04% <0.00%> (-1.25%)	⬇️
cmd/skaffold/app/cmd/config/list.go	65.21% <ø> (ø)
cmd/skaffold/app/cmd/config/set.go	88.72% <ø> (ø)
cmd/skaffold/app/cmd/config/util.go	54.28% <ø> (ø)
cmd/skaffold/app/cmd/credits.go	100.00% <ø> (ø)
cmd/skaffold/app/cmd/credits/export.go	0.00% <0.00%> (ø)
cmd/skaffold/app/cmd/deploy.go	40.90% <0.00%> (-12.94%)	⬇️
cmd/skaffold/app/cmd/generate_pipeline.go	60.00% <ø> (ø)
cmd/skaffold/app/cmd/inspect_modules.go	65.00% <ø> (ø)
cmd/skaffold/app/cmd/inspect_profiles.go	66.66% <ø> (ø)
... and 40 more

... and 416 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more