Conversation
|
Could a maintainer approve the workflow runs for this PR when you get a chance? CI, CI-TEST, CodeQL and DryRunGenPR have all been sitting in |
b83bab7 to
ed1e3ae
Compare
|
Rebased onto While rebasing I noticed this overlaps with #9360, which was opened about two weeks before mine and targets the same issue (#8148) and the same files. I'd rather not leave two PRs doing the same thing in your queue, so I'm happy to close this one if you'd prefer to take that one forward. One difference worth knowing before you decide: the comment on #9360 raises cache poisoning on transient, non-rate-limited API errors. This PR guards that case explicitly — Either way: the workflows on this PR have been at |
ed1e3ae to
981dca2
Compare
|
Heads up on one test change in the rebase. #9483 added "should not surface a non-size InvalidUserData.Malformed rejection as UserDataSizeLimitExceeded", which expected the reconcile to fail. This PR moves EC2 request rejections onto the validation condition instead of returning an error, so that case now reconciles cleanly and reports ValidationSucceeded=False with reason CreateLaunchTemplateValidationFailed. The intent of the test is unchanged: a malformed user data rejection is still never labelled as UserDataTooLarge, and I kept that assertion explicit by checking the new reason. Genuine size limit rejections still go through the IsUserDataTooLarge path first. Happy to split this into a separate test if you would rather keep the original one as is. |
Fixes #8148
Description
validateRunInstancesAuthorizationreturnsfmt.Errorf("validating ec2:RunInstances authorization, %w", err)for every error that isn'tUnauthorizedOperationorDryRunOperation. But EC2 also rejects a dry run over the request itself, so anInvalidBlockDeviceMapping(root volume smaller than the AMI's snapshot) becomes a reconcile error that is only ever logged, and logged as if it were an authorization problem. The EC2NodeClass stays atValidationSucceeded=Unknown/AwaitingReconciliationand retries forever, which is what #8148 reports.validateCreateFleetAuthorizationandvalidateCreateLaunchTemplateAuthorizationhave the same shape.All three now surface an AWS API error on
ValidationSucceededwith the error's code and message instead of returning it:Two things worth flagging, since they go a little past the surfacing change:
IsServerErrornever matched an EC2 error. It testedapiErr.ErrorFault() == smithy.FaultServer, but EC2's generated deserializers build asmithy.GenericAPIErrorwithout settingFault, so it always readFaultUnknown. That is why the 503Unavailablein the issue also landed in the authorization bucket. It now falls back to the response status code viaretry.DefaultRetryableHTTPStatusCodes, so it tracks the SDK rather than hardcoding a range.IsServerErroris only called fromvalidation.go, so the blast radius stays in this file.A cached validation failure lives for
ValidationTTL(30 minutes) and drivesReady=False, so only errors a retry cannot fix should reach it. Transient codes are filtered out first by a newIsNonTerminalError, which reuses the SDK's own throttle and retryable code sets and addsAuthFailure,RequestExpired,PendingVerification, plus the two launch template not-found codes (a template can be garbage collected betweenEnsureAlland the dry run). Those requeue exactly as before.CreateLaunchTemplateisn't actually a dry run, so its message says "request" instead.How was this change tested?
go test ./pkg/errors/... ./pkg/controllers/...andgolangci-lint run(v2.12.2, the versionhack/toolchain.shpins) both pass locally against envtest 1.34.1.New coverage:
DescribeTableentries, one per validator, asserting the reason and the full condition messageEC2ThrottledExceptionrequeue without caching or failing validationpkg/errors/errors_test.go, which the package didn't have, coveringIsServerError,IsNonTerminalErrorandToAPIErrorMessageI also confirmed the two key specs fail when the fix is reverted.
Does this change impact docs?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.