( aws-s3: ): addEventNotification() removal fails with AccessDenied on multi-bucket stacks #37667

Activity

1. abidhasan-aws

   added

   bugThis issue is a bug.This issue is a bug.

   needs-triageThis issue or PR still needs to be triaged.This issue or PR still needs to be triaged.

   on Apr 22, 2026
2. github-actions

   added

   @aws-cdk/aws-s3Related to Amazon S3Related to Amazon S3

   potential-regressionMarking this issue as a potential regression to be checked by team memberMarking this issue as a potential regression to be checked by team member

   on Apr 22, 2026
3. abidhasan-aws commented on Apr 22, 2026

   

   abidhasan-aws

   on Apr 22, 2026

   Last edited by abidhasan-aws

   ContributorAuthor

   More actions

   Adding possible workaround here until a permanent fix is prepared.

   WORKAROUND 1

   If you can downgrade to aws-cdk-lib v2.214.0, that should solve the problem — it is the last working version for this scenario.

   WORKAROUND 2 : Escape hatch to restore wildcard permission

   ⚠️ This grants the singleton BucketNotificationsHandler Lambda s3:PutBucketNotification on *, which is broader than CDK's default since v2.215.0. Step 3 removes it once you are done.

   This keeps you on your current CDK version.
   It is a 3-step process:

   1. Add the escape hatch
   2. Remove the notification
   3. Remove the escape hatch.

   The escape hatch attaches a standalone IAM Policy (s3:PutBucketNotification on *) that CloudFormation does not touch during the removal, so the handler Lambda keeps its permission. This uses the CDK escape hatch pattern.

   Step 1. Add the escape hatch and deploy.

   Using the reproduction stack as the starting point, add the escape hatch at the end of the stack.

     import * as cdk from 'aws-cdk-lib/core';
     import * as lambda from 'aws-cdk-lib/aws-lambda';
     import * as s3 from 'aws-cdk-lib/aws-s3';
     import * as s3n from 'aws-cdk-lib/aws-s3-notifications';
     import * as iam from 'aws-cdk-lib/aws-iam';                          // NEW
     import { Construct } from 'constructs';
   
     export class BucketNotificationRemovalTestStack extends cdk.Stack {
       constructor(scope: Construct, id: string, props?: cdk.StackProps) {
         super(scope, id, props);
   
         const newBucket1 = new s3.Bucket(this, '<somebucketname>1', {
           bucketName: '<somebucketname>1',
           removalPolicy: cdk.RemovalPolicy.DESTROY,
         });
         const newBucket2 = new s3.Bucket(this, '<somebucketname>2', {
           bucketName: '<somebucketname>2',
           removalPolicy: cdk.RemovalPolicy.DESTROY,
         });
         const lambdaDestinationFunction = new lambda.Function(this, 'MyLambda', {
           runtime: lambda.Runtime.PYTHON_3_13,
           handler: 'lambda_function.lambda_handler',
           code: lambda.Code.fromInline('foo'),
         });
   
         newBucket1.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaDestinationFunction));
         newBucket2.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaDestinationFunction));
   
         // --- ESCAPE HATCH START ---
         const handler = cdk.Stack.of(this)
           .node.tryFindChild('BucketNotificationsHandler050a0587b7544547bf325f094a3db834') as lambda.Function;
   
         if (!handler) {
           throw new Error('BucketNotificationsHandler singleton not found — ensure at least one addEventNotification() is declared before the escape hatch.');
         }
   
         handler.role!.attachInlinePolicy(new iam.Policy(this, 'HandlerWildcardFallback', {
           statements: [new iam.PolicyStatement({
             actions: ['s3:PutBucketNotification'],
             resources: ['*'],
           })],
         }));
         // --- ESCAPE HATCH END ---
       }
     }
   

   Notice that both addEventNotification() calls are still present. Do not remove anything yet, this step only installs the fallback policy.

   Optional sanity check: confirm the policy appears in the synthesized template:

     cdk synth | grep -A 5 HandlerWildcardFallback
   

   Deploy:

     cdk deploy
   

   Step 2. Remove the notification and deploy.

   Delete (or comment out) the addEventNotification() call for the bucket you no longer want monitored. For example, remove the line for newBucket2:

     newBucket1.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaDestinationFunction));
     // newBucket2.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaDestinationFunction));   // REMOVED
   

   Deploy:

     cdk deploy
   

   The per-bucket IAM statement for newBucket2 is removed from the template, but the HandlerWildcardFallback policy remains, so the handler Lambda retains permission and the cleanup succeeds without AccessDenied.

   Step 3. Remove the escape hatch and deploy (Strongly recommended).

   Once Step 2 has succeeded, delete the entire escape hatch block (everything between // --- ESCAPE HATCH START --- and // --- ESCAPE HATCH END ---) and also remove the import * as iam line if it is no longer used.

   Deploy:

     cdk deploy
   

   This returns you to a clean state with the notification removed from the stack.

   Hope this helps anyone else who hits this. Happy to clarify anything.
4. pahud commented on Apr 23, 2026

   

   pahud

   on Apr 23, 2026

   Last edited by pahud

   Contributor

   More actions

   Hi there,

   I was able to reproduce this locally on aws-cdk-lib v2.250.0 (latest) by synthesizing both the "before" (two buckets with notifications) and "after" (one notification removed) templates. I also cross-verified against v2.214.0, which uses a wildcard Resource: "*" and is unaffected.

   From my view, the root cause is in addHandlerPermissions() — the per-bucket ARN scoping introduced by PR #35334 is a valid security improvement, but it created a CloudFormation ordering gap. When a notification is removed, CloudFormation is free to update the shared IAM policy (revoking the bucket's ARN) before invoking the custom resource handler to clean up.

   Update: My initial thought was to add a DependsOn from each Custom::S3BucketNotifications to the handler's shared RoleDefaultPolicy. However, after live CloudFormation testing, I confirmed this does not work — DependsOn only controls create/delete ordering, not update ordering. Since the shared policy is updated (not deleted) when one of multiple buckets is removed, CloudFormation ignores the DependsOn and updates the policy first.

   The fix that works is using a dedicated IAM policy per bucket notification instead of the shared DefaultPolicy. When a notification is removed, its dedicated policy is deleted (not updated), and DependsOn correctly ensures the custom resource is deleted before its policy.

   PR: #37678
5. pahud

   added

   p0

   and removed

   needs-triageThis issue or PR still needs to be triaged.This issue or PR still needs to be triaged.

   on Apr 23, 2026
6. pahud

   added 2 commits that reference this issue on Apr 23, 2026

   fix(s3): use per-bucket IAM policies for BucketNotificationsHandler

   ...

   This issue will close once commit 87a6d7a is merged into the 'main' branch.

   87a6d7a

   fix(s3): use per-bucket IAM policies for BucketNotificationsHandler

   ...

   This issue will close once commit 3e71cd4 is merged into the 'main' branch.

   3e71cd4
7. mergify

   closed this as completedin #37708on Apr 28, 2026
8. mergify

   added a commit that references this issue on Apr 28, 2026

   fix(s3): resolve S3 notification removal race condition (#37708)

   ...

   This issue will close once commit dc5be98 is merged into the 'main' branch.

   dc5be98
9. github-actions commented on Apr 28, 2026

   

   github-actionsBot

   on Apr 28, 2026 – with GitHub Actions

   Contributor

   More actions

   Comments on closed issues and PRs are hard for our team to see.
   If you need help, please open a new issue that references this one.
10. github-actions

    locked as resolved and limited conversation to collaborators on Apr 28, 2026
11. ali-n4i

    added a commit that references this issue on Jul 29, 2026

    fix(s3): resolve S3 notification removal race condition (aws#37708)

    ...

    This issue will close once commit fd85f8a is merged into the 'main' branch.

    fd85f8a