(route53): cross account zone delegation fails sometimes

[phoefflin]

What is the problem?

Cross account zone delegation sometimes fails with an Access Denied error.

Reproduction Steps

1. create parent zones in parent_zone_account

change principle to sub_zone_account principle, deploy and get roleArns from stack outputs

import * as iam from 'aws-cdk-lib/aws-iam';
import * as route53 from 'aws-cdk-lib/aws-route53';
import { CfnOutput, Stack, StackProps, App } from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class ParentZones extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    const crossAccountZoneDelegationPrincipal = new iam.AccountPrincipal('1111111111')

    const parentZone1 = new route53.PublicHostedZone(this, 'HostedZone1', {
      zoneName: 'domain1.com',
      crossAccountZoneDelegationPrincipal,
    });

    const parentZone2 = new route53.PublicHostedZone(this, 'HostedZone2', {
      zoneName: 'domain2.com',
      crossAccountZoneDelegationPrincipal,
    });

    new CfnOutput(this, 'zone1RoleArn', { value: parentZone1.crossAccountZoneDelegationRole?.roleArn || '' });
    new CfnOutput(this, 'zone2RoleArn', { value: parentZone2.crossAccountZoneDelegationRole?.roleArn || '' });
  }
}

2. create subzones in parent_zone_account and delegate only one zone to the corresponding parent zone

update roleArns and deploy cdk app

import * as iam from 'aws-cdk-lib/aws-iam';
import * as route53 from 'aws-cdk-lib/aws-route53';
import { Construct } from 'constructs';
import { App, Stack, StackProps } from 'aws-cdk-lib';

export class SubZones extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    
    const zone1RoleArn = '<arn1 from stack output>';
    const zone2RoleArn = '<arn2 from stack output>';
    
    const subZone1 = new route53.PublicHostedZone(this, `zone1`, { zoneName: 'sub.domain1.com' });
    const subZone2 = new route53.PublicHostedZone(this, `zone2`, { zoneName: 'sub.domain2.com' });
    
    new ZoneDelegation(this, 'zone1Delegation', {
      delegatedZone: subZone1,
      roleArn: zone1RoleArn,
    })

    if (process.env.DELEGATE_ZONE2 !== undefined) {
      new ZoneDelegation(this, 'zone2Delegation', {
        delegatedZone: subZone2,
        roleArn: zone2RoleArn,
      })
    }
  }
}

export class ZoneDelegation extends Construct {
  public constructor(scope: Construct, id: string, props: {roleArn: string, delegatedZone: route53.PublicHostedZone}) {
    super(scope, id);
    
    const {roleArn, delegatedZone} = props
    new route53.CrossAccountZoneDelegationRecord(this, 'delegation', {
      delegatedZone,
      parentHostedZoneName: delegatedZone.zoneName.split('.').slice(1).join('.'),
      delegationRole: iam.Role.fromRoleArn(this, 'role', roleArn),
    });
  }
}

3. delegate also the second zone

rerun stack 2) with variable DELEGATE_ZONE2 set (ex: DELEGATE_ZONE2=true npm run cdk deploy

What did you expect to happen?

I expected both delegation NS records to be created in both parent zones

What actually happened?

Step 3) fails with an Access denied error:

SubZones: creating CloudFormation changeset...
4:25:30 PM | CREATE_FAILED        | Custom::CrossAccountZoneDelegation | zone2Delegationdel...omResourceA344A37B
Received response status [FAILED] from custom resource. Message returned: AccessDenied: User: arn:aws:sts::11111111111:assumed-role/SubZones-CustomCrossAccountZoneDelegationCustomRes-16PAYUODSOML0/SubZones-Cust
omCrossAccountZoneDelegationCustomRes-D8K6zY00FNXr is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1111111111:role/parentZones-HostedZone2CrossAccountZoneDelegationR-1SY04FS6EIRQV
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/query.js:50:29)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) (RequestId: 1bfdbae8-17e2-4888-b479-41e57eaa1b16)

CDK CLI Version

2.12.0 (build c9786db)

Framework Version

2.12.0

Node.js Version

v16.13.2

OS

linux

Language

Typescript

Language Version

3.9.10

Other information

No response

[phoefflin]

To my understanding the problem is that we are missing a dependency between the policy statement that is attached to the handler role (https://github.com/phoefflin/aws-cdk/blob/76b5c0d12e1e692efcf6a557ee4ddb6df3709e4d/packages/%40aws-cdk/aws-route53/lib/record-set.ts#L693) and the custom resource (https://github.com/phoefflin/aws-cdk/blob/76b5c0d12e1e692efcf6a557ee4ddb6df3709e4d/packages/%40aws-cdk/aws-route53/lib/record-set.ts#L699).

When additional zones are delegated the custom resource handler is triggered while the policy is still being created and therefore fails with an access denied error.

The solution could look something like that but unfortunately I currently fail to get a local build up for verification:
phoefflin@7c4ab54

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.