(cli): hotswap results in many call to ListStackResources of cloudformation API (with ThrottlingException)

[JPLemelin]

What is the problem?

We are using the hot-swap with lambda code change (cdk deploy --hotswap my-stack-with-lambda-functions --exclusively) .
Currently the stack have 68 lambdas, who are all using the same code asset, but with different entry point.

The problem we are facing is, the process take almost the exact same time of cloudformation deploy, i.e around 5-6mins 🥲

Reproduction Steps

Have a stack with many lambda, in my case 68 lambda

What did you expect to happen?

Only one call to ListStackResources of cloudformation API (or more if paged)

What actually happened?

Foreach lambdas, a ListStackResources of cloudformation API are done and some hit the ThrottlingException

CDK CLI Version

2.12.0

Framework Version

No response

Node.js Version

14

OS

MacOs

Language

Typescript

Language Version

No response

Other information

Hi!

We are using the hot-swap with lambda code change (cdk deploy --hotswap my-stack-with-lambda-functions --exclusively) .
Currently the stack have 68 lambdas, who are all using the same code asset, but with different entry point.

The problem we are facing is, the process take almost the exact same time of cloudformation deploy, i.e around 5-6mins 🥲

After some investigation we found the cdk do some (many many many 🙃) api call to cloudforamtion on ListStackResources, who take around 5mins.

On the 876 call to ListStackResources, 604 end with ThrottlingException: Rate exceeded

On first few second there are 68 call (same number of lambda) to ListStackResources who don’t have nextToken parameter, I think it’s the first call to ListStackResources, other call have a paging parameters (nextToken

aws-cdk/packages/aws-cdk/lib/api/evaluate-cloudformation-template.ts

Line 28 in 9858002

NextToken: nextToken,

)

So most of the hot-swap execution time is spent doing ListStackResources

Our cloudformation stack have 359 ressources

When we use the cli, aws cloudformation list-stack-resources --stack-name my-stack-with-lambda-functions , we did not hit any paging

I also add some screen shot from cloudtrail of CDK call during the hot-swap

[skinny85]

Hey @JPLemelin,

thanks for opening the issue. Is there any chance you can show us some of your CDK and runtime Lambda code? It will make it much easier to reproduce the issue locally.

Thanks,
Adam

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[JPLemelin]

@skinny85, I added a repo here https://github.com/JPLemelin/aws-cdk-issue-19021

In the repo I also add a cloud-trail export https://github.com/JPLemelin/aws-cdk-issue-19021/blob/master/doc/cloudtrail_event_history.json

Let's me know if you need more Information!

[skinny85]

Thanks, that's super helpful! I'll be diving into this today.

[skinny85]

@JPLemelin actually, I ran into a problem 😛.

Did you forget to include some code in your example app? I see this code in your CDK app: https://github.com/JPLemelin/aws-cdk-issue-19021/blob/6f115b8a97ce407eae2c7d35b614c63bd8fdf69f/lib/aws-cdk-issue-19021-stack.ts#L13, but the code does not contain a src directory...?

cdk synth fails with:

$ /Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/.bin/cdk synth

/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/aws-cdk-lib/core/lib/asset-staging.ts:109
      throw new Error(`Cannot find asset at ${this.sourcePath}`);
            ^
Error: Cannot find asset at /Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/src
    at new AssetStaging (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/aws-cdk-lib/core/lib/asset-staging.ts:109:13)
    at new Asset (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.ts:72:21)
    at AssetCode.bind (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/aws-cdk-lib/aws-lambda/lib/code.ts:180:20)
    at new Function (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/aws-cdk-lib/aws-lambda/lib/function.ts:350:29)
    at new AwsCdkIssue19021Stack (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/lib/aws-cdk-issue-19021-stack.ts:16:7)
    at Object.<anonymous> (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/bin/aws-cdk-issue-19021.ts:8:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Module.m._compile (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/ts-node/src/index.ts:1056:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Object.require.extensions.<computed> [as .ts] (/Users/adamruka/workplace/cloud-development-kit/oncall/hotswap-list-resources-spam/node_modules/ts-node/src/index.ts:1059:12)
Subprocess exited with error 1
error Command failed with exit code 1.

[skinny85]

Nvm, figured it out, PR has been opened with a fix 🙂.

[JPLemelin]

My bad, the src folder was missing due to my ignore file who was ignoring all *.js 😑

[skinny85]

No worries 🙂.

BTW - your setup with having to pass a -c envName =dev to execute any CDK command is not really idiomatic, and makes using this app harder than it has to be.

The idiomatic way is just to have 2 Stacks:

new AwsCdkIssue19021Stack(app, 'AwsCdkIssue19021StackDev', {
  env: {
    region: 'ca-central-1',
    account: 'abc',
  },
});

new AwsCdkIssue19021Stack(app, 'AwsCdkIssue19021StackProd', {
  env: {
    region: 'ca-central-1',
    account: 'xyz',
  },
});

More details: https://www.endoflineblog.com/cdk-tips-05-have-a-stack-instance-per-deployed-stack.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.