Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

custom_resources: infrequently invoked user function results in CodeArtifactUserPendingException exception in provider framework function #20123

Closed
vito-laurenza-zocdoc opened this issue Apr 28, 2022 · 4 comments · Fixed by #22612
Closed

custom_resources: infrequently invoked user function results in CodeArtifactUserPendingException exception in provider framework function #20123

vito-laurenza-zocdoc opened this issue Apr 28, 2022 · 4 comments · Fixed by #22612
Assignees
@TheRealAmazonKendra
Labels
@aws-cdk/custom-resources Related to AWS CDK Custom Resources bug This issue is a bug. effort/small Small work item – less than a day of effort needs-reproduction This issue needs reproduction. p1
Projects
AWS CDK Roadmap

Comments

@vito-laurenza-zocdoc
Copy link

Describe the bug

When using the custom resources provider framework, if a user function is not accessed often enough (?), that function can go inactive and the provider framework function code does not appear to handle waiting for the state to change to Active.

Expected Behavior

The provider framework function should properly handle waiting/retrying while an Inactive user function transitions state to Active.

Current Behavior

The provider framework function does not appear to handle waiting/retrying invoking an Inactive user function. The provider function will throw an error similar to the following:

Received response status [FAILED] from custom resource. Message returned: CodeArtifactUserPendingException: INFO: Lambda is initializing your function. It will be ready to invoke shortly. at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27) at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8) at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14) at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12) at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: <redacted>)

Reproduction Steps

This should be reproducible with any of the example custom resources in the custom_resources package.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.21.1 (build a6ee543)

Framework Version

No response

Node.js Version

v14.19.1

OS

Darwin Kernel Version 20.6.0: Tue Feb 22 21:10:41 PST 2022; root:xnu-7195.141.26~1/RELEASE_X86_64

Language

Typescript

Language Version

TypeScript (4.6.3)

Other information

No response
@vito-laurenza-zocdoc vito-laurenza-zocdoc added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Apr 28, 2022
@github-actions github-actions bot added the @aws-cdk/custom-resources Related to AWS CDK Custom Resources label Apr 28, 2022
@ryparker ryparker added p2 needs-reproduction This issue needs reproduction. labels Apr 28, 2022
@rix0rrr rix0rrr added effort/small Small work item – less than a day of effort good first issue Related to contributions. See CONTRIBUTING.md p1 and removed p2 needs-triage This issue or PR still needs to be triaged. labels May 9, 2022
@rix0rrr rix0rrr removed their assignment May 9, 2022
@rix0rrr
Copy link
Contributor

rix0rrr commented May 9, 2022

Yes, we need to catch this error and retry!

@comcalvi comcalvi self-assigned this Jun 16, 2022
@comcalvi
Copy link
Contributor

@vito-laurenza-zocdoc which cdk command did you run to produce the error?

@vito-laurenza-zocdoc
Copy link
Author

@vito-laurenza-zocdoc which cdk command did you run to produce the error?

No cdk commands involved. That error comes from the custom resource provider lambda function when it tries to invoke the user function IFF that user function is in an inactive state. As @rix0rrr mentioned, this needs a catch and retry (likely with backoff) to ensure that the user function is eventually invoked once it is ready again.

@comcalvi comcalvi mentioned this issue Jul 11, 2022
Closed
4 tasks
@comcalvi comcalvi removed the good first issue Related to contributions. See CONTRIBUTING.md label Jul 14, 2022
@comcalvi comcalvi mentioned this issue Jul 19, 2022
Closed
4 tasks
@evgenyka evgenyka added this to Working on it in AWS CDK Roadmap Sep 6, 2022
TheRealAmazonKendra added a commit that referenced this issue Oct 23, 2022
@TheRealAmazonKendra
fix(custom-resources): inactive lambda funtions fail on invoke
5a0a8e9 
closes #20123

All lambda functions can become inactive eventually. This will result in invocations failing.
This PR adds logic to wait for functions to become active on a failed invocation.
@TheRealAmazonKendra TheRealAmazonKendra mentioned this issue Oct 23, 2022
Merged
4 tasks
TheRealAmazonKendra added a commit to TheRealAmazonKendra/aws-cdk that referenced this issue Oct 26, 2022
@TheRealAmazonKendra
fix(custom-resources): inactive lambda funtions fail on invoke
4abbac9 
closes aws#20123

All lambda functions can become inactive eventually. This will result in invocations failing.
This PR adds logic to wait for functions to become active on a failed invocation.
@mergify mergify bot closed this as completed in #22612 Dec 9, 2022
mergify bot pushed a commit that referenced this issue Dec 9, 2022
@TheRealAmazonKendra
fix(custom-resources): inactive lambda functions fail on invoke (#22612)
def2971 
closes #20123

All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation.

----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Dec 9, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

brennanho pushed a commit to brennanho/aws-cdk that referenced this issue Dec 9, 2022
@TheRealAmazonKendra
fix(custom-resources): inactive lambda functions fail on invoke (aws#…
de4754e 
…22612)

closes aws#20123

All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation.

----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
brennanho pushed a commit to brennanho/aws-cdk that referenced this issue Jan 20, 2023
@TheRealAmazonKendra
fix(custom-resources): inactive lambda functions fail on invoke (aws#…
644045a 
…22612)

closes aws#20123

All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation.

----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@TheRealAmazonKendra TheRealAmazonKendra moved this from Working on it to Shipped in AWS CDK Roadmap Feb 7, 2023
brennanho pushed a commit to brennanho/aws-cdk that referenced this issue Feb 22, 2023
@TheRealAmazonKendra
fix(custom-resources): inactive lambda functions fail on invoke (aws#…
8fcb0e2 
…22612)

closes aws#20123

All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation.

----

### All Submissions:

* [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TheRealAmazonKendra TheRealAmazonKendra

Labels
@aws-cdk/custom-resources Related to AWS CDK Custom Resources bug This issue is a bug. effort/small Small work item – less than a day of effort needs-reproduction This issue needs reproduction. p1
Projects
No open projects
AWS CDK Roadmap
Shipped
Milestone
No milestone
5 participants
@rix0rrr @ryparker @vito-laurenza-zocdoc @TheRealAmazonKendra @comcalvi