Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code-Pipeline: update step delayed by missing files ('/root/.cdk/cache/notices.json', 'cdk.out/tree.json') #19201

Closed
erudisch opened this issue Mar 1, 2022 · 19 comments · Fixed by #19226
Assignees
Labels
@aws-cdk/aws-codepipeline Related to AWS CodePipeline bug This issue is a bug. in-progress This issue is being actively worked on. p1

Comments

@erudisch
Copy link

erudisch commented Mar 1, 2022

What is the problem?

The update step gets delayed, not aborted, by trying to open two missing files.

UpdateStack \| 29/57 \| 2:17:18 PM \| UPDATE_COMPLETE      \| AWS::CloudFormation::Stack  \| UpdateStack
--
[Container] 2022/02/28 14:14:24 Waiting for agent ping
[Container] 2022/02/28 14:14:25 Waiting for DOWNLOAD_SOURCE
[Container] 2022/02/28 14:14:26 Phase is DOWNLOAD_SOURCE
[Container] 2022/02/28 14:14:26 CODEBUILD_SRC_DIR=/codebuild/output/src677814557/src
[Container] 2022/02/28 14:14:26 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2022/02/28 14:14:26 Processing environment variables
[Container] 2022/02/28 14:14:27 No runtime version selected in buildspec.
[Container] 2022/02/28 14:14:29 Moving to directory /codebuild/output/src677814557/src
[Container] 2022/02/28 14:14:29 Configuring ssm agent with target id: codebuild:960179ab-f4cf-4c04-8582-5c6e4763d9fe
[Container] 2022/02/28 14:14:29 Successfully updated ssm agent configuration
[Container] 2022/02/28 14:14:29 Registering with agent
[Container] 2022/02/28 14:14:29 Phases found in YAML: 2
[Container] 2022/02/28 14:14:29  BUILD: 2 commands
[Container] 2022/02/28 14:14:29  INSTALL: 2 commands
[Container] 2022/02/28 14:14:29 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2022/02/28 14:14:29 Phase context status code:  Message: 
[Container] 2022/02/28 14:14:29 Entering phase INSTALL

[Container] 2022/02/28 14:14:29 Running command npm install -g aws-cdk
/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/aws-cdk/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ aws-cdk@2.14.0
added 1 package from 1 contributor in 1.358s

[Container] 2022/02/28 14:14:34 Phase complete: INSTALL State: SUCCEEDED
[Container] 2022/02/28 14:14:34 Phase context status code:  Message: 
[Container] 2022/02/28 14:14:34 Entering phase PRE_BUILD
[Container] 2022/02/28 14:14:34 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2022/02/28 14:14:34 Phase context status code:  Message: 
[Container] 2022/02/28 14:14:34 Entering phase BUILD

[Container] 2022/02/28 14:14:34 Running command cdk -a . deploy UpdateStack --require-approval=never --verbose
CDK toolkit version: 2.14.0 (build 762d71b)
Command line arguments: {
  _: [ 'deploy' ],
  a: '.',
  app: '.',
  'require-approval': 'never',
  requireApproval: 'never',
  verbose: 1,
  v: 1,
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  all: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  logs: true,
  '$0': '/usr/local/bin/cdk',
  STACKS: [ 'UpdateStack' ],
  'S-t-a-c-k-s': [ 'UpdateStack' ]
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: '.',
  context: {},
  debug: false,
  assetMetadata: true,
  requireApproval: 'never',
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [ '*' ],
  lookups: true
}
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to <REGION>
Resolving default credentials
Looking up default account ID from STS
Default account ID: <ACCOUNT>
Setting "CDK_DEFAULT_ACCOUNT" environment variable to <ACCOUNT>
context: {
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
--app points to a cloud assembly, so we bypass synth

✨  Synthesis time: 0.17s

UpdateStack: deploying...
Retrieved account ID <ACCOUNT> from disk cache
Assuming role 'arn:aws:iam::<ACCOUNT>:role/cdk-hnb659fds-deploy-role-<ACCOUNT>-<REGION>'.
Waiting for stack CDKToolkit to finish creating or updating...
[0%] start: Publishing c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311:<ACCOUNT>-<REGION>
Retrieved account ID <ACCOUNT> from disk cache
Assuming role 'arn:aws:iam::<ACCOUNT>:role/cdk-hnb659fds-file-publishing-role-<ACCOUNT>-<REGION>'.
[0%] check: Check s3://cdk-hnb659fds-assets-<ACCOUNT>-<REGION>/c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311.json
Call failed: getBucketEncryption({"Bucket":"cdk-hnb659fds-assets-<ACCOUNT>-<REGION>"}) => Access Denied (code=AccessDenied)
[0%] debug: Could not read encryption settings of bucket 'cdk-hnb659fds-assets-<ACCOUNT>-<REGION>': uploading with default settings ("cdk bootstrap" to version 9 if your organization's policies prevent a successful upload or to get rid of this message).
[0%] upload: Upload s3://cdk-hnb659fds-assets-<ACCOUNT>-<REGION>/c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311.json
[100%] success: Published c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311:<ACCOUNT>-<REGION>
UpdateStack: checking if we can skip deploy
UpdateStack: template has changed
UpdateStack: deploying...
Removing existing change set with name cdk-deploy-change-set if it exists
Attempting to create ChangeSet with name cdk-deploy-change-set to update stack UpdateStack
UpdateStack: creating CloudFormation changeset...
Initiated creation of changeset: arn:aws:cloudformation:<REGION>:<ACCOUNT>:changeSet/cdk-deploy-change-set/685a1ec5-7267-47f4-b859-30b5235c6e10; waiting for it to finish creating...
Waiting for changeset cdk-deploy-change-set on stack UpdateStack to finish creating...
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Initiating execution of changeset arn:aws:cloudformation:<REGION>:<ACCOUNT>:changeSet/cdk-deploy-change-set/685a1ec5-7267-47f4-b859-30b5235c6e10 on stack UpdateStack
Execution of changeset arn:aws:cloudformation:<REGION>:<ACCOUNT>:changeSet/cdk-deploy-change-set/685a1ec5-7267-47f4-b859-30b5235c6e10 on stack UpdateStack has started; waiting for the update to complete...
Waiting for stack UpdateStack to finish creating or updating...
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS (User Initiated))
UpdateStack |  0/57 | 2:15:44 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack  | UpdateStack User Initiated
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
 0/57 Currently in progress: UpdateStack
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset40 (PipelineAssetsFileAsset405A6FEB45) 
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset31 (PipelineAssetsFileAsset315BF305B1) 
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset33 (PipelineAssetsFileAsset331EA5BEFF) 
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset30 (PipelineAssetsFileAsset30AA2C9CCE) 
UpdateStack |  0/57 | 2:16:38 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset32 (PipelineAssetsFileAsset32B2F439EA) 
UpdateStack |  0/57 | 2:16:38 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset44 (PipelineAssetsFileAsset44F4DEECBF) 
UpdateStack |  0/57 | 2:16:38 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset28 (PipelineAssetsFileAsset28D7887C5A) 
UpdateStack |  0/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset35 (PipelineAssetsFileAsset359C5D1AAE) 
UpdateStack |  0/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset34 (PipelineAssetsFileAsset34304F56C6) 
UpdateStack |  0/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset37 (PipelineAssetsFileAsset37A82313CE) 
UpdateStack |  1/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset31 (PipelineAssetsFileAsset315BF305B1) 
UpdateStack |  2/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset40 (PipelineAssetsFileAsset405A6FEB45) 
UpdateStack |  3/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset30 (PipelineAssetsFileAsset30AA2C9CCE) 
UpdateStack |  3/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset36 (PipelineAssetsFileAsset36A9A3262B) 
UpdateStack |  4/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset32 (PipelineAssetsFileAsset32B2F439EA) 
UpdateStack |  5/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset33 (PipelineAssetsFileAsset331EA5BEFF) 
UpdateStack |  5/57 | 2:16:40 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset23 (PipelineAssetsFileAsset23A5E4949B) 
UpdateStack |  5/57 | 2:16:40 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset29 (PipelineAssetsFileAsset2994CB4E12) 
UpdateStack |  5/57 | 2:16:40 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset48 (PipelineAssetsFileAsset4856493151) 
UpdateStack |  6/57 | 2:16:40 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset44 (PipelineAssetsFileAsset44F4DEECBF) 
UpdateStack |  7/57 | 2:16:40 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset35 (PipelineAssetsFileAsset359C5D1AAE) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack |  8/57 | 2:16:40 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset34 (PipelineAssetsFileAsset34304F56C6) 
UpdateStack |  9/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset28 (PipelineAssetsFileAsset28D7887C5A) 
UpdateStack | 10/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset37 (PipelineAssetsFileAsset37A82313CE) 
UpdateStack | 11/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset36 (PipelineAssetsFileAsset36A9A3262B) 
UpdateStack | 11/57 | 2:16:41 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset24 (PipelineAssetsFileAsset240D3F5D3D) 
UpdateStack | 11/57 | 2:16:41 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset39 (PipelineAssetsFileAsset3939491230) 
UpdateStack | 12/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset29 (PipelineAssetsFileAsset2994CB4E12) 
UpdateStack | 13/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset48 (PipelineAssetsFileAsset4856493151) 
UpdateStack | 14/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset23 (PipelineAssetsFileAsset23A5E4949B) 
UpdateStack | 14/57 | 2:16:42 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset16 (PipelineAssetsFileAsset161A3A5D08) 
UpdateStack | 14/57 | 2:16:42 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset14 (PipelineAssetsFileAsset14A3CA232F) 
UpdateStack | 15/57 | 2:16:43 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset24 (PipelineAssetsFileAsset240D3F5D3D) 
UpdateStack | 16/57 | 2:16:43 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset39 (PipelineAssetsFileAsset3939491230) 
UpdateStack | 16/57 | 2:16:43 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset27 (PipelineAssetsFileAsset278581B380) 
UpdateStack | 16/57 | 2:16:43 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset2 (PipelineAssetsFileAsset24D2D639B) 
UpdateStack | 16/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset43 (PipelineAssetsFileAsset43E66E3741) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset16 (PipelineAssetsFileAsset161A3A5D08) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset42 (PipelineAssetsFileAsset4293256298) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset25 (PipelineAssetsFileAsset25C2F636DD) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset41 (PipelineAssetsFileAsset4148861B0A) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset15 (PipelineAssetsFileAsset15B22B8501) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset26 (PipelineAssetsFileAsset26CD3C46ED) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 18/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset42 (PipelineAssetsFileAsset4293256298) 
UpdateStack | 19/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset15 (PipelineAssetsFileAsset15B22B8501) 
UpdateStack | 20/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset41 (PipelineAssetsFileAsset4148861B0A) 
UpdateStack | 21/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset26 (PipelineAssetsFileAsset26CD3C46ED) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 21/57 | 2:16:55 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset47 (PipelineAssetsFileAsset47191A3E0D) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 22/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset2 (PipelineAssetsFileAsset24D2D639B) 
UpdateStack | 23/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset14 (PipelineAssetsFileAsset14A3CA232F) 
UpdateStack | 24/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset27 (PipelineAssetsFileAsset278581B380) 
UpdateStack | 25/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset43 (PipelineAssetsFileAsset43E66E3741) 
UpdateStack | 26/57 | 2:16:57 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset47 (PipelineAssetsFileAsset47191A3E0D) 
UpdateStack | 27/57 | 2:16:58 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset25 (PipelineAssetsFileAsset25C2F636DD) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 28/57 | 2:17:17 PM | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack  | UpdateStack 
Stack UpdateStack has completed updating

✅  UpdateStack

✨  Deployment time: 165.34s

Stack ARN:
arn:aws:cloudformation:<REGION>:<ARN>:stack/UpdateStack/f62b9db0-3e1d-11ec-a5dd-024bd07bc0e7

✨  Total time: 165.52s

Failed to load notices from cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json'
Error on request: Error: read ETIMEDOUT
Notices refreshed
Error on request: Error: read ETIMEDOUT
Failed to get tree.json file: Error: cdk.out/tree.json: ENOTDIR: not a directory, open 'cdk.out/tree.json'


[Container] 2022/02/28 14:33:13 Phase complete: BUILD State: SUCCEEDED

This error delays the Update Step by almost 16 minutes.

As source action we are using github action v2.

Do you have any idea whats going on here?

Regards

Reproduction Steps

start the pipeline using a github source action

What did you expect to happen?

no read timeouts for notice.json and tree.json

What actually happened?

a read timeout for notice.json and tree.json

CDK CLI Version

aws-cdk@2.15.0

Framework Version

No response

Node.js Version

14

OS

linux

Language

Typescript

Language Version

4.3.5

Other information

No response

@erudisch erudisch added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Mar 1, 2022
@github-actions github-actions bot added the @aws-cdk/aws-codepipeline Related to AWS CodePipeline label Mar 1, 2022
@skinny85
Copy link
Contributor

skinny85 commented Mar 1, 2022

@erudisch can you please show the relevant part of your CDK code?

@skinny85 skinny85 added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Mar 1, 2022
@clairemr
Copy link
Contributor

clairemr commented Mar 1, 2022

I'm having the same issue. Using the cdk pipelines module, the update step is taking a lot longer to run because of the below error

Failed to load notices from cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json'
--
990 | Error on request: Error: connect ETIMEDOUT 99.86.143.31:443
991 | Notices refreshed
992 | Error on request: Error: connect ETIMEDOUT 99.86.143.93:443
993 | Failed to get tree.json file: Error: cdk.out/tree.json: ENOTDIR: not a directory, open 'cdk.out/tree.json'

Looking at the logs, this error started appearing for me some time between 1:30-2pm Friday 25/3 AEST. The only change I made to my files between pipeline runs was commenting out a single line from my app code, so nothing that should have affected the self mutate step of the pipeline.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Mar 2, 2022
@NGL321 NGL321 added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Mar 2, 2022
@lennart-land
Copy link

I'm also having this issue.

This issues seems to be related to #18936 and cdk version 2.14.
We are using the pipelines.CodePipeline and part of this pipeline is the self mutating step.
This self mutating step executes the following command cdk -a . deploy Pipeline --require-approval=never --verbose.
Since we are behind a corporate proxy and we did not find a way to configure the proxy for fetching the notices.json it ends up in a timeout.

We also tried the flags --no-notices and notices = false but somehow they still got fetched and we had self mutating steps taking up to 18minutes.

An awesome fix would be to disable fetching of notices via environment variable.

@shindeja
Copy link

shindeja commented Mar 2, 2022

I'm also having this issue.
This issue is happening on cdk version 1.146.0.
We are using the pipelines.CodePipeline and part of this pipeline is the self mutating step. This self mutating step executes the following command cdk -a . deploy Pipeline --require-approval=never --verbose.

Failed to load notices from cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json' --| Failed to get tree.json file: Error: cdk.out/tree.json: ENOTDIR: not a directory, open 'cdk.out/tree.json'

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Mar 2, 2022
@erudisch
Copy link
Author

erudisch commented Mar 2, 2022

I'm having the same issue. Using the cdk pipelines module, the update step is taking a lot longer to run because of the below error

Failed to load notices from cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json'
--
990 | Error on request: Error: connect ETIMEDOUT 99.86.143.31:443
991 | Notices refreshed
992 | Error on request: Error: connect ETIMEDOUT 99.86.143.93:443
993 | Failed to get tree.json file: Error: cdk.out/tree.json: ENOTDIR: not a directory, open 'cdk.out/tree.json'

Looking at the logs, this error started appearing for me some time between 1:30-2pm Friday 25/3 AEST. The only change I made to my files between pipeline runs was commenting out a single line from my app code, so nothing that should have affected the self mutate step of the pipeline.

I was also just change a type of an input value for my lambda. nothing pipeline related

@erudisch
Copy link
Author

erudisch commented Mar 2, 2022

I'm also having this issue.

This issues seems to be related to #18936 and cdk version 2.14. We are using the pipelines.CodePipeline and part of this pipeline is the self mutating step. This self mutating step executes the following command cdk -a . deploy Pipeline --require-approval=never --verbose. Since we are behind a corporate proxy and we did not find a way to configure the proxy for fetching the notices.json it ends up in a timeout.

We also tried the flags --no-notices and notices = false but somehow they still got fetched and we had self mutating steps taking up to 18minutes.

An awesome fix would be to disable fetching of notices via environment variable.

This exactly reflects our setup

@jdornieden
Copy link

Same here!

@mrgum
Copy link

mrgum commented Mar 2, 2022

same here, updating to cdk 2.15 didn't change it

adding an install step of
"mkdir -p /root/.cdk/cache/"
before using cdk
fixed it for me

        pipeline = pipelines.CodePipeline(
            self,
            'pipeline',
            pipeline_name='myPipeline',
            synth=pipelines.CodeBuildStep(
                "Synth",
                input=source,
                install_commands=[
                    "mkdir -p /root/.cdk/cache/",
                    "npm install -g aws-cdk",
                    "python -m pip install --upgrade pip",
                    "pip install -r requirements.txt"
                ],
                commands=[
                    "npx cdk version",
                    "node -v",
                    "npx cdk synth -v"
                ]
                ]
            )
        )

@erudisch
Copy link
Author

erudisch commented Mar 2, 2022

same here, updating to cdk 2.15 didn't change it

adding an install step of "mkdir -p /root/.cdk/cache/" before using cdk fixed it for me

        pipeline = pipelines.CodePipeline(
            self,
            'pipeline',
            pipeline_name='myPipeline',
            synth=pipelines.CodeBuildStep(
                "Synth",
                input=source,
                install_commands=[
                    "mkdir -p /root/.cdk/cache/",
                    "npm install -g aws-cdk",
                    "python -m pip install --upgrade pip",
                    "pip install -r requirements.txt"
                ],
                commands=[
                    "npx cdk version",
                    "node -v",
                    "npx cdk synth -v"
                ]
                ]
            )
        )

doesn't work for me

@einstein-deploy
Copy link

einstein-deploy commented Mar 2, 2022

Same issue in CodeBuild here, cdk v1.147.0:

Resolving default credentials
905 | Failed to store notices in the cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json'
906 | Notices refreshed

@lennart-land
Copy link

As far i can tell the PR wont help to resolve our issue, since we are using the pipelines.CodePipeline and we are not able to change the cdk command for the self mutating step.

@NGL321 NGL321 added in-progress This issue is being actively worked on. p1 and removed needs-triage This issue or PR still needs to be triaged. labels Mar 4, 2022
@mergify mergify bot closed this as completed in #19226 Mar 7, 2022
mergify bot pushed a commit that referenced this issue Mar 7, 2022
The `--no-notices` flag is only being considered for actually displaying the notices, but not in the refresh step, used to mask latency.

Also handling request timeout on the `Promise` level. If the promise doesn't resolve within the time limit, resolve with an empty array, no matter what.

Fixes #19201.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Mar 7, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

TheRealAmazonKendra pushed a commit to TheRealAmazonKendra/aws-cdk that referenced this issue Mar 11, 2022
…19226)

The `--no-notices` flag is only being considered for actually displaying the notices, but not in the refresh step, used to mask latency.

Also handling request timeout on the `Promise` level. If the promise doesn't resolve within the time limit, resolve with an empty array, no matter what.

Fixes aws#19201.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@55Cancri
Copy link

It's been three months since this issue was opened, has anyone been able to solve this and get a pipeline successfully pass through the UpdatePipeline - SelfMutate step?

The error I receive is Failed to store notices in the cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json' and then a few lines later ENOENT: no such file or directory, open 'manifest.json'.

My pipeline exists in a monorepo setup like:

packages/
    client/
    server/
        ...cdk stuff
    shared/

where the cdk is in the server folder. The error seems to be searching for the cdk output after synth in the root of this monorepo instead of packages/server, despite setting the primaryOutputDirectory to "packages/server". Here is how my cdk stack looks:

   const pipeline = new cdk.pipelines.CodePipeline(this, "Pipeline", {
            pipelineName: app_name + "pipeline",
            // stop $1/month kms key charge
            crossAccountKeys: false,
            // the purpose of this step is to create a cdk.out folder via cdk synth command
            synth: new cdk.pipelines.ShellStep("Synth", {
                input: cdk.pipelines.CodePipelineSource.gitHub(
                    props!.repo_name,
                    props!.branch_name,
                    { authentication: github_token }
                ),
                commands: [
                    "yarn install",
                    "cd packages/server",
                    "yarn install",
                    "yarn build",
                    "npx cdk synth --all -c stage=" + env_name,
                    "mv cdk.out ../..",
                ],
                primaryOutputDirectory: "packages/server",
            }),
        })
       // ...
       pipeline.addStage(server_stage)
       pipeline.addStage(client_stage)

You can see the primaryOutputDirectory is set to "packages/server", and also the last step of the commands array is "mv cdk.out ../..", which moves the generated cdk/ folder up two directories to the root.
It looks like this step is not persisted to the UpdatePipeline - SelfMutate step because the error doesn't seem to find the folder anymore.
Any help on this would be great.

@einstein-aws
Copy link

Still having this issue. CDK 2.25.0 (build ae1cb4b)

@zdenkoe
Copy link

zdenkoe commented May 31, 2022

Hi, have you tried to enable

dockerEnabledForSelfMutation: true,

before enabling the docker configuration? (e.g.)

codeBuildDefaults: {
                buildEnvironment: {
                    buildImage: LinuxBuildImage.

@55Cancri
Copy link

Hi, have you tried to enable

dockerEnabledForSelfMutation: true,

before enabling the docker configuration? (e.g.)

codeBuildDefaults: {
                buildEnvironment: {
                    buildImage: LinuxBuildImage.

This did not work, and also, I am not using docker.
I'm trying to simply create a ci/cd pipeline for apps within a monorepo.

Has anyone discovered any workarounds or any sample repos with a cdk codepipeline working for a monorepo?
Do the maintainers of aws-cdk see these issues? A few cdk versions have been released in the last month but its still an issue.

@zdenkoe
Copy link

zdenkoe commented Jul 27, 2022

Hi, just noticed your setting primaryOutputDirectory: "packages/server",

Actually, mine looks like primaryOutputDirectory: "packages/server/cdk.out",

@froilan
Copy link

froilan commented Oct 31, 2022

@55Cancri
I was encountering the same issue as yours and managed to make it work.
zdenkoe was right, the primaryOutputDirectory was set without */cdk.out.

Reason for the issue:
The generated codebuild relies on the artifact typed CODEPIPELINE (docs). Meaning it will come from the build output of the CodePipeline. And not adding the cdk.out in the output directory means the build output of CodePipeline is empty. Resulting the self mutate step to have no inputs, thus the error.

Hope that helps.

@orestislykos
Copy link

In my case it had nothing to do with the output directory, I just had not pushed my pipeline change to the remote! Stupid error on my part, but the error log is not helpful so adding this here in case it helps others.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-codepipeline Related to AWS CodePipeline bug This issue is a bug. in-progress This issue is being actively worked on. p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.