(bootstrap): Bootstrapping with CDK v1.139.0 removes IAM policy from the ECR asset repository

[MousaZeidBaker]

What is the problem?

Bootstrapping from 1.138.0 to v1.139.0 removes IAM policy from the cdk-hnb659fds-container-assets-1234567890-eu-central-1 ECR repository resulting in Lambdas being unable to run due to not able to access their images. The following policy was removed:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "LambdaECRImageRetrievalPolicy",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": [
        "ecr:BatchGetImage",
        "ecr:DeleteRepositoryPolicy",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:SetRepositoryPolicy"
      ],
      "Condition": {
        "StringLike": {
          "aws:sourceArn": "arn:aws:lambda:eu-central-1:1234567890:function:*"
        }
      }
    }
  ]
}

resulting in the following Lambda error:

Failed to restore the function MyLambda: The function does not have permission to access the specified image.

Reproduction Steps

1. Bootstrap account with v1.138.0
2. Add permission policy (or just deploy a Docker Lambda which will add a policy automatically)
3. Bootstrap account with v1.139.0 and check the IAM policy gets removed from ECR cdk-hnb659fds-container-assets-1234567890-eu-central-1 repository

What did you expect to happen?

Bootstrap process should NOT remove IAM policy from ECR cdk-hnb659fds-container-assets-1234567890-eu-central-1 repository

What actually happened?

Bootstrap process removed IAM policy from ECR cdk-hnb659fds-container-assets-1234567890-eu-central-1 repository

CDK CLI Version

1.139.0

Framework Version

No response

Node.js Version

v14.15.0

OS

Ubuntu 20.04

Language

Python

Language Version

No response

Other information

No response

[scedward]

We also saw this unexpected change after upgrading to the latest bootstrap stack. For a quick fix, making a trivial change to the Dockerfile of the affected Lambda was enough to change the asset hash and led to restoration of the lost policy statement.

[polothy]

@NGL321 hello - we ran into this today and found the source of the problem. See this lambda doc.

Lambda needs ECR to have a policy on it to allow it to pull images. The "funny" thing that lambda does though is automatically add the policy to ECR:

If the Amazon ECR repository does not include these permissions, Lambda adds ecr:BatchGetImage and ecr:GetDownloadUrlForLayer to the container image repository permissions. Lambda can add these permissions only if the Principal calling Lambda has ecr:getRepositoryPolicy and ecr:setRepositoryPolicy permissions.

The problem is, on CDK bootstrap upgrade, the ECR policy that lambda added can be removed, thus breaking things for lambda (it cannot pull).

The fix seems to be to add this policy to ECR in the CDK bootstrap.

[ievgeniiskliarenko]

Hi @NGL321,
We got into the same issue with CDK v2 updating the bootstrap version from 9 to 10. Unfortunately, that caused a severe outage on our side.

[RomainMuller]

Seems the change is a collateral from the enablement of ECR Scan by default on the repository. This likely caused the repository policy to be re-written, and the Lambda-authored change to be removed. Definitely an awkward situation.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.