aws-codepipeline: cross region support creates huge inline policy document for the pipeline role #16244

adigopv · 2021-08-26T17:30:45Z

We have a code pipeline setup with cross region support. The pipeline creates replication buckets in cross regions and pipeline internally adds permissions for these s3 buckets in the pipeline role here.

Something like this for each cross region:

{
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*",
                "s3:PutObject*",
               "s3:Abort*"
            ],
            "Resource": [
                "arn:aws:s3:::<replicationbucket>",
                "arn:aws:s3:::<replicationbucket>/*"
            ],
            "Effect": "Allow"
        },

Since an addAction is called for each region, we add a new policy statement every time with the same permissions, which is causing the pipeline role inline policy to exceed the iam policy size limit causing below error.

Maximum policy size of 10240 bytes exceeded for role <pipelinerole>

Similar issue is seen for the sts:AssumeRole permissions added here.

Reproduction Steps

Create a codepipeline with multiple stages and ~ 15 regions.

What did you expect to happen?

The role that pipeline created should be reduced / compacted. Since there are many duplicate actions for each resource, we can have a single policy statement with all resources.

We were adding new stages and regions in our pipeline when we observed this.

What actually happened?

Got this error when deploying the stack.

Maximum policy size of 10240 bytes exceeded for role <pipelinerole>

Environment

CDK CLI Version : 1.23, I think the latest version has this bug too.
Framework Version: 1.23
Node.js Version: 12.x
OS : MacOS
Language (Version): TypeScript (3.8.3)

This is 🐛 Bug Report

The text was updated successfully, but these errors were encountered:

skinny85 · 2021-08-26T20:59:05Z

Thanks for reporting @adigopv. I think this is another need for #14713; I'm not sure there's much the CodePipeline module itself can do here.

One way you can try to unblock yourself is to create a Role explicitly, add it all of the permissions (in a more efficient way), and then pass an immutable copy of it into the CodePipeline construct(s): https://docs.aws.amazon.com/cdk/api/latest/docs/aws-iam-readme.html#opting-out-of-automatic-permissions-management

Thanks,
Adam

adigopv · 2021-08-27T08:20:07Z

Hi Adam, thanks for the suggestion ! But, we do not have knowledge about some of the resource ARNs like the replicationbucket ids for each region, as pipeline internally creates these s3 buckets in cross regions. Similarly the actionrole ARN for the sts:AssumeRole permissions. So, to overcome that we would have to use wildcards * in our policy to some extent which might be over permissive. But this can definitely be used as a temporary workaround.

Thanks,
Sai

skinny85 · 2021-08-27T18:45:03Z

@adigopv any chance you could supply me with a small CDK app (maybe hosted in a GitHub public repository) that reproduces this error? It would really help me trying to devise a workaround for this problem.

…size limit When we generate CodePipelines, we need to add an `sts:AssumeRole` statement for each Action in the pipeline, and a `Bucket.grantReadWrite()` statement for each region the pipeline is in, to the policy statement of the pipeline's Role. For pipelines with many Actions and/or regions, this makes the policy exceed IAM limit of 10240 bytes. Extract a new class from the CodePipeline CloudFormation Actions that caches the statements added to a given Principal by the 'Action' field, and groups the statements with the same 'Actions' by adding elements to the 'Resource' field. This dramatically reduces the duplication in the statement, and increases the chances of it being smaller than the limit. Use this new class in the `Pipeline` construct. Fixes aws#16244

moltar · 2021-09-30T05:22:09Z

This is a real blocker!

The whole idea of infra-as-code is that it can be scaled and generated.

But this becomes an impediment and an antithesis to infra-as-code.

We were testing on a small set of stages to make sure it all worked. When we added other stages, everything crumbled and the pipeline is now broken.

mmuller88 · 2021-09-30T07:33:55Z

@moltar how about destroying and redeploying your pipeline with alle needed stages included? A pipeline should be loosly coupled to the "golden" cdk stacks

moltar · 2021-09-30T09:37:38Z

how about destroying and redeploying your pipeline with alle needed stages included?

That's not always possible, as we add new stages.

Also, I am not sure how it'd help?

The problem is the pipeline generates a policy that is unnecessarily large. It lists permissions for every single asset, which there can be many of.

    {
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::11111111:role/ProjectPipelineSourceaatlanti-17INJW07P0N4J",
      "Effect": "Allow"
    },
    {
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::11111111:role/ProjectPipelineBuildSynthShellStepCod-1JX5K4YAETR30",
      "Effect": "Allow"
    },
    {
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::11111111:role/ProjectPipelineUpdatePipelineSelfMuta-4OFIXOVQVY50",
      "Effect": "Allow"
    },
    {
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::11111111:role/ProjectPipelineAssets1DockerAsset1Cod-NJA8YN3R8EW1",
      "Effect": "Allow"
    }

and on and on and on

These assets aren't stage-specific. If we have 1 or 3 stages, the assets do not change.

80% of the policy could be replaced with:

    {
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::11111111:role/ProjectPipeline*",
      "Effect": "Allow"
    },

moltar · 2021-09-30T12:29:13Z

@adigopv any chance you could supply me with a small CDK app (maybe hosted in a GitHub public repository) that reproduces this error? It would really help me trying to devise a workaround for this problem.

Hey, I don't have a repo, but a simple repro maybe is to loop over lambda function construct and create 50 lambdas with some dummy asset (but new asset for each).

This will produce insanely long policy which sets a new role for each asset, and then grants sts:AssumeRole in the main pipeline policy for each one of those roles.

moltar · 2021-09-30T12:36:23Z

Here's the full (sanitized) policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*",
                "s3:PutObject*",
                "s3:Abort*"
            ],
            "Resource": [
                "arn:aws:s3:::ad-mainline-mainlinepipelineartifactsbucket32b9fb-xyz",
                "arn:aws:s3:::ad-mainline-mainlinepipelineartifactsbucket32b9fb-xyz/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey",
                "kms:Encrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*"
            ],
            "Resource": "arn:aws:kms:eu-west-1:111111111111:key/00000000-0000-0000-0000-000000000000",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineSourceScaleLeapatlanti-17INJW07P0N4J",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineBuildSynthShellStepCod-1JX5K4YAETR30",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineUpdatePipelineSelfMuta-4OFIXOVQVY50",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1DockerAsset1Cod-NJA8YN3R8EW1",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1DockerAsset2Cod-11OHUPSDDCAOZ",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset1CodeP-1DT95S0C165LP",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset10Code-1PTENGEIA6IGN",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset11Code-3SID6KWFXP4S",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset12Code-HASQGQSED9L8",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset13Code-10NEHGGVXQ07O",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset14Code-H4V83IMXBFH",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset15Code-1SZJY4YGP1Z9N",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset16Code-1FDTTZBNXIHNY",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset17Code-7ZEOMOE4FDLL",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset18Code-DGD5JWHHVVT4",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset19Code-100WNYVJLJF4U",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset2CodeP-194SY99PL17D1",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset20Code-5441XLBRNDZS",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset21Code-QF6TINYYHRL2",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset22Code-OUNVGMQ7TMSC",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset23Code-1UD2HYLSYXKNV",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset24Code-664Y8WHH7JZ9",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset25Code-AUVHQCS529T1",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset26Code-RV8CCCNDZDO6",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset27Code-XNU1I9IQRC30",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset28Code-1G05MOM2IF9QS",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset29Code-1D2F1LBMS773H",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset3CodeP-USV2ATZ3VAV1",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset30Code-BBKO6T4MZN8J",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset31Code-9XQBKA45886R",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset32Code-1VOHI3BAKRZZT",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset33Code-D1YDOXD4VB59",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset34Code-17HGFP2KLBNP7",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset35Code-LRF74E107MG5",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset36Code-1SZIOI7KG0KKR",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset37Code-7IBADQWRWOHE",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset38Code-KZS13S4OV0M5",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset39Code-IL8N1ZSU0ULL",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset4CodeP-1X172A6SCOVKE",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset40Code-TPAUSL1GFGD8",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset41Code-AXBUOGR42Z0",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset42Code-1I1USSIDU605G",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset43Code-1FTYT0NU8CVOV",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset44Code-2H0PIC97UOWQ",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset45Code-XJ5LHVI7WD8U",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset46Code-74644R1KCC6V",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset47Code-1OEKGUKDNVPIY",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset48Code-KIZEO8Y5F7ZI",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset49Code-3MF3M1C0MRS5",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset5CodeP-ULQ1QL1DBXAP",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset6CodeP-1JYS6M3MOWBQ9",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset7CodeP-1T53GD4PRG3J7",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets1FileAsset8CodeP-LUNA0SYKXWCP",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/AD-Mainline-MainlinePipelineAssets2FileAsset9CodeP-1BISQRL8S9XVK",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::111111111111:role/cdk-hnb659fds-deploy-role-111111111111-eu-west-1",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::222222222222:role/cdk-hnb659fds-deploy-role-222222222222-eu-west-1",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*",
                "s3:PutObject*",
                "s3:Abort*"
            ],
            "Resource": [
                "arn:aws:s3:::ad-mainline-support-ap-soeplicationbucket11c23xyz",
                "arn:aws:s3:::ad-mainline-support-ap-soeplicationbucket11c23xyz/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey",
                "kms:Encrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::333333333333:role/cdk-hnb659fds-deploy-role-333333333333-ap-southeast-1",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*",
                "s3:PutObject*",
                "s3:Abort*"
            ],
            "Resource": [
                "arn:aws:s3:::ad-mainline-support-ap-noeplicationbucket590d26xyz",
                "arn:aws:s3:::ad-mainline-support-ap-noeplicationbucket590d26xyz/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::44444444444444:role/cdk-hnb659fds-deploy-role-44444444444444-ap-northeast-1",
            "Effect": "Allow"
        }
    ]
}

bilalq · 2021-12-07T18:00:59Z

Could the CodePipeline construct run an aspect on itself to do the optimization on its end? It seems there's reluctance to make sweeping IAM optimization changes based on comments in #14713, but this is a real show-stopper for production usage of CDK.

skinny85 · 2021-12-07T18:53:12Z

You're right @bilalq. I've started working on this in #16350, but had not had the time to pick that back up.

meeber · 2022-01-21T18:04:31Z

We just ran into this. Glad to see it has recent attention in #18549 and #16350. It'd be great if we could pass a parameter like wildcard_policy = True to CodePipeline so that it automatically generates a less granular policy using wildcards that'll scale to deployments with thousands of target environments. In the meantime, we'll explore the suggestion to "pass a codepipeline.Pipeline to CodePipeline" with a custom role & policy. Thanks!

gottkanzler-rgb · 2022-01-25T13:31:05Z

We are running into similar issues with the bucket policy in cross region support stacks.
The created Bucket policy contains a huge amount of "redundant" statements, leading the deployment to fail with this error:
Normalized policy document exceeds the maximum allowed size of 20480 bytes.
So there would be an urgent need for a similiar mechanic as for the role, e.g. manually passing a custom Bucket policy to the CodePipeline.
The option with passing a codepipeline.Pipeline to CodePipelineis not convinient at all in this case, as we do not want to handle all the KMS configurations manually.

skinny85 · 2022-01-25T17:17:25Z

@gottkanzler-rgb can you specify your exact problem with passing codepipeline.Pipeline to CodePipeline in #18549?

gottkanzler-rgb · 2022-01-26T08:25:55Z

Initializing the Pipeline myself seems to have no effect. I have an artifact Bucket with a suitable resource policy, same for the KMS key, but when synthesizing the stack it automatically creates the redundant policy statements nevertheless.
I end up with 3000 lines of KMS Key policy and 7000 lines of bucket policy, resulting the Cfn template being over 1mb.

meeber · 2022-02-03T00:05:17Z

I stopped using CDK Pipelines, not so much because of this issue, but rather because of the non-adjustable AWS CodePipeline limits. In particular, the "50 actions per stage" and "50 stages per pipeline" limits are troublesome for deployments targeting hundreds or thousands of environments since multiple actions are created per environment.

I'm currently using the L2 CodePipeline and CodeBuild constructs and just running cdk synth and cdk deploy from a CodeBuild project. This required a bit more code than CDK Pipelines, and the CodePipeline execution dashboard isn't as informational (since all the deployments are in a single action), but it seems to have resolved the scaling problems caused by this issue and the underlying service limits.

ekeyser · 2022-02-21T23:43:11Z

@meeber Yes some limits seem very arbitrary. I'm sure it's defined for a reason but leaves a bad taste and doesn't generate a ton of confidence developing solutions relying on this service/tooling.

am29d · 2022-06-01T17:23:11Z

Hey guys,

I recently had the same issue #20565 when deploying to all AWS regions and luckily there was a recent fix to prevent 10k exceed limit for IAM policies. In addition, adding { "context": { "@aws-cdk/aws-iam:minimizePolicies": true }} to your cdk.json will reduce the number of generated policies and its size significantly significantly.

ekeyser · 2022-06-01T17:29:51Z

@am29d It does not help. I have that directive active. Indeed it does minimize IAM policy documents but the issue is much more widespread. There are many open issues in this project regarding the 10K limit being exceeded under what appear to be many scenarios.

am29d · 2022-06-01T17:39:09Z

@am29d It does not help. I have that directive active. Indeed it does minimize IAM policy documents but the issue is much more widespread. There are many open issues in this project regarding the 10K limit being exceeded under what appear to be many scenarios.

I see, I am not aware of all the cases, are there any other cases for 10k limit? I could only find three issues.

At least the recent PR 75bfce7 helped to reduce the policy size by creating overflow policies for cross region support stacks.

Can you point me to other issues, curious to see what blockers I might encounter further down the road.

ekeyser · 2022-06-01T17:58:14Z

#19835 lists a few issues. The main issue is closed but it references other issues some of which are probably still open.

https://github.com/aws/aws-cdk/issues?q=is%3Aissue+10240+is%3Aopen are open cases but I seriously doubt it covers all issues.

here's an issue I opened up back in Dec #18167 which won't show up in a search for 10240. Issue is due to pipelines/CodePipeline constructs not enforcing or honoring the withoutPolicyUpdates method (and therefore still generating policy documents that exceed the limit). So, different issue but still same underlying root cause.

Somewhere (maybe even in this ticket?) @rix0rrr mentions the challenge involved which has to do with the fact that the policy document is inline and not managed. I've kinda always wondered why someone can't split the policy up into multiple managed policies and then attach those multiple policies to the role in question. I'm sure it's easier said than done so I don't want to sound flippant or anythign. Just kinda wondered that's all. And no I'm not about to take a break from my multiple clients and 55 hours / week to help fix this issue (even though i sort of want to) :-) I just don't have time and I'm pretty sure you folks at AWS have the resources if you really wanted to figure it out ;-)

am29d · 2022-06-02T08:15:39Z

Hey @ekeyser ,

thanks for the pointers and the clarification, this helps to see the broader picture and other issues I might run into later. I agree that it's probably not a trivial challenge to solve, and with the recent PR the policies are chunked into an overflow policy document and are attached to the role.

I will take a look at your mentioned issue and check if the recent changes fixed the problem.

ekeyser · 2022-06-02T19:04:47Z

@am29d not to put too fine a point on it but the broader picture is this project and aws is failing users. You all have put v1 into maintenance mode without resolving problems that allow people to migrate to v2. Not sure what kind of logic that is but doesn't seem like very good logic.

adigopv added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 26, 2021

github-actions bot added the @aws-cdk/aws-codepipeline Related to AWS CodePipeline label Aug 26, 2021

github-actions bot assigned skinny85 Aug 26, 2021

skinny85 added effort/large Large work item – several weeks of effort p1 and removed needs-triage This issue or PR still needs to be triaged. labels Aug 26, 2021

skinny85 removed their assignment Aug 26, 2021

skinny85 mentioned this issue Sep 2, 2021

fix(codepipeline): large cross-region pipelines exceed IAM policy size limit #16350

Closed

skinny85 added the in-progress This issue is being actively worked on. label Sep 2, 2021

bilalq mentioned this issue Dec 7, 2021

(aws-iam): policy document optimization #14713

Closed

2 tasks

peterwoodworth mentioned this issue Jan 19, 2022

aws-cdk_pipelines: Maximum policy size of 10240 bytes exceeded for role #18531

Closed

skinny85 mentioned this issue Jan 20, 2022

(CDK Pipelines): allow passing a Role to CodePipeline #18549

Open

2 tasks

peterwoodworth mentioned this issue Feb 14, 2022

(aws_ecs): CDK insists on Modifying the ECS Execution/Task Role with duplicate permissions, resulting policy Is too big to deploy #18926

Closed

This was referenced Mar 4, 2022

CodePipeline: Maximum policy size of 10240 bytes exceeded for role xxx #19243

Closed

Tracking: Policy-generation creates oversized templates #19276

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws-codepipeline: cross region support creates huge inline policy document for the pipeline role #16244

aws-codepipeline: cross region support creates huge inline policy document for the pipeline role #16244

adigopv commented Aug 26, 2021 •

edited

skinny85 commented Aug 26, 2021

adigopv commented Aug 27, 2021 •

edited

skinny85 commented Aug 27, 2021 •

edited

moltar commented Sep 30, 2021

mmuller88 commented Sep 30, 2021

moltar commented Sep 30, 2021

moltar commented Sep 30, 2021

moltar commented Sep 30, 2021

bilalq commented Dec 7, 2021

skinny85 commented Dec 7, 2021

meeber commented Jan 21, 2022

gottkanzler-rgb commented Jan 25, 2022

skinny85 commented Jan 25, 2022

gottkanzler-rgb commented Jan 26, 2022

meeber commented Feb 3, 2022

ekeyser commented Feb 21, 2022

am29d commented Jun 1, 2022

ekeyser commented Jun 1, 2022

am29d commented Jun 1, 2022

ekeyser commented Jun 1, 2022

am29d commented Jun 2, 2022

ekeyser commented Jun 2, 2022

aws-codepipeline: cross region support creates huge inline policy document for the pipeline role #16244

aws-codepipeline: cross region support creates huge inline policy document for the pipeline role #16244

Comments

adigopv commented Aug 26, 2021 • edited

Reproduction Steps

What did you expect to happen?

What actually happened?

Environment

skinny85 commented Aug 26, 2021

adigopv commented Aug 27, 2021 • edited

skinny85 commented Aug 27, 2021 • edited

moltar commented Sep 30, 2021

mmuller88 commented Sep 30, 2021

moltar commented Sep 30, 2021

moltar commented Sep 30, 2021

moltar commented Sep 30, 2021

bilalq commented Dec 7, 2021

skinny85 commented Dec 7, 2021

meeber commented Jan 21, 2022

gottkanzler-rgb commented Jan 25, 2022

skinny85 commented Jan 25, 2022

gottkanzler-rgb commented Jan 26, 2022

meeber commented Feb 3, 2022

ekeyser commented Feb 21, 2022

am29d commented Jun 1, 2022

ekeyser commented Jun 1, 2022

am29d commented Jun 1, 2022

ekeyser commented Jun 1, 2022

am29d commented Jun 2, 2022

ekeyser commented Jun 2, 2022

adigopv commented Aug 26, 2021 •

edited

adigopv commented Aug 27, 2021 •

edited

skinny85 commented Aug 27, 2021 •

edited