CodePipeline: still getting 'Maximum policy size of 10240 bytes exceeded for role' for cross-account pipeline

[cprice404]

Describe the bug

We recently filed an issue about how CodePipeline was failing due to the generation of extremely large IAM policy documents:

#19243

Our issue was closed and it was suggested that this PR might fix it:

#19114

This tracking issue was closed via that PR:

#19276

We have since upgraded our CDK version to 2.19.0 and set @aws-cdk/aws-iam:minimizePolicies to true.

This did seem to decrease the size of some IAM policies, but we are still hitting the same failure.

Expected Behavior

Expected #19114 to reduce generated IAM policy sizes so that our pipeline would deploy.

Current Behavior

Pipeline still fails to deploy with Maximum policy size of 10240 bytes exceeded for role xxx.

See attachment for example of the offending IAM policy. The policy contains three statements, and the third one is the offending one. It is granting AssumeRole to all of the roles for all of the actions in all of the accounts that this pipeline is trying to deploy to.

See attachment for example (sanitized) policy.
pipelineofpipelinesstackcoreinfrastructurepipeline03AEF9CA.template.JUST_OFFENDING_POLICY.sanitized.json.gz

There are 97 resources in that statement, they look like this:

                {
                  "Fn::GetAtt": [
                    "pipelinecoreinfrastructureDeployProdcelluseast1produseast1synthCodePipelineActionRole8A60FDE1",
                    "Arn"
                  ]
                },

and this:

                {
                  "Fn::Join": [
                    "",
                    [
                      "arn:",
                      {
                        "Ref": "AWS::Partition"
                      },
                      ":iam::111111111111:role/core-infrastructure-pipelhangesactionroleb355ae8d7595154ac81b"
                    ]
                  ]
                },

Reproduction Steps

Create a CodePipeline that creates additional Pipelines. In one of those additional pipelines, create a large number of actions that target multiple AWS accounts.

It would take some non-trivial effort for me to extract a concise reproducer from our current (private) code, but pending the responses to the questions above about whether this is still expected behavior, how to override the Role, etc., I am willing to try to put one together!

Possible Solution

It's not clear to me whether this behavior is still expected after the fix in #19114 was released.

We are looking for guidance on:

• Whether this is still a known issue in CDK
• Whether it is expected that additional fixes will be merged to address this

Also, there seem to be a ton of other tickets related to this, and in several of them it is hinted that users with advanced use cases like ours may need to opt out of this automatic policy generation, and override the Role with one that we manage ourselves (which might use wildcards or multiple policy attachments, etc.). If that is the current prescription, it would be wonderful if there were some official docs or examples that illustrated how to do this; I haven't really seen anything so far that gives me a concrete idea of how to even attempt it.

Additional Information/Context

No response

CDK CLI Version

2.19.0

Framework Version

2.19.0

Node.js Version

v16.5.0

OS

Amazon Linux 2

Language

Typescript

Language Version

TypeScript 4.4.2

Other information

No response

[skinny85]

@rix0rrr you want to take a look at this one?

[cprice404]

To work around this, we had to do the following:

1. provide our own Role and Policy
2. Use withoutPolicyUpdates to prevent CDK from trying to generate the policy for it
3. Add a DependsOn from the Pipeline to the Policy to prevent CFN from trying to deploy them simultaneously
4. Manage our own cross-region replication buckets, due to the bug described in CodePipeline: when providing custom Pipeline Role, cross-region support stacks cannot be deployed #19881 .

Just in case it helps anyone else, here is roughly what we ended up with:

  const pipelineRolePolicyName = `pipelinepolicy-${pipelineName}`;
  const pipelineRolePolicy = new iam.Policy(scope, pipelineRolePolicyName, {
    policyName: pipelineRolePolicyName,
    // These policy statements were built up by hand after examining the
    // statements in the CDK-generated policy for several pipelines.
    statements: [
      // Grants the pipeline access to the s3 artifact buckets.
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: [
          's3:Abort*',
          's3:DeleteObject*',
          's3:GetBucket*',
          's3:GetObject*',
          's3:List*',
          's3:PutObject',
          's3:PutObjectLegalHold',
          's3:PutObjectRetention',
          's3:PutObjectTagging',
          's3:PutObjectVersionTagging',
        ],
        // Note that these bucket names correspond to our self-managed replication buckets, we are
        //  opting-out of CDK managing those for us.
        resources: [`arn:aws:s3:::pipe-artifacts-${pipelineAccount}-*`],
      }),
      
      // Grants the pipeline AssumeRole permissions into the various CDK-generated roles, for
      // cross-account / cross-region deploys etc.
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: ['sts:AssumeRole'],
        resources: targetAccounts.map(targetAccount => {
          // Here we are granting permissions to assume some roles that are generated by CDK
          // for use in pipelines.  We need to wild-card them so that the policy doesn't get
          // too large for IAM.  The roles will begin with a string like `cacheadmin-pipeline`,
          // unless the name of the pipeline is too long; CDK will truncate the prefix portion
          // after 25 chars, and the suffix varies per role, so we just use the first 25 chars.
          // NOTE: your pipeline names must all be lower case!
          const pipelineRoleNamePrefix = `${pipelineName}-pipeline`
            .substr(0, 25);
          return `arn:aws:iam::${targetAccount}:role/${pipelineRoleNamePrefix}*`;
        }),
      }),
    ],
  });

  const pipeline = new codepipeline.Pipeline(
    this,
    `pipeline-${props.pipelineName}`,
    {
      pipelineName: props.pipelineName,
      crossAccountKeys: true,
      reuseCrossRegionSupportStacks: true,

      // passing the role via `withoutPolicyUpdates` prevents CDK from trying to manually manage the policy.  See:
      //
      // https://github.com/aws/aws-cdk/issues/19835
      // https://github.com/aws/aws-cdk/issues/16244#issuecomment-906738231
      // https://docs.aws.amazon.com/cdk/api/v1/docs/aws-iam-readme.html#opting-out-of-automatic-permissions-management
      role: pipelineRole.withoutPolicyUpdates(),

      crossRegionReplicationBuckets: getCrossRegionReplicationBuckets(
        this,
        supportStackInfoForAllRegions
      ),
    }
  );

  // We need to enforce a `DependsOn` from the Pipeline to the Policy,
  // otherwise CFN will try to create them at the same time, and the Pipeline
  // will fail due to missing permissions.  However, if we just call the
  // main CDK `addDependency` function, it will add our policy as a dependency
  // to every resource that the pipeline creates, which will cause circular
  // dependency failures.  We need to use the "escape hatch" to access the CfnResources
  // directly and then add the dependency that way, so that it only gets added to the
  // Pipeline and no other resources.
  const pipelineCfnResource = pipeline.node.defaultChild as CfnResource;
  pipelineCfnResource.addDependsOn(
    pipelinePolicy.node.defaultChild as CfnResource
  );

[rix0rrr]

Can you give me a sense of how many stacks we’re talking about?

[cprice404]

Our biggest pipeline right now has 4 stacks, all of which are deployed to 4 different accounts (so, 16 stack deploys). We have about 5 actions per deploy (includes generating ChangeSet, and manual approval). So that's about 80 actions. In the generated policy, the AssumeRole policy statement contains one resource for each action. When we tipped the scales and started hitting the IAM policy size limit there were 97 resources in that policy statement.

[rix0rrr]

Best solution I can come up with here is to start reusing the same action role for multiple actions.

[cprice404]

Wouldn't it also be possible to split the policy into multiple separate policies, and attach multiple policies to the role?

[rix0rrr]

The limit is about the final policy document on the Role. Those fragments will all add up to the same total.

[cprice404]

Ah. Then perhaps providing an opt-in flag that would switch the AssumeRole policies to use a wildcard that will match all of the Action roles?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.