codepipeline: x-account AND x-region deployments are missing stack dependencies

[rix0rrr]

Describe the bug

A cross-region AND cross-account deployment where CodePipeline generates the support stacks for replication buckets and the support stacks for cross-account roles (which is the ideal situation), fails to deploy out of the box.

The cross-account and cross-region resources are created in separate stacks that don't have a dependency between them, but the resources on those stacks have bidirectional dependencies on each other:

• The Bucket and Key policies reference the Action Role
• The Role policy references the Bucket and Key (and deployment resources like CodeDeploy Deployment Groups)

Normally this wouldn't deploy, but we generate hard-coded resource names for the resources involved so that we can formulate policies anyway without having to have bidirectional cross-stack references.

---

The only order in which this deployment works is if we deploy the account stack (with the Role) before the replication stack (with the Bucket and Key), but there is no dependency between these stacks, so a naive cdk deploy may pick the wrong order and fail to deploy properly.

---

This is a tricky area to work in, as people may have built all kinds of elaborate constructions of stacks and resources and referenced resources, and anything that adds more dependencies is at risk of producing a cyclic dependency.

We are piloting a patchy fix right now for ECS CodeDeploy, which should be copied to all deployment actions if successful.

Expected Behavior

A cdk deploy should deploy in the right order.

Current Behavior

The replication bucket stack deploys before the role stack, causing a deployment failure.

Related issues

• Pilot PR: fix(codepipeline): x-env ECS deployment lacking support stack-dependency #24053
• Suggesting a thorough revamp of this mess: codepipeline: x-account AND x-region deployments are a mess #24051

[faridnsh]

I've been running into this issue with ecs deploy action. I tried the pilot PR on my own pipeline and there was no issues with it however. I was looking into the issues to see if anyone reported anything about the above the fix and indeed someone ran into a cyclic dependency issue(#26691). Looking at the project, quiet a lot was done and when dealing with cross stack complicated deployments it's easy to introduce cyclic dependencies.

What about making the KMS policy to actually have the role ARN in the principal but rather inside a condition? For example this is what we generate currently:

{
  "Action": ["kms:Decrypt","kms:DescribeKey"],
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::123456789012:role/multiregionteststack-suppdeployactionrole5fcf660dac560c995da5"
  },
  "Resource": "*"
}

If we do the following then AWS doesn't require the role to exist before the principal is deployed:

{
  "Action": ["kms:Decrypt","kms:DescribeKey"],
  "Effect": "Allow",
  "Principal": {
    "AWS": "123456789012"
  },
  "Resource": "*",
  "Condition": {
    "ArnEquals": {
      "aws:PrincipalArn": "arn:aws:iam::123456789012:role/multiregionteststack-suppdeployactionrole5fcf660dac560c995da5"
     }
  }
}

This way AWS doesn't require the role to exist before the stack is created. Although one could argue that this is less secure as if someone recreates the role, the KMS key allows access to the new role while if it was set in the Principal then AWS wouldn't allow access anymore. Given this security bit in mind, we could deploy the 3 different behaviours under a flag as props of the action, although not sure what would be a good name for it.

I would be happy to make a PR for this, if someone from CDK team would think this would be a good change.