feat(core): automatic cross stack, cross region references (under feature flag) #22008

corymhall · 2022-09-12T19:01:03Z

This PR adds the ability to automatically create references in cross-region stacks. You can now do something like

const stack1 = new Stack(app, 'Stack1', { env: { region: 'us-east-1' } });
const cert = new certificatemanager.Certificate(stack1, 'Cert', {...});

const stack2 = new Stack(app, 'Stack2', { env: { region: 'us-east-2' } });
new cloudfront.Distribution(stack2, 'Distro', {
  certificate: cert,
})

The above is a good example of the motivation behind this feature. A CloudFront distribution is a global resource and can be created in a CloudFormation stack in any region. Other resources, like the ACM certificate, that need to be attached to the CloudFront distribution can only be created in us-east-1. Another example is the CloudFront.EdgeFunction where we use a support stack and a custom resource to lookup the value.

To accomplish this, I've added two new constructs ExportsWriter & ExportReader. These constructs create Lambda backed custom resources.

ExportWriter responsibilities

Create/Update SSM parameters in the target region for each export
- Will first check to make sure that the export is not "imported" by the consuming stack. If it is, then it will not update the value. This is to mimic the behavior of CloudFormation stack exports.

ExportReader responsibilities

Tag/Untag parameter indicating whether the parameter has been "imported"
Delete all parameters if the stack is deleted.

I am currently using /cdk/exports/${consumingStackName}/ as the SSM path prefix to create all the exports.

Given the above example, this would create an output in stack1

{
  "ExportsWriteruseast2828FA26B": {                                                                                                                                                                                    
   "Type": "Custom::CrossRegionExportWriter",                                                                                                                                                                          
   "Properties": {                                                                                                                                                                                                     
    "ServiceToken": {                                                                                                                                                                                                  
     "Fn::GetAtt": [                                                                                                                                                                                                   
      "CustomCrossRegionExportWriterCustomResourceProviderHandlerD8786E8A",                                                                                                                                            
      "Arn"                                                                                                                                                                                                            
     ]                                                                                                                                                                                                                 
    },                                                                                                                                                                                                                 
    "Region": "us-east-2",                                                                                                                                                                                             
    "Exports": {                                                                                                                                                                                                       
     "/cdk/exports/East2Stack/East1Stackuseast1CertRefCert5C9FAEC135985652": {                                                                                                                                                                            
      "Ref": "Cert5C9FAEC1"                                                                                                                                                                                            
     }                                                                                                                                                                                                                 
    }                                                                                                                                                                                                                  
   }, 
}

And then an "import" in stack2 which is a dynamic ssm reference.

{
 "Resources": {
  "Distro87EBE6BA": {
   "Type": "AWS::CloudFront::Distribution",
   "Properties": {
    "DistributionConfig": {
     "ViewerCertificate": {
      "AcmCertificateArn": "{{resolve:ssm:/cdk/exports/East2Stack/East1Stackuseast1CertRefCert5C9FAEC135985652}}"
    }
   }
  }
 }
}

Currently this will create a single ExportsWriter per region, but we could potentially update this to just use a single ExportsWriter which can write exports to a list of regions.

Future extensions:

Could be updated to do cross account references as well
Could be used to implement general weak references

All Submissions:

Have you followed the guidelines in our Contributing guide?

Adding new Unconventional Dependencies:

This PR adds new unconventional dependencies following the process described here

New Features

Have you added the new feature to an integration test?
- Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

This PR adds the ability to automatically create references in cross-region stacks. You can now do something like ```ts const stack1 = new Stack(app, 'Stack1', { env: { region: 'us-east-1' } }); const cert = new certificatemanager.Certificate(stack1, 'Cert', {...}); const stack2 = new Stack(app, 'Stack2', { env: { region: 'us-east-2' } }); new cloudfront.Distribution(stack2, 'Distro', { certificate: cert, }) ``` To accomplish this, I've added a new construct `ExportsReader` which creates a Lambda backed custom resource. This custom resource will return all of the CloudFormation exports in a given region. Given the above example, this would create an output in `stack1` ```json { "Outputs": { "ExportsOutputRefCert5C9FAEC110AE5C6A": { "Value": { "Ref": "Cert5C9FAEC1" }, "Export": { "Name": "East1:ExportsOutputRefCert5C9FAEC110AE5C6A" } } } } ``` And then an "import" in `stack2` which references the `ExportReader` ```json { "Resources": { "Distro87EBE6BA": { "Type": "AWS::CloudFront::Distribution", "Properties": { "DistributionConfig": { "ViewerCertificate": { "AcmCertificateArn": { "Fn::GetAtt": [ "ExportsReaderuseast1D746CBDB", "East1:ExportsOutputRefCert5C9FAEC110AE5C6A" ] } } } } } } ``` Currently this will create a single ExportsReader per region, but we could potentially update this to just use a single ExportsReader which can list exports from a list of regions. Future extensions: - Could be updated to do cross account references as well - Could be used to implement weak references

gitpod-io · 2022-09-12T19:01:10Z

corymhall · 2022-09-15T12:44:08Z

packages/@aws-cdk/aws-cloudformation/test/integ.core-cross-region-references.ts

@@ -0,0 +1,65 @@
+import { Queue, IQueue } from '@aws-cdk/aws-sqs';


Put the integration test here since we can't have integration tests in core and there is a precedent for doing it this way.

corymhall · 2022-09-15T12:45:57Z

packages/@aws-cdk/aws-codepipeline-actions/test/integ.pipeline-with-replication.ts

+
+const artifact = new Artifact();
+const pipeline = new Pipeline(stack2, 'Pipeline', {
+  crossRegionReplicationBuckets: {


Created this integration test based off of a common cross region scenario

aws-cdk/packages/@aws-cdk/aws-codepipeline/README.md

Lines 320 to 374 in 1f03e8c

### Creating an encrypted replication bucket

If you're passing a replication bucket created in a different stack,

like this:

```ts

// Passing a replication bucket created in a different stack.

const app = new App();

const replicationStack = new Stack(app, 'ReplicationStack', {

env: {

region: 'us-west-1',

},

});

const key = new kms.Key(replicationStack, 'ReplicationKey');

const replicationBucket = new s3.Bucket(replicationStack, 'ReplicationBucket', {

// like was said above - replication buckets need a set physical name

bucketName: PhysicalName.GENERATE_IF_NEEDED,

encryptionKey: key, // does not work!

});

// later...

new codepipeline.Pipeline(replicationStack, 'Pipeline', {

crossRegionReplicationBuckets: {

'us-west-1': replicationBucket,

},

});

```

When trying to encrypt it

(and note that if any of the cross-region actions happen to be cross-account as well,

the bucket *has to* be encrypted - otherwise the pipeline will fail at runtime),

you cannot use a key directly - KMS keys don't have physical names,

and so you can't reference them across environments.

In this case, you need to use an alias in place of the key when creating the bucket:

```ts

// Passing an encrypted replication bucket created in a different stack.

const app = new App();

const replicationStack = new Stack(app, 'ReplicationStack', {

env: {

region: 'us-west-1',

},

});

const key = new kms.Key(replicationStack, 'ReplicationKey');

const alias = new kms.Alias(replicationStack, 'ReplicationAlias', {

// aliasName is required

aliasName: PhysicalName.GENERATE_IF_NEEDED,

targetKey: key,

});

const replicationBucket = new s3.Bucket(replicationStack, 'ReplicationBucket', {

bucketName: PhysicalName.GENERATE_IF_NEEDED,

encryptionKey: alias,

});

```

packages/@aws-cdk/core/lib/custom-resource-provider/export-reader-provider.ts

packages/@aws-cdk/core/README.md

packages/@aws-cdk/core/lib/custom-resource-provider/export-reader-provider.ts

…ion-provider

github-actions

The Pull Request Linter fails with the following errors:

❌ Features must contain a change to a README file.

PRs must pass status checks before we can provide a meaningful review.

Pull Request updated. Dissmissing previous PRLinter Review.

github-actions

The Pull Request Linter fails with the following errors:

❌ Features must contain a change to a README file.

PRs must pass status checks before we can provide a meaningful review.

Pull Request updated. Dissmissing previous PRLinter Review.

rix0rrr · 2022-10-13T09:10:47Z

packages/@aws-cdk/aws-cloudfront/test/integ.cloudfront-cross-region-cert.ts

+const account = process.env.CDK_INTEG_ACCOUNT ?? process.env.CDK_DEFAULT_ACCOUNT;
+const hostedZoneId = process.env.CDK_INTEG_HOSTED_ZONE_ID ?? process.env.HOSTED_ZONE_ID;
+if (!hostedZoneId) throw new Error('For this test you must provide your own HostedZoneId as an env var "HOSTED_ZONE_ID"');
+const hostedZoneName = process.env.CDK_INTEG_HOSTED_ZONE_NAME ?? process.env.HOSTED_ZONE_NAME;
+if (!hostedZoneName) throw new Error('For this test you must provide your own HostedZoneName as an env var "HOSTED_ZONE_NAME"');
+const domainName = process.env.CDK_INTEG_DOMAIN_NAME ?? process.env.DOMAIN_NAME;
+if (!domainName) throw new Error('For this test you must provide your own Domain Name as an env var "DOMAIN_NAME"');


Oof! Can we not create a zone as part of the test?

You need to have a valid public zone one way or another. We could create one in the test, but in order for it to be valid you would have to get the NS records and update the parent hosted zone.

rix0rrr · 2022-10-13T09:12:26Z

packages/@aws-cdk/core/README.md

+You can enable the feature flag `@aws-cdk/core:enableCrossRegionReferencesUsingCustomResources`
+in order to access resources in a different stack _and_ region. With this feature flag
+enabled it is possible to do something like creating a CloudFront distribution in `us-east-2` and
+an ACM certificate in `us-east-1`.


This should be in the CloudFront README (as well)

packages/@aws-cdk/core/README.md

packages/@aws-cdk/core/lib/custom-resource-provider/cross-region-ssm-reader-handler/index.ts

rix0rrr · 2022-10-13T09:16:55Z

packages/@aws-cdk/core/lib/custom-resource-provider/cross-region-ssm-reader-handler/index.ts

+      case 'Create':
+        console.info('Tagging SSM Parameter imports');
+        await addTags(ssm, imports, keyName);
+        return;


I feel like we need to read the value THROUGH a { Fn::GetAtt } of this resource, to make sure the tagging happens before the value is consumed.

I was thinking that we want to tag the resource after it is consumed. That way if it is not consumed (maybe the stack operation fails) we don't add the tags. Although I guess in that case we would rollback this resource as well which should remove the tags.

Also, if we read the value through this resource then in order for updates to be consumed then this resource needs to be updated. Right now I guess that is not a problem since we do not allow updates to exported values, but if we introduced weak references in the future that might be a problem.

Alth ough I guess in that case we would rollback this resource as well which should remove the tags.

👆 this

If consuming and tagging are not connected, then we might have a window where a parameter is consumed and not tagged, which is dangerous because it might be replaced in that window.

Also, if we read the value through this resource then in order for updates to be consumed then this resource needs to be updated.

We could solve that by sending the SSM values in as Properties -- that way if the value changes, the CR is invalidated.

We could solve that by sending the SSM values in as Properties -- that way if the value changes, the CR is invalidated.

genius!

Ok this is updated.

packages/@aws-cdk/core/lib/custom-resource-provider/cross-region-ssm-reader-handler/index.ts

rix0rrr · 2022-10-13T09:26:20Z

packages/@aws-cdk/core/lib/custom-resource-provider/cross-region-ssm-writer-handler/index.ts

+        await putParameters(ssm, newExports);
+        return;
+      case 'Delete':
+        // consuming stack will delete parameters


It should be safe to delete parameters that are not in use, right? I'm afraid we might leak them otherwise.

We would leak them until the consuming stack is deleted since the consuming stack will delete all of it's parameters (by path prefix). As it is we wouldn't have a list of all parameters that this resource has ever created, just what it had on the last update. We could store the state somewhere (another parameter?).

I think we would be able to delete the parameters from the producing stack if we only ever supported strong references (since the stack deployment will fail if you try to delete an imported parameter), but if we want to one day support weak references we need to "leak" the parameters from the producing stack.

We know which references are strong or weak beforehand, so we can apply different rules for them.

Right now, if the producing stack fails to create for some reason and rolls back (or the consumer fails to create) we never get to deploy the consumer and we will leak.

Ok I reworked this. Since we want to fail the stack operation if we are trying to update/delete exports that are in use, i've moved the deletion to the ExportWriter. The ExportReader no longer deletes parameters, it will only add/remove tags.

- moving feature flag to stack property - renaming some things

rix0rrr

Final change request: turn optInToCrossRegionReferences into crossRegionReferences ?

mergify · 2022-10-31T14:24:23Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2022-10-31T15:53:25Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2022-10-31T17:09:45Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 6197448
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2022-10-31T17:10:06Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

niebloomj · 2022-11-01T17:49:36Z

Where does it say what the feature flag is named?

kiernan · 2022-11-11T01:49:50Z

This sounds great. Could something like this also help us with cross-stage dependencies?

rittneje · 2023-01-19T16:23:39Z

Will first check to make sure that the export is not "imported" by the consuming stack. If it is, then it will not update the value. This is to mimic the behavior of CloudFormation stack exports.

@corymhall Standard CloudFormation exports also cannot be deleted if they are in use. (If you try, the entire stack update/delete is immediately aborted.) However, since your approach models these exports as custom resources, that means they won't be deleted until the end of the deployment. Furthermore, CloudFormation basically ignores failures that occur during the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS phase, meaning that the custom resource (and anything it manages) will just get leaked. Is this analysis correct?

On a related note, I think you may need something like Stack.export_value in order to address the "deadly embrace" issue that used to plague regular exports.

DimitriosKay · 2023-12-13T18:15:11Z

I take it this does not apply if the two cross-region stacks are deployed through separate CDK apps ?

mergify bot added the contribution/core This is a PR that came from AWS. label Sep 12, 2022

aws-cdk-automation requested a review from a team September 12, 2022 19:01

github-actions bot added the p2 label Sep 12, 2022

corymhall added 4 commits September 13, 2022 16:41

fixing build

4ce8290

adding some tests and integration tests

97416cc

fixing integration tests

cfcdd08

hopefully fixing integ tests

7a31868

corymhall marked this pull request as ready for review September 14, 2022 18:00

Merge branch 'main' into corymhall/cross-region-provider

b54bd80

corymhall requested a review from rix0rrr September 14, 2022 18:52

corymhall commented Sep 15, 2022

View reviewed changes

packages/@aws-cdk/core/lib/custom-resource-provider/export-reader-provider.ts Outdated Show resolved Hide resolved

corymhall changed the title ~~feat(core): automatic cross stack, cross region references~~ feat(core): automatic cross stack, cross region references (under feature flag) Sep 15, 2022

corymhall added pr-linter/exempt-readme The PR linter will not require README changes and removed pr-linter/exempt-readme The PR linter will not require README changes labels Sep 15, 2022

updating README

80f4092

rix0rrr reviewed Sep 22, 2022

View reviewed changes

corymhall added 5 commits September 23, 2022 17:45

switching to ssm writer

783cb02

Merge remote-tracking branch 'upstream/main' into corymhall/cross-reg…

104a249

…ion-provider

updating integration tests

49944e6

renaming

a4ac262

updating docs

3f99d97

github-actions bot previously requested changes Sep 23, 2022

View reviewed changes

fixing build

13b831b

github-actions bot previously requested changes Sep 23, 2022

View reviewed changes

corymhall and others added 5 commits October 3, 2022 13:24

fixing version of aws-sdk

f9cbab7

fix test

3a1c86b

fixing formatting in integ test

38b6779

Merge branch 'main' into corymhall/cross-region-provider

2c1a37b

Merge branch 'main' into corymhall/cross-region-provider

500ea0c

rix0rrr requested changes Oct 13, 2022

View reviewed changes

corymhall added 4 commits October 13, 2022 18:10

updates based on review comments

7ffd80b

- moving feature flag to stack property - renaming some things

updating integ-tests and readme

5fe1d7b

updating based on review comments

01f3366

fixing tests

9e1c2dc

corymhall requested a review from rix0rrr October 20, 2022 19:43

updating docs to indicate that this feature is experimental

6b9660d

rix0rrr added the pr/do-not-merge This PR should not be merged at this time. label Oct 31, 2022

rix0rrr approved these changes Oct 31, 2022

View reviewed changes

removing optIn from property name

fdea663

corymhall removed the pr/do-not-merge This PR should not be merged at this time. label Oct 31, 2022

Merge branch 'main' into corymhall/cross-region-provider

3c1f2ff

Merge branch 'main' into corymhall/cross-region-provider

6197448

mergify bot merged commit f1b5497 into aws:main Oct 31, 2022

rv2673 mentioned this pull request Nov 7, 2022

(core): crossRegionReference resources cirmcumvent Aspects application #22820

Open

pahud mentioned this pull request Mar 6, 2023

(core): crossRegionReference only works for first destination region #24464

Closed

mrgrain mentioned this pull request Oct 27, 2023

Weak references aws/aws-cdk-rfcs#82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): automatic cross stack, cross region references (under feature flag) #22008

feat(core): automatic cross stack, cross region references (under feature flag) #22008

corymhall commented Sep 12, 2022 •

edited

gitpod-io bot commented Sep 12, 2022

corymhall Sep 15, 2022

corymhall Sep 15, 2022

github-actions bot left a comment

github-actions bot left a comment

rix0rrr Oct 13, 2022

corymhall Oct 13, 2022

rix0rrr Oct 13, 2022

rix0rrr Oct 13, 2022

corymhall Oct 13, 2022

rix0rrr Oct 14, 2022 •

edited

corymhall Oct 18, 2022

corymhall Oct 20, 2022

rix0rrr Oct 13, 2022

corymhall Oct 13, 2022

corymhall Oct 13, 2022

rix0rrr Oct 14, 2022 •

edited

corymhall Oct 20, 2022

rix0rrr left a comment

mergify bot commented Oct 31, 2022

mergify bot commented Oct 31, 2022

aws-cdk-automation commented Oct 31, 2022

mergify bot commented Oct 31, 2022

niebloomj commented Nov 1, 2022

kiernan commented Nov 11, 2022

rittneje commented Jan 19, 2023 •

edited

DimitriosKay commented Dec 13, 2023

		@@ -0,0 +1,65 @@
		import { Queue, IQueue } from '@aws-cdk/aws-sqs';

	### Creating an encrypted replication bucket

	If you're passing a replication bucket created in a different stack,
	like this:

	```ts
	// Passing a replication bucket created in a different stack.
	const app = new App();
	const replicationStack = new Stack(app, 'ReplicationStack', {
	env: {
	region: 'us-west-1',
	},
	});
	const key = new kms.Key(replicationStack, 'ReplicationKey');
	const replicationBucket = new s3.Bucket(replicationStack, 'ReplicationBucket', {
	// like was said above - replication buckets need a set physical name
	bucketName: PhysicalName.GENERATE_IF_NEEDED,
	encryptionKey: key, // does not work!
	});

	// later...
	new codepipeline.Pipeline(replicationStack, 'Pipeline', {
	crossRegionReplicationBuckets: {
	'us-west-1': replicationBucket,
	},
	});
	```

	When trying to encrypt it
	(and note that if any of the cross-region actions happen to be cross-account as well,
	the bucket has to be encrypted - otherwise the pipeline will fail at runtime),
	you cannot use a key directly - KMS keys don't have physical names,
	and so you can't reference them across environments.

	In this case, you need to use an alias in place of the key when creating the bucket:

	```ts
	// Passing an encrypted replication bucket created in a different stack.
	const app = new App();
	const replicationStack = new Stack(app, 'ReplicationStack', {
	env: {
	region: 'us-west-1',
	},
	});
	const key = new kms.Key(replicationStack, 'ReplicationKey');
	const alias = new kms.Alias(replicationStack, 'ReplicationAlias', {
	// aliasName is required
	aliasName: PhysicalName.GENERATE_IF_NEEDED,
	targetKey: key,
	});
	const replicationBucket = new s3.Bucket(replicationStack, 'ReplicationBucket', {
	bucketName: PhysicalName.GENERATE_IF_NEEDED,
	encryptionKey: alias,
	});
	```

feat(core): automatic cross stack, cross region references (under feature flag) #22008

feat(core): automatic cross stack, cross region references (under feature flag) #22008

Conversation

corymhall commented Sep 12, 2022 • edited

All Submissions:

Adding new Unconventional Dependencies:

New Features

gitpod-io bot commented Sep 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr Oct 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr Oct 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr left a comment

Choose a reason for hiding this comment

mergify bot commented Oct 31, 2022

mergify bot commented Oct 31, 2022

aws-cdk-automation commented Oct 31, 2022

AWS CodeBuild CI Report

mergify bot commented Oct 31, 2022

niebloomj commented Nov 1, 2022

kiernan commented Nov 11, 2022

rittneje commented Jan 19, 2023 • edited

DimitriosKay commented Dec 13, 2023

corymhall commented Sep 12, 2022 •

edited

rix0rrr Oct 14, 2022 •

edited

rix0rrr Oct 14, 2022 •

edited

rittneje commented Jan 19, 2023 •

edited