feat(cli): bundle dependencies

[iliapolo]

Use esbuild via a custom new tool to bundle CLI dependencies and release a package with no runtime dependencies.

More details as to reasoning and implementation here.

Note

This PR has some implications on programmatic usage of the CLI.

Namely, deep imports like so:

import { PluginHost } from 'aws-cdk/lib/plugin'

Will no longer be available. These imports are considered private and should not have been used in the first place.
Instead, switch to:

import { PluginHost } from 'aws-cdk'

If your import isn't available from the top-level, it means that export is actually private, and should be avoided.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[gitpod-io]

[RomainMuller]

As discussed yesterday, there might be a middle ground here... We don't necessarily need to roll up the whole CLI, we could also only roll up each of the third party dependencies separately (i.e: esbuild the minimatch library to vendor/minimatch.js, etc...).

This is a little more work to get set-up, but would save you from having to deal with esModuleInterop stuff as well as import cycles.

It is also possible to configure esbuild to produce more than one entry point, which can also be used to make import cycles work better (although this is not a silver bullet).

---

Apparently, using import type for "type-only imports" (i.e: when you don't use the type's value, for instantiation, extension, instanceof, etc...) breaks the cycles (the imports/no-cycles rule will not consider statements that are not emitted as JS).

We should also frown on/prohibit the use of instanceof as it is very much a 🦶🏻🔫 in JavaScript. Although our dependency modeling TRIES to guarantee there is only ever ONE version of a given library... Sometimes npm does weird things... Using instanceof almost guarantees it won't work in that scenario, whereas using a dedicated API (i.e: symbol property check or something) could work (so this is a slightly better posture).

[mergify]

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[iliapolo]

@RomainMuller Anything else to add here?

[mergify]

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
• Commit ID: a1c8a5f
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[mergify]

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[stevehodgkiss]

Unfortunately, this disables the use of aws-cdk in a pragmatic way (i.e. aws/aws-cdk-rfcs#300). We rely on invoking CDK programatically from a custom CLI library, so it looks like we'll be stuck on aws-cdk 2.13.0 until a solution is found. EDIT: Actually, this still appears to work, but dependencies of aws-cdk need to be manually pulled in to the custom CLI package (@aws-cdk/region-info, cdk-assets, proxy-agent etc), since they're bundled into aws-cdk/lib/index.js with this change as far as I can tell.

Following on from what @RomainMuller mentioned, is there a middle ground solution available (that doesn't disable programatic usage of aws-cdk)?

[stevehodgkiss]

One approach might be to separate the bundled package (aws-cdk) from a package with the previous npm dependency management (aws-cdk-cli?). The code for aws-cdk would move to aws-cdk-cli, aws-cdk would depend on aws-cdk-cli and become a package to bundle it up with it's dependencies and CLI entrypoint. Normal/direct usage of aws-cdk would use the bundled package with strict control of dependencies, and pragmatic use of CDK could make use of aws-cdk-cli with normal npm dependency management. Thoughts?

Another approach would be to expose a supported programmatic API for CDK deployment functionality: aws/aws-cdk-rfcs#300

While it's still technically possible to use aws-cdk as a library, all of it's (now bundled) dependencies need to be defined on the package that uses it, since they're bundled into the aws-cdk entrypoint now and not defined as package dependencies. That's not ideal.

[mnapoli]

This is a huge BC break, it's very problematic for programmatic usage that was done of the CDK.

Would it be possible at least to export CloudFormationDeployments?

Currently aws-cdk exports the low-level deployStack function, but a lot of the CDK deployment logic is in CloudFormationDeployments.deploy (which calls deployStack).

We use the CDK programmatically in https://github.com/getlift/lift to integrate CDK with Serverless Framework (and we want to continue promoting CDK). I'm sure others have plenty of other use cases, as we can see by the number of upvotes in aws/aws-cdk-rfcs#300:

[Tanemahuta]

The problem seems to be the lack of exposure for the libraries used by the cdk deployer.

But I found workaround.
The following example is working for version 2.17.0.

Please note: you need to install some dependencies:

npm i --save-dev @aws-cdk/cloud-assembly-schema@2.17.0 @aws-cdk/cloudformation-diff@2.17.0 @aws-cdk/cx-api@2.17.0

After that you'll be basically using the same approach, but "reloading" the assembly and

import * as cdk from 'aws-cdk-lib';
import {SdkProvider} from "aws-cdk/lib";
import {CloudFormationDeployments} from "aws-cdk/lib/api/cloudformation-deployments";
import * as cxapi from '@aws-cdk/cx-api';

const app = new cdk.App();
const stack = new Stack(app, "my-stack");

// Synthesize the app
const synthesized = app.synth();

// Reload the synthesized artifact for stack using the cxapi from dependencies
const assembly = new cxapi.CloudAssembly(synthesized.directory);
const stackArtifact = cxapi.CloudFormationStackArtifact.fromManifest(
    assembly, stack.artifactId,
    synthesized.getStackArtifact(stack.artifactId).manifest
) as cxapi.CloudFormationStackArtifact;

// Create a provider using set profile and use it for the CFN deployments service
const sdkProvider = await SdkProvider.withAwsCliCompatibleDefaults({profile: process.env.AWS_PROFILE});
const cfn = new CloudFormationDeployments({sdkProvider: sdkProvider});
// Deploy the stack
const deployResult = await cfn.deployStack({
    stack: stackArtifact, force: true,
});

Hope that helps.

[cgarvis]

@mnapoli sorry for the breaking change. The colors.js incident highlighted an attack vector for our customers that we need to close. With programmatic access being an unsupported use case, we hadn't considered it in our solution. We have a small team focused 100% on community PRs at the moment so the fastest way to unblock Lift would be to create a PR with your suggestions. I'll work from my side to get eyes on it.

For everyone else, messages and upvotes in closed PRs are hard for us to see. Please add your upvote to aws/aws-cdk-rfcs#300 to help us priortize.