feat(core): add volumes-from option to docker run command for bundling

[webratz]

relates to #8799
follow up to stale #21660

Describe the feature

Ability to add --volumes-from flag when bundling assets with docker.
This enabled people using Docker in Docker to use CDKs bundling functionality, which is currently not possible.

Use Case

CICD systems often run within a docker container already. Many systems mount the /var/run/docker.sock from the host system into the CICD container. When running bundling within such a container it currently breaks, as docker assume the path is from the host system, not within the CICD container.
The options allows to mount the data from any other container. Very often it will be the current one which can be used by using the HOSTNAME environment variable

Proposed Solution

Add optional property to DockerRunOptions and BundlingOptions that would translate into --volumes-from {user provided option}

This change would not reflect in any CloudFormation changes, but only with the docker commands performed when bundling.

Due to using the --volumes-from option, docker will instead of trying to find the path on the host (where it does not exist) try to use the volume that is created by the container C1 that is actually running the CDK. With that it is able to access the files from CDK and can continue the build.

The following plain docker steps show how this works from the docker side, and why we need to adjust the --volumes-from parameter.

docker volume create builds
docker run -v /var/run/docker.sock:/var/run/docker.sock -v builds:/builds -it docker

Now within the just created docker container, run the following commands.

echo "testfile" > /builds/my-share-file.txt
docker run --rm --name DinDContainer --volumes-from="${HOSTNAME}" ubuntu bash -c "ls -hla /builds"

We see that the second container C2 (here DinDContainer) has the same files available as the container C1.

Alternative solutions

I'm not aware of alternative solutions for this docker in docker use cases, besides of not relying on docker at all, which is out of scope for this MR.

---

All Submissions:

• Have you followed the guidelines in our Contributing guide?

Adding new Unconventional Dependencies:

• This PR adds new unconventional dependencies following the process described here

New Features

• Have you added the new feature to an integration test?
  • Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?
    I ran it, but it seems not to have generated something, i might need some guidance there.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[gitpod-io]

[aws-cdk-automation]

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

[webratz]

The failure is sort of expected:

@aws-cdk/aws-lambda-python: docker: Error response from daemon: No such container: 941e5a523af5.

This due to the fact, that the CodeBuild pipeline does not work like the CICD systems that this PR is targeting to enable to build.
Is there a way to either influence the calls to docker in the integration test or similar things?
Edit: found a way in the meanwhile. All tests are passing and are actually testing what they should

Pull request has been modified.

[mrgrain]

Glad you made it working, but I really don't see any point in the test. It literally just tests docker. Sorry about that!

To get this one merged asap, let's remove the integ test and revert customCommands as well.

I'd also think we should create parity of bundling options for all three functions we have (node, go, python). So what I'm pretty much asking you is to revert all python changes here and than create second PR that adds unified support for that as well as some of the other Docker options. Hope that's okay.

Pull request has been modified.

[aws-cdk-automation]

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

[webratz]

this is all stripped down now to the minimum.
As discussed the integration tests and changes in the python module are now all removed.
This makes the PR checker unhappy, but i assume you know what to do it to grant exemptions there.

[mrgrain]

Thanks @webratz , I appreciate you working through this with me. If you're interested, you could look at extending/aligning the function bundling interfaces next.

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

Pull request has been modified.

[mrgrain]

@webratz I'll take it from here. No need to keep updating the branch (in fact makes everything slower).

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: 6c21217
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).