(core): (asset-staging) Avoid docker bind mounts in asset staging / bundling to allow for more use cases

[webratz]

Describe the feature

Docker based bundling should optionally allow for other options than bind mounts for passing the input and output data for bundling to the container.

Use Case

My overall use case is bundling of (python) lambda function within a CICD environment which is within a container, and only has access to the /var/run/docker.sock mounted in from the host system.

I have worked on the PRs #23330 #23318 and #22829 to resolve this issue, by allowing to mount in the volume from the container running CDK into the bundling container. Sadly the last bit seems not work, due to the bind mounts being dependent on host paths.

https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/core/lib/asset-staging.ts#L436

Proposed Solution

There is different options, which of none I think are a great fit, so I'm happy to hear other ideas.

Option A

Pass environment variables with sourcePath and bundleDir into the container, so the user is able manually make these files available, eg by mounting the outer Volume and providing symlinks.

Option B

Offer an alternative variant where a docker cp is run before to populate the input dir, and again at the end to copy out the data from the output dir.
Docker copy is usually not very fast and performant, but would even work with remote sockets on other hosts.

Option C

Somehow prepend the bind mount path with the correct path to the volume from the outer container

Other Information

The code that creates the bind mount is at https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/core/lib/asset-staging.ts#L435

The directories /asset-input and /asset-output are visible as empty within the bundling container. This comes from the "wrong" bind mount (see below).

Output of docker inspect of the Mounts section of the bundling container. It shows that the volumesFrom mounting of /builds works correctly (check source path), but the paths for the bind mounts are wrong (would need the volume path as a prefix).

 [
            {
                "Type": "bind",
                "Source": "/builds/sre/cdk-demo/ops/cdk/src/resources/jwt-lambda",
                "Destination": "/asset-input",
                "Mode": "delegated,shared",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/builds/sre/cdk-demo/ops/cdk/cdk.out/asset.a3db61addee4a8185740ba792365a233b3bdee504d6b7e126cec64f273cf73e1",
                "Destination": "/asset-output",
                "Mode": "delegated,shared",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "runner-g8tpkuey-project-1490-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70",
                "Source": "/nvme/docker/volumes/runner-g8tpkuey-project-1490-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70/_data",
                "Destination": "/cache",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "runner-g8tpkuey-project-1490-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8",
                "Source": "/nvme/docker/volumes/runner-g8tpkuey-project-1490-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data",
                "Destination": "/builds",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ]

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.55.1

Environment details (OS name and version, etc.)

Amazon Linux 2

[webratz]

ping @jogold and @mrgrain as you seem to have worked on these topics before, and might have a deeper knowledge

[webratz]

related issue #8799

[mrgrain]

Thanks for the detailed write up and linking the related issue. I'm closing this one here, so we can keep track better . You're very welcome to post more details on the other one.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.