[@aws-cdk/aws-lambda-python] dockerExec() asset Bundling does not work in GitHub Actions

[pgollucci]

Example: https://github.com/p6m7g8/p6-namer/runs/1278777487?check_suite_focus=true
See #projen thread about this in CDK.dev slack

Reproduction Steps

What did you expect to happen?

The Lambda layer to be bundled successfully like it is locally

What actually happened?

Something inside dockerExec() failed and no output was made to bundle the asset. I can't tell if this is a docker volume mount
or just an error in docker.

Environment

• **CLI Version :1.68.0+
• **Framework Version:1.68.0+
• Node.js Version: uses: actions/setup-node@v1 from Github Actions
• **OS : ubuntu-latest container on Github Actions as of 10/26-present
• Language (Version): TS CDK code, Lambda is in Python

Other

---

This is 🐛 Bug Report

[eladb]

@jogold any ideas?

[jogold]

I'm not sure that volume mounting works in the Docker-in-Docker of GitHub Actions...

[corymhall]

@pgollucci can you let me know if you still see this issue on the latest version? I think this might be fixed with the recent changes we've done with docker bundling in aws-lambda-python.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[trobert2]

I am facing a similar issue. I see also that #12940 got closed.
Is there any resolution in sight for this problem? Could it be possible to implement a local bundling solution, similar to what esbuild does for the nodeJS lambda equivalent?

EDIT: @corymhall, I'm using aws_cdk.aws_lambda_python_alpha==2.9.0a0 and the issue persists

[corymhall]

@trobert2 Can you provide a reproducible example along with the error that you are receiving? I've tried and been unable to reproduce this using GitHub Actions.

[sarah-story]

@corymhall I'm facing a similar issue with Gitlab CI, which is also using docker-in-docker. I have a PythonFunction defined similar to this:

const lambda = new PythonFunction(this, "Lambda Function", {
      entry: 'lambda/',
      runtime: Runtime.PYTHON_3_8,
      handler: 'handle',
      role: lambdaRole,
});

When I run cdk deploy --stack=${STACK_NAME} in the docker container, it downloads all the requirements from my requirements.txt, bundles the lambda, then claims the bundle is empty. I assume this is because mounting doesn't work correctly in dind.

The exact error message is: Bundling did not produce any output. Check that content is written to /asset-output.

I am using typescript CDK version 2.10.0.

[trobert2]

hey @corymhall, thanks for the reply.

I was having problems with using the python lambda function:

        handler_function = PythonFunction(
            self, "handlingFunction",
            entry=os.path.join(os.path.dirname(
                os.path.realpath(__file__)), "../src/lambda"),
            runtime=Runtime.PYTHON_3_9,
            index="lambda_function.py",
            environment={
                "LOGLEVEL": "DEBUG"
            },
            timeout=Duration.seconds(20),
            memory_size=512,
            tracing=Tracing.ACTIVE
        )

This works fine on my computer.
The GH actions declaration would be as follows:

on:
  push:
    branches: [ main ]

jobs:
  aws_cdk:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: cdk diff
        uses: youyo/aws-cdk-github-actions@v2
        with:
          cdk_subcommand: 'diff'
          cdk_version: '2.9.0'
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: 'eu-central-1'

I opend an issue on the GH actions maintainer repo (youyo/aws-cdk-github-actions#50) but since, I've moved to installing cdk and executing the commands on the runner myself and got the same issue.

Here is the output after the GH action execution:

npm WARN saveError ENOENT: no such file or directory, open '/github/workspace/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/github/workspace/package.json'
npm WARN workspace No description
npm WARN workspace No repository field.
npm WARN workspace No README data
npm WARN workspace No license field.

+ typescript@4.5.5
added 1 package from 1 contributor and audited 1 package in 2.346s
found 0 vulnerabilities

Install aws-cdk 2.9.0
Successful install aws-cdk 2.9.0
Install requirements.txt
Successful install requirements.txt
Run cdk diff  "*"
Sending build context to Docker daemon   55.3kB

Step 1/8 : ARG IMAGE=public.ecr.aws/sam/build-python3.7
Step 2/8 : FROM $IMAGE
latest: Pulling from sam/build-python3.9
1533ccbbac79: Pulling fs layer
f4222fa12a28: Pulling fs layer
ec1a8451d297: Pulling fs layer
e6574f313133: Pulling fs layer
04953c722c8b: Pulling fs layer
8e410e941e98: Pulling fs layer
11d61a0ecb59: Pulling fs layer
46e549523509: Pulling fs layer
f2c9996e16c5: Pulling fs layer
866e7f50ff17: Pulling fs layer
e6574f313133: Waiting
04953c722c8b: Waiting
8e410e941e98: Waiting
11d61a0ecb59: Waiting
46e549523509: Waiting
f2c9996e16c5: Waiting
866e7f50ff17: Waiting
f4222fa12a28: Verifying Checksum
f4222fa12a28: Download complete
ec1a8451d297: Verifying Checksum
ec1a8451d297: Download complete
1533ccbbac79: Verifying Checksum
1533ccbbac79: Download complete
e6574f313133: Verifying Checksum
e6574f313133: Download complete
8e410e941e98: Verifying Checksum
8e410e941e98: Download complete
11d61a0ecb59: Verifying Checksum
11d61a0ecb59: Download complete
46e549523509: Verifying Checksum
46e549523509: Download complete
f2c9996e16c5: Verifying Checksum
f2c9996e16c5: Download complete
866e7f50ff17: Verifying Checksum
866e7f50ff17: Download complete
1533ccbbac79: Pull complete
04953c722c8b: Verifying Checksum
04953c722c8b: Download complete
f4222fa12a28: Pull complete
ec1a8451d297: Pull complete
e6574f313133: Pull complete
04953c722c8b: Pull complete
8e410e941e98: Pull complete
11d61a0ecb59: Pull complete
46e549523509: Pull complete
f2c9996e16c5: Pull complete
866e7f50ff17: Pull complete
Digest: sha256:31621deaf6d0e4097f718f8cdbdb53a9a9e3422b1c32bce811f38d2703c47922
Status: Downloaded newer image for public.ecr.aws/sam/build-python3.9:latest
 ---> 7925e2bf1015
Step 3/8 : ARG PIP_INDEX_URL
 ---> Running in 70d252af0aa7
Removing intermediate container 70d252af0aa7
 ---> 535ab05ca23f
Step 4/8 : ARG PIP_EXTRA_INDEX_URL
 ---> Running in 9b14e9268969
Removing intermediate container 9b14e9268969
 ---> 9f05cd0c8acd
Step 5/8 : ARG HTTPS_PROXY
 ---> Running in 3e02af5c9fd3
Removing intermediate container 3e02af5c9fd3
 ---> e2cf08c62f59
Step 6/8 : RUN pip install --upgrade pip
 ---> Running in e985238ef3cd
Requirement already satisfied: pip in /var/lang/lib/python3.9/site-packages (21.2.4)
Collecting pip
  Downloading pip-22.0.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-22.0.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container e985238ef3cd
 ---> 34b027d4bab2
Step 7/8 : RUN pip install pipenv poetry
 ---> Running in 16c650c9a013
Collecting pipenv
  Downloading pipenv-2022.1.8-py2.py3-none-any.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 55.3 MB/s eta 0:00:00
Collecting poetry
  Downloading poetry-1.1.12-py2.py3-none-any.whl (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.0/175.0 KB 34.5 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=36.2.1 in /var/lang/lib/python3.9/site-packages (from pipenv) (58.1.0)
Requirement already satisfied: pip>=18.0 in /var/lang/lib/python3.9/site-packages (from pipenv) (22.0.2)
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Collecting virtualenv
  Downloading virtualenv-20.13.0-py2.py3-none-any.whl (6.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 74.9 MB/s eta 0:00:00
Collecting certifi
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.2/149.2 KB 32.7 MB/s eta 0:00:00
Collecting pexpect<5.0.0,>=4.7.0
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.0/59.0 KB 15.8 MB/s eta 0:00:00
Collecting poetry-core<1.1.0,>=1.0.7
  Downloading poetry_core-1.0.7-py2.py3-none-any.whl (424 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 424.8/424.8 KB 50.3 MB/s eta 0:00:00
Collecting cleo<0.9.0,>=0.8.1
  Downloading cleo-0.8.1-py2.py3-none-any.whl (21 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.9
  Downloading CacheControl-0.12.10-py2.py3-none-any.whl (20 kB)
Collecting cachy<0.4.0,>=0.3.0
  Downloading cachy-0.3.0-py2.py3-none-any.whl (20 kB)
Collecting html5lib<2.0,>=1.0
  Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.2/112.2 KB 15.7 MB/s eta 0:00:00
Collecting pkginfo<2.0,>=1.4
  Downloading pkginfo-1.8.2-py2.py3-none-any.whl (26 kB)
Collecting requests<3.0,>=2.18
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 KB 15.1 MB/s eta 0:00:00
Collecting shellingham<2.0,>=1.1
  Downloading shellingham-1.4.0-py2.py3-none-any.whl (9.4 kB)
Collecting requests-toolbelt<0.10.0,>=0.9.1
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 KB 14.8 MB/s eta 0:00:00
Collecting clikit<0.7.0,>=0.6.2
  Downloading clikit-0.6.2-py2.py3-none-any.whl (91 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.8/91.8 KB 25.2 MB/s eta 0:00:00
Collecting tomlkit<1.0.0,>=0.7.0
  Downloading tomlkit-0.8.0-py3-none-any.whl (33 kB)
Collecting keyring<22.0.0,>=21.2.0
  Downloading keyring-21.8.0-py3-none-any.whl (32 kB)
Collecting crashtest<0.4.0,>=0.3.0
  Downloading crashtest-0.3.1-py3-none-any.whl (7.0 kB)
Collecting packaging<21.0,>=20.4
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 KB 11.6 MB/s eta 0:00:00
Collecting msgpack>=0.5.2
  Downloading msgpack-1.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 322.2/322.2 KB 49.6 MB/s eta 0:00:00
Collecting lockfile>=0.9
  Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting pylev<2.0,>=1.3
  Downloading pylev-1.4.0-py2.py3-none-any.whl (6.1 kB)
Collecting pastel<0.3.0,>=0.2.0
  Downloading pastel-0.2.1-py2.py3-none-any.whl (6.0 kB)
Collecting six>=1.9
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting SecretStorage>=3.2
  Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting jeepney>=0.4.2
  Downloading jeepney-0.7.1-py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.1/54.1 KB 14.3 MB/s eta 0:00:00
Collecting pyparsing>=2.0.2
  Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 20.7 MB/s eta 0:00:00
Collecting ptyprocess>=0.5
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 KB 14.7 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.7/138.7 KB 24.2 MB/s eta 0:00:00
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.11-py3-none-any.whl (39 kB)
Collecting filelock<4,>=3.2
  Downloading filelock-3.4.2-py3-none-any.whl (9.9 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.4-py2.py3-none-any.whl (461 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 461.2/461.2 KB 51.9 MB/s eta 0:00:00
Collecting platformdirs<3,>=2
  Downloading platformdirs-2.4.1-py3-none-any.whl (14 kB)
Collecting cryptography>=2.0
  Downloading cryptography-36.0.1-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 63.8 MB/s eta 0:00:00
Collecting cffi>=1.12
  Downloading cffi-1.15.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (444 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 444.3/444.3 KB 30.0 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 32.3 MB/s eta 0:00:00
Installing collected packages: webencodings, pylev, ptyprocess, pkginfo, msgpack, lockfile, distlib, certifi, virtualenv-clone, urllib3, tomlkit, six, shellingham, pyparsing, pycparser, poetry-core, platformdirs, pexpect, pastel, jeepney, idna, filelock, crashtest, charset-normalizer, cachy, virtualenv, requests, packaging, html5lib, clikit, cffi, requests-toolbelt, pipenv, cryptography, cleo, cachecontrol, SecretStorage, keyring, poetry
Successfully installed SecretStorage-3.3.1 cachecontrol-0.12.10 cachy-0.3.0 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.11 cleo-0.8.1 clikit-0.6.2 crashtest-0.3.1 cryptography-36.0.1 distlib-0.3.4 filelock-3.4.2 html5lib-1.1 idna-3.3 jeepney-0.7.1 keyring-21.8.0 lockfile-0.12.2 msgpack-1.0.3 packaging-20.9 pastel-0.2.1 pexpect-4.8.0 pipenv-2022.1.8 pkginfo-1.8.2 platformdirs-2.4.1 poetry-1.1.12 poetry-core-1.0.7 ptyprocess-0.7.0 pycparser-2.21 pylev-1.4.0 pyparsing-3.0.7 requests-2.27.1 requests-toolbelt-0.9.1 shellingham-1.4.0 six-1.16.0 tomlkit-0.8.0 urllib3-1.26.8 virtualenv-20.13.0 virtualenv-clone-0.5.7 webencodings-0.5.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 16c650c9a013
 ---> 0bc6d930f4e0
Step 8/8 : CMD [ "python" ]
 ---> Running in f889f79064b1
Removing intermediate container f889f79064b1
 ---> 8ba814a0e72b
Successfully built 8ba814a0e72b
Successfully tagged cdk-5bf8e17a0eeb506ba04d24eda67daebef697aa067d69b13cdc66c501f6b40536:latest
Bundling asset CdkPythonStack/handlingFunction/Code/Stage...
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
jsii.errors.JavaScriptError: 
  Error: Failed to bundle asset CdkPythonStack/handlingFunction/Code/Stage, bundle output is located at /github/workspace/cdk.out/asset.e46407f680efdc5e6249cb50a9f81c5d996affe99241b32077a6b2d6cf11998e-error: Error: docker exited with status 1
      at AssetStaging.bundle (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:672)
      at AssetStaging.stageByBundling (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:4168)
      at stageThisAsset (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:1675)
      at Cache.obtain (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/core/lib/private/cache.js:1:242)
      at new AssetStaging (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:2070)
      at new Asset (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.js:1:620)
      at AssetCode.bind (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/aws-lambda/lib/code.js:1:3503)
      at new Function (/tmp/jsii-kernel-kDPKEj/node_modules/aws-cdk-lib/aws-lambda/lib/function.js:1:2024)
      at new PythonFunction (/tmp/jsii-kernel-kDPKEj/node_modules/@aws-cdk/aws-lambda-python-alpha/lib/function.js:36:9)
      at /tmp/tmps01xffrt/lib/program.js:8432:58

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 8, in <module>
    CdkPythonStack(app, "CdkPythonStack",
  File "/usr/lib/python3.8/site-packages/jsii/_runtime.py", line 86, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/github/workspace/cdk_a_python/cdk_a_python_stack.py", line 39, in __init__
    a_handler_function = PythonFunction(
  File "/usr/lib/python3.8/site-packages/jsii/_runtime.py", line 86, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/aws_cdk/aws_lambda_python_alpha/__init__.py", line 513, in __init__
    jsii.create(self.__class__, self, [scope, id, props])
  File "/usr/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 290, in create
    response = self.provider.create(
  File "/usr/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 344, in create
    return self._process.send(request, CreateResponse)
  File "/usr/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 326, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: Failed to bundle asset CdkPythonStack/handlingFunction/Code/Stage, bundle output is located at /github/workspace/cdk.out/asset.e46407f680efdc5e6249cb50a9f81c5d996affe99241b32077a6b2d6cf11998e-error: Error: docker exited with status 1
 Subprocess exited with error 1
CDK subcommand diff for stack * has failed. See above console output for more details.

Please note that requirements.txt is in the correct path, where the lambda function code is. The requirements.txt file for CDK (root of the repo) also gets read and installed properly as you can see from the command output. I have double checked the workspace to be the root of the code (the same location I execute the cdk command locally from, but with success.) and I can ls and see the lambda requirements file in the correct place and cat the file.

What other options did I try?
This: https://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_lambda/README.html#bundling-asset-code (no success)

I ended up using subprocess to pip3 install -t locally and create a Lambda Layer myself. That worked in the end. I find that to be suboptimal though.

[corymhall]

@trobert2 are you able to create a sample project that can reproduce your issue? I create a simple project and I was able to synth successfully without any errors. You can see the action run here

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[pchernikov-sxm]

Has anybody been able to figure this out? Facing same issue here. I believe this has to do with the fact that our runners are running in docker (with mounting the socket to make them docker capable).

[dragan-test-and-trace]

Same here, and @Pavel-Chernikov I think you may be onto something. I am trying to build from a custom Azure DevOps pipeline agent, that runs inside a docker container and I am also binding a volume to the host's docker-engine Unix socket.

[sbritocorral]

Any updates on this?

[pchernikov-sxm]

This is happening when using DinD because containers don't share the root filesystem.

Separate Filesystem: Each container, including the DinD container, runs in its own isolated environment with its own root filesystem.

Context Mismatch: When you create a volume in the job container, you're actually instructing the Docker engine of the host. But when you're running a command inside a DinD container, the Docker engine being used is the one within that DinD container. These are two different contexts.

[NickDarvey]

@sbritocorral, did you find any workarounds?

[RichardSieg]

@NickDarvey Not quite sure whether this helps in this use case but this is how we bundle our python lambdas on CI

https://github.com/1davidmichael/cdk-local-bundling-example

[NickDarvey]

@RichardSieg, that does help. That's what we're doing now :) thanks