lambda: corrupt zip archive asset produced when using node v15.6

[machielg]

Update from the CDK team

When using local assets with the lambda module using the Code.fromAsset() API, deployment of new assets (during cdk deploy) will fail with the error - "Uploaded file must be a non-empty zip"

We have confirmed this issue manifests when using node engine v15.6. The root cause is being tracked by node - nodejs/node#37027. For the time being, use a node engine < 15.6.

If you have encountered this issue, you need to do three things to resolve this -

1. Downgrade to a node version < 15.6. You can see what you're current node version is by running node --version.
2. Delete the cached asset in the cdk.out/ directory. You could safely delete the entire directory, and the correct assets and templates will be re-generated during the next run of cdk synth or cdk deploy.
3. Delete the previously uploaded asset from the staging bucket. The S3 key of the asset will be prefixed with asset/ and have the hash of the asset. The hash of the asset can be found in your application's CloudFormation template.
   The staging bucket can be obtained by running the following command -

 aws cloudformation describe-stack-resources --stack-name CDKToolkit --logical-resource-id StagingBucket  --query 'StackResources[].PhysicalResourceId'

---

Original issue filed below

When doing

const fillFn = new lambda.Function(this, "Zip2GzipSQSFill", {
    code: lambda.Code.fromAsset("lib/python"),
    runtime: lambda.Runtime.PYTHON_3_7,
    handler: "mylambda.handler"
    }
);

my file is here (relative to root of the CDK project):
lib/python/mylambda.py

its contents:

def handler(event, context):
    print("HELLO AWS")

I get the error:

Uploaded file must be a non-empty zip (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: fcfaf553-70d3-40b8-85d2-a15f6c3bcef0; Proxy: null)

Reproduction Steps

run cdk deploy

What did you expect to happen?

the Lambda is created in AWS

What actually happened?

An zip file is uploaded to the CDK staging bucket, there is a zip file there with my python file, but that file has no contents

Environment

• CDK CLI Version : 1.85.0 or 1.83.0
• Framework Version: 1.83.0
• Node.js Version: 15.6.0
• OS : MacOS
• Language (Version): TypeScript (4.1.3)

Other

I'm suspecting it's a nodejs/library problem, in that some library is producing this invalid zip file but I have no evidence of this.

---

This is 🐛 Bug Report

[joel-aws]

@machielg -- could you update the title, maybe to "Uploaded Zips of Functions are Empty".

I'm running into the same issue and want to ensure it's visible.

[joel-aws]

Running into the same issue now with aws_s3_deployment on 1.84.0 and 1.85.0:

//EDIT:

I can see the asset was bundled locally (e.g., asset.90b...), however, when browsing to the s3://cdk-hnb659fds-assets-*-*/assets/ prefix, I can see that the zip corresponding to asset.90b has the correct files in it, but they are entirely empty.

OS: Catalina

[hoegertn]

I had the same issue today with a customer using the serverless framework. So it might be something outside CDK, also.

[joel-aws]

I had the same issue today with a customer using the serverless framework. So it might be something outside CDK, also.

I think you're right... Although I just tried it with us-east-1 and us-west-1 to no avail; so I don't think it's with AWS infra either. Curious where the issue is, then.

[buddhabuddy]

Very similar issue, zip downloaded shows CRC errors but files are present and 7zip can recover them.

Error is "Uploaded file must be a non-empty zip"

[DrCake]

I didn't have issues with deploying the aws_s3_deployment function itself on 1.83, but did find that the files uploaded to S3 were all empty. Updating to 1.85 I now get the error in #12536 (comment)

[hoegertn]

What versions of Node do y'all use? As a wild guess, I think it might be a node 15 issue.

[joel-aws]

What versions of Node do y'all use? As a wild guess, I think it might be a node 15 issue.

v15.6.0

[machielg]

I updated to node 15.6.0 just yesterday when also this bug started so it’s a likely suspect. Downgrading to node 14 didn’t seem to help though. I’m planning to run it in a docker container to isolate the library responsible.

[DrCake]

What versions of Node do y'all use? As a wild guess, I think it might be a node 15 issue.

I’m having the same issue on Node 12

[machielg]

I tested with several node versions, I can't get it to work with any version (10, 14, 15).

It's clear that zipping of my asset dir with the python code isn't triggered, the zip file with the code isn't created.

I just updated all packages and it seems to work again.

[joel-aws]

I just updated all packages and it seems to work again.

Which packages did you update and using what commands?

[machielg]

I just updated all packages and it seems to work again.

Which packages did you update and using what commands?

I got an update for @types/node to version 14.14.21 which was released very recently.

[danilobuerger]

I had the same issue after upgrading to 15.6.0. Downgrading (and deleting cdk.out/.cache and deleting the zip files on the s3 bucket) worked!

[machielg]

I had the same issue after upgrading to 15.6.0. Downgrading (and deleting cdk.out/.cache and deleting the zip files on the s3 bucket) worked!

Perhaps you can list your relevant package versions?

[danilobuerger]

@machielg Which package version?

[danilobuerger]

@machielg They have nothing to do with it... as I wrote downgrading node itself and deleting the caches is enough.

[machielg]

@danilobuerger I removed the cache and s3 assets but that didn’t work for me so I’m trying to find out what fixed it

[danilobuerger]

@machielg you probably did not clear the caches.

[danilobuerger]

@machielg please read above, I mentioned it in my first comment here....

[therealsebo]

I had the same issue after upgrading to 15.6.0. Downgrading (and deleting cdk.out/.cache and deleting the zip files on the s3 bucket) worked!

Thx @danilobuerger ! Downgrading to Node@14, deleting the cdk dir and Remove Zip from S3 worked for me

[danilobuerger]

@therealsebo Awesome! It seems to me that everything including 15.5.1 is working.

[PierreKiwi]

I am also facing this issue.
I managed to make it work by clearing the asset in S3 and deleting cdk.out (not sure what .cache is) but it s failing again.

[machielg]

I am also facing this issue.
I managed to make it work by clearing the asset in S3 and deleting cdk.out (not sure what .cache is) but it s failing again.

it was working for me yesterday, but this morning it failed again. The zip file created by CDK contains all the files, but they are all empty. The issue seems to be coming from the archiver lib:
archiverjs/node-archiver#491

[machielg]

Downgrading to Node v15.5.1 and doing a proper clean seems to fix the problem.

[PierreKiwi]

Thanks @machielg. I downgraded to Node v14.15.4 and to node-archiver 5.1.0. So far it is looking OK now.

[cmckni3]

Ran into this issue today with node v12.20.0 and CDK version 1.84.0. Downgraded to CDK version 1.74.0 to get it working again.

[zxkane]

I met the error Could not unzip uploaded file. Please check your file, then try to upload again. for updating a Lambda layer after bumping CDK from 1.91.0 to 1.95.1. The node version is v12.21.0.

Looks like the fix still has problem in case updating the existing stack. The stack updating succeeded after using above script to clean all assets in cdktoolkit bucket.

[hugomallet]

Same issue on Mac 11.2.3 with Node 15.14.0 and cdk 1.97.0

Sometimes I have another error :
Error: ENOENT: no such file or directory, open 'node_modules/.bin/semver'

[bgshacklett]

My currently working configuration is Node v14.16.1 (LTS) and CDK version 1.100.0.

As others have stated, clearing the staging bucket of affected assets is required to fix this problem. After becoming aware of this issue, running cdk deploy ... -vvv indicated that CDK was detecting a pre-existing asset within the staging bucket. With this asset already uploaded, CDK skipped uploading the current copy (leaving a corrupted zip file in place) and proceeded to attempt to use the corrupted zip file in the following Lambda function deployment.

My issue was resolved after updating to the previously mentioned Node and CDK versions, then removing the assets which were indicated in the verbose output produced by the -vvv flag.

[SamuelCabralCruz]

Encountered the present issue in the past and dev team's response did work, but now I faced the problem again following update of aws-cdk from 1.113.0 to 1.114.0 with node 14.17.3. After downgrading aws-cdk back to 1.113.0 and cleaning the staging bucket, everything worked fine again.

Hope you guys will find a real fix soon. 🙏

[amirfireeye]

This ugly script fixes assets after deploy so you can deploy again. Run it in the root folder of your project so it can find cdk.out. It iterates over assets listed in manifest.json, downloads them from S3, extracts to a temporary folder, re-zips, and finally uploads back to S3.

#!/bin/bash
set -ex

ASSEMBLY_DIRECTORY=`jq -r '.artifacts[] | select(.type == "cdk:cloud-assembly") | .properties.directoryName' cdk.out/manifest.json`
ASSET_MANIFESTS=`jq -r '.artifacts[] | select(.type == "cdk:asset-manifest") | .properties.file' cdk.out/$ASSEMBLY_DIRECTORY/manifest.json`
cd cdk.out/$ASSEMBLY_DIRECTORY
ASSETS=`jq -r '.files[].destinations[] | "s3://" + .bucketName + "/" + .objectKey' $ASSET_MANIFESTS | grep zip`

TMP=`mktemp -d`
cd $TMP

for ASSET in $ASSETS
do
    if aws s3 ls $ASSET; then
        aws s3 cp $ASSET pkg.zip
        mkdir s
        cd s
        if ! unzip ../pkg.zip; then echo bad zip; fi
        rm ../pkg.zip
        zip -r ../pkg.zip * .gitempty
        aws s3 cp ../pkg.zip $ASSET
        cd ..
        rm -rf s
    fi
done

rm -rf $TMP

[timmattison]

I'm seeing this in 1.121.0 today.

[bgshacklett]

@timmattison does anyone else perform deployments who might be using an older version? Are you able to replicate after clearing out the affected assets from the bucket? Is it possible for you to share the log output using the -vvv flag?

[robertd]

@timmattison can you share you package.json and package-lock.json files?

[timmattison]

Of course I fixed this before I went on vacation and now I don't remember what I changed. It was an error on my end but the error message didn't help me diagnose what was wrong. I'll take a look and see what I can find and report back either way.

[timmattison]

I can't find it now. I want to say that this was an issue where there was a require that was referencing a file that didn't exist but in CDK 122 it seems to throw an error early saying that the module couldn't be found. If I ever figure it out I'll post it here but for now I think we will have to ignore it without reproduction steps. :-/

[SamuelCabralCruz]

just tried with node 16.13.0 and cdk 132 and everything seems fine! thx yall!

[rtatton]

For future reference, I'm adding my experience with this issue.

Configuration

• CDK: 1.152.0
• CDK CLI: 2.20.0 (build 738ef49)
• Node: 12.22.12
• npm: 6.14.16
• Gradle: 7.4.2
• Java: 11
• OS: Mac OS X 12.3.1 aarch64

I had a Gradle project for each Lambda handler and a Gradle project for CDK code, i.e.,

root
|
--- cdk-code
|
---
    |
    --- cdk.json
    --- cdk.out
|
--- lambda-handler-1
|
--- lambda-handler-2
|
.
.
.
--- gradlew

Each Lambda handler project used the Java application Gradle plugin, so when bundling locally, I had to use

../gradlew clean check distZip

since BundlingOutput.ARCHIVED requires exactly one archive file in the directory (i.e., no .tar archive). Really, only the distZip is necessary, but cleaning and running checks is a safe option.

So what I would try to run is cdk bootstrap, then cdk synth, and then cdk deploy. Bootstrapping and synthesizing would complete successfully, and I could verify that all handler ZIP archives were properly being picked up by CDK and hashed (using AssetHashType.OUTPUT) to cdk.out. However, when I would run cdk deploy, I noticed that an empty ZIP file would appear in cdk.out.

The key observation that ultimately helped me was the following. Upon running cdk synth, I confirmed that each handler aws:asset:path matched what was in cdk.out. However, upon running cdk deploy, the value of aws:asset:path (located under "Metadata") for all handlers changed to the hash of the empty ZIP archive. From what I could tell, it's as if CDK didn't recognize that the handlers were already bundled, and so cdk deploy "bundled the bundled assets."

To fix the issue, I simply deleted cdk.out after running cdk synth and then ran cdk deploy. I haven't confirmed, but I think running cdk synth --no-staging would save you the trouble of deleting cdk.out.

This is only the second project I have worked with CDK, so I may be missing some critical detail about how to properly use CDK or the CDK CLI that would've avoided this issue, but it didn't seem to be caused by any partcular version of software I was using.