Problems creating sharp Lambda layer package with custom libvips

[Cruaier]

Possible install-time or require-time problem

• I have read and understood all of the documentation relating to installation.
• I have searched for known bugs relating to this problem in my choice of package manager.

You must confirm both of these before continuing.

Are you using the latest version of sharp?

• I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

If you cannot confirm this, please upgrade to the latest version and try again before opening an issue.

If you are using another package which depends on a version of sharp that is not the latest,
please open an issue against that package instead.

Are you using a supported runtime?

• I am using Node.js with a version that satisfies ^18.17.0 || ^20.3.0 || >=21.0.0
• I am using Deno
• I am using Bun

If you cannot confirm any of these,
please upgrade to the latest version
and try again before opening an issue.

Are you using a supported package manager and installing optional dependencies?

• I am using npm >= 9.6.5 with --include=optional
• I am using yarn >= 3.2.0
• I am using pnpm >= 7.1.0 with --no-optional=false
• I am using Deno
• I am using Bun

If you cannot confirm any of these, please upgrade to the latest version of your chosen package manager
and ensure you are allowing the installation of optional or multi-platform dependencies before opening an issue.

What is the complete error message, including the full stack trace?

{
  "errorType": "Error",
  "errorMessage": "source: bad seek to 293616\nheif: Unsupported feature: Unsupported codec (4.3000)",
  "trace": [
    "Error: source: bad seek to 293616",
    "heif: Unsupported feature: Unsupported codec (4.3000)",
    "    at Sharp._read (/opt/nodejs/node_modules/sharp/lib/output.js:1441:19)",
    "    at Readable.read (node:internal/streams/readable:737:12)",
    "    at resume_ (node:internal/streams/readable:1255:12)",
    "    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)"
  ]
}

What is the complete output of running npm install --verbose --foreground-scripts sharp in an empty directory?

output.log

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

  System:
    OS: Linux 6.8 Amazon Linux 2023
    CPU: (12) x64 Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
    Memory: 12.20 GB / 31.28 GB
    Container: Yes
    Shell: 5.2.15 - /bin/bash
  Binaries:
    Node: 20.9.0 - /var/lang/bin/node
    npm: 10.1.0 - /var/lang/bin/npm

The problem

I am not sure if that is the right place to ask, please let me know if this is more related to libvips or something else.

The error above is happening when running sharp in a Lambda layer to convert a HEIC file into a JPG.

I am compiling libde265, libheif, libwebp, libvips to use with sharp in a custom Docker container. Installing sharp and running it there works well with HEIC files. My goal is to generate a zip file with all dependencies for a Lambda layer.

The below code is installing sharp, copying the libraries and zipping all together for use in a Lambda layer.

mkdir -p ${LAMBDA_TASK_ROOT}/lib
mkdir -p ${LAMBDA_TASK_ROOT}/nodejs
npm install node-addon-api node-gyp
npm --prefix ${LAMBDA_TASK_ROOT}/nodejs install --cpu=x64 --os=linux --verbose --foreground-scripts sharp@${SHARP_VERSION}
rm -rf ${LAMBDA_TASK_ROOT}/nodejs/node_modules/\@img
ldd ${LAMBDA_TASK_ROOT}/nodejs/node_modules/sharp/src/build/Release/sharp-linux-x64.node | sed -nE "s/^[^\/\n]*(\/[^ ]+)(.*)/\1/p" | xargs -I {} sh -c 'if ! [ -f ${LAMBDA_TASK_ROOT}/lib/$(basename {}) ]; then cp {} ${LAMBDA_TASK_ROOT}/lib; fi'
cd ${LAMBDA_TASK_ROOT}
zip -ry /tmp/sharp.zip lib
zip -ru /tmp/sharp.zip nodejs

I assume the issue is related to dynamically linked libraries; though the error message from sharp seems to indicate that libvips has not been compiled with libheif.

Any help or hint into the right direction would be greatly appreciated.

[lovell]

Did you see https://github.com/zoellner/sharp-heic-lambda-layer ?

[Cruaier]

Thanks for the prompt answer. It was just updated using the latest versions...I'll try it out; even though the compile steps are very similar to mine. Will update this ticket when I confirm it works.

[Cruaier]

Unfortunately, the Lambda layer from the other project seems not to work. The error I am getting is below.

{
  "errorType": "Error",
  "errorMessage": "Input buffer contains unsupported image format",
  "trace": [
    "Error: Input buffer contains unsupported image format",
    "    at Sharp.metadata (/opt/nodejs/node_modules/sharp/lib/input.js:486:17)",
    "    at getFileMetadata (/var/task/index.js:1351:40)",
    "    at Runtime.handler (/var/task/index.js:1377:26)",
    "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
  ]
}

I created a ticket there (zoellner/sharp-heic-lambda-layer#30); maybe the issue with building both Lambda layers is related.

[zoellner]

Is there any documentation on which exact files are needed at runtime after the compilation from source is done?

[lovell]

@zoellner You can use ldd to list the shared libraries required by a compiled binary, e.g.:

$ ldd node_modules/sharp/src/build/Release/sharp-linux-x64.node

[Cruaier]

@lovell got it fixed. The issue for me was related to a wrong preset during the libheif compilation. Code is available here: https://github.com/Dobe-Solutions/sharp-heif-lambda-layer