Unable to load verdaccio-aws-s3-storage plugin on 6-next

[noahjohnhay]

Your Environment

• verdaccio version: 6.0.0-6-next.48 (docker:nightly-master)
• node version [12.x.x, 14.x.x]: 16? (docker:nightly-master)
• package manager: [npm@7, pnpm@6, yarn@2] (docker:nightly-master)
• os: [mac, windows@10, linux] [mac, linux] (docker:nightly-master)
• platform: [npm, docker, helm, other] docker

Describe the bug

Plugin fails to load using either 6-next of latest version of the plugin, using the example docker installation for plugin, works on 5 but is failing on nightly-master

To Reproduce

FROM node:lts-alpine as a builder

WORKDIR /verdaccio/plugins

RUN npm install --global-style --no-bin-links --omit=optional verdaccio-aws-s3-storage@6-next

# Have tried using latest ^ as well

# Final build image
FROM verdaccio/verdaccio:nightly-master

ADD config.yaml /verdaccio/conf/config.yaml

ADD htpasswd /verdaccio/htpasswd

COPY --chown=$VERDACCIO_USER_UID:root --from=builder \
  /verdaccio/plugins/node_modules/verdaccio-aws-s3-storage \
  /verdaccio/plugins/verdaccio-aws-s3-storage

Expected behavior

plugin loading as expected

Configuration File (cat ~/.config/verdaccio/config.yaml)

storage: /verdaccio/storage/data

plugins: /verdaccio/plugins

web:
  enable: true
  title: Verdaccio
  darkMode: true

auth:
  htpasswd:
    file: /verdaccio/htpasswd
    algorithm: bcrypt

store:
  aws-s3-storage:
    bucket: S3_BUCKET

uplinks:
  npmjs:
    url: https://registry.npmjs.org/
    cache: true
    maxage: 30m
  nexus:
    url: REDACTED
    cache: true
    maxage: 30m

packages:
  '@*/*':
    access: $all
    proxy: nexus npmjs
  '**':
    access: $all
    proxy: nexus npmjs

server:
  keepAliveTimeout: 60

middlewares:
  audit:
    enabled: true

log: { type: stdout, format: pretty, level: debug }

Environment information

Debugging output

➜  verdaccio-docker git:(try-6) ✗ pnpm build

> @cvent/verdaccio-docker@0.1.8 build /Users/nhay/code/verdaccio/packages/verdaccio-docker
> run-s build:*


> @cvent/verdaccio-docker@0.1.8 build:docker /Users/nhay/code/verdaccio/packages/verdaccio-docker
> docker-cvent build

[+] Building 5.6s (13/13) FINISHED
 => [internal] load build definition from Dockerfile                                                                                             0.0s
 => => transferring dockerfile: 502B                                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                  0.0s
 => [internal] load metadata for docker.io/verdaccio/verdaccio:nightly-master                                                                    4.7s
 => [internal] load metadata for docker.io/library/node:lts-alpine                                                                               4.8s
 => [internal] load build context                                                                                                                0.0s
 => => transferring context: 5.32kB                                                                                                              0.0s
 => [builder 1/3] FROM docker.io/library/node:lts-alpine@sha256:88d9d8da697877a4a771a40e5cbc10a12c2ad959e82f3b0f36ef35635e17f693                 0.0s
 => CACHED [stage-1 1/4] FROM docker.io/verdaccio/verdaccio:nightly-master@sha256:8ea65206deb9b88c3c992b0e5b4486db7b3e3631c41512c65e78dc101fa56  0.0s
 => CACHED [builder 2/3] WORKDIR /verdaccio/plugins                                                                                              0.0s
 => CACHED [builder 3/3] RUN npm install --global-style --no-bin-links --omit=optional verdaccio-aws-s3-storage@6-next                           0.0s
 => [stage-1 2/4] ADD config.yaml /verdaccio/conf/config.yaml                                                                                    0.0s
 => [stage-1 3/4] ADD htpasswd /verdaccio/htpasswd                                                                                               0.0s
 => [stage-1 4/4] COPY --chown=10001:root --from=builder   /verdaccio/plugins/node_modules/verdaccio-aws-s3-storage   /verdaccio/plugins/verdac  0.3s
 => exporting to image                                                                                                                           0.4s
 => => exporting layers                                                                                                                          0.4s
 => => writing image sha256:8c024a23004dd18525b37606e42861cefb3189f1f702a4be570313b5ab7303ba                                                     0.0s
 => => naming to docker.io/library/verdaccio-docker                                                                                              0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
➜  verdaccio-docker git:(try-6) ✗ docker run -it verdaccio-docker
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
 warn --- Cannot read properties of undefined (reading 'aws-s3-storage') on loading plugins at /verdaccio/plugins for aws-s3-storage
 error--- package not found, try to install verdaccio-aws-s3-storage with a package manager
 info -=- local storage path /verdaccio/storage/data/.verdaccio-db.json
 warn -=- no private database found, recreating new one on /verdaccio/storage/data/.verdaccio-db.json
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
 info --- using htpasswd file: /verdaccio/conf/verdaccio/htpasswd
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
 info --- http address http://0.0.0.0:4873/
 info --- version: 6.0.0-6-next.48
 info --- server started
^C info --- received shutdown signal - closing server gracefully...
 info --- server closed.

Contribute to Verdaccio

• I'm willing to fix this bug 🥇

[noahjohnhay]

Looking at this PR line appears to be where the error it being thrown https://github.com/verdaccio/verdaccio/pull/3370/files#diff-3163017865a47f9bfea6e6e6e4a6cbfbff4a9b0afea35c1365614bd48f7f168aR91

I did try adding

server:
  pluginPrefix: verdaccio

But that didn't appear to have any effect

[noahjohnhay]

Found out how to run with debugging enabled

➜  verdaccio-docker git:(try-6) ✗ docker run --env "DEBUG=verdaccio*" -it verdaccio-docker
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  verdaccio:config:parse parse config file /verdaccio/conf/config.yaml +0ms
  verdaccio:config:parse parsing config file: '/verdaccio/conf/config.yaml' +7ms
  verdaccio:logger logging stdout enabled +0ms
  verdaccio:logger setup logger +4ms
  verdaccio:logger has prettifier? true +0ms
  verdaccio:server start server +0ms
  verdaccio:config config path: /verdaccio/conf/config.yaml +0ms
  verdaccio:server loaded filter plugin +14ms
  verdaccio:logger trace (10) was changed to trace (10) +50ms
  verdaccio:logger trace (10) was changed to trace (10) +10ms
  verdaccio:logger trace (10) was changed to trace (10) +7ms
  verdaccio:storage uplinks available [ 'npmjs', 'nexus' ] +0ms
  verdaccio:server storage init start +31ms
  verdaccio:storage:local local storage created +0ms
  verdaccio:logger trace (10) was changed to trace (10) +15ms
  verdaccio:plugin:loader:async plugin aws-s3-storage +0ms
  verdaccio:plugin:loader:async plugin path /verdaccio/plugins +0ms
  verdaccio:plugin:loader:utils loading plugin /verdaccio/plugins/verdaccio-aws-s3-storage +0ms
  verdaccio:plugin:loader:async plugin is ES6 +1s
  verdaccio:plugin:loader:async is scoped plugin false +5ms
  verdaccio:plugin:loader:async plugin pkg name verdaccio-aws-s3-storage +1ms
  verdaccio:plugin:loader:utils loading plugin verdaccio-aws-s3-storage +980ms
  verdaccio:plugin:loader:utils plugin verdaccio-aws-s3-storage not found +2ms
  verdaccio:plugin:loader:async plugin found 0 +3ms
  verdaccio:storage:local no custom storage found, loading default storage @verdaccio/local-storage +1s
  verdaccio:plugin:local-storage config path '/verdaccio/conf/config.yaml' +0ms
  verdaccio:plugin:local-storage plugin storage path '/verdaccio/storage/data/.verdaccio-db.json' +2ms
  verdaccio:storage:local storage plugin init +3ms
  verdaccio:plugin:local-storage plugin init +1ms
  verdaccio:plugin:local-storage local packages loaded +40ms
  verdaccio:plugin:local-storage sync database started +6ms
  verdaccio:plugin:local-storage creating folder '/verdaccio/storage/data' +1ms
  verdaccio:plugin:local-storage creating folder '/verdaccio/storage/data' created succeed +63ms
  verdaccio:plugin:local-storage sync write succeeded +276ms
  verdaccio:storage:local storage plugin initialized +406ms
  verdaccio:storage local init storage initialized +1s
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
 warn --- Cannot read properties of undefined (reading 'aws-s3-storage') on loading plugins at /verdaccio/plugins for aws-s3-storage
 error--- package not found, try to install verdaccio-aws-s3-storage with a package manager
 info -=- local storage path /verdaccio/storage/data/.verdaccio-db.json
 warn -=- no private database found, recreating new one on /verdaccio/storage/data/.verdaccio-db.json
  verdaccio:config check secret key +2s
  verdaccio:config generate a new key +9ms
  verdaccio:plugin:local-storage sync database started +66ms
  verdaccio:plugin:local-storage creating folder '/verdaccio/storage/data' +2ms
  verdaccio:plugin:local-storage creating folder '/verdaccio/storage/data' created succeed +12ms
  verdaccio:plugin:local-storage sync write succeeded +7ms
  verdaccio:storage local storage secret initialized +67ms
  verdaccio:plugin:loader:async plugin found 0 +480ms
  verdaccio:storage filters available [] +3ms
  verdaccio:server storage init end +2s
  verdaccio:logger trace (10) was changed to trace (10) +2s
  verdaccio:plugin:loader:async plugin htpasswd +23ms
  verdaccio:plugin:loader:async plugin path /verdaccio/plugins +1ms
  verdaccio:plugin:loader:utils loading plugin /verdaccio/plugins/verdaccio-htpasswd +515ms
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
  verdaccio:plugin:loader:utils plugin /verdaccio/plugins/verdaccio-htpasswd not found +49ms
  verdaccio:plugin:loader:async is scoped plugin false +59ms
  verdaccio:plugin:loader:async plugin pkg name verdaccio-htpasswd +1ms
  verdaccio:plugin:loader:utils loading plugin verdaccio-htpasswd +1ms
  verdaccio:plugin:loader:async plugin is ES6 +30ms
  verdaccio:plugin:htpasswd password hash algorithm: bcrypt +0ms
  verdaccio:plugin:htpasswd file: /verdaccio/htpasswd +0ms
  verdaccio:plugin:htpasswd config path: /verdaccio/conf/config.yaml +0ms
  verdaccio:plugin:htpasswd htpasswd path: /verdaccio/conf/verdaccio/htpasswd +2ms
  verdaccio:plugin:loader:async plugin found 1 +5ms
  verdaccio:auth auth plugins found 1 +0ms
  verdaccio:plugin:loader:async plugin audit +32ms
  verdaccio:plugin:loader:async plugin path /verdaccio/plugins +1ms
  verdaccio:plugin:loader:utils loading plugin /verdaccio/plugins/verdaccio-audit +70ms
  verdaccio:plugin:loader:utils plugin /verdaccio/plugins/verdaccio-audit not found +1ms
  verdaccio:plugin:loader:async is scoped plugin false +4ms
  verdaccio:plugin:loader:async plugin pkg name verdaccio-audit +0ms
  verdaccio:plugin:loader:utils loading plugin verdaccio-audit +2ms
  verdaccio:plugin:loader:async plugin is ES6 +3ms
  verdaccio:plugin:loader:async plugin found 1 +1ms
  verdaccio:auth jwt middleware +55ms
 info --- using htpasswd file: /verdaccio/conf/verdaccio/htpasswd
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
  verdaccio:web:render static path '/opt/verdaccio/packages/plugins/ui-theme/static' +0ms
  verdaccio:web:api:package initialized package web api +0ms
  verdaccio:web:api:sidebar initialized sidebar web api +0ms
  verdaccio:web:api:readme initialized readme web api +0ms
  verdaccio:node-api http enabled +0ms
 info --- http address http://0.0.0.0:4873/
 info --- version: 6.0.0-6-next.48
 info --- server started

[noahjohnhay]

verdaccio:plugin:loader:async plugin aws-s3-storage +0ms
verdaccio:plugin:loader:async plugin path /verdaccio/plugins +0ms
verdaccio:plugin:loader:utils loading plugin /verdaccio/plugins/verdaccio-aws-s3-storage +0ms
verdaccio:plugin:loader:async plugin is ES6 +1s
verdaccio:plugin:loader:async is scoped plugin false +5ms
verdaccio:plugin:loader:async plugin pkg name verdaccio-aws-s3-storage +1ms
verdaccio:plugin:loader:utils loading plugin verdaccio-aws-s3-storage +980ms
verdaccio:plugin:loader:utils plugin verdaccio-aws-s3-storage not found +2ms
verdaccio:plugin:loader:async plugin found 0 +3ms

this snippet looks to be https://github.com/verdaccio/verdaccio/pull/3370/files#diff-3163017865a47f9bfea6e6e6e4a6cbfbff4a9b0afea35c1365614bd48f7f168aR71-R120

where it's trying to load the plugin twice, once by its full path and then again just by name?

I updated the Dockerfile to also copy it in the default node_modules directory

FROM node:lts-alpine as builder

WORKDIR /verdaccio/plugins

RUN npm install --global-style --no-bin-links --omit=optional verdaccio-aws-s3-storage@6-next

# Final build image
FROM verdaccio/verdaccio:nightly-master

ADD config.yaml /verdaccio/conf/config.yaml

ADD htpasswd /verdaccio/htpasswd

COPY --chown=$VERDACCIO_USER_UID:root --from=builder \
  /verdaccio/plugins/node_modules/verdaccio-aws-s3-storage \
  /verdaccio/plugins/verdaccio-aws-s3-storage

COPY --chown=$VERDACCIO_USER_UID:root --from=builder \
  /verdaccio/plugins/node_modules/verdaccio-aws-s3-storage \
  /opt/verdaccio/node_modules/verdaccio-aws-s3-storage

This seems to get past the second tryLoad call but now I'm hitting a new exception

➜  verdaccio-docker git:(try-6) ✗ docker run --env "DEBUG=verdaccio*" -it verdaccio-docker
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  verdaccio:config:parse parse config file /verdaccio/conf/config.yaml +0ms
  verdaccio:config:parse parsing config file: '/verdaccio/conf/config.yaml' +7ms
  verdaccio:logger logging stdout enabled +0ms
  verdaccio:logger setup logger +4ms
  verdaccio:logger has prettifier? true +0ms
  verdaccio:server start server +0ms
  verdaccio:config config path: /verdaccio/conf/config.yaml +0ms
  verdaccio:server loaded filter plugin +14ms
  verdaccio:logger trace (10) was changed to trace (10) +49ms
  verdaccio:logger trace (10) was changed to trace (10) +8ms
  verdaccio:logger trace (10) was changed to trace (10) +6ms
  verdaccio:storage uplinks available [ 'npmjs', 'nexus' ] +0ms
  verdaccio:server storage init start +31ms
  verdaccio:storage:local local storage created +0ms
  verdaccio:logger trace (10) was changed to trace (10) +17ms
  verdaccio:plugin:loader:async plugin aws-s3-storage +0ms
  verdaccio:plugin:loader:async plugin path /verdaccio/plugins +1ms
  verdaccio:plugin:loader:utils loading plugin /verdaccio/plugins/verdaccio-aws-s3-storage +0ms
  verdaccio:plugin:loader:async plugin is ES6 +968ms
  verdaccio:plugin:loader:async is scoped plugin false +5ms
  verdaccio:plugin:loader:async plugin pkg name verdaccio-aws-s3-storage +0ms
  verdaccio:plugin:loader:utils loading plugin verdaccio-aws-s3-storage +944ms
  verdaccio:plugin:loader:async plugin is ES6 +995ms
uncaught exception, please report (https://github.com/verdaccio/verdaccio/issues) this:
Error: TypeError: Cannot read properties of undefined (reading 'aws-s3-storage')
    at startServer (/opt/verdaccio/packages/server/express/build/server.js:154:11)
    at async /opt/verdaccio/packages/node-api/build/server.js:155:13
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
 warn --- Cannot read properties of undefined (reading 'aws-s3-storage') on loading plugins at /verdaccio/plugins for aws-s3-storage
 error--- storage has failed: undefined

[juanpicado]

Hi, I haven't read the whole comments, just one note:

6.0.0-6-next.48 (docker:nightly-master)

This is not ready for plugins with docker yet, those changes are quiet recent, the current aws-s3-storage is not ready for v6 #2218 and also documentation is not ready yet which is something working on #3424 lately

but thanks let me know I'll keep this thread alive and eventually all will works just fine. Keep on mind the nightly-master still very fresh and breaking changes arrives with no warning.

[noahjohnhay]

Thank you so much for the quick reply! I suspected that to be the case but wanted to share just in case and see if I may be able to help. I was just playing around with 6 as it has some great new features we're looking for, but absolutely understand its actively being worked on and is volatile. Thank you again for all your hard work on this project!

[juanpicado]

verdaccio:plugin:loader:async plugin aws-s3-storage +0ms
verdaccio:plugin:loader:async plugin path /verdaccio/plugins +0ms
verdaccio:plugin:loader:utils loading plugin /verdaccio/plugins/verdaccio-aws-s3-storage +0ms
verdaccio:plugin:loader:async plugin is ES6 +1s
verdaccio:plugin:loader:async is scoped plugin false +5ms
verdaccio:plugin:loader:async plugin pkg name verdaccio-aws-s3-storage +1ms
verdaccio:plugin:loader:utils loading plugin verdaccio-aws-s3-storage +980ms
verdaccio:plugin:loader:utils plugin verdaccio-aws-s3-storage not found +2ms
verdaccio:plugin:loader:async plugin found 0 +3ms

this snippet looks to be https://github.com/verdaccio/verdaccio/pull/3370/files#diff-3163017865a47f9bfea6e6e6e4a6cbfbff4a9b0afea35c1365614bd48f7f168aR71-R120

where it's trying to load the plugin twice, once by its full path and then again just by name?

I updated the Dockerfile to also copy it in the default node_modules directory

FROM node:lts-alpine as builder

WORKDIR /verdaccio/plugins

RUN npm install --global-style --no-bin-links --omit=optional verdaccio-aws-s3-storage@6-next

# Final build image
FROM verdaccio/verdaccio:nightly-master

ADD config.yaml /verdaccio/conf/config.yaml

ADD htpasswd /verdaccio/htpasswd

COPY --chown=$VERDACCIO_USER_UID:root --from=builder \
  /verdaccio/plugins/node_modules/verdaccio-aws-s3-storage \
  /verdaccio/plugins/verdaccio-aws-s3-storage

COPY --chown=$VERDACCIO_USER_UID:root --from=builder \
  /verdaccio/plugins/node_modules/verdaccio-aws-s3-storage \
  /opt/verdaccio/node_modules/verdaccio-aws-s3-storage

This seems to get past the second tryLoad call but now I'm hitting a new exception

➜  verdaccio-docker git:(try-6) ✗ docker run --env "DEBUG=verdaccio*" -it verdaccio-docker
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  verdaccio:config:parse parse config file /verdaccio/conf/config.yaml +0ms
  verdaccio:config:parse parsing config file: '/verdaccio/conf/config.yaml' +7ms
  verdaccio:logger logging stdout enabled +0ms
  verdaccio:logger setup logger +4ms
  verdaccio:logger has prettifier? true +0ms
  verdaccio:server start server +0ms
  verdaccio:config config path: /verdaccio/conf/config.yaml +0ms
  verdaccio:server loaded filter plugin +14ms
  verdaccio:logger trace (10) was changed to trace (10) +49ms
  verdaccio:logger trace (10) was changed to trace (10) +8ms
  verdaccio:logger trace (10) was changed to trace (10) +6ms
  verdaccio:storage uplinks available [ 'npmjs', 'nexus' ] +0ms
  verdaccio:server storage init start +31ms
  verdaccio:storage:local local storage created +0ms
  verdaccio:logger trace (10) was changed to trace (10) +17ms
  verdaccio:plugin:loader:async plugin aws-s3-storage +0ms
  verdaccio:plugin:loader:async plugin path /verdaccio/plugins +1ms
  verdaccio:plugin:loader:utils loading plugin /verdaccio/plugins/verdaccio-aws-s3-storage +0ms
  verdaccio:plugin:loader:async plugin is ES6 +968ms
  verdaccio:plugin:loader:async is scoped plugin false +5ms
  verdaccio:plugin:loader:async plugin pkg name verdaccio-aws-s3-storage +0ms
  verdaccio:plugin:loader:utils loading plugin verdaccio-aws-s3-storage +944ms
  verdaccio:plugin:loader:async plugin is ES6 +995ms
uncaught exception, please report (https://github.com/verdaccio/verdaccio/issues) this:
Error: TypeError: Cannot read properties of undefined (reading 'aws-s3-storage')
    at startServer (/opt/verdaccio/packages/server/express/build/server.js:154:11)
    at async /opt/verdaccio/packages/node-api/build/server.js:155:13
 debug--- plugins folder defined, loading plugins from /verdaccio/plugins
 warn --- Cannot read properties of undefined (reading 'aws-s3-storage') on loading plugins at /verdaccio/plugins for aws-s3-storage
 error--- storage has failed: undefined

Here is an example of how the API would look like

https://verdaccio.org/docs/next/api/core/interfaces/pluginUtils.IPackageStorage#savepackage

Eventually, this would be linked to a guide to upgrading plugins, etc, just a small start to expose internals easily. Feedback is welcome.