Cannot list packages on main screen with verdaccio-aws-s3-storage configured

[sromaniuk-s]

Your Environment

• verdaccio version: 5.18.0
• node version: from official docker image verdaccio/verdaccio:5.18.0
• package manager: npm, from official docker image verdaccio/verdaccio:5.18.0
• os: docker image verdaccio/verdaccio:5.18.0
• platform: kubernetes (helm)

Describe the bug

After installing chart into kubernetes cluster with preprovisioned aws-s3-storage plugin, application cannot list packages, but can reach them via direct URL.

To Reproduce

1. Configuration block from mostly base helm values.
2. Install into cluster with following values,
3. Upload packages into S3 Verdaccio/storage/ (backup from working old version 4.x.x). Couple GBs.
4. Proceed into UI.

Expected behavior
Everything ok on old 4.x.x set up, but...

Screenshots, server logs, package manager log
UI reachable, but 0 packages (changed host due security reasons):

Can reach a package (changed host and package name due security reasons):

Configuration files
Part of very basic values:

 configMap: |
  # necessary (see https://github.com/verdaccio/verdaccio/issues/673)
  storage: /verdaccio/storage/data
  
  store:
    aws-s3-storage:
      keyPrefix: Verdaccio/storage/
      bucket: ${bucket}
      region: ${region}
      accessKeyId: ${access_key_id}
      secretAccessKey: ${secret_access_key}

  auth:
    htpasswd:
      file: /verdaccio/storage/htpasswd
      max_users: 4
      algorithm: bcrypt
      rounds: 10

  uplinks:
    npmjs:
      url: https://registry.npmjs.org/
      max_fails: 100

  packages:
    '@*/*':
      access: $all
      publish: $authenticated
      proxy: npmjs

    '**':
      access: $all
      publish: $authenticated
      proxy: npmjs

  middlewares:
    audit:
      enabled: true

  max_body_size: 100m

  server: 
    keepAliveTimeout: 60

  logs: {type: stdout, format: pretty, level: debug}

Dockerfile

FROM verdaccio/verdaccio:5.18.0

ENV NODE_ENV=production
USER root
RUN npm i -g verdaccio-aws-s3-storage@10.3.0

Environment information
EKS 1.22, Helm 3.8.2, Chart 4.10.3, verdaccio-aws-s3-storage@10.3.0, verdaccio@5.18.0

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)
• $ DEBUG=verdaccio* verdaccio enable extreme verdaccio debug mode (verdaccio api)
• $ npm -ddd prints:
• $ npm config get registry prints:

Plugin init starts (changed bucket name due security reasons):

verdaccio:plugin:loader loading plugin /usr/local/lib/node_modules/verdaccio/build/plugins/aws-s3-storage
verdaccio:plugin:loader plugin /usr/local/lib/node_modules/verdaccio/build/plugins/aws-s3-storage not found
verdaccio:plugin:loader loading plugin verdaccio-aws-s3-storage
verdaccio:plugin:loader plugin aws-s3-storage is an npm package
erdaccio:plugin:loader plugin is ES6
{"level":30,"time":1670257501552,"pid":8,"hostname":"verdaccio-64b7b6bf68-txp4f","content":"aws-s3-storage","prefix":"verdaccio","msg":"plugin successfully loaded: @{prefix}-@{content}"}
{"level":20,"time":1670257501552,"pid":8,"hostname":"verdaccio-64b7b6bf68-txp4f","keyPrefix":"Verdaccio/storage/","bucket":"MY_BUCKET","msg":"s3: [_getData] bucket: @{bucket} prefix: @{keyPrefix}"}
{"level":20,"time":1670257502155,"pid":8,"hostname":"verdaccio-64b7b6bf68-txp4f","keyPrefix":"Verdaccio/storage/","bucket":"MY_BUCKET","msg":"s3: [_sync] bucket: @{bucket} prefix: @{keyPrefix}"}
{"level":20,"time":1670257502285,"pid":8,"hostname":"verdaccio-64b7b6bf68-txp4f","msg":"s3: [_sync] sucess"}

Also found empty response from /-/verdaccio/data/packages:

[sromaniuk-s]

@juanpicado Please, assist if you have time.

[juanpicado]

On my list, but I need some time, I have little time now. But I'll take a look and reply soon as I have an answer. Thanks for the detailed info always appreciated users taking the time for describing as much as they can 💯 .

Questions:

• https://github.com/verdaccio/verdaccio/tree/master/docker-examples/v4/amazon-s3-docker-example This should work but is v4 (old) but I'll try to migrate it to v5, have you tried it out with Localstak (probably not the same but helps a bit)
• Are you migrating from v4 to v5 or is new installation?

I don't see at first glance anything bad in your conf, but let's see.

[sromaniuk-s]

Hello @juanpicado !
Thanks for your response.
Yes, I'm migrating from v4 to v5.

[juanpicado]

Sure, just fyi don't expect much movement on this until the end of the year, but I have the feeling you have permission issues
maybe you can check this in the meantime https://verdaccio.org/blog/2021/04/14/verdaccio-5-migration-guide specially new permissions section for Docker

[sromaniuk-s]

Yeah, main problem of /packages request that didn't register all of the paackages in S3.

[juanpicado]

It would be great if someone invest some time to migrate https://github.com/verdaccio/verdaccio/tree/master/docker-examples/v4/amazon-s3-docker-example to v5

[sromaniuk-s]

@juanpicado I've just woundered could it be a problem of Nginx Ingress Controller. Tested root-path nginx conf, tested 4.13.2 and got same result.

[juanpicado]

There are some debug techniques that could be implemented in your side https://github.com/verdaccio/verdaccio/wiki/Debugging-Verdaccio in order to collect more data . The example could be outdated since is just an example and does not receive maintenance, afk, it did work well for long time, but if someone make it works correctly would be great.

[sromaniuk-s]

@juanpicado Hello buddy! I hope you're well.
Is there any progress on this ?

[juanpicado]

@juanpicado Hello buddy! I hope you're well.
Is there any progress on this ?

I'm not actively working on this, I have no AWS or any way to test to locally with version 5. Maybe someone else wants to ...