Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add optional peer dependencies (yarn2/pnpm) #2606

Merged
merged 1 commit into from Nov 5, 2020

Conversation

andreialecu
Copy link

@andreialecu andreialecu commented Nov 2, 2020

Description

It appears that aws4 was being required without being listed as a dependency. This results in errors when using strict package managers such as Yarn 2 (berry) and pnpm.

Additionally, it seems a non-standard way of requiring optional modules was being used.

All recent package managers support optional peer dependencies using the peerDependenciesMeta field:

npm as of: npm/cli#224
Yarn 1: https://classic.yarnpkg.com/en/docs/package-json#toc-peerdependenciesmeta
Yarn 2: https://yarnpkg.com/configuration/manifest/#peerDependenciesMeta
pnpm: https://pnpm.js.org/en/package_json#peerdependenciesmeta

What changed?

Removed require_optional dependency and used standard optional peer dependency definitions.

Added aws4 as an optional peer dependency. See:
https://yarnpkg.com/advanced/rulebook/#packages-should-only-ever-require-what-they-formally-list-in-their-dependencies

Are there any files to ignore?

No

@andreialecu
Copy link
Author

For clarity, aws4 is being required here:

aws4 = require('aws4');

@mbroadst
Copy link
Member

mbroadst commented Nov 4, 2020

Hi @andreialecu, thanks for your contribution. Unfortunately, for the 3.6.x series we need to continue to support Node Argon so I don't think this approach will work. I just tried it locally, and npm automatically installs all of the peer dependencies which is explicitly what we're trying not to do. Are you unable to use the driver with yarn, or is it just giving you a warning message? We can consider this approach on the master branch (upcoming 4.0 release), where we have already removed require_optional

@andreialecu
Copy link
Author

Hey @mbroadst, thank you for explaining. I didn't consider the need to support npm 2.x, which was the last version to auto install peer dependencies.

I have simplified the PR to only add the peerDependenciesMeta field to package.json. This will keep Yarn 2 and pnpm users happy because they both auto-imply that something listed in peerDependenciesMeta is a peer of the package, and will install things properly as a result.

See:
https://pnpm.js.org/en/package_json#peerdependenciesmeta-pkg-optional

// express is not specified in the peerDependencies field,
// so pnpm will assume that any version of express is fine
"express": {
"optional": true
}

@andreialecu andreialecu changed the title refactor: remove require_optional and add aws4 fix: add optional peer dependencies (yarn2/pnpm) Nov 5, 2020
@andreialecu
Copy link
Author

Are you unable to use the driver with yarn, or is it just giving you a warning message? We can consider this approach on the master branch (upcoming 4.0 release), where we have already removed require_optional

pnpm and yarn2+ users using the PnP installation strategy will not be able to use MONGODB-AWS authentication without this change because the dependency isn't properly defined.

Yarn 2 users can use an override mechanism called packageExtensions to work around it, but it's not ideal.

See: https://yarnpkg.com/getting-started/migration/#fix-dependencies-with-packageextensions

Copy link
Member

@mbroadst mbroadst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification @andreialecu , this looks good to me. Hopefully we'll all be able to move to v4 shortly, where we won't have this issue!

@mbroadst mbroadst merged commit 186090e into mongodb:3.6 Nov 5, 2020
@gruckion
Copy link

gruckion commented Sep 25, 2022

Hi Team, sounds like you resolved the issue using packageExtensions since the library is not explicit with its peer dependencies. I can't find any examples of how to use node-mongo with Yarn PnP.

I did some up with this.

packageExtensions:
  "mongodb@*":
    dependencies:
      bson-ext: "*"
      kerberos: "*"
      "@mongodb-js/zstd": "*"
      aws4: "*"
      mongodb-core: "*"
      bson: "*"
      node-gyp: "*"
      "mongodb-client-encryption": "*"
    peerDependenciesMeta:
      "@mongodb-js/zstd":
        optional: true
      snappy:
        optional: true
      bson-ext:
        optional: true
      kerberos:
        optional: true

This got me past a few errors but now I just get this crazy mess.

❯ yarn package
➤ YN0000:
➤ YN0000: Packaging seeder-quote for stage stage (eu-west-2)
➤ YN0000:
➤ YN0000: Warning: in ../../.yarn/__virtual__/mongodb-virtual-2f9c25f79b/0/cache/mongodb-npm-4.10.0-99903c2c4a-4847fe69b6.zip/node_modules/mongodb/lib/utils.js 993:38-93
➤ YN0000:   Critical dependency: the request of a dependency is an expression
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 17:28-65
➤ YN0000:   Module not found: Error: Can't resolve './zstd.win32-x64-msvc.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 19:28-70
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-win32-x64-msvc' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 35:28-61
➤ YN0000:   Module not found: Error: Can't resolve './zstd.darwin-x64.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 37:28-66
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-darwin-x64' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 47:28-63
➤ YN0000:   Module not found: Error: Can't resolve './zstd.darwin-arm64.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 65:28-64
➤ YN0000:   Module not found: Error: Can't resolve './zstd.linux-x64-gnu.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 67:28-69
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-linux-x64-gnu' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 77:28-66
➤ YN0000:   Module not found: Error: Can't resolve './zstd.linux-arm64-gnu.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 79:28-71
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-linux-arm64-gnu' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 89:28-70
➤ YN0000:   Module not found: Error: Can't resolve './zstd.linux-arm-gnueabihf.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd/index.js 91:28-75
➤ YN0000:   Module not found: Error: Can't resolve '@mongodb-js/zstd-linux-arm-gnueabihf' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/@mongodb-js-zstd-npm-1.0.0-1751370982-73e82a30be.zip/node_modules/@mongodb-js/zstd'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:569-607
➤ YN0000:   Module not found: Error: Can't resolve './snappy.android-arm64.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:622-662
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-android-arm64' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:811-852
➤ YN0000:   Module not found: Error: Can't resolve './snappy.android-arm-eabi.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:867-910
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-android-arm-eabi' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1159-1198
➤ YN0000:   Module not found: Error: Can't resolve './snappy.win32-x64-msvc.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1213-1254
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-win32-x64-msvc' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1403-1443
➤ YN0000:   Module not found: Error: Can't resolve './snappy.win32-ia32-msvc.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1458-1500
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-win32-ia32-msvc' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1651-1692
➤ YN0000:   Module not found: Error: Can't resolve './snappy.win32-arm64-msvc.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1707-1750
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-win32-arm64-msvc' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:1997-2032
➤ YN0000:   Module not found: Error: Can't resolve './snappy.darwin-x64.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2047-2084
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-darwin-x64' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2231-2268
➤ YN0000:   Module not found: Error: Can't resolve './snappy.darwin-arm64.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2626-2662
➤ YN0000:   Module not found: Error: Can't resolve './snappy.freebsd-x64.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2677-2715
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-freebsd-x64' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2900-2939
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-x64-musl.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:2954-2995
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-x64-musl' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3131-3169
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-x64-gnu.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3184-3224
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-x64-gnu' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3389-3430
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-arm64-musl.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3445-3488
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-arm64-musl' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3626-3666
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-arm64-gnu.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3681-3723
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-arm64-gnu' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3876-3920
➤ YN0000:   Module not found: Error: Can't resolve './snappy.linux-arm-gnueabihf.node' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000:
➤ YN0000:   in ../../.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy/index.js 1:3935-3981
➤ YN0000:   Module not found: Error: Can't resolve '@napi-rs/snappy-linux-arm-gnueabihf' in '/Users/sigex/workdir/backend/lambda/.yarn/cache/snappy-npm-7.1.2-2cf48206bf-7381485fdd.zip/node_modules/snappy'
➤ YN0000: Environment: darwin, node 16.13.2, framework 3.22.0 (local), plugin 6.2.2, SDK 4.3.2
➤ YN0000: Docs:        docs.serverless.com
➤ YN0000: Support:     forum.serverless.com
➤ YN0000: Bugs:        github.com/serverless/serverless/issues
➤ YN0000:
➤ YN0000: Error:
➤ YN0000: Webpack compilation failed:
➤ YN0000:
➤ YN0000: in ../../.yarn/unplugged/@mongodb-js-zstd-darwin-arm64-npm-1.0.0-df74982a20/node_modules/@mongodb-js/zstd-darwin-arm64/zstd.darwin-arm64.node 1:0
➤ YN0000:   Module parse failed: Unexpected character '�' (1:0)
➤ YN0000:   You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
➤ YN0000:   (Source code omitted for this binary file)
➤ YN0000:
➤ YN0000:   in ../../.yarn/unplugged/@napi-rs-snappy-darwin-arm64-npm-7.1.2-605592bf69/node_modules/@napi-rs/snappy-darwin-arm64/snappy.darwin-arm64.node 1:0
➤ YN0000:   Module parse failed: Unexpected character '�' (1:0)
➤ YN0000:   You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
➤ YN0000:   (Source code omitted for this binary file)
➤ YN0000:
➤ YN0000: 1 deprecation found: run 'serverless doctor' for more details
➤ YN0000: Done in 16s 172ms

Notice how it's just trying to load every possible operating systems match for both snappy and mongodb-js/zstd.

How has anyone managed to get this library to work with Yarn PnP? Can we please get some examples documentation.

I submitted an issue here as this repo doesn't have a issue section?

yarnpkg/berry#4904

@mbroadst @andreialecu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants