build(salesforcedx-vscode): update node to v16 and switch from npm to yarn

[mohanraj-r]

What does this PR do?

• Update node to Active LTS v16 (EOL 2024-04-30)
• Switch from npm to yarn due to issues with npm 8 and vsce Update to npm v8 which is bundled with Node v16
• Rebased on build: add typescript project references #4055

What issues does this PR fix or reference?

@W-11004197@

Functionality Before

• Node v14 was used with older lockfile format

Functionality After

• Node v16 used with yarn npm v8 and lockfile format v3

Notes

• Run npm run reinstall after merge (remind team in slack)

Summary

• vsce packaging doesn't work with
  • npm v8 / v7 (even on node 14) (Support npm workspaces microsoft/vscode-vsce#580, Add --ignoreDependencies to package command microsoft/vscode-vsce#439, Support relative paths in depedencies microsoft/vscode-vsce#673, Peer/Dev Dependency Resolution Error lerna/lerna#2822)
  • yarn classic / berry (with legacy node_modules)
    • vsce packaging doesn't error but doesn't produce valid working extensions (Created VSIX via vsce package --yarn does not include not-hoisted deep dependency microsoft/vscode-vsce#432)
      • 114 MB vs 476 MB with npm

Solutions

• use our own bundling ([WIP] build: use a bundler (esbuild) along with vsce to package vsix files #4087)
  • esbuild
    • doesn't support dynamic imports
  • rollup
    • has a plugin for dynamic imports that I am trying out right now
• Downgrade to using npm v6 with node v16 (build: update to node v16 with npm v6 for packaging #4092)

PNPM

Pros

• Built-in phantom dep, peer dep checks
• Built-in support for mono-repo
  • Handles dep locks for all packages in the mono-repo
• Almost drop-in replacement for npm commands
  • supports node_modules by default unlike latest yarn
• Closest drop-in replacement for lerna with similar commands (run, exec) and options (concurrency, filtering/scope)
  • bootstraps the mono-repo packages
• Easier to read output / mental model
• Lock file doesn't get updated implicitly on each install
• Faster than npm, yarn

TODO

Packaging vsix fix

• Bundling Extensions | Visual Studio Code Extension API
  • esbuild - An extremely fast JavaScript bundler
• clean-modules - npm
• Update to latest vsce
• Turn on debug log for esbuild and fix reported issues as much as feasible
• When loading the extension and investigate with the developer tools to check errors
• Unable to bundle extension with ESBUILD · Issue #147551 · microsoft/vscode
  • [Feature] Support dynamic imports via plugins · Issue #700 · evanw/esbuild
    • ⭐ esbuild/NETLIFY.md at netlify · netlify/esbuild fork with dynamic import support
  • esbuild/community-plugins: Community plugins for esbuild
  • esbuild dynamic import - npm search
  • i18n dynamic require for default locale
    • i18n dynamic require for non-default locales
  • Messages.importMessagesDirectory(__dirname); from node_modules/@salesforce/core/lib/exported.js
    • esbuild-plugin-fileloc - npm
• bundling doesn't pull in all required dependencies · Issue #1619 · evanw/esbuild
  • --main-fields=module,main
  • esbuild-node-externals - npm
• Cannot resolve dependency 'vscode' · Issue #3034 · parcel-bundler/parcel
• Rollup
  • plugins/packages/dynamic-import-vars at master · rollup/plugins
    • dynamic locale message loading
• refactor:dist/ for rollup, out/ for tsc ?

Build fix

• install yarn in CI and dev workflow
• npm ci in yarn
• unit/integ test failure
  • Error: Cannot find module
  • yarn path issues
  • Unit tests: Path contains invalid characters: C:\Users\circleci\project\packages\salesforcedx-utils-vscode\coverage\lcov-report\out\src\cli\webpack:\salesforcedx-utils-vscode
  • Integ tests: Error: ENOENT: no such file or directory, stat 'c:\Users\circleci\project\packages\system-tests\assets\sfdx-simple\.vscode'
• e2e test vsix files
  • upgrade vsce
  • npm prune -> yarn install
    • yarn install --production --ignore-scripts --prefer-offline
    • yarn install --production=true --ignore-optional
    • yarn install --focus
    • node-prune - npm
    • clean-modules - npm
  • upgrade lerna
  • upgrade yarn
    • Lerna Bootstrap Breaks When Using Yarn Version 2+ · Issue #2910 · lerna/lerna
  • vsce-yarn-patch
  • npm with yarn mixed mode
    • yarn for regular install, npm prune for packaging
      • with/without vsce --yarn
      • npm v8
      • npm v6
  • chore: first try with esbuild to reduce extension size @W-6980161@ - WIP by jag-sfdc · Pull Request #3748 · forcedotcom/salesforcedx-vscode
  • pnpm
    • Spike Background / Motivation
      • A story of how we migrated to pnpm - ‹div›RIOTS
      • The Case for pnpm Over npm or Yarn | by Jacky Efendi | Better Programming
      • NPM vs PNPM vs Yarn | Rush
    • Support pnpm · Issue #421 · microsoft/vscode-vsce
    • The inferred type of "X" cannot be named without a reference to "Y". This is likely not portable. A type annotation is necessary. · Issue #42873 · microsoft/TypeScript
    • Fix missing peer deps

Refactor

• Migrate npm commands
• Update dev docs
• remove lerna bootstrap?
  • fix called twice
• re-eval added @tsignore
• npmrc -> yarnrc
• refactor clean and reinstall script

Document / share

• visual studio code - Where are the VSCode error logs for extensions? - Stack Overflow
• Open Extensions Log Folder

Explore

• yarn install --flat

  • Install all the dependencies, but only allow one version for each package.

• Yarn v3 with node_modules
• yarn install --frozen-lockfile

  • Don’t generate a yarn.lock lockfile and fail if an update is needed.

• yarn audit | Yarn
• yarn upgrade-interactive

[gbockus-sf]

Cool looks good 👍 Any ideas on why the circle build is failing? The error looks familiar but I don't remember the details of what results in that error.

npm WARN config production Use `--omit=dev` instead.
 ERROR  Command failed: npm list --production --parseable --depth=99999 --loglevel=error

Think we need to update the packing script to account for the above warning?

[mohanraj-r]

vsce package doesn't work with npm 8 or 7 (microsoft/vscode-vsce#580, microsoft/vscode-vsce#439, microsoft/vscode-vsce#673, lerna/lerna#2822) even on node 14.
Also tried

• with and without npm workspaces
• updating vsce to latest version (2.x).

lerna ERR! npm run vscode:package stderr:
npm WARN config production Use `--omit=dev` instead.
 ERROR  Command failed: npm list --production --parseable --depth=99999 --loglevel=error
npm ERR! code ELSPROBLEMS
npm ERR! missing: @salesforce/salesforcedx-test-utils-vscode@54.9.0, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: @types/async-lock@0.0.20, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: @types/chai@4.3.0, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: @types/mocha@^5, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: @types/node@12.0.12, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: @types/rimraf@^2.0.2, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: @types/sinon@^2.3.7, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: chai@^4.0.2, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: cross-env@5.2.0, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: mocha-junit-reporter@^1.23.3, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: mocha-multi-reporters@^1.1.7, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: mocha@^5, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: mock-spawn@0.2.6, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: nyc@^13, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: rimraf@2.6.1, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: sinon@^13.0.1, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: typescript@3.8.3, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: vscode-debugadapter-testsupport@1.28.0, required by salesforcedx-apex-debugger@npm:@salesforce/salesforcedx-apex-debugger@54.9.0
npm ERR! missing: @salesforce/ts-sinon@1.3.21, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/chai@4.3.0, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/cross-spawn@6.0.0, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/mkdirp@0.5.2, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/mocha@^5, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/proxyquire@1.3.28, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/shelljs@0.7.9, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/sinon@^2.3.7, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: @types/vscode@1.49.0, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: chai@^4.0.2, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: mocha-junit-reporter@^1.23.3, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: mocha-multi-reporters@^1.1.7, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: mocha@^5, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: mock-spawn@0.2.6, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: nyc@^13, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: proxyquire@2.1.3, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: request-light@0.2.4, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: sinon@^13.0.1, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0
npm ERR! missing: typescript@3.8.3, required by salesforcedx-utils-vscode@npm:@salesforce/salesforcedx-utils-vscode@54.9.0

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2022-04-19T18_36_10_092Z-debug-0.log

[mohanraj-r]

Bundle size / validity

Baseline (vsce + npm prune)

To compare with baseline, the vsix files buit with vsce & npm (from develop@ 6782d42)

 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-debugger/salesforcedx-vscode-apex-debugger-54.11.0.vsix (605 files, 1.24MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-replay-debugger/salesforcedx-vscode-apex-replay-debugger-54.11.0.vsix (4830 files, 6.93MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex/salesforcedx-vscode-apex-54.11.0.vsix (5193 files, 22.06MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-core/salesforcedx-vscode-core-54.11.0.vsix (16070 files, 24.6MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-expanded/salesforcedx-vscode-expanded-54.11.0.vsix (6 files, 12.04KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lightning/salesforcedx-vscode-lightning-54.11.0.vsix (5524 files, 8.6MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lwc/salesforcedx-vscode-lwc-54.11.0.vsix (20187 files, 30.9MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-soql/salesforcedx-vscode-soql-54.11.0.vsix (3231 files, 4.84MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-visualforce/salesforcedx-vscode-visualforce-54.11.0.vsix (757 files, 3.22MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode/salesforcedx-vscode-54.11.0.vsix (7 files, 21.57KB)

Yarn

vsce packaging seems to work with yarn (without any patches) - but the vsix bundle sizes are significantly lower than those produced with npm. Quick smoke test with the vsix files shows errors and extensions not working as intended.

8:scode-apex-debugger/salesforcedx-vscode-apex-debugger-54.10.0.vsix (28 files, 48.31KB)
13:scode-apex-replay-debugger/salesforcedx-vscode-apex-replay-debugger-54.10.0.vsix (52 file
18:scode-apex/salesforcedx-vscode-apex-54.10.0.vsix (109 files, 14.85MB)
22: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-core/salesforcedx-vscode-core-54.10.0.vsix (295 files, 373.02KB)
25: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-expanded/salesforcedx-vscode-expanded-54.10.0.vsix (6 files, 12.03KB)
28: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lightning/salesforcedx-vscode-lightning-54.10.0.vsix (19 files, 55.78KB)
31: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lwc/salesforcedx-vscode-lwc-54.10.0.vsix (131 files, 166.16KB)
34: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-soql/salesforcedx-vscode-soql-54.10.0.vsix (9 files, 402.21KB)
37: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-visualforce/salesforcedx-vscode-visualforce-54.10.0.vsix (22 files, 45.4KB)
40: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode/salesforcedx-vscode-54.10.0.vsix (7 files, 21.48KB)

With esbuild

 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-debugger/salesforcedx-vscode-apex-debugger-54.11.0.vsix (28 files, 61.62KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-replay-debugger/salesforcedx-vscode-apex-replay-debugger-54.11.0.vsix (52 files, 1.23MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex/salesforcedx-vscode-apex-54.11.0.vsix (109 files, 16.15MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-core/salesforcedx-vscode-core-54.11.0.vsix (295 files, 3.69MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-expanded/salesforcedx-vscode-expanded-54.11.0.vsix (6 files, 12.05KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lightning/salesforcedx-vscode-lightning-54.11.0.vsix (19 files, 801.67KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lwc/salesforcedx-vscode-lwc-54.11.0.vsix (131 files, 1.22MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-soql/salesforcedx-vscode-soql-54.11.0.vsix (9 files, 391.29KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-visualforce/salesforcedx-vscode-visualforce-54.11.0.vsix (22 files, 73.99KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode/salesforcedx-vscode-54.11.0.vsix (7 files, 21.58KB)

Update vsce

Using latest vsce version 2.7.0 (or 2.6.1) results in error from package that doesn't need to be packaged - microsoft/vscode-vsce#675

$ lerna run vscode:package --concurrency 1 && node scripts/reformat-with-prettier
lerna notice cli v3.22.1
lerna info Executing command in 10 packages: "yarn run vscode:package"
lerna ERR! yarn run vscode:package exited 1 in 'salesforcedx-vscode-apex-debugger'
lerna ERR! yarn run vscode:package stdout:
$ vsce package --yarn
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

lerna ERR! yarn run vscode:package stderr:
 ERROR  Extension entrypoint(s) missing. Make sure these files exist and aren't ignored by '.vscodeignore':
  extension/out/src.js
error Command failed with exit code 1.

lerna ERR! yarn run vscode:package exited 1 in 'salesforcedx-vscode-apex-debugger'
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.
error Command failed with exit code 1.

Other potential solutions

• npm prune equivalent behavior · Issue #696 · yarnpkg/yarn doesn't help
• upgrading yarn, vsce, lerna or using vsce-yarn-patch individually or in combinations doesn't help

[mohanraj-r]

Broken up into smaller PRs