update major versions for npm deps + remove log option

[lukekarrys]

This PR makes breaking changes to the following workspaces (all except libnpmfund):

@npmcli/arborist
libnpmaccess
libnpmdiff
libnpmexec
libnpmhook
libnpmorg
libnpmpack
libnpmpublish
libnpmsearch
libnpmteam
libnpmversion

arborist should be published first, and then bumped inside libnpmexec and libnpmfund before those are published.

npm@8.5.0 /Users/lukekarrys/projects/npm/cli
├── @npmcli/arborist@4.3.1 -> ./workspaces/arborist
├── libnpmaccess@5.0.1 -> ./workspaces/libnpmaccess
├── libnpmdiff@3.0.0 -> ./workspaces/libnpmdiff
├─┬ libnpmexec@3.0.3 -> ./workspaces/libnpmexec
│ └── @npmcli/arborist@4.3.1 deduped -> ./workspaces/arborist
├─┬ libnpmfund@2.0.2 -> ./workspaces/libnpmfund
│ └── @npmcli/arborist@4.3.1 deduped -> ./workspaces/arborist
├── libnpmhook@7.0.1 -> ./workspaces/libnpmhook
├── libnpmorg@3.0.1 -> ./workspaces/libnpmorg
├── libnpmpack@3.1.0 -> ./workspaces/libnpmpack
├── libnpmpublish@5.0.1 -> ./workspaces/libnpmpublish
├── libnpmsearch@4.0.1 -> ./workspaces/libnpmsearch
├── libnpmteam@3.0.1 -> ./workspaces/libnpmteam
└── libnpmversion@2.0.2 -> ./workspaces/libnpmversion

[npm-robot]

found 20 benchmarks with statistically significant performance improvements

• app-large: clean, lock-only, cache-only, cache-only:peer-deps, modules-only, no-lock, no-cache, no-modules, no-clean, no-clean:audit
• app-medium: clean, lock-only, cache-only, cache-only:peer-deps, modules-only, no-lock, no-cache, no-modules, no-clean, no-clean:audit

timing results

app-large	clean	lock-only	cache-only	cache-only peer-deps	modules-only	no-lock	no-cache	no-modules	no-clean	no-clean audit
npm@8	53.734 ±0.60	32.257 ±0.24	52.811 ±47.53	22.225 ±1.03	3.316 ±0.04	3.308 ±0.01	2.777 ±0.10	12.829 ±0.16	2.677 ±0.02	3.762 ±0.00
#4427	0.487 ±0.01	0.487 ±0.01	0.493 ±0.01	0.489 ±0.00	0.486 ±0.00	0.485 ±0.01	0.483 ±0.00	0.482 ±0.00	0.482 ±0.00	0.487 ±0.00

app-medium	clean	lock-only	cache-only	cache-only peer-deps	modules-only	no-lock	no-cache	no-modules	no-clean	no-clean audit
npm@8	39.059 ±1.40	25.581 ±0.10	14.313 ±0.18	15.488 ±0.02	3.045 ±0.03	2.998 ±0.01	2.689 ±0.03	9.578 ±0.10	2.512 ±0.03	3.434 ±0.02
#4427	0.491 ±0.01	0.507 ±0.01	0.484 ±0.00	0.481 ±0.00	0.488 ±0.01	0.480 ±0.00	0.491 ±0.00	0.491 ±0.02	0.486 ±0.00	0.489 ±0.00

[nlf]

this one is a doozy lol

we looked at this together during our pairing session, i then checked out this branch and kicked the tires briefly and saw no issues. i think this is good to land!

[XhmikosR]

@lukekarrys I don't want to open a new issue, but happy to do it if you think it's worth it.

There are still some duplicate packages that in theory could be deduplicated by updating the deps where needed upstream:

C:\Users\xmr\Desktop\cli>git checkout release-next
Already on 'release-next'
Your branch is up to date with 'origin/release-next'.

C:\Users\xmr\Desktop\cli>find-duplicate-dependencies
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!
This package has the following duplicate dependencies:

lru-cache:
[
  {
    name: 'lru-cache',
    version: '6.0.0',
    from: 'lru-cache@6.0.0',
    path: 'npm/cacache'
  },
  {
    name: 'lru-cache',
    version: '7.3.1',
    from: 'lru-cache@7.3.1',
    path: 'npm/make-fetch-happen'
  }
]

make-fetch-happen:
[
  {
    name: 'make-fetch-happen',
    version: '10.0.3',
    from: 'make-fetch-happen@10.0.3',
    path: 'npm'
  },
  {
    name: 'make-fetch-happen',
    version: '9.1.0',
    from: 'make-fetch-happen@9.1.0',
    path: 'npm/node-gyp'
  }
]

ms:
[
  {
    name: 'ms',
    version: '2.1.2',
    from: 'ms@2.1.2',
    path: 'npm/make-fetch-happen/agentkeepalive/debug'
  },
  {
    name: 'ms',
    version: '2.1.3',
    from: 'ms@2.1.3',
    path: 'npm/make-fetch-happen/agentkeepalive/humanize-ms'
  }
]

http-proxy-agent:
[
  {
    name: 'http-proxy-agent',
    version: '5.0.0',
    from: 'http-proxy-agent@5.0.0',
    path: 'npm/make-fetch-happen'
  },
  {
    name: 'http-proxy-agent',
    version: '4.0.1',
    from: 'http-proxy-agent@4.0.1',
    path: 'npm/node-gyp/make-fetch-happen'
  }
]

@tootallnate/once:
[
  {
    name: '@tootallnate/once',
    version: '2.0.0',
    from: '@tootallnate/once@2.0.0',
    path: 'npm/make-fetch-happen/http-proxy-agent'
  },
  {
    name: '@tootallnate/once',
    version: '1.1.2',
    from: '@tootallnate/once@1.1.2',
    path: 'npm/node-gyp/make-fetch-happen/http-proxy-agent'
  }
]

Should save a few bytes/files if these are deduped :)

EDIT: these deps might be unused or used as devDependencies (needs a closer look):

• chownr
• write-file-atomic

[XhmikosR]

It seems there are plenty of useless files still included too (untested):

Files that could possibly be removed

 node_modules/ansicolors/test/ansicolors.js         |   71 --
 node_modules/ansistyles/test/ansistyles.js         |   15 -
 node_modules/archy/examples/beep.js                |   24 -
 node_modules/archy/examples/multi_line.js          |   25 -
 node_modules/archy/test/beep.js                    |   40 -
 node_modules/archy/test/multi_line.js              |   45 -
 node_modules/archy/test/non_unicode.js             |   40 -
 node_modules/colors/examples/normal-usage.js       |   82 --
 node_modules/colors/examples/safe-string.js        |   79 --
 node_modules/concat-map/example/map.js             |    6 -
 node_modules/concat-map/test/map.js                |   39 -
 node_modules/delegates/test/index.js               |   94 --
 node_modules/dezalgo/test/basic.js                 |   29 -
 node_modules/encoding/test/test.js                 |   49 -
 node_modules/err-code/test/test.js                 |  159 ---
 node_modules/function-bind/test/index.js           |  252 -----
 node_modules/gauge/lib/demo.js                     |   45 -
 node_modules/has/test/index.js                     |   10 -
 node_modules/ip/test/api-test.js                   |  407 -------
 node_modules/is-core-module/test/index.js          |  130 ---
 node_modules/isexe/test/basic.js                   |  221 ----
 node_modules/jsonparse/examples/twitterfeed.js     |   30 -
 node_modules/jsonparse/test/big-token.js           |   24 -
 node_modules/jsonparse/test/boundary.js            |  110 --
 node_modules/jsonparse/test/offset.js              |   67 --
 node_modules/jsonparse/test/primitives.js          |   57 -
 node_modules/jsonparse/test/surrogate.js           |   26 -
 node_modules/jsonparse/test/unvalid.js             |   15 -
 node_modules/jsonparse/test/utf8.js                |   38 -
 node_modules/minipass-sized/test/basic.js          |   83 --
 .../node-gyp/gyp/tools/emacs/testdata/media.gyp    | 1105 -------------------
 .../gyp/tools/emacs/testdata/media.gyp.fontified   | 1107 --------------------
 node_modules/node-gyp/test/common.js               |    3 -
 .../test/fixtures/VS_2017_BuildTools_minimal.txt   |    1 -
 .../test/fixtures/VS_2017_Community_workload.txt   |    1 -
 .../node-gyp/test/fixtures/VS_2017_Express.txt     |    1 -
 .../node-gyp/test/fixtures/VS_2017_Unusable.txt    |    1 -
 .../test/fixtures/VS_2019_BuildTools_minimal.txt   |    1 -
 .../test/fixtures/VS_2019_Community_workload.txt   |    1 -
 .../node-gyp/test/fixtures/VS_2019_Preview.txt     |    1 -
 node_modules/node-gyp/test/fixtures/ca-bundle.crt  |   40 -
 node_modules/node-gyp/test/fixtures/ca.crt         |   21 -
 .../test/fixtures/nodedir/include/node/config.gypi |    6 -
 node_modules/node-gyp/test/fixtures/server.crt     |   21 -
 node_modules/node-gyp/test/fixtures/server.key     |   27 -
 .../node-gyp/test/fixtures/test-charmap.py         |   31 -
 node_modules/node-gyp/test/process-exec-sync.js    |  140 ---
 node_modules/node-gyp/test/simple-proxy.js         |   27 -
 node_modules/node-gyp/test/test-addon.js           |  150 ---
 .../node-gyp/test/test-configure-python.js         |   82 --
 .../node-gyp/test/test-create-config-gypi.js       |   70 --
 node_modules/node-gyp/test/test-download.js        |  207 ----
 .../node-gyp/test/test-find-accessible-sync.js     |   84 --
 .../node-gyp/test/test-find-node-directory.js      |  119 ---
 node_modules/node-gyp/test/test-find-python.js     |  226 ----
 .../node-gyp/test/test-find-visualstudio.js        |  676 ------------
 node_modules/node-gyp/test/test-install.js         |   46 -
 node_modules/node-gyp/test/test-options.js         |   31 -
 node_modules/node-gyp/test/test-process-release.js |  434 --------
 .../npm-normalize-package-bin/test/array.js        |   37 -
 .../npm-normalize-package-bin/test/nobin.js        |   35 -
 .../npm-normalize-package-bin/test/object.js       |  141 ---
 .../npm-normalize-package-bin/test/string.js       |   37 -
 node_modules/promise-all-reject-late/test/index.js |   88 --
 node_modules/promise-retry/test/test.js            |  263 -----
 node_modules/promzard/example/buffer.js            |   12 -
 node_modules/promzard/example/index.js             |   11 -
 .../promzard/example/npm-init/init-input.js        |  191 ----
 node_modules/promzard/example/npm-init/init.js     |   37 -
 .../promzard/example/npm-init/package.json         |   10 -
 node_modules/promzard/example/substack-input.js    |   61 --
 node_modules/promzard/test/basic.js                |   91 --
 node_modules/promzard/test/buffer.js               |   84 --
 node_modules/promzard/test/exports.input           |    5 -
 node_modules/promzard/test/exports.js              |   48 -
 node_modules/promzard/test/fn.input                |   18 -
 node_modules/promzard/test/fn.js                   |   56 -
 node_modules/promzard/test/simple.input            |    8 -
 node_modules/promzard/test/simple.js               |   30 -
 node_modules/promzard/test/validate.input          |    8 -
 node_modules/promzard/test/validate.js             |   20 -
 node_modules/qrcode-terminal/example/basic.js      |    2 -
 node_modules/qrcode-terminal/example/basic.png     |  Bin 46426 -> 0 bytes
 node_modules/qrcode-terminal/example/callback.js   |    4 -
 .../qrcode-terminal/example/small-qrcode.js        |    6 -
 node_modules/qrcode-terminal/test/main.js          |   63 --
 node_modules/retry/example/dns.js                  |   31 -
 node_modules/retry/example/stop.js                 |   40 -
 node_modules/retry/test/common.js                  |   10 -
 .../retry/test/integration/test-forever.js         |   24 -
 .../retry/test/integration/test-retry-operation.js |  258 -----
 .../retry/test/integration/test-retry-wrap.js      |  101 --
 .../retry/test/integration/test-timeouts.js        |   69 --
 node_modules/socks/docs/examples/index.md          |   17 -
 .../docs/examples/javascript/associateExample.md   |   90 --
 .../socks/docs/examples/javascript/bindExample.md  |   83 --
 .../docs/examples/javascript/connectExample.md     |  258 -----
 .../docs/examples/typescript/associateExample.md   |   93 --
 .../socks/docs/examples/typescript/bindExample.md  |   86 --
 .../docs/examples/typescript/connectExample.md     |  265 -----
 node_modules/text-table/example/align.js           |    8 -
 node_modules/text-table/example/center.js          |    8 -
 node_modules/text-table/example/dotalign.js        |    9 -
 node_modules/text-table/example/doubledot.js       |   11 -
 node_modules/text-table/example/table.js           |    6 -
 node_modules/text-table/test/align.js              |   18 -
 node_modules/text-table/test/ansi-colors.js        |   32 -
 node_modules/text-table/test/center.js             |   18 -
 node_modules/text-table/test/dotalign.js           |   20 -
 node_modules/text-table/test/doubledot.js          |   24 -
 node_modules/text-table/test/table.js              |   14 -
 node_modules/unique-filename/test/index.js         |   23 -
 node_modules/unique-slug/test/index.js             |   13 -
 .../validate-npm-package-name/test/index.js        |  109 --
 node_modules/wcwidth/test/index.js                 |   64 --
 115 files changed, 10140 deletions(-)

BTW, nice to see that some work is done already in regards to the file size/number of files! https://packagephobia.com/result?p=npm%406.14.16%2Cnpm%407.24.2%2Cnpm%408.5.1

[wraithgar]

Most of those are because we're waiting on node-gyp to land its next semver major nodejs/node-gyp#2603

[XhmikosR]

Both issues?

BTW when I do npm install I get thousands of modified files on my machine.

[lukekarrys]

It seems there are plenty of useless files still included too (untested)

This is interesting, but I'd be concerned with removing files from any valid dependencies due to the nature of JavaScript, dynamic requires, etc. Not saying we would never do that, but it seems like not the lowest hanging fruit vs the effort/impact.

There are still some duplicate packages that in theory could be deduplicated by updating the deps where needed upstream:

• lru-cache: This will be handled by the next cacache release (which is on our list).
• make-fetch-happen: waiting on node-gyp@9
• ms: These should get deduped somehow. I'd be open to a PR/issue about this.
• http-proxy-agent: waiting on node-gyp@9
• @tootallnate/once: waiting on node-gyp@9

EDIT: these deps might be unused or used as devDependencies (needs a closer look):

I just checked both of those and they are deduped in production deps but not dev/docs. We're not as concerned about those.

[lukekarrys]

BTW when I do npm install I get thousands of modified files on my machine.

@XhmikosR That shouldn't be happening 😄 I just tested on my machine with a brand new clone of the repo and npm i and got nothing modified. It might be worth opening an issue with all the details so we can see what's going on.

[XhmikosR]

Thanks for the replies!

It seems there are plenty of useless files still included too (untested)

This is interesting, but I'd be concerned with removing files from any valid dependencies due to the nature of JavaScript, dynamic requires, etc. Not saying we would never do that, but it seems like not the lowest hanging fruit vs the effort/impact.

There are still some duplicate packages that in theory could be deduplicated by updating the deps where needed upstream:

* `lru-cache`: This will be handled by the next cacache release (which is on our list).

* `make-fetch-happen`: waiting on `node-gyp@9`

* `ms`: These should get deduped somehow. I'd be open to a PR/issue about this.

* `http-proxy-agent`: waiting on `node-gyp@9`

* `@tootallnate/once`: waiting on `node-gyp@9`

EDIT: these deps might be unused or used as devDependencies (needs a closer look):

I just checked both of those and they are deduped in production deps but not dev/docs. We're not as concerned about those.

AFAICT you are already ignoring plenty of dev-related files from production deps hence why I suggested this. I'm not saying these are super important, I agree with you. I'm just saying it should be looked at later, because the gains should be significant :)

@XhmikosR That shouldn't be happening 😄 I just tested on my machine with a brand new clone of the repo and npm i and got nothing modified. It might be worth opening an issue with all the details so we can see what's going on.

I suspect it's something related to Windows and symlinks, so as long as you get a clean repo that's enough for me :)

[lukekarrys]

AFAICT you are already ignoring plenty of dev-related files from production deps hence why I suggested this.

I think it's a good suggestion, but I'm not aware of anywhere we are already doing this. We try to do it in all our own modules via package.json#files so only the minimum necessary files are in the tarball. And we bundle 3rd party deps, but try to not ever change code inside node_modules/ outside of deduping. For most of the files you listed I think the best course of action would be PRs implementing package.json#files.

[XhmikosR]

#3362 and https://github.com/npm/cli/blob/latest/node_modules/.gitignore

I do open upstream PRs when possible and I agree that is the optimal solution, but until then you could also leverage the current solution :)

[lukekarrys]

Thank you for showing me that we have a current solution already. I must've missed that we are doing that! 😄