ws node module throws warnings when webpack'd

[shaper]

webpack step fails at ToT after a clean (rm node_modules, npm install, ./tools/sigh) today with error below. It's not immediately clear to me if this is due to a storageNG ramdisk dep, or devtools, or something else, see error detail below. Will seek thoughts from @piotrswigon but FYI @shans

> arcs@0.0.0 build:webpack-tools /usr/local/google/home/wkorman/workspace/arcs
> webpack --config ./src/tools/webpack.config.js

Hash: e278cb663a1b8aa70f39
Version: webpack 4.41.2
Time: 11125ms
Built at: 11/22/2019 11:48:30 AM
        Asset     Size  Chunks             Chunk Names
bundle-cli.js  938 KiB       0  [emitted]  bundle-cli
Entrypoint bundle-cli = bundle-cli.js
 [13] external "crypto" 42 bytes {0} [built]
 [15] external "fs" 42 bytes {0} [built]
 [16] external "path" 42 bytes {0} [built]
 [18] external "util" 42 bytes {0} [built]
 [21] external "stream" 42 bytes {0} [built]
 [22] external "events" 42 bytes {0} [built]
 [28] external "os" 42 bytes {0} [built]
 [32] external "buffer" 42 bytes {0} [built]
 [54] external "assert" 42 bytes {0} [built]
 [56] external "http" 42 bytes {0} [built]
 [57] external "url" 42 bytes {0} [built]
 [96] external "https" 42 bytes {0} [built]
 [97] external "net" 42 bytes {0} [built]
[104] (webpack)/buildin/harmony-module.js 573 bytes {0} [built]
[105] ./build/tools/bundle-cli.js + 173 modules 1.65 MiB {0} [built]
      | ./build/tools/bundle-cli.js 1.34 KiB [built]
      | ./build/tools/bundle.js 3.85 KiB [built]
      | ./build/platform/loader.js 379 bytes [built]
      | ./build/runtime/manifest.js 54.5 KiB [built]
      | ./build/platform/assert-node.js 424 bytes [built]
      | ./build/runtime/flags.js 1.92 KiB [built]
      | ./build/runtime/interface-info.js 17.4 KiB [built]
      | ./build/runtime/schema.js 8.05 KiB [built]
      | ./build/runtime/id.js 4.18 KiB [built]
      | ./build/runtime/storageNG/drivers/driver-factory.js 2.04 KiB [built]
      | ./build/runtime/recipe/comparable.js 1.78 KiB [built]
      | ./build/runtime/storageNG/storage-key.js 804 bytes [built]
      | ./build/runtime/storageNG/store.js 2.88 KiB [built]
      | ./build/runtime/type.js 24 KiB [built]
      | ./build/runtime/recipe/type-checker.js 13.3 KiB [built]
      |     + 159 hidden modules
    + 91 hidden modules

WARNING in ./node_modules/ws/lib/buffer-util.js
Module not found: Error: Can't resolve 'bufferutil' in '/usr/local/google/home/wkorman/workspace/arcs/node_modules/ws/lib'
 @ ./node_modules/ws/lib/buffer-util.js
 @ ./node_modules/ws/lib/sender.js
 @ ./node_modules/ws/index.js
 @ ./build/platform/devtools-channel-node.js
 @ ./build/devtools-connector/devtools-connection.js
 @ ./build/devtools-connector/devtools-arc-inspector.js
 @ ./build/runtime/runtime.js
 @ ./build/runtime/storageNG/drivers/ramdisk.js
 @ ./build/runtime/manifest.js
 @ ./build/tools/bundle.js
 @ ./build/tools/bundle-cli.js

WARNING in ./node_modules/ws/lib/validation.js
Module not found: Error: Can't resolve 'utf-8-validate' in '/usr/local/google/home/wkorman/workspace/arcs/node_modules/ws/lib'
 @ ./node_modules/ws/lib/validation.js
 @ ./node_modules/ws/lib/sender.js
 @ ./node_modules/ws/index.js
 @ ./build/platform/devtools-channel-node.js
 @ ./build/devtools-connector/devtools-connection.js
 @ ./build/devtools-connector/devtools-arc-inspector.js
 @ ./build/runtime/runtime.js
 @ ./build/runtime/storageNG/drivers/ramdisk.js
 @ ./build/runtime/manifest.js
 @ ./build/tools/bundle.js
 @ ./build/tools/bundle-cli.js

[sjmiles]

I had this problem and it was fixed by npm i.

I notice your output is from webpack, so I wonder if the bug report should read webpack failure related to ws.

Is the idea that tools/sigh is suppose to take care of updating npm for you? I'm not clear on what the guarantees are.

[sjmiles]

Ok, so my last comment was wrong.

The error is from webpackTools section of tools/sigh I think that's why I made a bad deduction (I was running other directives).

npm i didn't fix it at all.

[sjmiles]

Ok, these are warnings that ws causes to happen because it supports optional binary packages.

websockets/ws#1220

I'll look into tweaking webpack.config to squelch the warnings.

[piotrswigon]

I'll write down what I understood:

WS library is used to connect to Arcs Explorer when running Arcs in Node (browser uses browser-provided WebSocket implementation). WS can be greatly sped up by including these 2 libraries. However both require a native component that can fail to compile depending on the developer's machine, so they are not required. @sjmiles change to the webpack config makes it explicit that we don't want webpack to include these libraries, which stops it from generating a warning. This is fine as we don't care so much about performance of the Runtime-Explorer on Node right now.

[shaper]

@piotrswigon what caused these errors to arise, when we hadn't seen them previously? Do you know whether an involved module was updated?

[piotrswigon]

I was not able to correlate this warning with the version of webpack or ws. Are we confident these warnings just appeared now?

@shaper how did you discover them? Did you notice them in the command line, or are they blocking travis? I want to understand whether they only appeared now or whether they were there for months / days.

[shaper]

They appeared last week when I was building on my workstation. I don't think these blocked Travis as they were just warnings. I saw them on the command line and hadn't seen them previously, but had been building mainly on my Pixelbook for prior week or two.

[sjmiles]

I thought this was all figured out re: my comment from 3 days ago and my patch.

Why are we still talking about versions?

[piotrswigon]

My understanding is that we know how to fix it, but we don't know why it surfaced only now. I presume @shaper would like to understand the relationship between this issue and the lack of package-lock.json.