Compile with warnings #1220

albandaft · 2017-10-20T10:19:08Z

Hi guys
I have integrated the tool inside my project and I get this warning while compiling:

This issue is coming from the zalmoxisus/remote-redux-devtools#102, but is a WS library issue.
Basically I notice that the BufferUtil.js check if the library is already present to use it, other use it implements it. The problem is that this will cause require to fire an warning.

WARNING in ./node_modules/ws/lib/BufferUtil.js
Module not found: Error: Can't resolve 'bufferutil' in '/Users/alban/projects/frontend-base/node_modules/ws/lib'
 @ ./node_modules/ws/lib/BufferUtil.js 35:21-42
 @ ./node_modules/ws/lib/Receiver.js
 @ ./node_modules/ws/index.js
 @ ./node_modules/socketcluster-client/lib/sctransport.js
 @ ./node_modules/socketcluster-client/lib/scsocket.js
 @ ./node_modules/socketcluster-client/index.js
 @ ./node_modules/remote-redux-devtools/lib/devTools.js
 @ ./node_modules/remote-redux-devtools/lib/index.js
 @ ./src/store/configureStore.js
 @ ./src/index.server.js
 @ multi ./src/index.server.js

WARNING in ./node_modules/ws/lib/Validation.js
Module not found: Error: Can't resolve 'utf-8-validate' in '/Users/alban/projects/frontend-base/node_modules/ws/lib'
 @ ./node_modules/ws/lib/Validation.js 10:22-47
 @ ./node_modules/ws/lib/Receiver.js
 @ ./node_modules/ws/index.js
 @ ./node_modules/socketcluster-client/lib/sctransport.js
 @ ./node_modules/socketcluster-client/lib/scsocket.js
 @ ./node_modules/socketcluster-client/index.js
 @ ./node_modules/remote-redux-devtools/lib/devTools.js
 @ ./node_modules/remote-redux-devtools/lib/index.js
 @ ./src/store/configureStore.js
 @ ./src/index.server.js
 @ multi ./src/index.server.js

Wouldn't it be better to just include these libraries in the package.json or remove totally them from loading?

The text was updated successfully, but these errors were encountered:

lpinca · 2017-10-20T11:00:40Z

This is done on purpose as both bufferutil and utf-8-validate have a native component which can fail to compile. Originally they were included in package.json but they have been removed to prevent installation from failing when compilation was not successful.
Removing them completely is not an option as they greatly improve the library performances.

lpinca · 2017-10-21T05:30:52Z

@albandaft one way to get rid of the warnings is to exclude bufferutil and utf-8-validate when bundling.

In the browser you must use the native WebSocket object, not ws.

lpinca · 2017-10-21T05:40:37Z

To be honest I'm not even sure why you get the warnings as it should use the shimmed version provided by socketcluster-client: https://github.com/SocketCluster/socketcluster-client/blob/87ac534521471e8da6cc95967a7af059601fdd2a/package.json#L30-L32

albandaft · 2017-10-23T08:30:09Z

@lpinca The reason why the warnings appear is simple: To test if bufferutil or utf-8-validate is installed you just call require('bufferutil') which will through a warning if the library is not there.

How to exclude the bufferutil from building?

lpinca · 2017-10-23T08:48:49Z

remote-redux-devtools uses socketcluster-client which in turns uses ws. socketcluster-client uses this shim for the browser when it is bundled so you shouldn't get any warning.

albandaft · 2017-10-23T08:53:27Z

@lpinca I get this warning during the react command line building, not in the browser. It is not a matter of shim, it is simply a matter of require, which is used to test the presence of the library:

From https://github.com/websockets/ws/blob/master/lib/BufferUtil.js#L35

try {
  const bufferUtil = require('bufferutil');

  module.exports = Object.assign({ concat }, bufferUtil.BufferUtil || bufferUtil);
} catch (e) /* istanbul ignore next */ {
...
}

lpinca · 2017-10-23T09:10:42Z

How to exclude the bufferutil from building?

In Browserify you can use the --exclude option or Browserify#exclude(). Other bundlers have similar options/plugins.

I get this warning during the react command line building, not in the browser

Yes but I guess you are creating a bundle for the browser right?

albandaft · 2017-10-23T09:15:26Z

On the browser everything works fine. I need to remove the warnings because eventually a pipeline build process will fail to pass test if the build exit code is not correct.

I am actually running the bundle in memory in watch mode, dev mode.
I will try to workaround to exclude them from the building.

lpinca · 2017-10-23T09:25:57Z

It is not a matter of shim, it is simply a matter of require, which is used to test the presence of the library

try {
  const bufferUtil = require('bufferutil');

  module.exports = Object.assign({ concat }, bufferUtil.BufferUtil || bufferUtil);
} catch (e) /* istanbul ignore next */ {
...
}

That code along with all ws code should be completely ignored and replaced by the bundler with the shim code when instructed to do that.

The browser field is where the module author can hint to the bundler which elements (other modules or source files) need to be replaced when packaging.

albandaft · 2017-10-26T13:16:46Z

None of the above worked.

lpinca · 2017-10-26T13:38:56Z

index.js

'use strict';

module.exports = require('remote-redux-devtools');

webpack.config.js

'use strict';

module.exports = {
  entry: './index.js',
  output: {
    libraryTarget: 'umd',
    filename: 'bundle.js',
    library: 'enhancer'
  }
};

This doesn't generate any warning with webpack@3.8.1.

albandaft · 2017-10-26T14:12:25Z

@lpinca With your example I do not get either the warnings. Here we get those warning in different machines. We will keep trying to understand...

lpinca · 2017-10-26T14:20:35Z

Are you requiring ws in other places or is remote-redux-devtools the only module that uses it via socketcluster-client?

waheed25 · 2017-11-13T11:24:08Z

is there any solution to remove these warnings???
because none of the above works.

syang · 2018-10-02T01:37:29Z

Is the warning negligible? Or should we install bufferutil ?

gosp · 2018-10-15T13:21:57Z

@syang
webpack.config.js :: externals: [/node_modules/, 'bufferutil', 'utf-8-validate'],

for example:

module.exports = {
  entry: { server: path.join(__dirname, 'src', 'script', 'index.ts') },
  resolve: { extensions: ['.js', '.ts'] },
  target: 'node',
  mode: 'none',
  // this makes sure we include node_modules and other 3rd party libraries
  externals: [/node_modules/, 'bufferutil', 'utf-8-validate'],
  output: {
    path: path.join(__dirname, '..', 'dist'),
    filename: 'server.js'
  },
  module: {
    rules: [{ test: /\.ts$/, loader: 'ts-loader' }]
  },
  plugins: []
};

KATT · 2018-10-25T14:11:21Z

Or:

module.exports = {
  // [..]
  externals: {
    bufferutil: 'commonjs bufferutil',
    'utf-8-validate': 'commonjs utf-8-validate',
  },
  // [..]
};

u-bits · 2019-01-28T22:04:36Z

I solved this warning by just changing the target in my webpack.config.js to web
https://webpack.js.org/configuration/target/

WARNING in /Users/moox/Sync/Development/phenomic/node_modules/ws/lib/BufferUtil.js Module not found: Error: Can't resolve 'bufferutil' in '/Users/moox/Sync/Development/phenomic/node_modules/ws/lib' WARNING in /Users/moox/Sync/Development/phenomic/node_modules/ws/lib/Validation.js Module not found: Error: Can't resolve 'utf-8-validate' in '/Users/moox/Sync/Development/phenomic/node_modules/ws/lib' Ref websockets/ws#1220

ozyman42 · 2019-04-27T23:26:18Z

Seems a bit weird that if I want to bundle socket.io-client via webpack then I need to explicitly externalize the bufferutil and utf-8-validate packages in my webpack config because of this flaw in the transitive dependency ws. Either socket.io-client should stop using ws, or ws should find/invent alternatives to bufferutil and utf-8-validate that it would be comfortable using. Those try catch require blocks in lib/buffer-util.js and lib/validation.js seem like an undesirable solution to the problem.

ozyman42 · 2019-04-27T23:55:20Z

Removing them completely is not an option as they greatly improve the library performances.

@lpinca I don't understand why that is the case. If the two packages are listed in package.json under devDependencies, then clients of the ws package will never download bufferutil and utf-8-validate, and thus will never experience the performance increase you speak of.

lpinca · 2019-04-28T05:06:12Z

@alexleung if you want to use bufferutil utf-8-validate (recommended), you have to explicitly add them to your dependencies:

npm install ws bufferutil utf-8-validate

or

npm install ws
npm install --save-optional bufferutil utf-8-validate

ws will use them when available.

ozyman42 · 2019-04-28T17:32:23Z

Why aren't bufferutil and utf-8-validate saved as optionalDependencies in this repo?
I think that would at least make more sense than devDependencies, and wouldn't require a code change besides to package.json, then webpack core could be updated to know not to throw warnings when transitive dependencies' optionalDependencies are not installed, and just auto externalize them.

lpinca · 2019-04-28T17:43:48Z

They are dev dependencies because tests must pass with them. If we move them to optional dependencies they are no longer opt-in.

Out of curiosity, are you making a bundle for the browser or Node.js?

ozyman42 · 2019-04-28T17:46:53Z

I thought the definition of something being optional is that it is opt-in.
I am creating a bundle for Node.js

lpinca · 2019-04-28T17:59:07Z

I thought the definition of something being optional is that it is opt-in.

Not for npm, you have to use the --no-optional flag to not install optional dependencies.

ozyman42 · 2019-04-28T19:13:27Z

npm will try and install those optional deps, but if the install fails for those, it wouldn't fail the overall install, which is what you were originally trying to do.

they have been removed to prevent installation from failing when compilation was not successful.

lpinca · 2019-04-28T19:42:21Z

they have been removed to prevent installation from failing when compilation was not successful.

Yes, exactly. See 49b1109. This was done because install failed even though they were optional dependencies.

Now npm has probably fixed the the issue but I see no good reasons to add them back. What's wrong with externalizing them in webpack config?

ozyman42 · 2019-04-29T00:18:49Z

I see. Failing in optional dependencies should not fail the entire installation, but if it was still failing in this case I can understand why it was placed in devdeps.

Externalizing these in the webpack config is undesirable because the point of a package/library is to abstract away / encapsulate behavior so you don't need to know how it works, you just use it. Needing to externalize these couple of packages because ws is a transitive dependency of many other libraries (some of which end up being bundled), seems to break the encapsulation that a package typically would provide. Worse it breaks this encapsulation all the way up the dependency tree such that no package which depends on ws can hide this from its own clients.

Maybe we could reopen this issue and mark as in-progess until utf-8-validate and bufferutil are replaced, or until fixes are made to those so they don't fail.

lpinca · 2019-04-29T05:20:43Z

Nothing breaks, the warnings are just that and can be safely ignored.

The same happen if you try to bundle a library which uses external dependencies in the same way ws does so in my opinion it's more an "issue" with bundlers than ws.

ozyman42 · 2019-05-01T04:17:26Z

Yes, exactly. See 49b1109. This was done because install failed even though they were optional dependencies.

This sounds like a bug within npm.

etienne-martin · 2019-12-09T16:09:20Z

For me the issue was that I had target: "node" in my webpack.config.js while I was trying to bundle for the web.

Switching to target: "web" removed the warnings.

ozyman42 · 2019-12-14T02:29:48Z

Now npm has probably fixed the the issue but I see no good reasons to add them back [to optionalDependencies]. What's wrong with externalizing them in webpack config?

Nothing breaks, the warnings are just that and can be safely ignored.

In general it's bad practice for a library to cause warnings if there is no actual problem, it clutters the console and wastes everyone's time searching for the root cause of a non-issue. This is good enough reason to add back to optionalDependencies.

ozyman42 · 2019-12-14T02:31:50Z

@etienne-martin I'm surprised that you saw the issue at all since #1626 should have fixed the problem. I'd be cautious of switching target to web, because that could cause greater issues if your bundle really is supposed to target node.

Dahkenangnon · 2020-10-28T08:43:16Z

@alexleung if you want to use bufferutil utf-8-validate (recommended), you have to explicitly add them to your dependencies:
npm install ws bufferutil utf-8-validate
or
npm install ws
npm install --save-optional bufferutil utf-8-validate
ws will use them when available.

Using this, i solve the warning

WARNING in /Users/moox/Sync/Development/phenomic/node_modules/ws/lib/BufferUtil.js Module not found: Error: Can't resolve 'bufferutil' in '/Users/moox/Sync/Development/phenomic/node_modules/ws/lib' WARNING in /Users/moox/Sync/Development/phenomic/node_modules/ws/lib/Validation.js Module not found: Error: Can't resolve 'utf-8-validate' in '/Users/moox/Sync/Development/phenomic/node_modules/ws/lib' Ref websockets/ws#1220

lpinca closed this as completed Oct 22, 2017

lpinca mentioned this issue Nov 13, 2017

WARNING in ./node_modules/ws/lib/BufferUtil.js on using of Websocket in storybook #1236

Closed

ozyman42 mentioned this issue Aug 4, 2019

optional dependencies when using es6 import/export webpack/webpack#7713

Closed

lpinca mentioned this issue Sep 10, 2019

fix: List optional peer dependencies #1626

Merged

sjmiles mentioned this issue Nov 22, 2019

ws node module throws warnings when webpack'd PolymerLabs/arcs#4133

Closed

jaime-ez mentioned this issue Jul 12, 2020

Problem of usage in nodejs 14.5 with the lastest version of @deepstream/client@5.1.2 deepstreamIO/deepstream.io-client-js#534

Closed

ozyman42 mentioned this issue Nov 30, 2020

Unable to bundle socket.io client due to undeclared dependency on base64-arraybuffer socketio/engine.io-parser#116

Closed

This was referenced Aug 4, 2023

소켓 사용시 컴파일 warning 발생 해결 august-thirty-first/frontend#34

Closed

Feat 소켓 사용시 컴파일 warning 발생 해결 august-thirty-first/frontend#35

Merged

prince272 mentioned this issue Sep 6, 2023

Compile with warnings hosseinmd/react-signalr#31

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compile with warnings #1220

Compile with warnings #1220

albandaft commented Oct 20, 2017 •

edited

lpinca commented Oct 20, 2017

lpinca commented Oct 21, 2017 •

edited

lpinca commented Oct 21, 2017

albandaft commented Oct 23, 2017

lpinca commented Oct 23, 2017

albandaft commented Oct 23, 2017 •

edited

lpinca commented Oct 23, 2017 •

edited

albandaft commented Oct 23, 2017

lpinca commented Oct 23, 2017 •

edited

albandaft commented Oct 26, 2017

lpinca commented Oct 26, 2017 •

edited

albandaft commented Oct 26, 2017

lpinca commented Oct 26, 2017

waheed25 commented Nov 13, 2017 •

edited

syang commented Oct 2, 2018

gosp commented Oct 15, 2018

KATT commented Oct 25, 2018

u-bits commented Jan 28, 2019 •

edited

ozyman42 commented Apr 27, 2019 •

edited

ozyman42 commented Apr 27, 2019 •

edited

lpinca commented Apr 28, 2019

ozyman42 commented Apr 28, 2019 •

edited

lpinca commented Apr 28, 2019 •

edited

ozyman42 commented Apr 28, 2019

lpinca commented Apr 28, 2019 •

edited

ozyman42 commented Apr 28, 2019

lpinca commented Apr 28, 2019

ozyman42 commented Apr 29, 2019

lpinca commented Apr 29, 2019 •

edited

ozyman42 commented May 1, 2019

etienne-martin commented Dec 9, 2019 •

edited

ozyman42 commented Dec 14, 2019

ozyman42 commented Dec 14, 2019

Dahkenangnon commented Oct 28, 2020

Compile with warnings #1220

Compile with warnings #1220

Comments

albandaft commented Oct 20, 2017 • edited

lpinca commented Oct 20, 2017

lpinca commented Oct 21, 2017 • edited

lpinca commented Oct 21, 2017

albandaft commented Oct 23, 2017

lpinca commented Oct 23, 2017

albandaft commented Oct 23, 2017 • edited

lpinca commented Oct 23, 2017 • edited

albandaft commented Oct 23, 2017

lpinca commented Oct 23, 2017 • edited

albandaft commented Oct 26, 2017

lpinca commented Oct 26, 2017 • edited

albandaft commented Oct 26, 2017

lpinca commented Oct 26, 2017

waheed25 commented Nov 13, 2017 • edited

syang commented Oct 2, 2018

gosp commented Oct 15, 2018

KATT commented Oct 25, 2018

u-bits commented Jan 28, 2019 • edited

ozyman42 commented Apr 27, 2019 • edited

ozyman42 commented Apr 27, 2019 • edited

lpinca commented Apr 28, 2019

ozyman42 commented Apr 28, 2019 • edited

lpinca commented Apr 28, 2019 • edited

ozyman42 commented Apr 28, 2019

lpinca commented Apr 28, 2019 • edited

ozyman42 commented Apr 28, 2019

lpinca commented Apr 28, 2019

ozyman42 commented Apr 29, 2019

lpinca commented Apr 29, 2019 • edited

ozyman42 commented May 1, 2019

etienne-martin commented Dec 9, 2019 • edited

ozyman42 commented Dec 14, 2019

ozyman42 commented Dec 14, 2019

Dahkenangnon commented Oct 28, 2020

albandaft commented Oct 20, 2017 •

edited

lpinca commented Oct 21, 2017 •

edited

albandaft commented Oct 23, 2017 •

edited

lpinca commented Oct 23, 2017 •

edited

lpinca commented Oct 23, 2017 •

edited

lpinca commented Oct 26, 2017 •

edited

waheed25 commented Nov 13, 2017 •

edited

u-bits commented Jan 28, 2019 •

edited

ozyman42 commented Apr 27, 2019 •

edited

ozyman42 commented Apr 27, 2019 •

edited

ozyman42 commented Apr 28, 2019 •

edited

lpinca commented Apr 28, 2019 •

edited

lpinca commented Apr 28, 2019 •

edited

lpinca commented Apr 29, 2019 •

edited

etienne-martin commented Dec 9, 2019 •

edited