Does this loader supports Node.js Worker threads?

[yvele]

Documentation Is:

• Missing
• Needed
• Confusing
• Not Sure?

Please Explain in Detail...

I'm not sure but I think this loader is not compatible with Node.js Worker threads.

I'm using Webpack to pack an AWS Lambda bundle that runs on Node.js and I came across this project.. And it looks like this project only handles Web Workers on the browser.

Your Proposal for Changes

Add a note specifying that this projects only works for Web Workers and is not compatible with Node.js Worker threads. And possibly add a link to a recommended project that handles Node.js Worker threads.

[alexander-akait]

No, PR welcome to docs

[tchakabam]

@yvele I am using this with NodeJS target, it is absolutely no problem.

It works just the same as for web platform workers. The APIs are compatible, even if NodeJS provides other tools on top,but that shouldnt be concern of this loader here.

What exactly is the problem you face? I can provide my config as example in case needed.

[tchakabam]

Should be needless to say this requires obviously to shim the global namespace like so first of all:

const {Worker: NodeJSWorker} = require('worker_threads');
const {resolve: resolvePath} = require('path');

global.Worker = function NodeJSWorkerThreadsPathResolveWrap(path, options) {
  // this __dirname will travel into the bundle and only resolved at
  // runtime there, so it will be the path to the dist bin
  // and we can assume to find the worker there.
  // generally, this has to be an absolute path with NodeJS worker-threads,
  // so this is why we use resolve here and not join.
  return new NodeJSWorker(resolvePath(__dirname, path), options);
}

This is TypeScript  plain JS, and the symbol name NodeJSWorkerThreadsPathResolveWrap is completely arbitrary choice.

EDIT:

and inside webpack conf:

    node: {
      __dirname: false, // this is important so that __dirname
                        // gets into bundle as vanilla and only resolved at runtime
    },

Very simple setup, and harmless if you dont need __dirname resolving by webpack (which is ugly as hell to do anyway I think).

Otherwise, everything works just fine like for webworker, since it is API compatible.

[alexander-akait]

Just note - webpack@5 support workers (web workers and Node.js worker threads) out of box, you don't need this loader, it was deprecated someday (need improve docs)

/cc @chenxsan Do we have examples/guide for web workers and etc stuff?

[tchakabam]

Just note - webpack@5 support workers (web workers and Node.js worker threads) out of box

Really? Then what is this project still about? It should indeed be shown first thing in readme I guess :)

[alexander-akait]

Really?

Yep

Then what is this project still about?

For migration purpose, also some features like inline (Blob) is not implemented

It should indeed be shown first thing in readme I guess

I want to do it after improving docs on site with link

[tchakabam]

For migration purpose, also some features like inline (Blob) is not implemented

if the webpack v5 built-in thing doesnt support inline, then it is not that useful :) and this plugin here is still not replaceable by anything else indeed for me.

[chenxsan]

Just note - webpack@5 support workers (web workers and Node.js worker threads) out of box, you don't need this loader, it was deprecated someday (need improve docs)

/cc @chenxsan Do we have examples/guide for web workers and etc stuff?

I'm afraid no.

[alexander-akait]

@tchakabam It is very easy to implement, anyway why you need inline?

@chenxsan Do you need help?

[chenxsan]

@alexander-akait I've created an issue to track it.

[tchakabam]

It is very easy to implement, anyway why you need inline?

I think it is just nicer to ship one single file, but that might be an outdated point of view :)

EDIT: obviously shipping libs as bundles is not outdated, but i guess seeing a worker like another chunk that gets loaded on demand might be a valid approach sometimes better than having to ship a single large file ... nevertheless many projects using worker-loader use inline, and thus the expectation on the distro end is that there is one single JS file. changing that might cause breakage potentially if that is the assumptions made on some consuming sides...

[alexander-akait]

Example from source https://github.com/webpack/webpack/tree/master/examples/worker

[tranvansang]

I have added node's worker_threads support in this PR #313.

I was able to build and run locally, but haven't added any test case to the PR.

Comments and collaboration are welcome.

[alexander-akait]

@tranvansang What is webpack version? webpack v5 supported worker (browser/node) out of box

[tranvansang]

@alexander-akait

I am using the quite new one ^5.28.0.

I gave it a try but couldn't succeed with webpack 5 natively.

So I modify the loader to first make things work in my project.

As a note that in my project, the worker using typescript and shares several module (in code) with the main thread.
And in the compiler config's public path is not the absolute path in the file system. I have to set publicPath to the same value as of the frontend's config.

If someone have ever tried and succeed, I'd like to give it another try.

I just have a question that: should I use new Worker('./my.worker') or new Worker(new URL('./my.worker', import.meta.url))?
I remember that I got stuck at this import.meta.url stuff.

[alexander-akait]

new Worker(new URL('./my.worker', import.meta.url))

Should work

This loader doesn't support worker_threads and will not work in right way even with your PR

[tranvansang]

In the branch that I customized for my project.

master...tranvansang:worker_threads

It works out of the box, and I have already released it for my use.

Thanks for the answer. If webpack means to support it. I definitely will make a try.

[alexander-akait]

code splitting will not work, also caching can be broken

[tranvansang]

new Worker(new URL('./my.worker', import.meta.url))

Should work

This loader doesn't support worker_threads and will not work in right way even with your PR

So, now I remembered how I gave up with native webpack support.

In my project, I set public path to the same value with the frontend config. For example: in production build, publicPath will be https://cdn.mywebsite.com/bundles. This config is used extensively for assets import URL, and several URL manipulation routines which use __webpack_public_path__.

While with the new URL, and import.meta.url, my assumption is that publicPath must be the file system path. Is this correct?

Please share me anything if you know how to customize the public path of the worker threads.

[alexander-akait]

Can you share your problem (simple example)? Hard to say how better to solve for you without example(s)

[tranvansang]

the migration/usage of webpack 5 was quite straightforward. I was able to use worker thread natively without the loader.

The only thing I had to do is to leave the publicPath config empty (which made me to manually set publicPath options in file-loader loader).

Now, I have another problem that jest doesn't work. Previously, I wrote my own transformer to fake Worker class and it ran perfectly. Now, it doesn't work with webpack-like syntax.
Because babel currently can't parse import.meta.url, and, it doesn't spawn a child transpiler causing worker code in typescript.

I am not yet familiar with syntax like babel plugin development, so my test is currently not available for modules that use worker threads.

I will appreciate if anyone has an idea on this.

[alexander-akait]

The only thing I had to do is to leave the publicPath config empty

No need to do it, please prefer to use auto

Because babel currently can't parse import.meta.url, and, it doesn't spawn a child transpiler causing worker code in typescript.

You need plugin for this