Sharing complex objects across worker threads?

[mayurk]

Is there a way to share DB pool object across the worker threads. Any attempts to send such object to the worker thread from the main thread via the pool.exec(...) call gives a DataCloneError since that object has a function reference. Also even though the worker threads are part of the same process ( i might be wrong here ) every time we "require" a module in the worker it gets a fresh copy instead getting a same copy from cache. This means we cannot use this module where DB connectivity is required.

[shlomiah]

+1
I try to use variables that was required in the main thread and don't know how to inject them into the worker pool.

[josdejong]

@mayurk worker threads are really isolated from each other, just search some more explanation about it on the internet if you want to know more.

@shlomiah I'm not sure if your remark is about the same topic. Can you explain your use case?

[jfoclpf]

@josdejong there's no way to use some sort of SharedArrayBuffer?

I need to share (not clone, not transfer, just share) a big heavy JS object created at the parent thread, and share it amongst children threads.

[jfoclpf]

@josdejong
I know this does not work, but something like this:

const workerpool = require('workerpool');
const pool = workerpool.pool();

const sharedObj = {a: 1, b: 2}

function add(a, b) {
  return a + b + sharedObj.a + sharedObj.b;
}

for (let i=0; i < 10; i++) {
  pool.exec(add, [i, i+1])
    .then(function (result) {
      console.log('result', result); // outputs 7
    })
    .catch(function (err) {
      console.error(err);
    })
    .then(function () {
      pool.terminate(); // terminate all workers when done
    });
}

[josdejong]

Sounds like you would like to use a SharedWorker. The workerpool library doesn't support that though.

And on a side note: options to transfer an object without copying are discussed in #3.

[jfoclpf]

@josdejong do you recommend me any npm library for nodeJs?

[josdejong]

I don't know, it depends on your use case. I'm sure you can find something or can just use native API's.

[normancapule]

@josdejong there's no way to use some sort of SharedArrayBuffer?

I need to share (not clone, not transfer, just share) a big heavy JS object created at the parent thread, and share it amongst children threads.

In my case I had to pass an image buffer from main node process to the worker processes. I used base64 encoding to share the buffer object

[Mae6e]

@josdejong there's no way to use some sort of SharedArrayBuffer?
I need to share (not clone, not transfer, just share) a big heavy JS object created at the parent thread, and share it amongst children threads.

In my case I had to pass an image buffer from main node process to the worker processes. I used base64 encoding to share the buffer object

hi. can you write a sample for that here?