Worker Threads and Shared Memory

[nathanforce]

Note: Repost of a comment in #2605, @novemberborn requested a discussion thread. 😃

I'll start this all of with a bug caveat that I don't have much experience with workers, memory sharing, or any of the other nitty gritty stuff involved here.

My ultimate goal is to create a jsdom instance for each test in my frontend application and avoid polluting the global namespace. I want to do this so that I can benefit from Ava's concurrency.

To run my app in jsdom I'm bundling the whole thing up using esbuild. When a <script> on the page requests my app bundle jsdom will respond with a Buffer created from esbuild's OutputFile.contents, a Uint8Array.

https://github.com/evanw/esbuild/blob/4e336e4f313a1f8a4bd6032f037d23f725f52e75/lib/types.ts#L101

This all works nicely and, importantly, it avoids serializing the contents of the JS file.

This works fine but has a scaling issue as each test suite/process needs to build the app. I wondered if workers might help solve this by building once and providing the output to all tests that need it.

I've put together a rudimentary example using workers that does indeed reduce the runs of esbuild to just 1, but the test time is far slower than multiple runs. My guess is that this is because the communication between the workers is serializing the build output.

This is where my limited knowledge of this type of thing starts to fade. It seems based on the docs for postMessage that the expected solution here would be to use SharedArrayBuffer to avoid copying the data. Would it be possible to expose that API in Ava's shared worker API or is it purposefully not exposed?

Sorry for the long winded backstory! Thanks for all your work.

[novemberborn]

The tests run in separate processes, so message sharing isn't possible. It's conceivable that passing a large buffer from the worker thread to the main process, down to the worker processes, is excessively slow.

Perhaps you could try writing it to a temporary file path and communicating that path with the tests?

[nathanforce]

😞 I also had the idea of writing a file but thought I'd try to avoid it if possible. Thanks for the suggestion.

Just to confirm my own understanding here, when you say tests run in separate processes you're referring to each individual test(), right?

[novemberborn]

Just to confirm my own understanding here, when you say tests run in separate processes you're referring to each individual test(), right?

No. A different process is used for each test file, but within that the test()s all run in the same process.