How to send SharedArrayBuffer from main process to Window processes

[nidin]

I am working on an offline renderer for three.js for this project i need to share SharedArrayBuffer between main and window processes. I can create SharedArrayBuffer in main process and window processes but no way to share the reference to other processes, how can it be achieved?

OS : macOS Sierra 10.12.6 (16G29)
Electron : Built from master #cc666c7

[welcome]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can.

To help make it easier for us to investigate your issue, please follow the contributing guidelines.

[codebytere]

Thanks for reaching out!

Because we treat our issues list as the team's backlog, we close issues that are questions since they don't represent a task needing to be completed. For most questions about Electron there are a lot of options. Check out the Electron community. There are also a bunch of helpful people in this community forum that should be willing to point you in the right direction.

[asineth0]

Did you ever end up figuring this out?

[ymeine]

Hi,

This may come in a bit late, but I did stumble upon this closed issue right here when doing researches on the exact same topic, so hopefully it could help other people looking for an answer.

TL;DR: you can't use SharedArrayBuffer between processes, only between threads. But there are other ways to share memory.

---

The MDN documentation

So, if we refer to the MDN documentation about SharedArrayBuffer, there's no mention at all about "processes". However, there's a lot of mentions about "threads", and of course "workers" since they are their abstraction in the JavaScript extended ecosystem (I say extended because workers are not a JavaScript thing, they belong to the platform embedding a JavaScript engine). They do mention a "main program", which is to me a very confusing term here:

To share memory using SharedArrayBuffer objects from one agent in the cluster to another (an agent is either the web page's main program or one of its web workers), postMessage and structured cloning is used.

So to clarify this description:

• cluster: a single process with multiple threads
• agent: a thread within that process
• main program: the main thread within that process
• web worker: a thread other than the main thread within that process

Now, after checking a bit the ECMAScript specifications, I realize "cluster" and "agent" are terms coming from there, and I may not completely understand those concepts yet. But given this description on MDN and my different tests, the mapping I made to "process" and "thread" should be correct in this context.

The ECMAScript specifications

Now if we look at the ECMAScript standard, first we see only a few mentions of threads, when describing execution Agents. Honestly, I didn't take the time to analyze that so I don't fully understand it.

Now, about the mentions of OS processes (by looking for the word "process" and excluding all of the irrelevant occurrences):

• the Data Blocks section describes shared data blocks and does mention the concept of processes
• another occurrence appears in the section about Agent Clusters, with an interesting quote

Programs within different agents may share memory by unspecified means. At a minimum, the backing memory for SharedArrayBuffers can be shared among the agents in the cluster.

And regarding mentions of SharedArrayBuffer, I looked for sections which refer to it alone, not along with ArrayBuffer. Indeed, those two abstractions share a lot, they serve as backing buffers for TypedArray and they handle the allocation and deallocation of a buffer in memory. But what imports to us here is this particular aspect of shared buffer, which can't be linked to ArrayBuffer.

The dedicated section about SharedArrayBuffer is really about the JavaScript API implementation, so it's not relevant here.

It's only when we reach the section about Memory Model that things could start getting interesting, but it's way too long to read right now, and the only thing I retain is: SharedArrayBuffer is meant to share memory between agents.

So we are back to the Agents section.

Unfortunately, as of right now I can't really provide an exact interpretation of these specifications.

How to share memory: using OS native APIs

So, nothing prevents you from writing a Node.js native addon (Node-API preferred) that would allocate and deallocate a block of shared memory. For instance on Windows:

• Creating Named Shared Memory
• CreateFileMapping
• OpenFileMapping
• MapViewOfFile

You can load that native addon on the main process side of Electron, and using node integration options in WebPreferences you can load that same native addon on the renderer process and its threads (Web Workers) as well, even though they advise against it.

You would have several instances of your addon loaded, which is a duplication of data, but only for your addon state. The block of shared memory you would allocate, deallocate and pass around using your addon's exposed API would be the same and it would be shared memory.

Now, I didn't mention the access part: reading and writing from that block of memory. You theoretically have two options, but you will see that there's a HUGE limitation with Electron which may bum you.

Option 1: add custom APIs to read and write to the buffer. That's not standard, and only your code will know how to interact with it. Passing the buffer wrapper to 3rd-parties may not work unless they accept adapters.

Option 2: wrap it in an ArrayBuffer, to then be able to wrap with TypedArray and use its standard APIs. That is theoretically possible because the Node-API allows attaching external buffers (understand: buffers you manage on your own) to a new ArrayBuffer instance. But the corresponding documentation section does even tell you explicitly that you won't be able to use it in Electron:

Some runtimes other than Node.js have dropped support for external buffers.
On runtimes other than Node.js this method may return napi_no_external_buffers_allowed to indicate that external buffers are not supported. One such runtime is Electron as described in this issue electron/issues/35801.

Your only hope if you want to go that route is either to have issue #35241 resolved or to build Electron yourself without that mentioned memory cage.

References:

• [Bug]: Electron 21 breaks node-api stability guarantees #35801
• [Feature Request]: Provide builds without the memory cage #35241
• https://www.electronjs.org/blog/v8-memory-cage

Conclusion

It's unfortunate that Electron resembles more and more to a black box webview, while it's probably the only viable option to build applications a bit more interesting than yet another TODO list or documentation websites wrappers.

Of course, having SharedArrayBuffer directly usable across processes would be the best, and the most logical thing to do. But I don't even know if it would ever be planned.

The route towards using a native addon is more complex, and if you want compatibility with ArrayBuffer/TypedArray then it becomes WAY more complex since you have to build Electron yourself and maintain that build. To be honest I'm still suck at that stage but I have other projects preventing me from exploring this further right now. If your application use case permits it, you can accommodate with just a full custom native API.

[ymeine]

Regarding the custom build of Electron, I managed to make one, but there are cons:

• while electron/build-tools is a precious gem that automates almost everything, it was not complete for my needs (knowing I fully automated the build):
  • I had to trick to move some cache folders to another location (too big for my main drive), by patching the generated configuration
  • I had to try multiple times before properly syncing and building everything properly, because of gaps in the documentation/CLI help
  • I didn't find a way to properly use a custom Electron commit ID or even a custom fork, despite the documentation: so I have to patch the code and re-run a sync command which is quite long no matter what
• the whole download took between 30 minutes and 1h with a fair network connection
• the whole custom build took 3h on a reasonable computer spec
• I am not sure the generated build is stable since changing some build variables may not be enough: the rest of the build, as well as the code itself, must make sure it's compatible with those variables changes

Otherwise the idea is fairly simple:

• when downloading the code and preparing the build, all the content goes into one folder
• under this folder, entry point build configurations are in src/electron/build/args
• depending on how you created your configuration, it could point to release.gn or testing.gn
• you need to either change one of those files or create a new one and point to it (update the configuration), which is what I did with a file I named custom.gn

The content of custom.gn is:

import("testing.gn")
v8_enable_sandbox = false
v8_enable_pointer_compression = false

Adapt the import depending on your needs. The important part here is the customization of the V8 options.

With the generated build, I could use the solution I wrote in my previous comment and wrap my instances in ArrayBuffer and that way use all the standards APIs. I still need to see how reliable this build will be on the longer term.