worker: improve integration with native addons #23319

addaleax · 2018-10-07T23:07:19Z

Native addons are now unloaded if all Environments referring to it
have been cleaned up, except if it also loaded by the main Environment.

/cc @gabrielschulhof Who I remember had some suggestions for improving upon this approach

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

nodejs-github-bot · 2018-10-07T23:07:20Z

@addaleax build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1159/pipeline

Native addons are now unloaded if all Environments referring to it have been cleaned up, except if it also loaded by the main Environment. Thanks go to Alexey Orlenko, Olivia Hugger and Stephen Belanger for reviewing the original version of this change. Refs: ayojs/ayo#118

richardlau · 2018-10-07T23:30:06Z

Does this need updates to the docs? I'm assuming it only applies to context-aware addons: https://github.com/nodejs/node/blob/master/doc/api/addons.md#context-aware-addons

addaleax · 2018-10-07T23:52:16Z

@richardlau I’ve updated the documentation a bit (and added the WIP label here, this isn’t fully working yet)

Trott · 2018-11-20T21:08:19Z

@addaleax Still working on this?

gabrielschulhof · 2018-11-25T06:03:23Z

@addaleax I don't believe we should be storing dlopen handles in an associative array where the key is the file name. We should just perform a dlopen() every time we need a module, and dlclose all the handles we've collected in an environment when we clean up the environment. The reason is that internally dlopen() already has an associative array where the key is the file name and thus it already caches the handles and increases a given handle's refcount if dlopen() is called again and the corresponding .so is already mapped into the process.

Thus, I believe it is sufficient to add a cleanup hook when the environment is first created which simply closes all the handles which accumulate during the life cycle of that environment. If any of the addons add their own environment cleanup hooks, as advised in the documentation, then those will fire before the one that closes all the handles.

So, I guess this also means that we should probably be storing the list of modules that were loaded into an environment on that environment so that the cleanup hook will find them when it fires.

gabrielschulhof · 2018-12-01T19:49:15Z

test/addons/worker-addon/binding.cc

+  node::RemoveEnvironmentCleanupHook(context->GetIsolate(), Dummy, nullptr);
+}
+
+NODE_MODULE(NODE_GYP_MODULE_NAME, Initialize)


This should be NODE_MODULE_CONTEXT_AWARE because Initialize makes use of the context.

gabrielschulhof · 2018-12-01T20:14:38Z

test/addons/dlopen-ping-pong/test.js

-const re = /^Error: Module did not self-register\.$/;
-assert.throws(() => require(`./build/${common.buildType}/binding`), re);
+// This second `require()` call should not throw an error.
+require(`./build/${common.buildType}/binding`);


This is a pretty strong change in behaviour. NODE_MODULE and even NODE_MODULE_CONTEXT_AWARE were so far expected to fail to register a second time around. Developers should explicitly move to modules that can be loaded multiple times by replacing those two macros with NODE_MODULE_INIT(/*exports, module, context*/) {...} as per our docs, while ensuring that they address the concerns raised there.

If we require that modules move to NODE_MODULE_INIT then that's another reason we don't need to store dlopen handles in a file-name-keyed table.

Trott · 2018-12-04T23:04:48Z

Needs a rebase.

Originally from portions of nodejs#23319.

This is an alternative to nodejs#23319 which attaches the loaded addons to the environment and closes them when the environment is destroyed.

addaleax · 2018-12-10T21:31:15Z

This has been superseded by @gabrielschulhof’s #24861 :)

Originally from portions of #23319. PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

This is an alternative to #23319 which attaches the loaded addons to the environment and closes them when the environment is destroyed. PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Originally from portions of #23319. PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

This is an alternative to #23319 which attaches the loaded addons to the environment and closes them when the environment is destroyed. PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Originally from portions of nodejs#23319. PR-URL: nodejs#24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

This is an alternative to nodejs#23319 which attaches the loaded addons to the environment and closes them when the environment is destroyed. PR-URL: nodejs#24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Originally from portions of #23319. Backport-PR-URL: #25258 PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

This is an alternative to #23319 which attaches the loaded addons to the environment and closes them when the environment is destroyed. Backport-PR-URL: #25258 PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Originally from portions of nodejs#23319. PR-URL: nodejs#24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

This is an alternative to nodejs#23319 which attaches the loaded addons to the environment and closes them when the environment is destroyed. PR-URL: nodejs#24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Unloading native addons from the main thread was an (presumably unintended) significant breaking change, because addons may rely on their memory being available after an `Environment` exits. This patch only restricts this to Worker threads, at least for the time being, and thus matches the behaviour from nodejs#23319. Refs: nodejs#24861 Refs: nodejs#23319

Unloading native addons from the main thread was an (presumably unintended) significant breaking change, because addons may rely on their memory being available after an `Environment` exits. This patch only restricts this to Worker threads, at least for the time being, and thus matches the behaviour from #23319. PR-URL: #25577 Refs: #24861 Refs: #23319 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

Allow loading add-ons from multiple Node.js instances if they are declared context-aware; in particular, this applies to N-API addons. Also, plug a memory leak that occurred when registering N-API addons. Refs: nodejs#23319

Allow loading add-ons from multiple Node.js instances if they are declared context-aware; in particular, this applies to N-API addons. Also, plug a memory leak that occurred when registering N-API addons. Refs: #23319 PR-URL: #26175 Fixes: #21481 Fixes: #21783 Fixes: #25662 Fixes: #20239 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

addaleax requested a review from gabrielschulhof October 7, 2018 23:07

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. node-api Issues and PRs related to the Node-API. labels Oct 7, 2018

addaleax force-pushed the worker-addons branch from 654099c to efc57aa Compare October 7, 2018 23:07

addaleax added addons Issues and PRs related to native addons. worker Issues and PRs related to Worker support. and removed node-api Issues and PRs related to the Node-API. labels Oct 7, 2018

fixup! worker: improve integration with native addons

f93bca3

addaleax added the wip Issues and PRs that are still a work in progress. label Oct 7, 2018

squash! worker: improve integration with native addons

b7685b6

addaleax force-pushed the worker-addons branch from 1eb48bb to b7685b6 Compare October 7, 2018 23:52

jasnell approved these changes Oct 8, 2018

View reviewed changes

gabrielschulhof reviewed Dec 1, 2018

View reviewed changes

gabrielschulhof mentioned this pull request Dec 6, 2018

src: unload native addons when the environment is destroyed #24861

Closed

4 tasks

gabrielschulhof pushed a commit to gabrielschulhof/node that referenced this pull request Dec 7, 2018

test,doc: add tests and docs for addon unloading

c991280

Originally from portions of nodejs#23319.

gabrielschulhof pushed a commit to gabrielschulhof/node that referenced this pull request Dec 7, 2018

src: unload addons when environment quits

005cbc9

This is an alternative to nodejs#23319 which attaches the loaded addons to the environment and closes them when the environment is destroyed.

addaleax closed this Dec 10, 2018

addaleax deleted the worker-addons branch December 10, 2018 21:31

addaleax added the superseded label Dec 10, 2018

gabrielschulhof pushed a commit that referenced this pull request Dec 20, 2018

test,doc: add tests and docs for addon unloading

622e348

Originally from portions of #23319. PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

MylesBorins pushed a commit that referenced this pull request Dec 25, 2018

test,doc: add tests and docs for addon unloading

8e8c651

Originally from portions of #23319. PR-URL: #24861 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

addaleax mentioned this pull request Jan 18, 2019

src: restrict unloading addons to Worker threads #25577

Closed

3 tasks

addaleax mentioned this pull request Feb 17, 2019

worker: improve integration with native addons (take II) #26175

Closed

4 tasks

tshemsedinov mentioned this pull request Mar 2, 2019

Require for native add-ons in worker_threads metarhia/Metarhia#34

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

worker: improve integration with native addons #23319

worker: improve integration with native addons #23319

addaleax commented Oct 7, 2018 •

edited

nodejs-github-bot commented Oct 7, 2018

richardlau commented Oct 7, 2018

addaleax commented Oct 7, 2018

Trott commented Nov 20, 2018

gabrielschulhof commented Nov 25, 2018

gabrielschulhof Dec 1, 2018

gabrielschulhof Dec 1, 2018 •

edited

Trott commented Dec 4, 2018

addaleax commented Dec 10, 2018

worker: improve integration with native addons #23319

worker: improve integration with native addons #23319

Conversation

addaleax commented Oct 7, 2018 • edited

Checklist

nodejs-github-bot commented Oct 7, 2018

richardlau commented Oct 7, 2018

addaleax commented Oct 7, 2018

Trott commented Nov 20, 2018

gabrielschulhof commented Nov 25, 2018

gabrielschulhof Dec 1, 2018

Choose a reason for hiding this comment

gabrielschulhof Dec 1, 2018 • edited

Choose a reason for hiding this comment

Trott commented Dec 4, 2018

addaleax commented Dec 10, 2018

addaleax commented Oct 7, 2018 •

edited

gabrielschulhof Dec 1, 2018 •

edited