Are custom loaders run on their own realm always?

[dead-claudia]

In the "How do loaders declare what hooks they implement?" part of the loaders design document, it makes several claims regarding loaders imported by third parties:

Loaders can expose and exported constructor that creates an object with functions matching the names the hooks they implement.

• This allows Loaders to function even within Realms that have frozen globals.
  • Globals would not work if Loader has a frozen realm.
• This avoids potential overriding by dependencies by 3rd party modules.
• This guarantees that 3rd party modules could import the hook if they know the specifier of the loader.
  • Use policies or similar to restrict this capability.
  • This does not allow them to use the same instance of loader object as the runtime since an independent call only made by the runtime can be used to obtain a unique object.

Two claims stick out to me in this mix:

• This avoids potential overriding by dependencies by 3rd party modules.
• This does not allow them to use the same instance of loader object as the runtime since an independent call only made by the runtime can be used to obtain a unique object.

Suppose we have three loaders root_app/loaders/alice.mjs, root_app/loaders/bob.mjs, and node_modules/dep/eve.mjs, each loaded in that order.

// root_app/loaders/alice.mjs
export default class Alice {
	// ...
}

// root_app/loaders/bob.mjs
import Alice from "./alice.mjs"
export default class Bob {
	// ...
}

// node_modules/dep/eve.mjs
import Alice from "../../loaders/alice.mjs"
let loaders = new Set()
let prev = Alice.prototype.resolve
Alice.prototype.resolve = function (...args) {
	loaders.add(this)
	return Reflect.apply(prev, this, args)
}
export default class Eve {
	// ...
}

What would eve.mjs see here? If eve.mjs is in the same realm as the alice.mjs and bob.mjs loaders, or at least share the same module cache, both of the claims in question are false. And based on my reading, it's unclear whether this is the case or not, and I can't find any language explicitly stating one way or the other.

[jkrems]

I would expect them to not share the same loader cache if they're supposed to be isolated. But I agree that it should be called out either way. In general I think module caches should be per realm but that may be beyond this discussion.

[devsnek]

loader hooks don't have caches, they respond to the main loader needing to fill in its own cache.

[jkrems]

@devsnek If they try to use import/require/etc. would it then fail? Or do you mean "they share the cache" by "they don't have caches"?

[devsnek]

@jkrems i'm not sure what you mean by "try to use import/require/etc". each realm has a loader. each loader can have hooks, which run in separate realms/threads.

[jkrems]

The loader hooks are implemented in files. Those files will hopefully be able to run as a module. So import may be valid syntax inside of the hook implementation. But what does it do? See the original post in this thread for why sharing a require or ESM cache between the loader and the targeted realm is problematic.

Something needs to happen when we encounter require or import within the loader hook code. Right now (according to @isiahmeadows's investigation) we don't define what happens.

[devsnek]

ahhhhh i understand now. those imports would be subject to the hooks of the current realm, of which there are none. whether or not we modify the hook behaviour there, the caches of each realm will never be shared. v8 can't link modules from different realms.

[jkrems]

Okay, so we agree I think. Each realm/context should have its own module map. And each hook should be considered running in its own context, with its own isolated module map (and require cache). We should call that out in our design docs.

[Jamesernator]

If loaders live in their own realm isn't this a bit of a hazard with dynamicInstantiate?

e.g.:

export function dynamicInstantiate(url) {
  return {
    exports: ['foo'],
    execute: (exports) => {
      // Array from wrong realm
      exports.foo.set([1,2,3]);
    },
  }
}

[devsnek]

the larger hazard is that we want to run loaders in their own threads, so that would be impossible anyway. i was talking about this with bradley at some point but i don't remember where that conversation ended...

[jkrems]

Afaik dynamicInstantiate isn’t likely to end up in the final loader API, among other things for these reasons.

…

On Mon, Sep 16, 2019 at 7:46 PM Gus Caplan ***@***.***> wrote: the larger hazard is that we want to run loaders in their own threads, so that would be impossible anyway. i was talking about this with bradley at some point but i don't remember where that conversation ended... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#384?email_source=notifications&email_token=AAEKR5HPCRYFWP5IGPKCMGLQKBAHHA5CNFSM4IWZ3NNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD63DK3A#issuecomment-532034924>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEKR5C4LRBSMCDLYFBPUADQKBAHHANCNFSM4IWZ3NNA> .

[SMotaal]

@jkrems I think it is fair to question the current form of dynamicInstantiate.

Worth noting though that in order to deal with the prototypical realities of the language when aligning loader hooks with forms of dynamic namespace mutations of more popular demand, similar dynamic instantiation mechanisms will be inevitable.

Those hooks apply directly on the instantiated namespace of a module which is bound to specific context (or root realm) primordials… and so this entails that (regardless of technical detail) not all hooks will apply equally… some are pre-cache, some are post, and from those some will likely need to be in realm and operating on respective namespace object instances, where some even operate on a compartment level — imho at least.