Add hook for dynamic entry chunk emission

[lukastaegert]

This PR contains:

• bugfix
• feature
• refactor
• documentation
• other

Are tests included?

• yes (bugfixes and features will not be merged without tests)
• no

Breaking Changes?

• yes (breaking changes will not be merged unless absolutely necessary)
• no

List any relevant issue numbers:
Will resolve #2742

Description

This PR will allow the emission of additional chunks and refine a lot of related logic.

Emitting new chunks

With this PR, a new plugin context function this.emitChunk(moduleId: string, options?: {name?: string}): string is added. This function allows to both split chunks by specifying a moduleId that resolves to a module that is already in the graph as well as add new chunks if the moduleId resolves to a new module. If Rollup has already completed the loading and parsing phase, emitting new chunks will fail with an error. With regard to tree-shaking, all exports of the module are automatically included.

Emitted chunks will follow the naming scheme for auto-generated chunks. By supplying a name option, the [name] part of the generated chunk name can be set. If this name is already assigned to a chunk with a different entry module or if there is already another chunk name assigned to the entry module (e.g. via the object form of the input option), bundling will fail with an error.

The function returns a reference id that can be used to reference the chunk later via this.getChunkFileName(referenceId: string): string on the plugin context (can only be called during bundle.generate), or via ROLLUP_CHUNK_URL_referenceId from code generated in load and transform plugin hooks.

Note that if you do not need a new module to be an entry point that creates a chunk that can be referenced, the recommended way of adding new modules to the graph is still creating virtual modules via the resolveId/load/transform hooks as this will result in fewer chunks and better tree-shaking.

Internally, a separate ModuleLoader has been extracted that accepts additional entry points and manual chunks before and while the module loading is running.

How to use this for workers

One idea behind this PR is the dynamic creation of workers. Depending on the format, there are different possible approaches.

ES modules

To my knowledge there is NO BROWSER that supports ES modules in workers (i.e. `new Worker(..., {type: 'module'})). Still, this is how you could do this in a plugin:

export default function workerPlugin() {
  let workerId;
  return {
    load(id) {
      if (id === 'get-worker') {
        if (!workerId) {
          workerId = this.emitChunk('worker');
        }
        return `export default new Worker(import.meta.ROLLUP_CHUNK_URL_${workerId}, {type: 'module'});`;
      }
    },
    resolveId(id) {
      if (id === 'get-worker') {
        return id;
      }
      return null;
    }
  };
}

AMD and SystemJS

What does work today is AMD and SystemJS support. However here, you need a proxy chunk that takes care the corresponding module loader is loaded first:

export default function workerPlugin() {
  let workerId;
  let proxyId;
  return {
    load(id) {
      if (id === 'get-worker') {
        if (!workerId) {
          workerId = this.emitChunk('worker');
          proxyId = this.emitChunk('worker-proxy');
        }
        return `export default new Worker(import.meta.ROLLUP_CHUNK_URL_${proxyId});`;
      }
      if (id === 'worker-proxy') {
        // this chunks is completely replaced but we add a reference to the URL so that the
        // chunk hash reflects the URL
        return `PLACEHOLDER(import.meta.ROLLUP_CHUNK_URL_${workerId})`;
      }
    },
    renderChunk(code, chunk, options) {
      if (chunk.facadeModuleId === 'worker-proxy') {
        if (options.format === 'system') {
          return `importScripts('path/to/systemjs/dist/s.js');System.import('./${this.getChunkFileName(workerId)}');`;
        }
        if (options.format === 'amd') {
          return `importScripts('path/to/requirejs/require.js');requirejs(['./${this.getChunkFileName(workerId)}']);`;
        }
      }
    },
    resolveId(id) {
      if (id === 'get-worker' || id === 'worker-proxy') {
        return id;
      }
      return null;
    }
  };
}

AMD improvements:

AMD chunks will no longer use the .js extension for relative dependencies. This will make it possible to use a baseUrl for a project as this changes resolution behaviour for AMD modules: https://requirejs.org/docs/api.html#jsfiles

There are now also some tests that actually run an AMD module loader to test the chunking

Bugfix: Empty dynamic entry points

In this PR, a bug was discovered where Rollup would fail if a dynamic entry point would contain nothing but reexports. This is fixed now.

New resolveFileUrl hook

This combines both the existing resolveAssetUrl hook but also resolves chunk URLs. To discern them, a type parameter is provided.

Named and unnamed entry points

To refine the chunking behaviour and especially the interaction with manualChunks, the logic follows a concept of named and unnamed entry points:

• Named entry points are created by using the object form of input
• Unnamed entry points are created by using the array or string form of input
• Dynamically created entry points are named if a name is provided, otherwise they are unnamed.

There no longer is a hard restriction preventing duplicate entry points, i.e. having more than one entry point referencing the same module. On the contrary, this is now permitted to e.g. allow a dynamically emitted entry point to coincide with an existing one. This means:

• If all entry points referencing the same entry module are either unnamed or have the same name, only one chunk is created for all of them.
• If there are entry points with different names references the same entry module, an error is thrown.

Also, dynamic entry points will now receive the name of the imported module as [name] in the file name pattern.

ManualChunks improvements

Manual chunks have been very much refined:

• Manual chunks can now be created using modules that depend on each other (this is important for allowing much more free manual chunk creation in the future, e.g. via a function)
• Different manual chunks that depend on each other no longer require careful ordering of their declarations
• Different manual chunks that reference the same entry module will throw an error
• If a manual chunk contains an entry point, one of two things will happen:
  • If the manual chunk alias and exposed exports match, then the manual chunk will become the entry chunk (this also includes using the naming scheme for entry chunks)
  • otherwise a facade for the entry chunk is created

[lukastaegert]

@guybedford Not yet ready for review but this may still be interesting for you to have a look at and maybe add some comments. I will update this PR until everything is resolved.

[guybedford]

Looks amazing. A second argument seems like it could permit the alias of the chunk nicely. I've also been wanting to see entry chunks with the same modules permitted (effectively just referencing the same shared chunk right?).

Seems like a lot to review in one PR (manual chunks + entry chunks work), but I can aim to take a look at the manual chunks approach too when its ready.

[lukastaegert]

I've also been wanting to see entry chunks with the same modules permitted

As stated in the description, this should work, but I am actually not sure if I added a test for this, will need to check.

Looks amazing. A second argument seems like it could permit the alias of the chunk nicely

Thanks. I have been wondering about this, another way of approaching it could be to allow the same syntax as for input where you can supply also an object or an array. Technically it would be very little work to allow this. But then again I am not sure if we need that. Allowing named chunks would also incur the risk of conflicts with other chunks with the same alias but a different entry module. At the moment, this will fail the build, so plugin creators would need to check for this. So my thought was to not allow this yet until we are sure we have a use case?

[lukastaegert]

On second thought, having more than one entry would cause problems as to how the ids should be returned, so yes, a second parameter would make more sense.

[lukastaegert]

@guybedford @surma @jakearchibald @philipwalton This is now mostly ready from my side (except documentation updates, hopefully I can finish them as well soon). Please give this a spin. In the PR description, I added a suggestion for how to use this with workers. Note that ESM support for workers in browsers is rather poor so I also outlined how this could be used with AMD or SystemJS (basically a proxy chunk is added that loads the runtime).

[jakearchibald]

Really excited to hear about this. A few of us are on deadlines for Google I/O right now, so don't take slow reviews as a sign of disinterest 😄. Looking forward to giving this a spin.

[jakearchibald]

(I'm also excited about how this could allow HTML to be an entry point, as the plugin could scan for <script> and add chunks)

[lukastaegert]

Documentation is now updated as well

[lukastaegert]

(I'm also excited about how this could allow HTML to be an entry point, as the plugin could scan for <script> and add chunks)

Interesting point. You could already achieve this via the options hook today but of course this API is "cleaner". I made sure that it is possible to bundle successfully without providing an input option if after the buildStart hook there is at least one entry point defined.

[philipwalton]

Hey @lukastaegert I'm in the same situation as Jake (don't have a lot of time to really dig in) but after an initial look, I'm pretty happy with these changes, and I think the emitChunk() function would solve the use case I mentioned in the issue.

I'm also excited to read about the ManualChunks improvements, as hopefully this will solve the feature request I proposed in #2688.

Nice work!

[lukastaegert]

Thanks!

I'm also excited to read about the ManualChunks improvements, as hopefully this will solve the feature request I proposed in #2688.

Not yet but it is now only a small step that is left and I hope I manage to add it (or to a second PR) and release this together.