Initial exploration of startService API

[ggoodman]

This PR contains:

• bugfix
• feature
• refactor
• documentation
• other

Are tests included?

• yes (bugfixes and features will not be merged without tests)
• no

Not yet

Breaking Changes?

• yes (breaking changes will not be merged unless absolutely necessary)
• no

List any relevant issue numbers:

N/A - Experimental feature exploration

Description

This PR is designed to add a new operating paradigm ('service mode') to Rollup to allow it to fulfill the requirements of tools like Vite and WMR's dev mode such that those tools can use Rollup directly instead of re-implementing parts of it that are specific to the dev workflow.

To be useful to tools like Vite, I extrapolate the following top-level requirements (but would love it if others can fine-tune this):

1. An application can request the bundling a specific source file on-demand and obtain a result that could be generated in a target module format. That result would be functionally equivalent to how the module would have behaved in the context of a Bundle with preserveModules and preserveEntrySignatures set to strict or allow-extension. For example, this might be used to translate an http request for a source file into code suitable for serving to the browser. It might also be used to create a dynamic Node module usable in the same way ViteDevServer::ssrLoadModule is used in Vite.
2. An application can obtain assets that have been produced as a side-effect of bundling other source files. In the situation where a Plugin has used emitFile to produce a new asset (or module), the application must be able to determine if the requested asset exists and obtain a reference to it. For example, a dev server would want to distinguish between urls it controls and 404s. If the Rollup Service has an asset at the requested path, then that is a 'controlled' url and the app would want to serve the asset owned by Rollup.
3. Changes to the host environment will correctly invalidate outdated internal state. The addition of new files to directories might invalidate module resolutions. Changes to config files might invalidate transformed modules. Changes to the source code will invalidate the corresponding Modules and perhaps any emitted Assets. Invalidation should happen immediately (ideally synchronously) but no work would need to be done to re-create invalidated state until it is re-requested.
4. An application can observe invalidations. When modules and assets are invalidated, the application can subscribe to these events. For this to be useful, the application must be able to query the dependency Graph. For example, if a source file is modified, the application could use information about changes in exported symbols and their linkage to perform very nuanced hot-module replacement (HMR). More coarse-grained HMR would also be possible by querying the module-level dependency graph.
5. An application can determine the granularity of output production. An application could indicate to the Rollup Service how much of the graph it wants to include in a given bundling request. This would be conceptually similar to manualChunks where the app could indicate that it wants to treat react as a black-box. The app would ask Rollup to traverse the graph and include everything within the app-defined boundary (in this case ./node_modules/react) in a single output unit. The app might say that a user's source files should be 1:1 with output artifacts but a users's dependencies would be n:1.

I suspect that Rollup has almost all of the essential bits to satisfy these requirements but that it would take some refactoring. I expect that the change of paradigm from run-to-completion to also supporting a long-lived, incremental service would be tricky. Assumptions about internal state at different phases of the pipeline might be broken. To attack this problem will mean finding those and coming up with strategies to support re-entrancy.

The hardest parts that I anticipate are related to invalidation tracking. Specifically, invalidation of module resolution. To support this sort of invalidation would require the cooperation of module-resolution plugins and perhaps new APIs in Rollup's Plugin Context to support tracking changes to the contents (or existence of) directories. Invalidating internal state will also expose a lot of assumptions about statefulness.

Having talked through a lot of this with @lukastaegert at a high level on Discord, we think that tree-shaking (as a concept) would be counter-productive to the effort. Perhaps this Service mode would force disable tree-shaking.

We also talked about how to deal with some conventions that would affect performance in real-world use-cases, such as those for which Vite is designed. Notably, libraries like react use the process.env.NODE_ENV convention to package both dev and production builds in the same module. A tool using the Service mode of Rollup would want to be able to build plugins that could prune branches of the AST based on this convention and have graph building reflect this.

Thoughts?

[lukastaegert]

I think this has a lot of potential, will have a deeper look later this week.

[lukastaegert]

Just convert this.modules to a Set instead of doing this, should also have better performance for large bundles. We are mostly iterating over them, which works just as well for a Set, and it is a private variable anyway.

[SIMJO7747]

https://www.youtube.com/@seventysevenrecordztv