Allow more programmatic control over manual chunks

[philipwalton]

Feature Use Case

Note: I'm trying to do something with Rollup that I don't think is possible, so consider this a feature request. If it is possible, please let me know how it can be done.

What I want to do is use Rollup to generate multiple output files (chunks) from a single input file. The reason I want to do this, is I want to deploy and version my application code separately from my third-party dependency code.

As an example, imagine my app has entry point ./main.mjs, and this file loads other modules both local to my application as well as some in node_modules. Here's some super simplified code:

import {vendor1} from 'npm-pkg-1';
import {vendor2a, vendor2b} from 'npm-pkg-2';

import {localA} from './localA.mjs';
import {localB} from './localB.mjs';

// ...

After running this through Rollup, I want my output files to look like this:

build/
  npm-pkg-1-XXXXXXXX.mjs
  npm-pkg-2-XXXXXXXX.mjs
  main-XXXXXXXX.mjs

And I want to load them using a single script tag like this:

<script type="module" src="./main-XXXXXXXX.mjs"></script>

And inside ./main-XXXXXXXX.mjs it would look like this:

import { vendor1 } from './npm-pkg-1-XXXXXXXX.mjs';
import { vendor2a, vendor2b } from './npm-pkg-2-XXXXXXXX.mjs';

// The rest of the `main.mjs` bundle goes here...

The point here is that all modules from npm-pkg-1 get collapsed into a single module (with a version string), and all modules from npm-pkg-2 get collapsed into a different module (with its own version string as well).

Or, if you wanted, you should be able to configure rollup to collapse all third-party code into a single vendor-XXXXXXXX.mjs module.

Note, webpack already supports a feature similar to this using its optimization.splitChunks and optimization.splitChunks.cacheGroups options. For example, on my blog I have the following config that allows me to independently version all my npm dependencies:

splitChunks: {
  chunks: 'all',
  maxInitialRequests: Infinity,
  minSize: 0,
  cacheGroups: {
    npm: {
      test: /node_modules/,
      name: (mod) => {
        const pkgName = mod.context.match(/node_modules\/([^/]+)/)[1];
        return `npm-${pkgName}`;
      },
    },
  },
},

The only difference between what I'm suggesting here and what I can do with webpack, is with webpack I have to load the scripts individually in my HTML (and also include the webpack runtime):

<script type="module" src="./runtime-XXXXXXXX.mjs"></script>
<script type="module" src="./npm-pkg-1-XXXXXXXX.mjs"></script>
<script type="module" src="./npm-pkg-2-XXXXXXXX.mjs"></script>
<script type="module" src="./main-XXXXXXXX.mjs"></script>

While listing all modules in the HTML isn't that much of a burden, my preference would be to just leverege the native module loader and not have to include the runtime.

Another side benefit of a feature like this, is you can dramatically reduce the number of total import statements your production code makes, which should help a lot with performance when using native modules in production.

Does something like this seem feasible?

Feature Proposal

I expected the manualChunks option to work this this already, but it doesn't seem like it does. If that option could be extended (or a new option creating) to take a function that gets invoked with the module ID and returns a chunk name, that seems like the simplest API to me:

manualChunks: (moduleID) => {
  if (moduleID.includes('node_modules')) {
    const packageName = moduleID.match(/node_modules\/([^/]+)/)[1];
    return packageName;
  } else {
    return 'main';
  }
},

[lukastaegert]

I believe there are several options of achieving what you are after, let me explain.

The simplest option is to just designate npm-pkg-1 and npm-pkg-2 as additional entry points. This will

• guarantees that all code imported by these files ends up in separate chunks from your remaining code
• does not guarantee that it will be only two chunks, depending on how many other chunks are created importing only some of the dependencies.

Example:
Config file:

import resolve from 'rollup-plugin-node-resolve';

export default ({
  // to even better control the generated names and folders, use an object, e.g.
  // {main: 'main.js', 'vendor/pkg1': 'npm-pkg-1', 'vendor/pkg2': 'npm-pkg-2'}
  input: ['main.js', 'npm-pkg-1', 'npm-pkg-2'],
  plugins: [resolve()],
  output: [{
    dir: 'dist',
    format: 'esm',

    // fine-grained control over the generated names; note that the "entryFileNames" option
    // is now the more relevant one
    entryFileNames: '[name]-[hash].js',
    chunkFileNames: '[name]-[hash].js'
  }]
});

Example where two chunks are created

Example where more chunks are created

If you want to make sure that no additional chunks are created, manualChunks are the way to go. It would be interesting to know why they did not work for you, this is what worked for me:

import resolve from 'rollup-plugin-node-resolve';

export default ({
  input: 'main.js',
  plugins: [resolve()],
  manualChunks: {'pkg-1': ['npm-pkg-1'], 'pkg-2': ['npm-pkg-2']}
  output: [{
    dir: 'dist',
    format: 'esm'
  }]
});

Note that you definitely need the node-resolve plugin!

[philipwalton]

Thanks for looking into this:

The simplest option is to just designate npm-pkg-1 and npm-pkg-2 as additional entry points.

In my case this either doesn't work or isn't ideal for a few reasons:

• It requires me to know and enumerate all of my node_module dependencies. I'd prefer to be able to construct that list at runtime.
• It requires all my dependencies to themselves be entry points, but that's not the case for all npm dependencies, as this error shows (it's an npm package with multiple entry points, so there's no "main" or "module" field):

[!] Error: Could not resolve entry (idlize)
Error: Could not resolve entry (idlize)
    at error (/Users/philipwalton/Projects/philipwalton/blog/node_modules/rollup/dist/rollup.js:3597:30)
    at /Users/philipwalton/Projects/philipwalton/blog/node_modules/rollup/dist/rollup.js:17134:17

It would be interesting to know why they did not work for you, this is what worked for me:

This is a nice example (I wish the docs had examples!). I wasn't actually aware that you could list raw package names in the module list. But even then this unfortunately doesn't quite work for me.

The problems here are effectively the same as the problems above: first, It requires me to know and enumerate all of my node_module dependencies. And even if I do know that, I have to know the order in which these dependencies resolve, or I get an error like this:

[!] Error: Cannot assign node_modules/dom-utils/index.js to the "npm.dom-utils" chunk as it is already in the "npm.autotrack" chunk.
Try defining "npm.dom-utils" first in the manualChunks definitions of the Rollup configuration.

While it's possible for me to manually work around this particular problem, it'd be a lot nicer if I could declare my manual chunks programmatically.

And the second problem, again, is not all my modules have entry points, so if they don't then I have to individually list the modules I want to belong to a particular chunk.

All this requires me to know a lot about my dependencies (and their dependencies).

---

Would it be possible to add the feature proposal I outlined above? Where the manualChunks option could take a function that gets called whenever a new module dependency is discovered, and the return value of that function would be the chunkAlias (and not returning anything could default to the entry chunk, so another dynamic chunk).

[philipwalton]

It's it's helpful, I was able to programmatically get Rollup to do what I want, but in order to do it I had to run Rollup twice: a first time to generate a module graph I could use to create manualChunks, and then a second time using the manualChunks object I'd just created.

Here's an example I though together (not super optimized, but maybe it'll help clarify what I'm trying to do):

const {rollup} = require('rollup');
const resolve = require('rollup-plugin-node-resolve');

const tSort = (nodes) => {
  const sorted = [];
  const seen = new Set();
  const visit = (node) => {
    if (!seen.has(node)) {
      for (const dep of node.deps) {
        visit(dep);
      }
      seen.add(node);
      sorted.unshift(node);
    }
  }
  for (const node of nodes.values()) {
    visit(node);
  }
  return sorted;
}

const tSortRollupModules = (rollupModules) => {
  // Reverse the dependency graph generated by Rollup.
  // I.e. rather that a list of modules and their dependencies, make a list
  // of modules and the other modules who depend on them.
  const modules = new Map();
  const getOrCreateModule = (id) => {
    if (!modules.has(id)) {
      modules.set(id, {id, deps: new Set()})
    }
    return modules.get(id);
  };
  for (const {id, dependencies} of rollupModules) {
    for (const dep of dependencies) {
      getOrCreateModule(dep).deps.add(getOrCreateModule(id));
    }
  };
  return tSort(modules);
}

(async () => {
  // Do the prebundle to generate the dependency graph.
  const prebundle = await rollup({
    input: 'assets/javascript/main.js',
    plugins: [resolve()],
  });

  const modules = tSortRollupModules(prebundle.cache.modules);
  const manualChunks = {};

  const NODE_MODULE = /node_modules\/([^/]+)/;
  for (const mod of modules) {
    if (mod.id.match(NODE_MODULE)) {
      const chunk = RegExp.$1;
      let chunkMods = manualChunks[chunk] || [];

      // Delete the key, so when it's re-added it goes at the end.
      // A bit of a hack, but keys in `manualChunks` need to be ordered.
      if (chunk in manualChunks) {
        delete manualChunks[chunk];
      }
      manualChunks[chunk] = chunkMods
      chunkMods.push(mod.id);
    }
  }

  const bundle = await rollup({
    input: 'assets/javascript/main.js',
    plugins: [
      resolve(),
    ],
    manualChunks,
  });

  await bundle.write({
    dir: 'dist',
    format: 'esm',
    chunkFileNames: '[name]-[hash].mjs',
    entryFileNames: '[name]-[hash].mjs',
  });
})();

It's a fairly involved process to generate manualChunks correctly because Rollup is very picky about both the ordering of keys and values in that object (as you can see I had to sort the dependencies myself). It'd be great if correct ordering could be inferred by Rollup.

And ideally, it'd be possible to do this in a single invocation of rollup.bundle().

In case you're curious, here's what that object looked like when run for my blog:

{ 'workbox-core':
   [ '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-core/_version.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-core/_private/logger.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-core/_private/Deferred.mjs' ],
  idlize:
   [ '/Users/philipwalton/Projects/philipwalton/blog/node_modules/idlize/lib/queueMicrotask.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/idlize/lib/now.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/idlize/idle-callback-polyfills.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/idlize/IdleValue.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/idlize/IdleQueue.mjs' ],
  'dom-utils':
   [ '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/lib/parse-url.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/lib/parents.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/lib/matches.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/lib/get-attributes.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/lib/dispatch.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/lib/closest.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/lib/delegate.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/dom-utils/index.js' ],
  autotrack:
   [ '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/event-emitter.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/method-chain.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/constants.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/usage.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/tracker-queue.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/utilities.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/store.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/session.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/provide.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/url-change-tracker.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/page-visibility-tracker.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/media-query-tracker.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/impression-tracker.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/outbound-link-tracker.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/max-scroll-tracker.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/clean-url-tracker.js',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/autotrack/lib/plugins/event-tracker.js' ],
  'workbox-window':
   [ '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-window/_version.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-window/utils/WorkboxEvent.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-window/utils/urlsMatch.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-window/utils/EventTargetShim.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-window/messageSW.mjs',
     '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-window/Workbox.mjs' ] }

[Xazzzi]

Rollup is very picky about both the ordering of keys and values in that object

So i recently ran into the same issue when trying to factor-out my vendor chunk in react application, like so:

import { dependencies } from './package.json';
// and then later in rollup config
manualChunks: {
  dependencies: Object.keys(dependencies),
},

Error was Cannot assign node_modules/prop-types/index.js to the "dependencies" chunk as it is already in the "dependencies" chunk.. This makes no sense to me, as if something is already in the chunk - why would bundler complain about it and not just reuse what's already there?
So i peeked into the source for where the error was thrown and made a small change to skip entries with chunkAlias that is the same as one we'd like to assign:

    if (manualChunkModules) {
        for (var _i = 0, _d = Object.keys(manualChunkModules); _i < _d.length; _i++) {
            var chunkName = _d[_i];
            currentEntryHash = randomUint8Array(10);
            for (var _e = 0, _f = manualChunkModules[chunkName]; _e < _f.length; _e++) {
                currentEntry = _f[_e];

+                if (currentEntry.chunkAlias === chunkName) {
+                  continue;
+                }

                if (currentEntry.chunkAlias) {
                    error({
                        code: 'INVALID_CHUNK',
                        message: "Cannot assign " + path.relative(process.cwd(), currentEntry.id) + " to the \"" + chunkName + "\" chunk as it is already in the \"" + currentEntry.chunkAlias + "\" chunk.\nTry defining \"" + chunkName + "\" first in the manualChunks definitions of the Rollup configuration."
                    });
                }
                currentEntry.chunkAlias = chunkName;
                modulesVisitedForCurrentEntry = (_a = {}, _a[currentEntry.id] = true, _a);
                addCurrentEntryColourToModule(currentEntry);
            }
        }
    }

My app seems to build and work just file with this tweak, but i'm worried that next two lines maybe did something important:

modulesVisitedForCurrentEntry = (_a = {}, _a[currentEntry.id] = true, _a);
addCurrentEntryColourToModule(currentEntry);

But on a first glance, if we already marked currentEntry with same chunkAlias earlier, then theese two lines were already executed earlier as well, and it seems more or less safe to just skip an iteration.

[guybedford]

@Xazzzi that does sound like a bug, please post a PR if you can.

Ideally we should make sure we are doing a unique comparison though, not based on chunkAlias check, but rather checking the chunk by its unique Id.

[lukastaegert]

@xazzi #2809 should also solve your issue with manual chunks

[lukastaegert]

@philipwalton I created #2831 which allows manual chunks to be defined as a function. This should hopefully allow you to solve your use case.

[fabd]

EDIT: I solved my question. I was getting duplicate entries. The solution to assign some modules to the entry where they are used, is to explicitly return the name of the entry as a string, for those root level modules. If I used just return when there are no importers, they are somehow assigned a different file.

../web/build/dist/assets/entry-account.fdfe0768.js    0.18kb / brotli: 0.15kb
../web/build/dist/assets/entry-account.a66d065d.js    3.45kb / brotli: 1.30kb

edit: correct code below in case that is useful to anybody, it creates a VENDOR, and a COMMONS bundle. COMMONS includes all shared code, instead of having multiple smaller files. So I end up for any given page in my old php app, I have a VENDOR, COMMONS, an entry-something.js, and the corresponding CSS. This keeps the javascript/stylesheet includes to a minimum, while taking advantage of shared bundles.

      input: [
        "./src/entry-account.ts",
        "./src/entry-landing.ts",
        // etc
      ],

manualChunks: (id, { getModuleInfo }) => {
          if (/\/node_modules\//.test(id)) {
            console.log('%d : "%s" goes into VENDOR', nr, id);
            return "VENDOR";
          }

          const entryPoints = [];

          // We use a Set here so we handle each module at most once. This
          // prevents infinite loops in case of circular dependencies
          const idsToHandle = new Set(getModuleInfo(id).importers);

          for (const moduleId of idsToHandle) {
            const { isEntry, importers } = getModuleInfo(
              moduleId
            );
            if (isEntry) {
              entryPoints.push(moduleId);
            }

            // The Set iterator is intelligent enough to iterate over elements that
            // are added during iteration
            for (const importerId of importers) idsToHandle.add(importerId);
          }

          // This is an entry (root level)
          if (entryPoints.length === 0) {
            let entryName = `${id.split('/').slice(-1)[0].split('.')[0]}`;
            console.log('%d : "%s" is the ENTRY %s', nr, id, entryName);
            return entryName;
          }

          // If there is a unique entry, we bundle the code with that entry
          if (entryPoints.length === 1) {
            let entryName = `${entryPoints[0].split('/').slice(-1)[0].split('.')[0]}`;
            console.log('"%s" goes into UNIQUE ENTRY %s', id, entryName);
            return entryName;
          }

          // For multiple entries, we put it into a "shared code" bundle
          if (entryPoints.length > 1) {
            console.log('"%s" goes into COMMONS (non-vendor) chunk', id);
            return 'common';
          }
        },

[lukastaegert]

You need preserveEntrySignatures: "allow-extension", otherwise facade chunks are created.

[murtuzaalisurti]

@lukastaegert thanks, this worked for me!

import resolve from 'rollup-plugin-node-resolve';

export default ({
  input: 'main.js',
  plugins: [resolve()],
  output: [{
    dir: 'dist',
    format: 'esm',
    // note that manualChunks should now be used inside the output options!
    manualChunks: {'pkg-1': ['npm-pkg-1'], 'pkg-2': ['npm-pkg-2']},
    chunkFileNames: "[name].js"
  }]
});