Concurrent generate calls share a mutable plugin driver

[marijnh]

• Rollup Version: 1.25.0
• Operating System (or Browser): Any
• Node Version: 12.10

How Do We Reproduce?

Create a plugin with a generateBundle hook that calls this.emitFile({type: "asset", fileName: "myfile", source: "abc"}).

Enable the plugin and run rollup on some code with multiple output targets. For example with this config file:

export default {
  input: "./bar.js",
  output: [
    {
      file: "bar1.js",
      format: "cjs"
    },
    {
      file: "bar2.js",
      format: "esm"
    }
  ],
  plugins: [{
    generateBundle() {
      this.emitFile({type: "asset", fileName: "myfile", source: "abc"})
    }
  }]
}

Running rollup will produce this error:

[!] (plugin at position 1) Error: Could not emit file "myfile" as it conflicts with an already emitted file.
Error: Could not emit file "myfile" as it conflicts with an already emitted file.
    at error (/home/marijn/tmp/rollup-plugin-typescript2/node_modules/rollup/dist/rollup.js:10162:30)
    at reserveFileNameInBundle (/home/marijn/tmp/rollup-plugin-typescript2/node_modules/rollup/dist/rollup.js:16226:16)
    at FileEmitter.emitAsset (/home/marijn/tmp/rollup-plugin-typescript2/node_modules/rollup/dist/rollup.js:16350:17)
    at Object.FileEmitter.emitFile (/home/marijn/tmp/rollup-plugin-typescript2/node_modules/rollup/dist/rollup.js:16280:29)
    at Object.generateBundle (/home/marijn/tmp/rollup-plugin-typescript2/foo.js:21:12)
    at /home/marijn/tmp/rollup-plugin-typescript2/node_modules/rollup/dist/rollup.js:16635:25

Which, as per the discussion in #3150, is not the intended behavior in this case.

[marijnh]

Digging into this, I'm seeing the code for generating the different output targets run concurrently, even though they share things like the plugin driver, which are mutable stateful things that are reset at the start of generate.

Are these supposed to run concurrently? If so, I don't think we can have a shared mutable plugin driver. (Specifically, the driver's file emitter's bundle property.)

[lukastaegert]

Ah yes, I may have overlooked things here. There is a startOutput function to separate the outputs, but this would only work if they are not overlapping. Which they are...

So fixing this is a larger refactoring that may take some time. It is in fact a goal to have separate pluginDrivers for the outputs to be able to have output-specific plugins. This may just speed this up.

For the original issue linking here, I guess the ignoreConflict option (+ making conflicts a warning) are acceptable short-term solutions.

[marijnh]

If this is an issue that'll be fixed in a more fundamental way soon, would temporarily disabling the error entirely be a better stopgap measure? Introducing an option that'll later just be a rudiment (I can't really imagine intentionally emitting a file multiple times in a single bundle) might be messy.

[marijnh]

(Or just pass the actual current bundle along to FileEmitter.emitFile, and have it check that instead of its own bundle object. Possibly even remove that bundle object and always pass it to everyone who needs it.)

[lukastaegert]

Or just pass the actual current bundle along to FileEmitter.emitFile, and have it check that instead of its own bundle object.

Fair enough :)

So from my side, I have a talk upcoming next week which means it might take me up to two weeks to fix this, depending on how much refactoring is involved. Maybe an approach where bundles are passed along can be implemented more quickly. But in the long term, I do want per-module plugins, so there might be a chance I would rather go for the full solution myself if it is possible with reasonable effort.

Disabling the error would be ok for the time being, though I really would want my warning back at some point.

[lukastaegert]

PR welcome I guess.

[marijnh]

I'll happily set up a PR. So, concretely, what would you prefer I do...

• disable the error, with the understanding that you'll re-enable it when you do the refactor

• pass bundle objects down to the plugin driver

[lukastaegert]

If you can pull off the latter, that would be awesome and prefered. Otherwise the first option. As for testing, it should be easily possible to check this scenario in a chunking-form test.

[marijnh]

Okay, I gave up on the bundle-passing idea since, indeed, that's harder than I thought and would involve half the refactor you alluded to to get anything working at all. So PR #3175 crudely disables the error.

[lukastaegert]

Thanks! I'll not be able to check it today but if it looks good, I'll release it as a patch the next days.

[marijnh]

(Updated the issue title to point at the underlying problem, now that the manifestation is fixed/worked around.)

[marijnh]

There's another problematic manifestation of this bug: if two outputs emit to different directories, only one of the output dirs will receive the files emitted by generateBundle hooks. Again, due to the accidental object sharing, all the emitted files end up in one output's bundle. The other doesn't get them at all. You can reproduce with a config like this:

export default {
  input: "index.js",
  output: [
    {
      dir: "dist1",
      format: "cjs"
    },
    {
      dir: "dist2",
      format: "esm"
    }
  ],
  plugins: [{
    generateBundle() {
      this.emitFile({type: "asset", fileName: "myfile", source: "abc"})
    }
  }]
}

[marijnh]

Would serializing the output generation be a good idea until this is properly fixed? I.e. don't actually run them concurrently?

[lukastaegert]

Yes, that definitely makes sense. PR welcome. Will still need 1-2 weeks to have something ready.

[marijnh]

Done in PR #3201

[wessberg]

Fixing this will fix #3098