[webpack 5] rebuild is twice as slow comparing to webpack 4

[meskill]

After upgrading to webpack 5 in our project we noticed that rebuild time increased more than 2 times comparing to webpack 4.

I've created a test repo https://github.com/meskill/webpack5-performance-issue to show the difference in rebuild time for webpack4 and webpack5.

It shows more than 2-time slowness of rebuild in webpack5 compare to webpack4. Profiling shows similar info

Cpuprofile for both webpack4 and webpack5

[alexander-akait]

Thank you for creating this issue. However, issues need to follow one of our templates so that we can clearly understand your particular circumstances.

Please help us help you by recreating the issue using one of our templates.

Please respect time of other developers

[Thebarda]

Hello, I have the same issue here. I have compared build time with Webpack 4 and Webpack 5 on a project using Speed Measure Plugin and here is the result :

As you can see, the time took by plugins ands loaders is about the same but Webpack 5 is twice slower than Webpack 4 is my case.
I can provide more informations if needed.

[sokra]

@Thebarda please attach cpu profiles

--progress profile also displays some timing information.

[alexander-akait]

I think bug in HtmlWebpackHarddiskPlugin

[Thebarda]

Here is the cpuprofile for a Webpack 4 build

And the cpuprofile for Webpack 5 build.

Indeed @alexander-akait, for the webpack 5 build I'm using html-webpack-plugin on next branch (html-webpack-plugin@5 alpha) that supports Webpack 5 and html-webpack-harddisk-plugin@1.0.2

[alexander-akait]

What is version of terser-webpack-plugin, please provide configuration, it is really hard each time to ask you about versions, options and etc

[alexander-akait]

Why you run rebuild in production mode?

[alexander-akait]

What is node version? Node@10 has regression in perf

[sokra]

@Thebarda Seems like you are talking about build performance and not rebuild performance. This issue is about rebuild performance, so seem like your issue is not similar to @meskill's issue, and I beg you to open an separate issue for your unrelated issue. It looks strongly like you are using persistent cache for terser-webpack-plugin in webpack 4 and measuring the cached build. While in webpack 5 you have to opt-in into persistent caching. Please make sure to remove node_modules/.cache before measuring.

[sokra]

@meskill i looked at your profile and noticed a few things:

version	change to update time	compile time
webpack 4	~850ms	~500ms
webpack 5	~1300ms	~950ms

So we are talking about 450ms extra for webpack 5.

• module graph validation takes about ~100ms extra in webpack 5
  • webpack 5 is using a safe cache. webpack 4 uses an unsafe cache
  • You can change that with module.unsafeCache: true, but I don't recommend that
• ModuleInfoHeaderPlugin costs about ~100ms extra in webpack 5
  • It prints more info about exports, etc.
  • You can disable that with output.pathinfo: false
  • We should add caching to this plugin
  • Do we really need that detailed information by default? Maybe boolean | "verbose" and default to simple information.
• Stats.toString takes about ~140ms extra in webpack 5
  • Seems like webpack 4 skips cached modules early, while webpack 5 extracts data for them before skiping them later
  • You can disable modules with stats.modules: false, but this would hide all modules
  • We should not extra data for hidden modules during stats generation
• In both versions the change is pushed pretty late to the web socket
  • watchers are applied before that. They cost about ~50ms (webpack 5) resp. ~15ms (webpack 4)
  • node-notifier is triggered in webpack 4. That costs about ~35ms
  • I wonder if we should add a delay to the watchers?
• Both versions wait 300ms after a file change before starting the build
  • This happens to merges multiple files changes that happen in fast sequence (like batch replace or whatever)
  • You could change that with watchOptions.aggregateTimeout: 50. Not sure if this worth it.

[meskill]

@sokra Thanks, for the suggestions, I will check it.

Also, I've changed the initial issue description with some simple repo to neglect our project-specific config - in this repo webpack4 and webpack5 use the same config for the react example project.

P.S.: yes we already fixed the problem with long stats.toJSON by setting all: false and enabling only needed information about the build what made stats.JSON pretty fast now

[keepitterron]

This is compatible with what I'm seeing in a large, multi entries app when I upgrade. Building is a bit slower (~18s to ~22s), rebuilding in watch mode is up to 3x slower.

Development mode

  optimization: {
    moduleIds: 'hashed',
    splitChunks: {
      cacheGroups: {
        default: false,
        commons: {
          name: 'commons',
          chunks: 'all',
          minChunks: 3,
        },
      },
    },
  },

webpack 4

[webpack-cli] Compilation starting...
<i> [webpack.Progress] 216ms building
<i> [webpack.Progress] 12ms finish module graph
<i> [webpack.Progress] 155ms chunk graph
<i> [webpack.Progress] 183ms advanced chunk optimization
<i> [webpack.Progress] 84ms before module ids
<i> [webpack.Progress] 23ms module id optimization
<i> [webpack.Progress] 142ms hashing
<i> [webpack.Progress] 12ms additional chunk assets processing
<i> [webpack.Progress] 41ms emitting
[webpack-cli] Compilation finished
Hash: ac5359db523b6792fc80
Version: webpack 4.44.1
Time: 917ms

webpack 5

[webpack-cli] Compilation starting...
<i> [webpack.Progress]  | 20 ms setup > compile
<i> [webpack.Progress]  |  | 14 ms compilation > AssetModulesPlugin
<i> [webpack.Progress]  | 27 ms setup > compilation
<i> [webpack.Progress] 50 ms setup
<i> [webpack.Progress]  | 608 ms building > entries dependencies modules
<i> [webpack.Progress]  | 506 ms building > entries dependencies modules
<w> [webpack.Progress] 1116 ms building
<i> [webpack.Progress]  |  | 85 ms finish module graph > FlagDependencyExportsPlugin
<i> [webpack.Progress]  | 147 ms sealing > finish module graph
<i> [webpack.Progress]  | 181 ms sealing > chunk graph
<i> [webpack.Progress]  |  | 224 ms chunk optimization > SplitChunksPlugin
<i> [webpack.Progress]  | 232 ms sealing > chunk optimization
<i> [webpack.Progress]  |  | 34 ms module ids > HashedModuleIdsPlugin
<i> [webpack.Progress]  | 35 ms sealing > module ids
<i> [webpack.Progress]  | 303 ms sealing > module hashing
<i> [webpack.Progress]  | 35 ms sealing > code generation
<i> [webpack.Progress]  | 31 ms sealing > runtime requirements
<i> [webpack.Progress]  | 52 ms sealing > hashing
<i> [webpack.Progress]  |  | 22 ms asset processing > copy-webpack-plugin
<i> [webpack.Progress]  | 22 ms sealing > asset processing
<i> [webpack.Progress]  | 11 ms sealing > after asset optimization
<w> [webpack.Progress] 1079 ms sealing
<i> [webpack.Progress]  | 38 ms emitting > emit
<i> [webpack.Progress] 38 ms emitting
99% done plugins webpack-cli[webpack-cli] Compilation finished
assets by status 86.9 MiB [cached] 352 assets
cached modules 24.1 MiB (javascript) 147 KiB (runtime) [cached] 5480 modules
javascript modules 2.79 KiB
  ./assets/img/logos/ sync ^\.\/.*$ 870 bytes [built]
  ./src/client/js/components/ui-components/emoji/index.js 1.43 KiB [built]
  ./assets/img/video-player/ sync ^\.\/.*.*\.png$ 530 bytes [built]

webpack 5.10.0 compiled with 1 warning in 2283 ms

[alexander-akait]

Why you use splitChunks in development mode?

[keepitterron]

@alexander-akait I see non-trivial improvements in CPU and memory usage using split chunks in development with webpack4.

Anyways, taking optimization out in dev mode changes nothing (I rm -rf node_modules for good measure before running the test)

[webpack-cli] Compilation starting...
<i> [webpack.Progress]  | 23 ms setup > compile
<i> [webpack.Progress]  |  | 12 ms compilation > EntryPlugin
<i> [webpack.Progress]  | 15 ms setup > compilation
<i> [webpack.Progress] 38 ms setup
<i> [webpack.Progress]  | 763 ms building > entries dependencies modules
<i> [webpack.Progress]  | 34 ms building > entries dependencies modules
<i> [webpack.Progress] 797 ms building
<i> [webpack.Progress]  |  | 174 ms finish module graph > FlagDependencyExportsPlugin
<i> [webpack.Progress]  | 271 ms sealing > finish module graph
<i> [webpack.Progress]  |  | 38 ms plugins > WarnCaseSensitiveModulesPlugin
<i> [webpack.Progress]  | 39 ms sealing > plugins
<i> [webpack.Progress]  | 293 ms sealing > chunk graph
<i> [webpack.Progress]  |  | 12 ms chunk optimization > EnsureChunkConditionsPlugin
<i> [webpack.Progress]  |  | 343 ms chunk optimization > SplitChunksPlugin
<i> [webpack.Progress]  | 356 ms sealing > chunk optimization
<i> [webpack.Progress]  |  | 25 ms module ids > NamedModuleIdsPlugin
<i> [webpack.Progress]  | 26 ms sealing > module ids
<i> [webpack.Progress]  | 625 ms sealing > module hashing
<i> [webpack.Progress]  | 58 ms sealing > code generation
<i> [webpack.Progress]  | 104 ms sealing > runtime requirements
<i> [webpack.Progress]  | 90 ms sealing > hashing
<i> [webpack.Progress]  |  | 12 ms asset processing > copy-webpack-plugin
<i> [webpack.Progress]  | 12 ms sealing > asset processing
<i> [webpack.Progress]  | 18 ms sealing > after asset optimization
<w> [webpack.Progress] 1922 ms sealing
<i> [webpack.Progress]  | 29 ms emitting > emit
<i> [webpack.Progress] 29 ms emitting
99% done plugins webpack-cli[webpack-cli] Compilation finished
assets by status 622 MiB [cached] 354 assets
cached modules 24.1 MiB (javascript) 57.6 KiB (runtime) [cached] 5415 modules
./src/client/js/components/ui-components/emoji/index.js 1.43 KiB [built]
webpack 5.10.0 compiled successfully in 2787 ms

I'm really not sure which way to go. Webpack 4 is MUCH faster on rebuilds.
I will try to put together a reproducible repo but it's gonna be hard, this is a big private app.

[keepitterron]

On splitChunks in dev mode I see other people ran into the same issue and used the same solution:
#8644 (comment)

My reason was that when you have chunking-by-default enabled, a change of a single source file causes a single, small chunk+sourcemap to be re-rendered, emitting e.g. ~100KB of code.

On the other hand, when you have a single chunk which is e.g. 50MB and you always have to emit it (and its sourcemap, which is especially expensive). This also was problematic for browsers, as they would choke while loading huge sourcemaps like that.

[joshxyzhimself]

Aside from worse compile times, has anyone noticed the larger memory build-up?

On subsequent edits and re-bundling of my code my ram usage just keeps stacking up.

[alexander-akait]

@joeldenning with webpack-dev-server or simple bundle?

[joeldenning]

@joeldenning with webpack-dev-server or simple bundle?

I think you may have tagged me instead of @joshxyzhimself 😄

[joshxyzhimself]

😸 😸 😸

@alexander-akait just simple bundle sir

edit file -> recompile -> ram usage increase
edit file again -> recompile -> ram usage increase
edit file again -> recompile -> ram usage increase

current workaround i do is kill the process and rerun it

i was thinking of writing a plugin that exposes and calls global.gc() but there may be other ways i'm not aware of? 😸

const fs = require('fs');
const os = require('os');
const path = require('path');
const assert = require('assert');
const process = require('process');
const webpack = require('webpack');
const TerserPlugin = require('terser-webpack-plugin');

const process_cwd = process.cwd();

const chunks_path = path.join(process_cwd, '.chunks.json');
const output_path = path.join(process_cwd, '/client-dist');

class chunk_extract_plugin {
  constructor(mode) {
    assert(typeof mode === 'string');
    this.mode = mode;
  }
  apply(compiler) {
    compiler.hooks.done.tap('Hello World Plugin', (stats) => {

      const chunks_set = new Set();

      if (this.mode === 'production') {
        stats.compilation.chunks.forEach((chunk) => {
          chunk.files.forEach((file) => {
            if (path.extname(file) === '.js') {
              chunks_set.add(file);
            }
          });
        });
      }

      stats.compilation.entrypoints.forEach((entrypoint) => {
        entrypoint.chunks.forEach((chunk) => {
          chunk.files.forEach((file) => {
            if (path.extname(file) === '.js') {
              chunks_set.add(file);
            }
          });
        });
      });

      const chunks_array = Array.from(chunks_set);

      fs.writeFileSync(chunks_path, JSON.stringify(chunks_array, null, 2));
    });
  }
}

const Client = (env, argv) => {
  const { mode } = argv;
  return {
    devtool: mode === 'development' ? 'source-map' : false,
    entry: ['./client/client.jsx'],
    resolve: {
      extensions: ['.js', '.jsx'],
    },
    module: {
      rules: [
        {
          enforce: 'pre',
          test: /\.(js|jsx)$/,
          exclude: /node_modules/,
          use: [
            {
              loader: 'eslint-loader',
              options: {
                fix: true,
                cache: true,
                emitWarning: true,
              },
            },
          ],
        },
        {
          test: /\.(js|jsx)$/,
          use: 'babel-loader',
          exclude: /node_modules/,
        },
        {
          test: /\.(css)$/,
          use: [
            { loader: 'style-loader' },
            { loader: 'css-loader', options: { importLoaders: 1 } },
            { loader: 'postcss-loader' },
          ],
        },
        {
          test: /\.(woff(2)?|ttf|eot|otf|svg)(\?v=\d+\.\d+\.\d+)?$/,
          use: [
            {
              loader: 'file-loader',
              options: {
                name: '[name].[ext]',
                outputPath: 'fonts/',
              },
            },
          ],
        },
      ],
    },
    plugins: [
      new webpack.DefinePlugin({ ENVIRONMENT: JSON.stringify(mode) }),
      new chunk_extract_plugin(mode),
    ],
    optimization: {
      runtimeChunk: true,
      splitChunks: { chunks: 'all' },
      minimize: mode === 'production',
      minimizer: [
        new TerserPlugin({
          parallel: os.cpus().length,
          terserOptions: {
            output: { comments: false },
            compress: { dead_code: true },
            mangle: true,
          },
        }),
      ],
    },
    output: {
      filename: mode === 'production' ? '[chunkhash].js' : '[name].js',
      chunkFilename: mode === 'production' ? '[chunkhash].js' : '[name].js',
      path: output_path,
      publicPath: '/scripts/',
    },
    stats: 'minimal',
  };
};

module.exports = [Client];

[joshxyzhimself]

is webpack the new slack? 😸

[sokra]

@joshxyzhimself if you create a new issue with a repro repo, we can help to find the memory leak. Please create a new issue as this is unreleated to the rebuild performance issue.

[meskill]

@sokra hello, I've tested some of the settings you suggested and it improved time a little bit, but webpack4 still beats webpack5 regarding rebuild time. changes

Does webpack team have any plans to improve performance for the rebuild in near future?

[alexander-akait]

@meskill do you update webpack to the latest 5 version? Maybe you can provide profile file?

[alexander-akait]

@meskill can you try latest webpack version and put feedback more here?

[alexander-akait]

@keepitterron Can you do the same and put feedback here, thanks

[alexander-akait]

defaultWebpack: ['webpack/lib/'] why you use it? webpack invalidate caches after update version, you should not do it. Can you try to disable cacheUnaffected? backCompat can be disable if you use plugins/loaders for webpack v5.

But HRM looks for a lot of time in module building

How many?

[xyxiao001]

I tried to disable cacheUnaffected backCompat，waiting for me to try

[xyxiao001]

webpack 5.65.0
config

cache: isProd ? false : {
      name: `${createHash(includeChunks)}-${includeChunks}`,
      type: 'filesystem',
 },

config.experiments = {
      lazyCompilation: {
         entries: true,
          imports: true,
      },
      futureDefaults: true,
      outputModule: true,
      css: true,
 };
 config.module.unsafeCache = true;
 config.output.pathinfo = false;
 config.stats.all = false;
 config.optimization.splitChunks = false;
 config.optimization.sideEffects = false;
 config.optimization.providedExports = false;

devserve

DONE  Compiled successfully in 462273ms                                                                            
 SMP  ⏱  
General output time took 7 mins, 42.3 secs
 SMP  ⏱  Loaders
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
@vue/cli-service took 4 mins, 4.78 secs
  module count = 1769
babel-loader, and 
cache-loader, and 
@vue/cli-service took 3 mins, 10.77 secs
  module count = 1711
@vue/vue-loader-v15, and 
cache-loader, and 
@vue/cli-service took 3 mins, 2.9 secs
  module count = 3388
cache-loader, and 
@vue/cli-service took 2 mins, 16.96 secs
  module count = 1711
babel-loader took 1 min, 39.004 secs
  module count = 686
modules with no loaders took 1 min, 30.28 secs
  module count = 3075
svg-sprite-loader, and 
svgo-loader took 1 min, 25.69 secs
  module count = 172
css-loader, and 
postcss-loader, and 
sass-loader took 1 min, 3.94 secs
  module count = 8
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
cache-loader, and 
@vue/cli-service took 48.27 secs
  module count = 12
svgo-loader took 44.67 secs
  module count = 220
css-loader, and 
postcss-loader took 19.99 secs
  module count = 30
@vue/vue-loader-v15, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
@vue/cli-service took 3.45 secs
  module count = 3538
cache-loader, and 
font-cloak took 2.095 secs
  module count = 10
@vue/vue-loader-v15, and 
babel-loader, and 
cache-loader, and 
@vue/cli-service took 0.795 secs
  module count = 1711
vue-style-loader, and 
css-loader, and 
postcss-loader took 0.023 secs
  module count = 30
html-webpack-plugin took 0.016 secs
  module count = 1
@vue/vue-loader-v15, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
cache-loader, and 
@vue/cli-service took 0.012 secs
  module count = 24
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader took 0.009 secs
  module count = 8

hrm

 DONE  Compiled successfully in 6785ms                                                                                   
 SMP  ⏱  
General output time took 6.78 secs

 SMP  ⏱  Loaders
cache-loader, and 
@vue/cli-service took 1.29 secs
  module count = 1
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
@vue/cli-service took 0.434 secs
  module count = 1
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
cache-loader, and 
@vue/cli-service took 0.388 secs
  module count = 1
cache-loader, and 
@vue/vue-loader-v15, and 
cache-loader, and 
@vue/cli-service took 0.379 secs
  module count = 1
babel-loader, and 
cache-loader, and 
@vue/cli-service took 0.366 secs
  module count = 1

webpack 4.42.1
devserve

 DONE  Compiled successfully in 326894ms                                                                                                            下午9:05:35



 SMP  ⏱  
General output time took 5 mins, 26.91 secs
 SMP  ⏱  Plugins
ModuleFederationPlugin took 5 mins, 10.84 secs
 SMP  ⏱  Loaders
css-loader, and 
vue-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 5 mins, 4.26 secs
  module count = 1771
cache-loader, and 
vue-loader took 4 mins, 48.66 secs
  module count = 5108
vue-loader, and 
cache-loader, and 
babel-loader, and 
cache-loader, and 
vue-loader took 4 mins, 37.51 secs
  module count = 3428
cache-loader, and 
babel-loader took 2 mins, 28.69 secs
  module count = 696
svg-sprite-loader, and 
svgo-loader took 2 mins, 17.41 secs
  module count = 172
css-loader, and 
postcss-loader, and 
sass-loader took 1 min, 52.3 secs
  module count = 8
modules with no loaders took 1 min, 50.96 secs
  module count = 2338
css-loader, and 
vue-loader, and 
postcss-loader, and 
cache-loader, and 
vue-loader took 50.4 secs
  module count = 12
file-loader, and 
svgo-loader took 46.29 secs
  module count = 220
url-loader took 26.075 secs
  module count = 723
css-loader, and 
postcss-loader took 24.83 secs
  module count = 28
vue-loader, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 3.013 secs
  module count = 3542
url-loader, and 
cache-loader, and 
font-cloak took 1.38 secs
  module count = 10
raw-loader took 0.652 secs
  module count = 5
html-webpack-plugin took 0.06 secs
  module count = 1
vue-style-loader, and 
css-loader, and 
postcss-loader took 0.053 secs
  module count = 28
vue-loader, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
cache-loader, and 
vue-loader took 0.018 secs
  module count = 24
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader took 0.013 secs
  module count = 8

hrm

 DONE  Compiled successfully in 4424ms                                                                                                              下午9:06:30



 SMP  ⏱  
General output time took 4.42 secs
 SMP  ⏱  Plugins
ModuleFederationPlugin took 0.198 secs
 SMP  ⏱  Loaders
cache-loader, and 
vue-loader took 0.491 secs
  module count = 2
css-loader, and 
vue-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 0.312 secs
  module count = 2
cache-loader, and 
babel-loader, and 
cache-loader, and 
vue-loader took 0.201 secs
  module count = 1

After the upgrade, HRM spends more time in building module

[alexander-akait]

I don't know what is @vue/cli-service, but it is very slow on initial build, very... Also svgo-loader eats very many time (do you need it in develoment mode ? I think it is useful only for production). Also check the mode option? Do you use development for development, because production requires more time?

[alexander-akait]

Also name: ${createHash(includeChunks)}-${includeChunks}, is the same during multiple runs without modification configuration files? How you calculate it?

[xyxiao001]

@vue/cli-service is the cli provided by Vue, encapsulated some configuration items.

[xyxiao001]

Also name: ${createHash(includeChunks)}-${includeChunks}, is the same during multiple runs without modification configuration files? How you calculate it?

Cache by the imported module name, can start a module separately

[xyxiao001]

Thank you for answering, I am trying to remove the vue-cli, use webpack5 directly,

[alexander-akait]

Cache by the imported module name, can start a module separately

It should be deteriorated between different builds, i.e. if you run webpack and then stop and then run again, the value should be same

[VishalBhandare-Apertumo]

no recent communication on this issue.
is this fixed or will never be solved?
can someone suggest alternative for problem?
few configuration changes are not making webpack5 build faster than webpack4.

[alexander-akait]

@VishalBhandare-Apertumo Please provide profile as minimum (examples are above)

[VishalBhandare-Apertumo]

We downgraded to webpack4 as seems there was this issue with webpack5. i have moved to new work now. But yes i need to find solution for this problem soon or later. i will communicating in this thread when i start working on fixing this.

@alexander-akait you have been very actively helping people here. thanks a lot.

I will get back to you while debugging issue if required.

For Vue Cli users linking references
https://github.com/vuejs/vue-cli/issues/6212

[alexander-akait]

Yes, will be great to profile and see there is perf problem (maybe in some loaders/plugins?) Feel free to ping me when you will do it

[FishOrBear]

Does ts-loader not support cache?
If I remove the cache-loader, the build time becomes 30s, if I add the cache-laoder, it will be within 10s

   "webpack": "^5.72.0",
        "webpack-cli": "^4.9.2",
        "webpack-dev-server": "^4.8.1",
        "webpack-merge": "^5.8.0",
        "webpackbar": "^5.0.2",

        "ts-loader": "^9.2.8",

[vankop]

cache-loader is deprecated, you don't need it with webpack@5

[noppa]

@FishOrBear Have you tried with Webpack's own filesystem cache? It's off (or rather in-memory only) by default.

  cache: {
    type: 'filesystem',
    compression: 'gzip',
  },

should speed up subsequent starts considerably

[FishOrBear]

I have started in webpack.common.ts

const config: webpack.Configuration = {
     cache: {
         type: 'filesystem',
     },

[FishOrBear]

Thanks for your reply, I know it's been stopped for maintenance, but if I don't use it(cache-loader), it's slow.

[sibelius]

lazy compilation makes compilation faster

should we close this issue ?

and open new issues more specific ?

[alexander-akait]

@sibelius this issue for tracking perf problems, because in some places we can have potentials improvements or bugs/memory leaks (but yes lazy compilation do your compiler faster)

[meskill]

We noticed the big build/rebuild slowdowns on the big applications with a lot of dynamic imports.
The most noticeable time is spent in visitModules block (more than 4s).

Webpack logs with dynamic imports

      100% (31840 affected + 0 infected of 31840) modules flagged as affected (31840 new modules, 0 changed, 0 references changed, 0 unchanged, 0 were not built)
   compute affected modules: 445.484298 ms
   finish modules: 19.2084 ms
   report dependency errors and warnings: 427.454098 ms
   optimize dependencies: 0.3432 ms
   create chunks: 4727.013482 ms
      0% modules flagged as affected by chunk graph (31840 new modules, 0 changed, 0 unchanged)
   compute affected modules with chunk graph: 217.759399 ms
   optimize: 701.219197 ms
      27958 modules hashed, 0 from cache (0.88 variants per module in average)
   module hashing: 1225.826896 ms
      100% code generated (27956 generated, 0 from cache)
   code generation: 2767.025084 ms
   runtime requirements.modules: 229.206198 ms
   runtime requirements.chunks: 709.575096 ms
   runtime requirements.entries: 101.976499 ms
   runtime requirements: 1044.298593 ms
   hashing: initialize hash: 0.0097 ms
   hashing: sort chunks: 0.8831 ms
   hashing: hash runtime modules: 7641.072852 ms
   hashing: hash chunks: 2737.142484 ms
   hashing: hash digest: 1.9061 ms
   hashing: process full hash modules: 0.0078 ms
   hashing: 10394.245136 ms
      100% code generated (23 generated, 0 from cache)
   record hash: 0.1943 ms
   module assets: 23.545 ms
   create chunk assets: 23352.211563 ms
   process assets: 13429.300045 ms
LOG from webpack.Compiler

 make hook: 515618.904476 ms

 finish make hook: 320.076499 ms

 finish compilation: 894.234496 ms

 seal compilation: 58097.3293 ms

 afterCompile hook: 1.5714 ms

 emitAssets: 37881.879832 ms

 emitRecords: 2.1531 ms
LOG from webpack.ResolverCachePlugin

29% really resolved (46940 real resolves with 0 cached but invalid, 113980 cached valid, 0 concurrent)
LOG from webpack.buildChunkGraph

 visitModules: prepare: 43.4525 ms

 visitModules: visiting: 118.5766 ms

 visitModules: calculating available modules: 1.5126 ms

 visitModules: merging available modules: 0.5015 ms

 visitModules: check modules for revisit: 0.2055 ms

 visitModules: prepare: 43.6924 ms

 visitModules: visiting: 172.638099 ms

 visitModules: calculating available modules: 0.4296 ms

 visitModules: merging available modules: 0.7218 ms

 visitModules: check modules for revisit: 0.1626 ms

 visitModules: prepare: 98.670599 ms

 visitModules: visiting: 4121.603685 ms

 visitModules: calculating available modules: 129.659099 ms

 visitModules: merging available modules: 90.8755 ms

 visitModules: check modules for revisit: 0.0466 ms

 visitModules: prepare: 0.6327 ms

 visitModules: visiting: 7.5088 ms

2377412 queue items processed (1183284 blocks)

212 chunk groups connected

184 chunk groups processed for merging (212 module sets, 3 forked, 17887 + 0 modules forked, 25618 + 0 modules merged into fork, 17801 resulting modules)

184 chunk group info updated (0 already connected chunk groups reconnected)

 visitModules: 4646.700283 ms

 connectChunkGroups: 1.2897 ms

 cleanup: 0.5593 ms
LOG from webpack.FileSystemInfo

74917 new snapshots created

0% root snapshot uncached (0 / 113980)

0% children snapshot uncached (0 / 0)

0 entries tested

File info in cache: 68593 timestamps 0 hashes 0 timestamp hash combinations

File timestamp snapshot optimization: 82% (538990/660274) entries shared via 2865 shared snapshots (152815 times referenced)

Directory info in cache: 1 timestamps 0 hashes 0 timestamp hash combinations

Directory timestamp snapshot optimization: 0% (0/1) entries shared via 0 shared snapshots (0 times referenced)

Missing items snapshot optimization: 73% (238747/329206) entries shared via 4374 shared snapshots (60172 times referenced)

Managed items info in cache: 0 items
LOG from webpack.Watching

 done hook: 190.095494 ms

 storeBuildDependencies: 3.4207 ms

 beginIdle: 3.9957 ms

And if I disable any dynamic imports by using babel-plugin-dynamic-import-node then I'll get slightly less time spent in visitModules (less than 0.5 s)

Webpack logs with babel-dynamic-import

100% (31840 affected + 0 infected of 31840) modules flagged as affected (31840 new modules, 0 changed, 0 references changed, 0 unchanged, 0 were not built)
   compute affected modules: 296.223434 ms
   finish modules: 15.761897 ms
   report dependency errors and warnings: 307.241732 ms
   optimize dependencies: 0.3177 ms
   create chunks: 544.269179 ms
      0% modules flagged as affected by chunk graph (31840 new modules, 0 changed, 0 unchanged)
   compute affected modules with chunk graph: 213.570052 ms
   optimize: 643.424658 ms
      27958 modules hashed, 0 from cache (0.88 variants per module in average)
   module hashing: 1230.355427 ms
      100% code generated (27956 generated, 0 from cache)
   code generation: 4562.625918 ms
   runtime requirements.modules: 216.952694 ms
   runtime requirements.chunks: 38.561599 ms
   runtime requirements.entries: 4.5616 ms
   runtime requirements: 262.439893 ms
   hashing: initialize hash: 0.0094 ms
   hashing: sort chunks: 0.1772 ms
   hashing: hash runtime modules: 42.491898 ms
   hashing: hash chunks: 422.087988 ms
   hashing: hash digest: 0.9766 ms
   hashing: process full hash modules: 0.0032 ms
   hashing: 470.203986 ms
      100% code generated (16 generated, 0 from cache)
   record hash: 0.1962 ms
   module assets: 18.619199 ms
   create chunk assets: 2269.294433 ms
   process assets: 877.657474 ms
LOG from webpack.Compiler

 make hook: 538650.482171 ms

 finish make hook: 1342.499603 ms

 finish compilation: 620.990163 ms

 seal compilation: 11285.573424 ms

 afterCompile hook: 1.5306 ms

 emitAssets: 6284.810903 ms

 emitRecords: 1.3997 ms
LOG from webpack.ResolverCachePlugin

29% really resolved (46771 real resolves with 0 cached but invalid, 113632 cached valid, 0 concurrent)
LOG from webpack.buildChunkGraph

 visitModules: prepare: 227.601256 ms

 visitModules: visiting: 474.178495 ms

56451 queue items processed (27958 blocks)

0 chunk groups connected

0 chunk groups processed for merging (0 module sets, 0 forked, 0 + 0 modules forked, 0 + 0 modules merged into fork, 0 resulting modules)

0 chunk group info updated (0 already connected chunk groups reconnected)

 visitModules: 475.582295 ms

 connectChunkGroups: 0.1497 ms

 cleanup: 0.0644 ms
LOG from webpack.FileSystemInfo

74748 new snapshots created

0% root snapshot uncached (0 / 113632)

0% children snapshot uncached (0 / 0)

0 entries tested

File info in cache: 68593 timestamps 0 hashes 0 timestamp hash combinations

File timestamp snapshot optimization: 81% (535485/658517) entries shared via 2879 shared snapshots (148310 times referenced)

Directory info in cache: 1 timestamps 0 hashes 0 timestamp hash combinations

Directory timestamp snapshot optimization: 0% (0/1) entries shared via 0 shared snapshots (0 times referenced)

Missing items snapshot optimization: 73% (238554/327783) entries shared via 4342 shared snapshots (62039 times referenced)

Managed items info in cache: 0 items
LOG from webpack.Watching

 done hook: 6.8447 ms

 storeBuildDependencies: 4.1044 ms

 beginIdle: 2.016 ms

visitModules is spent mostly in this code
that have the comment about possibility of removing it. @sokra do you have any plans on improvements for this code? We really rely on dynamic imports nowadays and cannot use lazy compilation due to incopatibility with SSR

[sibelius]

enable lazyCompilation it will make everything faster

[mozhuanzuojing]

enable lazyCompilation it will make everything faster

very fast

[mozhuanzuojing]

In the last weeks we spend a bit time optimizing rebuild performance further.

• We added a new experimental flag experiments.cacheUnaffected which improves rebuild performance by detecting modules unaffected by the current change and skipping computation for them.
• We added a new faster hashing algorithm output.hashFunction: "xxhash64" which is implemented in assemblyscript and a bit faster compared to the default one
• We improved module.unsafeCache to be much faster
• We added experiments.futureDefaults which enables experiments.cacheUnaffected, output.hashFunction: "xxhash64" and a few other things.

With experiments.futureDefaults and module.unsafeCache we now see all of our benchmarks (https://webpack.github.io/benchmark/) being faster in webpack 5 compared to webpack 4.

If that's not enough, webpack 5 has optimization.sideEffects enabled by default, while webpack 4 had that disabled by default. You can use optimization.sideEffects: false to disable that for webpack 5 too.

Need more even more speed? You can also use optmization.providedExports: false to disable another expensive step in the build. (This might break some apps which use complex circular ESM reexports) (We eventually add caching for that step to cacheUnaffected, which would make it fast without the need to disable it)

Great work

[sibelius]

can we close this ?

[sibelius]

check this tips https://dev.to/woovi/web-dev-efficiency-achieving-faster-builds-and-lower-memory-load-with-webpack-4cm8

[alexander-akait]

I think yes, we can close, if somebody encounters to perfomance problems - feel free to open a new issue

[JerryLyu93]

We noticed the big build/rebuild slowdowns on the big applications with a lot of dynamic imports. The most noticeable time is spent in visitModules block (more than 4s).

Webpack logs with dynamic imports
And if I disable any dynamic imports by using babel-plugin-dynamic-import-node then I'll get slightly less time spent in visitModules (less than 0.5 s)

Webpack logs with babel-dynamic-import
visitModules is spent mostly in this code that have the comment about possibility of removing it. @sokra do you have any plans on improvements for this code? We really rely on dynamic imports nowadays and cannot use lazy compilation due to incopatibility with SSR

Did you resolved this not by use babel-plugin-dynamic-import-node?