[core] Flatten imports to speed up webpack build & node resolution

[anthonyalayo]

What's the problem? 🤔

Background

While playing around with Next.js, I installed a package that was using @mui. I immediately noticed huge delays running next dev and the large module count (10K+) listed in the terminal. Searching for an answer, I landed on a long list of requests for help:

• Super slow page load times in development environment vercel/next.js#17977
• SWC is slow when importing directly from @mui/material vercel/next.js#37614
• SWC doesn't tree shake library vercel/next.js#42343
• NextJs compiling extremely slow vercel/next.js#29559
• Slow perfomance & Compile zounds of modules vercel/next.js#43285
• https://www.reddit.com/r/nextjs/comments/swciuj/next_dev_is_slow/

But none of those had an actual answer with an actual solution, just guesses and workarounds. One of the popular answers was: https://mui.com/material-ui/guides/minimizing-bundle-size/

After attempting to fix the library I pulled in by following the first option on that guide:
https://mui.com/material-ui/guides/minimizing-bundle-size/#option-one-use-path-imports

I noticed that the module count was still in the thousands. Why?

After reading everything on webpack, modules, tree shaking, etc, I made a demo application using CRA and a single import to @mui to showcase the problem.

Problem

Reproduction Steps

1. Run npx create-react-app cra-test to get the latest with Webpack 5
2. Ejected from CRA so that I could modify the webpack config for more metrics
3. Using webpack dev server, I modified this to see what modules traversed in development
4. Using webpack prod builds, I modified this to see modules traversed in production

I used the following configurations for stats.toJson() to balance the verbosity of the output.
I tested with and without default imports for a single icon and component.

1. With default import {assets: false, chunks: false, modulesSpace: 6, nestedModules: false, reasonsSpace: 10}
2. With path import: {assets: false, chunks: false})

Demo App 1 - Default Import Icon

This is the scenario that we are told to avoid when using @mui, and for good reason.
I created this as a baseline to see what webpack had to traverse.

import { AbcRounded } from '@mui/icons-material'

function App() {
  return (
      <div>
        <AbcRounded />
      </div>
  );
}
export default App;

Demo App 1 - Webpack Metrics Output

As expected, using the default import for AbcRounded ended up pulling in all icons.

Webpack Module Summary

orphan modules 6.28 MiB [orphan] 10860 modules
runtime modules 28.5 KiB 14 modules

Demo App 2 - Path Import Icon

Following the docs, you would expect an import like this to be lightweight. It isn't.

import AbcRounded from '@mui/icons-material/AbcRounded'

function App() {
  return (
      <div>
        <AbcRounded />
      </div>
  );
}
export default App;

Demo App 2 - Webpack Metrics Output

This is the problem. Importing a single icon resulted in:

1. Importing ./utils/createSvgIcon
2. Importing @mui/material/utils
3. Importing a lot more...(doesn't fit in a single screenshot)

Webpack Module Summary

orphan modules 534 KiB [orphan] 274 modules
runtime modules 28.5 KiB 14 modules

Demo App 3 - Path Import Component

This problem isn't unique to @mui/icons-material either. Here's performing a path import of a button.

import Button from '@mui/material/Button'

function App() {
  return (
      <div>
          <Button>Hello World</Button>
      </div>
  );
}
export default App;

Demo App 3 - Webpack Metrics Output

Again, the problem. Importing a single button resulted in:

1. Importing ./buttonClasses
2. Importing @mui/utils
3. Importing a lot more...(doesn't fit in a single screenshot)

Webpack Module Summary

orphan modules 573 KiB [orphan] 291 modules
runtime modules 28.5 KiB 14 modules

We should not be importing hundreds of modules from a single icon or button.

But... Tree Shaking? Side Effects?

This was confusing for me too, and I had to go into the details to find the answer.

1. Webpack does not tree shake at the code level, it only tree shakes at the module level.
2. Terser performs dead code elimination when minifying, but webpack still has to traverse all those imports!
3. This happens whether you are building for development or production.
4. We "cue up" our code for deletion by using ESM, but it isn't done until the minification happens.

Yes the bundle will still be minimized successfully when following ESM practices, but thousands of modules being traversed bloats memory and slows down development servers.

What are the requirements? ❓

Importing from @mui should always result in a minimal dependency graph for that particular import.

What are our options? 💡

Option 1 - Proposal

Apply transformations to the @mui build process to ensure a minimal dependency graph for all files.

Option 2 - Alternative

Remove all barrel files from @mui. This option isn't great as the developer experience that they provide is desired by both library maintainers and library users alike.

Proposed solution 🟢

1. Apply import transformations within @mui packages.

Showcased above, even when importing a component or icon directly, thousands of downstream modules get pulled in. This happens because within @mui itself, barrel files are being utilized for their developer experience. In that case, why not follow the same recommendation that @mui gives, and add these transforms to the build process?

2. Make "modularize imports" work for all @mui packages

In [docs] Modularize Imports for Nextjs, the comment #35457 (comment) requested that the docs don't include @mui/material for import transformations via babel or swc, since there are outliers that cause the transformation to fail.

Instead of backing off here, the work should be put in to fix it. The same barrel files that @mui is using internally for better developer experience is what users of the library need as well. By fixing point 1, this will come for free.

Resources and benchmarks 🔗

Below are the webpack metrics I collected for the applications in the background statement:
mui-default-import-icon-truncated-metrics.txt
mui-path-import-button-metrics.txt
mui-path-import-icon-metrics.txt

[anthonyalayo]

@oliviertassinari this issue isn't specific to icons, please see "Demo App 3":

This problem isn't unique to @mui/icons-material either.

[flaviendelangle]

We could enforce a few rules to improve the amount of dependencies listed.
On of them being to never import from the root of @mui/utils and @mui/system.

[anthonyalayo]

@flaviendelangle that would be great!

[flaviendelangle]

I think Olivier added the icon label because it is the scenario with the most obvious gains.
An icon is a super small component and we are looking through a full package (@mui/system at least).
For regular component, I don't think you should expect the list of files visited to be small, but we can definitely make improvements.

By the way, on the X components (data grid and pickers), we are probably even worse than that.
But the components being big by themselves, it's less notceable.
We can still improve though.

[michaldudak]

Thanks for such a detailed report, @anthonyalayo! I agree with @flaviendelangle, we can start with creating an eslint rule to disallow imports from barrel files. It could be better than introducing a build transform, as anyone reading the source code will see the proper way of importing other modules.

@anthonyalayo, would you be interested in working on this?

[anthonyalayo]

@michaldudak thanks for going through it 😄 sure I'm interested, but I wanted to hit on that point you just mentioned:

we can start with creating an eslint rule to disallow imports from barrel files. It could be better than introducing a build transform, as anyone reading the source code will see the proper way of importing other modules.

I considered this (as it's the easier way to go code change wise), but I also noted in the proposed solution that the developer experience would be affected. A tangential discussion happened on a Next.js issue, and the audience was quite in favor of barrel files: vercel/next.js#12557 (comment)

On top of that, Next.js recently released modularizeImports for SWC, since babel-plugin-transform-imports was quite nice for developer experience.

While bundlers understand these barrel files and can remove unused re-exports (called "dead code elimination"), this process involves parsing/compiling all re-exported files. In case of published libraries some npm packages ship barrel files that have thousands of modules re-exported, which slows down compile times. These libraries recommended babel-plugin-transform-imports to avoid this issue, but for those using SWC, there was no previous support. We've added a new SWC transform built into Next.js called modularizeImports.

Leveraging this transform with @mui/icons-material or lodash allows skipping compilation of unused files.

In the absence of these features, I think the only solution would be an eslint rule to reject barrel file usage. But since we already are somewhat across the goal post for good DX and performance, why not take it all the way?

[michaldudak]

By "taking it all the way" you mean fixing the issues that prevent babel and modularizeImports transforms to work, right? I haven't looked into this much, but I fear we won't be able to change the structure of our packages without introducing breaking changes. We can certainly look into it for the next release, as we're going to change how things are imported anyway (by introducing ESM and import maps)

[anthonyalayo]

@michaldudak agreed, it would be a breaking change (since some import locations would move to be consistent), so I think looking into it for the next release sounds reasonable to me.

With that being said, I can definitely do the eslint rule and import fixes associated with it. I'll make a PR for it an attach it to this issue.

[anthonyalayo]

@michaldudak I did an initial attempt, but there's quite a bit of eslint configs/overrides setup already with 'no-restricted-imports': https://github.com/mui/material-ui/blob/master/.eslintrc.js

I also noticed that paths doesn't support wildcards, so it would have to look something like this (unless someone chimes in with a better solution):

    'no-restricted-imports': [
      'error',
      {
        "paths": ['@mui/material', '@mui/material/utils', '@mui/styles', '@mui/base', '@mui/utils'],
      }
    ],

Is there anyone at @mui that could join in the conversation for how they would like it ideally?

[anthonyalayo]

@michaldudak bumping the above message in case it got missed

[michaldudak]

Unfortunately, "patterns": ["@mui/*"] doesn't seem to do the trick here. Explicitly listing all the forbidden paths seems reasonable.

cc @mui/code-infra for visibility and perhaps other opinions

[anthonyalayo]

Sounds good, If no other opinions from @mui/code-infra i'll do that then

[oliviertassinari]

This problem isn't unique to @mui/icons-material either. Here's performing a path import of a button.

@anthonyalayo I think that the number of modules isn't this relevant. We should focus more on the metrics that directly impact developers:

1. built time in dev mode
2. load time in dev mode

I have added the icon's label because so far, I don't think that it was ever proven that the problem goes beyond icons. https://mui.com/material-ui/guides/minimizing-bundle-size/#development-environment mentions a build time of x6 for icons, but for a button, back then, it was like 50%, mostly negligible.

It could be great to measure again, it was a long time ago.

On of them being to never import from the root of @mui/utils and @mui/system.

👍 agree, to keep doing it (I think that we started doing this a long time ago).

[joshkel]

I've been looking into what I think is the same root issue: I'm trying to speed up our Jest test suite. Jest by default executes each of its suites in an isolated Node context, which means all of the tests' dependencies have to be re-loaded, parsed, and executed for every test module. Tree-shaking doesn't apply, and Babel typically isn't run on node_modules for Jest (so babel-plugin-import or babel-plugin-direct-import won't help), and the costs are incurred on every test execution (in watch mode, every time a file is saved, versus just when webpack-dev-server is starting up). MUI packages' use of barrel imports from other MUI packages seems to have a noticeable impact here, so I'm interested in this area of work as well.

Should @mui/base be added to the list of "don't do a root import" packages, especially as more of @mui/material and @mui/joy start using it? (In some crude local testing, replacing barrel imports of @mui/base speeds up a single test module by 2.5% - not much, but measurable, for an process that's run all the time during development.)

If this isn't the same issue or would be better tracked separately, please let me know.

[oliviertassinari]

I'm trying to speed up our Jest test suite.

@joshkel If you didn't read it before, check this out: https://blog.bitsrc.io/why-is-my-jest-suite-so-slow-2a4859bb9ac0, and search for "Barrel files".

Barrel files
Now that we have the inspector hooked up, we can immediately see the problem — almost all of our time loading the test file is spent loading the @mui/material library. Instead of loading only the button component we need, Jest is processing the entire library.

But yeah, the solution they propose is already applied by the developers that are landing on this issue.

---

I have added the good to take GitHub label, the problem, and solution are clear: the community can help. We need to flatten our internal imports, especially because competitive libraries that use one npm package per component are forced to flatten, and hence get a hedge. In our case, we can have the same forcing constraint with eslint.

A closing note, let's please benchmark, so we can have a nice Twitter/release note mention about the actual win this brings to the community. I'm sure developers are always happy to hear about efficiency wins 😁

[oliviertassinari]

Removing assignments as it could be confused into this is something the person is working on. I assume nobody is working on it right now.

[joacub]

Removing assignments as it could be confused into this is something the person is working on. I assume nobody is working on it right now.

is this going to be done by mui or are you asking developers to help on this ??, we are getting really slow development with nextjs 13, really really slow.

[kevcmk]

As of right now, I would urge anyone considering using NextJS 13 to avoid Material UI v5

[oliviertassinari]

As of right now, I would urge anyone considering using NextJS 13 to avoid Material UI v5

@kevcmk Which issue are you facing?

[kevcmk]

Material UI documentation will send you down the complete wrong path if you're using next.js.

@oliviertassinari After looking through your posts on other code reviews, I tracked down this code (specific to Next.js), which finally did the trick. Thank you for this ↓

const nextConfig = {
...
modularizeImports: {
    "@mui/material": {
      transform: "@mui/material/{{member}}",
    },
    "@mui/icons-material": {
      transform: "@mui/icons-material/{{member}}",
    },
  },
}

module.exports = nextConfig

and, if you're like me, you ran into issues with, for example

import { useTheme } from "@mui/material"; # Will fail given the above next.config modification.

Use this instead

import { useTheme } from "@mui/material/styles";

Also, thank you for your work on this project and your effort on the issues board.

[oliviertassinari]

This issue is up for grabs for the community. It should be relatively easy to make progress. It's a matter of have deeper imports, avoiding barrel index files.