[Rollup2.7+] vite build is slow ,vite transform 10000+ file( 2000+[vue] file), It takes 2 hours to build a mac os air

[gaocheng001]

Rollup Version

2.79.1

Operating System (or Browser)

macos

Node Version (if applicable)

16.9

Link To Reproduction

https://github.com/cayweb778/vite2rolluptest.git

Expected Behaviour

中文:
我需要帮助!!

问题描述: vite build,开发了 2000+ "vue file", 构建需要transform 10000+文件

问题节点: ”assignEntryToStaticDependencies 构建缓慢“

问题描述: macos air需要构建2小时, 低性能服务器需要6小时左右

rollup新版本有所改进么?

调试过程( 正在进行 )

1. 调试到assignEntryToStaticDependencies 函数,开始卡住,正在进一步调试
2. 看到里面判断 areEntryPointsContainedOrDynamicallyDependent 卡住,这里面有两层for
3. 有可能是这步get,add影响的,先排掉看看速度

// areEntryPointsContainedOrDynamicallyDependent 函数
    const dynamicallyDependentEntryPoints = dynamicallyDependentEntryPointsByDynamicEntry.get(entry);
                for (const dependentEntry of dynamicallyDependentEntryPoints) {
                    // entriesToCheck.add(dependentEntry);
                }

4. for 无内容,速度是非常快的,测试去掉add
   5.去掉add,速度还ok,定位到add引起的缓慢,看看add的变量到底存了什么
5. 查到了一次add的巨大量操作
   • assignEntryToStaticDependencies 一次 add了5亿次,
   • 我们的项目需要1000次,那就是5000亿add,这相当恐怖
6. 查一下为什么需要巨大量add
7. 尝试了 另一个小项目 transform4000+ file,发现只需要35次,add次数7629次,而这个问题项目1000次,add次数达到了5000亿次add
   9,
   dynamicallyDependentEntryPointsByDynamicEntry :
   entryPoints
   containedIn

每次modulesToHandle才1000左右,但areEntryPointsContainedOrDynamicallyDependent中的add进行了5亿次,
继续排查

11. 理解了下areEntryPointsContainedOrDynamicallyDependent,似乎只有index.html时才会returnfalse,

• 那么为什么,大部分动态导入的模块中,都有index.html这个子依赖呢?
• 如果仅判断index.html,那么在上层一次性查完.效率应该会更高吧

dependentEntryPointsByModule :

modulesToHandle 是 1000个入口模块中的一个

英文:
I need help!!

Problem description: vite build, developed 2000+"vue file", and build requires transform 10000+files

Problem node: "assignEntryToStaticDependencies are slowly constructed“

Problem description: It takes 2 hours to build a mac os air, and about 6 hours for a low performance server

Is there any improvement in the new version of rollup?

Debugging process (in progress)

1. Debug to the assignEntryToStaticDependencies function. It starts to get stuck and is being debugged further

2. It is judged that areEntryPointsContainedOrDynamically Dependent is stuck when you see it. There are two layers of for

3. This step of get and add may affect the speed

//AreEntryPointsContainedOrDynamically Dependent function

const dynamicallyDependentEntryPoints = dynamicallyDependentEntryPointsByDynamicEntry.get(entry);

for (const dependentEntry of dynamicallyDependentEntryPoints) {

// entriesToCheck.add(dependentEntry);

}

4. For has no content and is very fast. Test to remove add

5. Remove add, and the speed is OK. Locate the slow speed caused by add, and see what the add variable has

6. A massive add operation is found

-AssignEntryToStaticDependencies were added 500 million times at a time,

-Our project needs 1000 times, that is 500 billion add, which is quite terrible

7. Check why a large amount of add is needed

8. I tried another small project, transform4000+file, and found that only 35 times were needed, 7629 times were added, while this problem project had 1000 times, and the number of add times reached 500 billion times

9,

dynamicallyDependentEntryPointsByDynamicEntry :

entryPoints

containedIn

ten

Each time the modulesToHandle is only about 1000, but the add in areEntryPointsContainedOrDynamically Dependent has been performed 500 million times,

Continue troubleshooting

11. After understanding the following areEntryPointsContainedOrDynamically Dependent, it seems that only index.html returns false,

-So why do most dynamically imported modules have the subdependency index.html?

-If you only judge index.html, it will be more efficient to check it in the upper layer at one time

dependentEntryPointsByModule :

ModulesToHandle is one of 1000 entry modules

Actual Behaviour

我正在调试过程,
调试到assignEntryToStaticDependencies 函数,开始卡住,正在进一步调试

[lukastaegert]

From the other thread

What is areEntryPointsContainedOrDynamically Dependent used for? I removed it, reduced the construction time from 5 hours to less than 5 minutes, and simply tested the running status. It is good.

It is an optimization to reduce the emitted number of chunks in the following scenario

// main.js
import { a } from './a.js';
import('./b.js');
console.log('main', a);

// a.js
export const a = 'a';

// b.js
import { a } from './a.js';
console.log('b', a);

Output without this optimization: Three separate chunks
However when the dynamic import b.js is loaded, a.js has already been loaded and is in memory, so it can be merged into main.js to have two instead of three chunks.

But this optimization indeed could be potentially costly. I however did not have a large enough code-base yet to see the effect. One could potentially improve performance here by reducing the situations in which this optimization is applied, or other things.

I would like to work on this soon to improve the situation, and your reproduction could help a lot to try out some things.

Unfortunately I do not speak Chinese. Can you give me instructions in English how to "run" your reproduction to see the issue? What do I need to do after checking out the repository?

[gaocheng001]

git clone https://github.com/cayweb778/hello.git

cd hello

/*
 just one build, Most of the packages can be tested for `pnpm build`. They are all `vite build`
 `cd webs/packages/boozsoft-platform-stock`
 or `cd webs/packages/boozsoft-platform-system`
 or `cd webs/packages/boozsoft-platform-zhongzhang`
 or `cd webs/packages/boozsoft-platform-gdzc`
 or `cd webs/packages/boozsoft-platform-home`
 or `cd webs/packages/boozsoft-auth`
*/
cd webs/packages/boozsoft-platform-system/

npm install pnpm -g
export NODE_OPTIONS=--max-old-space-size=30000
pnpm install
pnpm build

[gaocheng001]

From the other thread

What is areEntryPointsContainedOrDynamically Dependent used for? I removed it, reduced the construction time from 5 hours to less than 5 minutes, and simply tested the running status. It is good.

It is an optimization to reduce the emitted number of chunks in the following scenario

// main.js
import { a } from './a.js';
import('./b.js');
console.log('main', a);

// a.js
export const a = 'a';

// b.js
import { a } from './a.js';
console.log('b', a);

Output without this optimization: Three separate chunks However when the dynamic import b.js is loaded, a.js has already been loaded and is in memory, so it can be merged into main.js to have two instead of three chunks.

But this optimization indeed could be potentially costly. I however did not have a large enough code-base yet to see the effect. One could potentially improve performance here by reducing the situations in which this optimization is applied, or other things.

I would like to work on this soon to improve the situation, and your reproduction could help a lot to try out some things.

Unfortunately I do not speak Chinese. Can you give me instructions in English how to "run" your reproduction to see the issue? What do I need to do after checking out the repository?

Thanks , As above

[lukastaegert]

Unfortunately, I did not manage to reproduce the problem because for me, the build always failed much earlier due to unrelated issues. Still, I took some time to rewrite the critical part of the chunking algorithm in a way that hopefully performs better in #4736.

As your reproduction still uses Vite 2 and an older version of Rollup, I created a beta version of the patch based on rollup@2.77.

Can you please try it out and give me feedback by installing rollup@2.77.4-0?

[gaocheng001]

```getAlreadyLoadedModulesByDynamicEntry``` Sorry, it is still very slow here. It seems that there is not much change

[gaocheng001]

Capture in debugging:

allModules 7200+ elements
dependentEntryPoints 1041 elements
dynamicEntriesToHandle 1400+ elements

// dynamicImportsByEntry.get(dynamicEntry) has 1400+ elements
dynamicEntriesToHandle.push(...dynamicImportsByEntry.get(dynamicEntry));

dynamicEntriesToHandle has 3731926 elements

[gaocheng001]

This is new single vite2 repo: https://github.com/cayweb778/vite2rolluptest.git
// rollup version: rollup@2.77.4-0
npm install pnpm -g
export NODE_OPTIONS=--max-old-space-size=3000000000
pnpm install
pnpm build

[gaocheng001]

At present, I am using this this for testing. Everything is normal

[gaocheng001]

[gaocheng001]

Seems to be slower, 4s * 7000/60/60=9-hour+, Even doing 24-hour+

-- dynamicEntriesToHandle has 3731926 elements

[lukastaegert]

Thanks for the new reproduction, I was actually able to work with this. Based on this, I decided to finally make the algorithm slightly simpler. Now it will not always create ideal chunks, but it still will in most case. For me, it now generates the chunks in around 30s.

You can try it out via rollup@2.77.4-1

[gaocheng001]

Thanks for the new reproduction, I was actually able to work with this. Based on this, I decided to finally make the algorithm slightly simpler. Now it will not always create ideal chunks, but it still will in most case. For me, it now generates the chunks in around 30s.

You can try it out via rollup@2.77.4-1

[lukastaegert]

If you want to say you still have performance issues there, I fear you will need to update to Rollup 3/Vite 4. This part has been completely reworked here and also a performance issue has been fixed. But I do not have the bandwidth to port a fix to Rollup 2.

[gaocheng001]

If you want to say you still have performance issues there, I fear you will need to update to Rollup 3/Vite 4. This part has been completely reworked here and also a performance issue has been fixed. But I do not have the bandwidth to port a fix to Rollup 2.

thank you very,base on this.i will update vite3 +,this may tak some time,hope on rollup 3.xx may use this improve,

Can you tell me the specific version of 3.xx or 4.xx

[gaocheng001]

funciton assignChunkIds need about 30min+,What can I do to improve the performance

[gaocheng001]

If possible, I will report to vite.

[gaocheng001]

A lot of processing has been done here,may be 5000 billions

[gaocheng001]

[gaocheng001]

Can I move this task to Rollup 3+, vite 4+. I have updated the repo dependency to expect future support,

repo: https://github.com/cayweb778/hello.git

rollup3.6 + vite4.x issue : #4740

[gaocheng001]

rollup3 + vite4.x issue : #4740

[rollup-bot]

This issue has been resolved via #4736 as part of rollup@3.7.2. You can test it via npm install rollup.