New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiples re-compile when using webpack-dev-server with project references #1157
Comments
This may be intended behaviour - but I'm not certain. The idea being that in watch mode all the projects that are in scope may need to be rebuilt to ensure a complete build. I'm not clear how "smart" we are in terms of knowing when recompilation is unnecessary. @sheetalkamat / @andrewbranch can you comment? |
Unfortunately I don’t know enough about how webpack-dev-server works to know why this is happening and how easy it is to avoid it. My first question would be whether the same happens with |
I dont know enough about dev server either but i was trying to clone and repro and i get error:
Note i will be out of office next week and might be busy catching with our 4.0 stuff week afterwards so will be some time before i take a look and comment on this one |
Sorry for the absolute path in the Yarn config, it just fixed it. Regarding I'm not sure how much |
I have loaded the repo and can confirm the multiple recompiles. Tracing the webpack watch code I understand why it is happening although I'm not sure there is a simple fix. After utils/src/index.ts is changed the following steps occur:
At step 7, any files which have been changed since step 2 (although see note below) will trigger a re-compile. Because the utils/build/*.js files were created after step 2, a new build will be triggered. The re-build does not cause the project references to be rebuilt because the utils/src/*ts files have not changed. Therefore no new files are created in utils/build and the build cycle ends after the second build. The second build is not necessary because ts-loader caused the project reference to be built at the start of the build process. By the time webpack included those files the new versions were already on disk. Webpack does not know this and assumed that since the files changed after the compilation was started that another compilation may be needed. A sidenote is that it is not quite true that files changed since step 2 will cause a recompile. Watchpack measures the accuracy of the filesystem and adds that figure to the modified time of the file to ensure that changes will cause a recompile. Watchpack initially sets this at 1000ms: But after a file is changed it is reduced. On my system under WSL this is 100ms but it could be 10ms under Windows. The FS_ACCURACY is not reduced from 1000ms until the first file is changed, so webpack will recompile until 1000ms after the start of the compilation. This is the cause of even more compiles on the first cycle. This could be fixed by a change to watchpack, but the double compile would remain. I'm not sure there is a simple solution to this. The guidelines for writing a webpack loader include:
Including projectReferences into ts-loader breaks these guidelines. ts-loader reads files directly from disk and writes files directly to disk. I could argue that ts-loader is not a normal loader. ProjectReferences are a great capability so perhaps it is worth breaking the guidelines. The second compile seems a small price to pay as it is very fast anyway. I do wonder it if would have been better to keep ts-loader simpler and benefit from the new features added to tsc (project references & watch api) by a separate mechanism, perhaps a plugin or a package such as lerna. |
Awesome write up @appzuka!
It's arguable that ts-loader has 2 modes:
I typically use 1. from the above for my own use cases. Another thing to consider is project references. It potentially makes use case 2. more scalable; speed even as your codebase grows by splitting up your codebase into multiple projects. Side note: since v5 |
@appzuka @johnnyreilly thanks for the amazing write-ups. After checking gazillions of ways to do monorepos, I find TS' project references the way to go and thus a must-have for my use cases. So I need project references in any way, the question is now how and I am wondering which of the following is the better option: 1. Compilation done by My gut feeling (but pls correct me): 2 has more optimization options (ts-fork-checker, parallelized builds, etc.) but it's not clear for me ATM if project references are properly implemented with I'd think that 1 even with just one compilation must be slower because of lacking optimization options, but I'd would have proper/solid project reference support. A clean build system which I could ELI5: tsc compiles, webpack bundles. And maybe it's not so slow because of incremental builds, so yeah. I tend to go for 1 but maybe I missed something and hence would love to hear your thoughts before I enter the rabbit hole of trying and benchmarking both 1 and 2. |
So, I tested option 1 (tsc transpiles, webpack bundles) with a mini hello world repo, so I can't say anything about speed. The setup was easy, it feels quite responsive but you have also double bundlings from webpack if you change a reference. Once you set an I am still curious to try option 2. This would mean I take the with webpack + ts-loader compiled folder out of the TS solution file's Edit: besides this performance topic; could you share your experience with ts-loader and fork-checker in the context of project references? Is is solid or not? Cmon guys, don't be so shy. |
Hey @desmap , Thanks for posting this feedback - it's hugely helpful! The project references support in ts-loader is fairly mature now thanks to the stellar efforts of @andrewbranch and @sheetalkamat 🌻❤️ One thing that we've never quite done is good documentation around how to use project references with ts-loader. What workflows are possible, what aren't. What's good practice. What doesn't work etc. If you were tempted to provide a docs PR to add your findings to PS I'm on my holidays so my replies may be sporadic PPS incidentally the |
I'd love to but ATM it would be just kind of a 'stay away' from To be fair, I just tried ts-loader with a perfectly working project references project (option 1) and still, after a lot tweaking, ts-loader couldn't import a project reference, e.g. a file containing I'd like to love ts-loader but this and people reporting a lot of issues here and severe issues on the fork-checker side makes using ts-loader a big ask. But here your comparison table in the context of project references:
[3] couldn't get it to work and ts-fork-checker has severe perf issues with project references reported by multiple people TypeStrong/fork-ts-checker-webpack-plugin#463 and TypeStrong/fork-ts-checker-webpack-plugin#453*
Did you check both issues above? Once you did there is no need to try fork-ts-checker anymore and again: ts-loader has to check anyway b/c of [1] Man, I am sorry and even if you found the bug why it doesn't find my references (or maybe I just didn't get it), tell me why should I use ts-loader with project references? Maybe I still missed something and that's the reason why I am here. Your feedback on the comparison table is welcome. |
Hey @desmap, I'm sorry to hear project references support in ts-loader and fork-ts-checker-webpack-plugin isn't covering your use case. |
You mean using "project references"? Yes, that's my use case which your README states as a feature of ts-loader. Hope that use case is not too exotic. Is there a reason you do not provide a working example with ts-loader and project references in the repo? This would be a good start and highly appreciated by all users (also as a signal that ts-loader really supports project references). However, that's your call. |
Hi @desmap, I have been spending a lot of time looking at project references and ts-loader recently. I agree that some documentation and examples are desperately needed and it is my intention to write some once I feel I understand the subject enough. I have a project using multiple project references, ts-loader, transpileOnly and fork-ts-checker-webpack-plugin. It works well and the performance seems good but there are a lot of options in tsconfig, webpack.config and ts-loader to get right and also you need to set up the project structure just right to get all the resolutions working (I use yarn workspaces). When I am done I will write this all up and provide an example project. In the meantime, if you have a repo to share with me that shows your issues I would be happy to take a look. |
hey @appzuka, thanks for the prompt/kind/empathetic reply, this was already quite helpful to move fwd.
I think this is the problem. Of course, I could use yarn workspaces to import packages within a monorepo yarn-workspaces-style but this is not needed with project references which makes things easier. Not that yarn workspaces is bad but it has its own warts. My feeling is that Is it a big deal? IDK, for some maybe, for some not. If this is really the case (I am too tired now to test this), I could make a PR for the README stating that resolving doesn't match the official specs and you still need kind of dedicated resolving solution such as a yarn workspaces or maybe even tsconfig paths (but again we now have project references and all this extra stuff shouldn't be required). [1] out of the scope of the current reference folder with its own tsconfig.json (not the solutions tsconf) |
In my project, yarn workspaces automatically creates a link to the referenced project in node_modules:
In the project reference directory I include a package.json which contains a "main" field which points to the transpiled code in workspace/reference/lib. Then, you can just import the reference from your main project:
The module resolution for both tsc and webpack correctly resolves this to the transpiled code via the symbolic link in node_modules. (Assuming you are using the 'node' module resolution strategy in tsconfig.json.) You need to make sure you are resolving to the transpiled js code, not the source ts code in the project reference, otherwise typescript complains that files are outside the root dir of the main project. You could do this without yarn workspaces. You could just create the links in node_modules yourself. Or you could do without the link and include a link to the reference in "paths" within tsconfig.json and also the resolve/modules setting in webpack config. I am not using yarn workspaces to build the referenced project, just to manage the link in node_modules. I hope this helps. |
Thanks again for the quick feedback. I found the reason: ts-loader CAN resolve project references without any extras, I needed to add FWIW, I didn't experience double compiles, so changing the comparison above. FWIW2, I tested compile times with |
FWIW3, tested also fork-ts-checker: there's no perf difference to before or before and FWIW4, I tested also compilation with So, same speed, two more dependencies but just one compile step. much better impression but still no clear advantage over option 1 but maybe I have to look further, I am continuing tmr, let's see |
I believe If you are not using In my project I only experience the double compiles in watch mode when I change one of the source files in a project reference. If I change a source file in the main projects I do not see a double compile. This behaviour is understandable. |
can confirm, it's super fast in the not referenced files (updating table). I couldn't get fork-ts-checker to work but I'll check tmr. However, the now working @johnnyreilly let me tinker around a bit more and I might PR an updated version of the comparison table, I mean now it looks way better than few hours before Edit: note to myself, I still need to test happypack and/or threaded-loader |
I think I needed to remove |
this is nuts. I have the feeling the perf is very much dependent if you do changes in non-referenced files vs referenced files (it's exactly the same with option 1). Latter make the biggest impact. The rest (transpilesOnly, fork-ts) doesn't make that much of an impact, i barely notice anything... strange |
fyi, 4-core vps[3] benchmarks, reference and non-reference files have each 1,000 functions tl;dr: option 1 is faster on reference builds and warm starts, option 2 on non-reference builds and cold starts
[1] Couldn't get thread-loader to work, got |
This is very interesting as I am currently working on something very similar. As you observe, tsc -b is very fast for subsequent builds. I am hoping to use this by creating a webpack plugin that runs tsc -b at the start of the build so you get this benefit but still a single step workflow. In the case of 'rebuild on 1st change in non-reference' the fastest build is by 'webpack-dev-server+tsloader, transpileOnly, fork-ts-checker'. This is probably the most important benchmark. Developers want a fast feedback loop when developing and it is probably worth sacrificing a slower production build and/or cold start for webpack-dev-server to achieve this. I am not surprised that rebuild after changing a reference is slower. ts-loader internally calls tsc -b on the project reference in this case so you get a time similar to the tsc -b plus webpack bundling. I would assume that the references are more stable projects and that most development goes on in the non-references, so maybe this is not so bad. The only thing that surprises me is that you are not seeing type checking in the case of webpack-dev-server+tsloader, transpileOnly, fork-ts-checker. I note that the times are similar to without fork-ts-checker so I wonder if fork-ts-checker is really working in this setup. I believe the best setup today is your Option 2 with transpileOnly added (assuming fork-ts-checker can be made to work). It could be that option 1 (tsc -b -w + webpack-dev-server) is a better option under some conditions (size of project, size of references, whether you are editing the main project or reference). Today that is a messier workflow and may be slower during development so it is hard to recommend. I'll copy you on my new plugin if it looks promising. It does offer much faster warm starts but I need to understand how to match the current speed for changing a single file in the main project as ts-loader does a great job there and it is probably the most critical use case. |
This is also what I didn't get: transpileOnly + fork-ts turns off type checking again. Once fork-ts is in place transpileOnly has to be omitted. I tested this multiple time with deliberate type errors which only popped up if transpileOnly wasn't omitted while fork-ts-checker being enabled. I also did a PR for the docs.
I tend also to option 2 but you could use both once you work a lot on refs but let's see.
I'd love to. |
Hi! All those diagnostics and investigations sounds very promising for the future of references support in I just didn't clearly understood: are you still experiencing the multiples re-compiles? |
i didn't experience any multi-compiles with ts-loader but my test repo is just two files (a main and a reference file) + the typical project reference boilerplate, so IDK |
Updated table (update in NEW OPTION 2, be aware that latter times were taken at another daytime, this can make some difference on a vps or fork-ts-checker struggles much more with project reference than the rest of the pack): fyi, 4-core vps[3] benchmarks, reference and non-reference files have each 1,000 functions
new tldr: there seems to be something wrong with how tsloader compiles references, so best are OPTION 3 and 1 check also #1174
[1] Couldn't get thread-loader to work, got |
final verdict: so after this benchmark marathon, I could think that fork-ts-checker might a deal-breaker in a proper DRY monorepo where a lot of references are used. pure tsc is speed-wise ok but still not great and you cannot turn off type-checks. Which you can with pure ts-loader. So maybe latter is the way to go with type checks in your editor only. Is this good, anyone experience with this setup? |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Closing as stale. Please reopen if you'd like to work on this further. |
Expected Behaviour
Build referenced projects, and then run a single Webpack compilation.
Actual Behaviour
Multiples Webpack compilations are ran, my guess is that it watch referenced project and re-compile for each emitted file by the referenced project instead of waiting for the referenced project to be compiled first.
Steps to Reproduce the Problem
yarn
at the root/packages/app
yarn webpack-dev-server
/packages/utils/src/index.ts
Location of a Minimal Repository that Demonstrates the Issue.
Linked above: monorepo-typescript-references
Additional notes
As you can see from the steps to reproduce, when running
webpack-dev-server
for the first time, 4 compilations are ran. My first guess is that it's due to thepackages/utils
project to output 4 files when built (index.js
,index.d.ts
,index.d.ts.map
,index.js.map
): the watcher would trigger 4 separate compilations for each of those files.When editing the file in the referenced project, two compilations are triggered (maybe because the
.d.ts
and the.d.ts.map
are not re-build when not changing types so only 2 files are updated in the build folder?).Note that, when running
webpack-dev-server
a second time when the referenced project is already built, only one compilation occurs.The text was updated successfully, but these errors were encountered: