best ts eslint setup for large monorepo?

[Zzzen]

Our monorepo is quite massive, and I wanted to share some statistics from the tsc to give you an idea of the scale:

Projects in scope:                         35
Projects built:                            33
Aggregate Files:                       116237
Aggregate Lines of Library:           1202224
Aggregate Lines of Definitions:       8076544
Aggregate Lines of TypeScript:         956954
Aggregate Lines of JavaScript:            844

To ensure code quality, I've been exploring options for linting within our monorepo. I've referred to the documentation (link), but I still have a few questions.

Switching to one root tsconfig.eslint.json (see One root tsconfig.json)

Since each package has its own compilerOptions, I'm unsure about the configuration details. Can someone provide guidance on the compilerOptions for the root tsconfig.eslint.json file?

Using a shell script to only lint one package at a time, using your existing config above.

It is helpful if we have enabled EXPERIMENTAL_useProjectService?

[bradzacher]

Context: in my day job work on a repo with >90k TS files.

At that scale it's unfortunately a physical impossibility to use the ESLint CLI out-of-the-box.

• ESLint itself scales linearly with the number of files because it runs single-threaded and synchronously.
• TypeScript similarly scales poorly because it too is single-threaded and synchronous.
• There are a few rough edges to the integration between ESLint and TypeScript and the larger the project the rougher those edges get.
• ESLint itself just doesn't support many of the concepts that we need to describe the shape of a codebase to make things work.

You can make it all work - however you will need to build out some infra based on your company's setup. For example you can build a layer on top of ESLint that runs each "package" in its own thread to isolate it and its memory from one another.

At Canva we don't use type-aware linting from the CLI because our repo doesn't have a nice "package" structure that would lend itself to clean isolation across threads. Instead we lint our files by roughly partitioning files across os.cpus().length-1 workers.

We do have some type-aware lint rules turned on via a custom TypeScript Language Server Plugin whose code is forked from this project. But we don't enforce type-aware rules at CI/CLI lint time.

[Zzzen]

For example you can build a layer on top of ESLint that runs each "package" in its own thread to isolate it and its memory from one another.

Wouldn't it potentially impact performance since each run requires TypeScript to perform a full project check?

[bradzacher]

Type Information is computed lazily. This means that for the most part TS does not calculate types unless they're asked for. So it doesn't really matter if you have 10 or 100000 files - at the start of the lint run there should be roughly the same number of types loaded (i.e. essentially none).

However what does happen is that TS will parse all files eagerly. So the impact you have is that you will likely re-read the same files repeatedly from disk and similarly re-parse them.
However by-and-large TS is bad at sharing data between projects - meaning even in a single thread you'll duplicate a lot of that work anyway.

Regardless though - you can either do the same work n times in parallel and then do n unique workloads on top of that - meaning your total runtime is essentially max(speed(n)).
Or you can do it all sequentially - in which case instead your total runtime would be sum(speed(n)). It's kind of a no-brainer to parallelise even if you're duplicating effort.

Additionally the alternative to parallelising is to do it in one thread - but you'll find that you'll OOM for a repo that big because for various reasons there's no mechanism in place for us to free memory. TS is a memory hog at scale.

[Zzzen]

Have you also experienced the issue of unresponsiveness in the tsserver and constant loading in the IDE as our TypeScript codebase grows larger? It seems that IntelliSense functionality is also not working most of the time.

[bradzacher]

Yes - typescript IDE performance scales horribly with the size of the codebase.

In particular it's the size of the dependency graph of the currently opened file(s) that impacts the performance. There's no way around this either because the TS server is single-threaded and it relies on the .ts source files, not the .d.ts files.

The best bet is to try and architect your codebase to keep dep graphs shallow where possible.