Feedback Wanted: Long-running ESLint sessions

[nzakas]

As we are continuing to gather requirements for the next version of ESLint, it's become clear that there are two primary ways ESLint is used:

1. Short-lived sessions - running from the CLI, lint-staged, and CI systems, where ESLint is run on a set number of files and then exits immediately.
2. Long-lived sessions - embedded inside of systems that run periodically as a result of watching files or polling. Examples include editor extensions and daemon processes (eslint_d).

Further, it's now very clear that the demands of each type of session are very different. For plugins like typescript-eslint and eslint-plugin-import, it's important to know when ESLint may be exiting so they can know whether or not to cache information. We're learning more about the difficulties of editor extensions, too, but we need your help.

Feedback Wanted

In this discussion, I'd like to get feedback from integrator and ecosystem authors to see how the differences between short-lived and long-lived sessions affect them. Specifically:

• What sort of data are you missing in order to do what you need?
• What functionality have you had to hack around to make things work?
• What could ESLint provide that would make your life easier?

We aren't talking deep implementation details or whether or not something is possible right now. What I'm hoping to get out of this is the wish list that everyone has in the back of their head. We are doing some serious rethinking of our APIs right now, so every seemingly outlandish wish is a possibility!

We want to make your life easier. Help us do it!

cc @bradzacher @ljharb @JoshuaKGoldberg @dbaeumer @alexander-doroshko

[ljharb]

The ability to know when to clear a cache would be very helpful. Since that isn't one single definition, providing more hooks or events that let me respond to different points in the linting lifecycle would help me define that for myself.

[nzakas]

Do you have an idea for what kind of hooks you would like? Off the top of my head I'm thinking:

1. Session start
2. Linting a file start
3. Linting a file end
4. Session end

Any others you can think of?

[JoshuaKGoldberg]

When a file CRUD takes place: would be good too. If file A depends on file B and file B changes, file A will likely need to be re-linted.

[bradzacher]

Yeah that's something that's missing - currently eslint assumes single-file linting, so there's no way to inform eslint of the dependencies between files.

This causes two immediate issues:

1. eslint's cache flag is broken if you use type-aware rules or many of eslint-plugin-import's rules
2. IDEs do not know that they should recalculate lints for an open file.

[alexander-doroshko]

From the editor integration point of view: maybe ESLint could behave as a long-running Language Server, providing highlighting and fixes via a well-known Microsoft LSP protocol?

[nzakas]

An LSP is a topic that has been brought up a few times in the past, but since other people rushed to create their own we never looked very closely at it.

I'll admit I'm a bit ignorant as to how LSPs work -- do all editors use the same format? Are there good example LSPs we can look at for guidance? (I"m guessing TypeScript is a bit heavy and maybe it would be easier to look at another example?)

[alexander-doroshko]

Yes, all editors use the same format. LSP server is a standalone tool that reads/writes JSONs according to the spec.

I think the starting point is this LSP guide. It even mentions ESLint LSP as an example :). So @dbaeumer can tell more about what it takes to create and maintain the server.

You don't need to maintain the LSP client and VS Code ESLint extension, you are only interested in the server-related docs from that page.

Technically, you don't have to use the TypeScript API that VS Code suggests. For example, it's possible to write a server in Java or Kotlin using Eclipse LSP4J library. You don't even have to use any existing library and can write LSP server from scratch in any programming language (like Rust and Dart support is written in Rust and Dart respectively). Though I think that using TypeScript API is the easiest path for ESLint.

At the moment, each editor is maintaining its own wrapper over ESLint engine that converts ESLint logic into some IDE-compatible long-running language server. At JetBrains, we are using some custom non-standard way to communicate with our custom long-running wrapper process, but we'd be happy to switch to a standard LSP protocol.

[bradzacher]

Yup, Alexander is correct.
LSP = "language server protocol". It's a spec which defines the API for language server implementations.
Though note that "language server" is a bit of a misnomer, a better name would be "IDE interaction protocol" or something. As an example - an ESLint LSP wouldn't deal with a language, instead it would define the set of interactions that ESLint enables within an IDE.

The idea is that you can define an LSP implementation and have any IDE instantiate that, talk to it using LSP JSON messages, and have the same set of behaviours everywhere.

Eg an ESLint LSP implementation would ask the IDE to tell it when a file is changed, and in response would inform the IDE of the lint results for the file (which the IDE can then choose how to display to the user).

[nzakas]

Awesome, thanks so much for all of the details @alexander-doroshko and @bradzacher. We can definitely look into creating an LSP server.

[dbaeumer]

One thing to add: VS Code maintains a library to write LSP servers in JS / TS and run them in the browser in a worker or in Node. The library is vscode-languageserver. Source code is here: https://github.com/microsoft/vscode-languageserver-node.git in the server directory.

[bradzacher]

What sort of data are you missing in order to do what you need?

• the ability to know when we are running in a persistent context
  • right now we assume we're always in a persistent context.
  • we have trialled some hacky logic to detect CLI runs by inspecting argv, but this doesn't handle many cases and doesn't let us detect fix runs where a file can be parsed 1..10 times.
  • knowing we're in a persistent run would allow us to attach file watchers - which would allow us to be more correct.
• lifecycle status to know when a file has been linted to completion (I.e. Parse, lint, fix, done)
• the ability to notify eslint of changes in dependencies.
  • for example say A depends on B. If you open A in your IDE, the IDE will lint that file and show the errors. If you open and change B - there's no way for the IDE to know that A should be re-linted. If the changes in B would have caused different lint results in A, then that means the IDE will show stale lint results until either (a) A is closed then re-opened or (b) A is edited (both cases causing a re-lint of A).

What functionality have you had to hack around to make things work?

As mentioned, right now we have some (experimental, off by default) logic that inspects process.argv to guess if we're running in the ESLint CLI. If we are, then we use a separate code path in the parser which treats the disk as constant - which is significantly faster!

For all other cases we assume we're in a persistent context and thus assume the disk could change - which has significant performance impacts.

Note that our logic does struggle a bit with --fix runs because we can't reliably detect them - so we have some (hacky) logic that says like "if we think we're in a CLI run AND a file is being parsed more than once - assume it's a fix run". For fix runs we need to use a separate code path which leverages some logic to handle changes at a smaller scale. This logic is much faster than the full-scale "anything could change" logic, but is slower than the "disk is constant" logic.

[bradzacher]

For context: right now fixes are actually multi-pass

Of course - which is entirely the problem.

In the CLI usecase this causes a problem for memory management.
If we're trying to figure out when we're done linting "project A" - we need to be sure that ESLint will not ask us to parse file X again.

However based on the current information that ESLint provides - all that we know is that ESLint wants us to parse file X. We don't know if parse 1 of file X is the first and last, or the first of 11 parses of the file.
This leads to a problem where we can never know for certain whether or not we're done with a file or not.

If we had some form of "lint phase" indicator for a file then we could know for sure when we're officially done with a file. Taken in culmination - when all files in project A are in the "lint complete" stage, then we can ditch project A from memory - helping prevent OOMs.

In terms of the persistent usecase - the lint phase would be less helpful overall, but could potentially unlock optimisations in future.

As a quick example - if we know that ESLint is going to ask us to parse / calculate types for file X up to 10 times in a row, then we may be able to optimise the incremental updates we're making to the types as the file contents are changed by the fixes.
We know that fixer changes will always apply just to the file being linted, and we also know that there's no way ESLint will ask us to lint a different file whilst it's fixing the current file.
With these two bits of information we could optimise the type updates so that we only do them "shallowly" during the fix passes. Put another way - instead of trying to update the types for file X and the entire tree of dependents on file X, we could cull that tree and just deal with file X. Oncethe file is "lint complete" then we know ESLint is about to lint the next file, so we would then do a complete typecheck pass on the final fixed text and update the entire tree of dependents with it.

Such an optimisation could theoretically improve autofix-on-save speed in codebases that use type-aware linting.

[bradzacher]

I'm assuming that A and B in your examples are source files in your project?

Yup, that's what I meant. Specifically thinking in terms of files

In the editor case, it seems like this would be a perfect example for an LSP to handle rather than a plugin?

Another usecase separate to IDEs is ESLint caching. Right now ESLint caching assumes all lint rules are single-file only and thus that changes to file A will never impact the lint results of file B.

If there was a way to tell ESLint "hey we know that file A depends on file B", then ESLint can do its own change-analysis to determine side-effects for things like invalidating caching correctly, or to inform IDE consumers that changes to file A will require re-linting of file B.

In terms of "should the LSP handle this" - it's difficult because you need to define what it means for files A and B to depend on one another.
On the surface it might seem simple for the pure JS usecase "does A import B?". But even then it can get complicated because you could have a case like this in B: export { val } from 'C'. Now A directly depends on B and transitively depends on C - meaning changes to C might also trigger re-lints on A.

But for TS it's much the same for values and types, but there are some extra cases to be aware of like global types declared in file D are used in file A, thus file A depends directly on file D.

For JS-like languages like angular or vue with their own templating languages, the template "html" files they use (which can be linted in ESLint today with @angular-eslint or vue-eslint-parser) will depend on the backing JS files that define their models.

For JS/TS there's also considerations about import mapping in tools (eg tsconfig paths or webpack loaders or webpack resolve aliases or rollup aliases, etc, etc).

All of this is a lot of information to encode into an LSP that is there to run ESLint on files. Some of it may even be impossible to properly encode without the user specifying their config in some way!

Instead it would probably make more sense for plugins to define this information and inform ESLint of these dependencies. For example:

• eslint-plugin-import has its own "plugin" system called "import resolvers" which allow the user to teach eslint-plugin-import how to resolve imports to files on disk so it can do its dependency analysis (eg eslint-import-resolver-typescript can be used to handle tsconfig paths and ensure the plugin looks for .ts files, not .js files when the extension isn't specified).
• When doing type-aware linting. @typescript-eslint knows all of the dependencies for any given file being linted because we've already computed the types for the entire project.
• Angular/vue eslint plugins have intricate knowledge of how to relate a template file to its model file.

If not, can you explain how you envision this communication?

I think there's probably a few ways you could go about this, but the simplest of ways would be to make this a first-class concept of ESLint "plugins". I.e. when ESLint asks a plugin to parse a file, the plugin parses it and also includes a list of the file's known dependencies.

That way ESLint has that information to act on for its caching, and it could pass it on to LSP consumers so that they can correctly re-lint files in response to changes.

---

Worth noting that for pure JS files doing purely single-file linting, this sort of system would be useless because two files cannot influence one-another. This sort of system is only interesting when there is cross-file linting in some way (type-aware linting, export shape analysis, etc).

[nzakas]

If we're trying to figure out when we're done linting "project A" - we need to be sure that ESLint will not ask us to parse file X again.

Ah! 💡 I get it now. We assumed re-parsing the file was a zero-cost activity, and it is for Espree and other single-file parsers, but not for TypeScript. So it would be useful to be notified each step of the way. This is very helpful, thank you.

when ESLint asks a plugin to parse a file, the plugin parses it and also includes a list of the file's known dependencies.

This is something I've been thinking about as well, but I don't see a way of doing this without making import resolvers part of the JS language core for ESLint. Because dependencies can be packages and not just files, we'd need some way to resolve them into files to be actually useful.

[bradzacher]

without making import resolvers part of the JS language core for ESLint

I don't think you specifically need import resolvers - probably just doing "dependency resolvers" would be fine.

For example a plugin could export an API like

function resolveDependencies(
  ast: ESTree.Program,
  filePath: string,
): string[];

I.e. ESLint can ask for the list of absolute paths to files that are dependencies of this file. This would be a completely language or import resolution strategy agnostic thing - how the plugin arrives at the list of files is up to it.

• ESLint core could ship with a default no-op resolver - because ESLint core's rules all assume files are safe to parse in complete isolation.
• eslint-plugin-import would return the set of files resolved by its import resolver tooling.
• @typescript-eslint would resolve the set of files resolved by TS.
• @angular-eslint and Vue would return the model/controller file(s) for the template file.

ESLint can use these lists of files to build a dependency graph for the entire project which will let it correctly handle its cache invalidation, as well as triggering IDE re-linting.

Just spitballing - there may be better ways to do this!

[nzakas]

Fair point. Putting the responsibility on the plugin makes sense -- it still seems like we'd want to be able to specify dependency resolvers as part of config, which would mean having it be a feature of the JavaScript language instead of asking every plugin to implement their own versions.

Definitely worth thinking more about this.

[dbaeumer]

Besides was has already been said here are the findings from maintaining the ESLint extension for VS Code:

• although the extension can't work around it plugins that assume that they are only running in a CLI context and store global state cause problems since these don't behave correctly in an editor session hence causing bugs we are usually filed again the extension. Besides having means to detect in which context a plugin runs it is IMO important to educate ESLint plugin providers that these two modes exists and that global state has to be managed keeping this in mind.
• ESLint and all its plugins are currently very disk bound which makes it very hard to use ESLint in a context were files are not stored on disk. It even needs some hacks to validate untitled files to get the right starting point for config files lookup. So one wish for a new ESLint version would be a FS abstraction that ESLint and its plugins can use which an host could change. This for example would allow to use ESLint in browser environments as well.

[bradzacher]

One issue that exists for a stateful parser in the IDE world is the dirty state of a file.

IDEs lint a files raw, unsaved text as the user types. This text is "dirty" until such time that the user saves the file and commits the text to disk.

Currently though stateful tooling doesn't know if it's parsing dirty state from an IDE or clean state from the disk because ESLint just says "parse this text".

The tricky thing is that you have to be careful about updating the internal state based on dirty text. If the user chooses not to save and instead closes the file without saving, then we will not be told about it by ESLint or the IDE - so we can't correctly purge the dirty state.

So if ESLint is looking to build tooling specifically for IDEs to integrate with - being able to have the IDE tell eslint "this file was opened", "this file was saved", "this file was closed" would be really vital in terms of knowing how to manage state.

[nzakas]

So if ESLint is looking to build tooling specifically for IDEs to integrate with - being able to have the IDE tell eslint "this file was opened", "this file was saved", "this file was closed" would be really vital in terms of knowing how to manage state.

Excellent point. @dbaeumer does such a communication mechanism exist in VS Code?

[dbaeumer]

@nzakas yes, such a communication mechanism does exist in VS Code and the LSP (https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_synchronization)

[dbaeumer]

Would you be able to share more details about the problems you have around this? I'm wondering if the new config system makes your life easier or worse? This is particularly useful to understand because a lot of the internal file resolution in the old config system has been removed in favor of a JS-only config file, and I'm hoping that makes it easier for editors to integrate with?

Yes, it has mostly to do with how ESLint reads and finds config files. Here is an example to check if the config files can be parse correctly https://github.com/microsoft/vscode-eslint/blob/53cbf1b315ef3ae898d4ca6f1313272c05746a5c/server/src/eslintServer.ts#L510 which I do by validating an empty content with a fake file location.

Same trick is used here: https://github.com/microsoft/vscode-eslint/blob/53cbf1b315ef3ae898d4ca6f1313272c05746a5c/server/src/eslint.ts#L1065

I think the flat config files will ease that, but the validating of the config file is still something I would need to fake since I haven't found API to do this.

[nzakas]

Thanks for those details! When you say that you're validating the config file, does that mean you're just validating its existence? Or are you actually wanting to make sure it doesn't cause an errors as its loaded?

For flat config, we could potentially expose a config file loader API. I need to think a bit more about that.

And do you have any examples of when plugins didn't work in VS Code due to the long-running session? If you have any issues you could point me to, that would be helpful.