Towards plugin 1.0.0: BABLR as the Babel core

[conartist6]

Babel has several significant architectural problems that this proposal aims to address together:

1. Babel only provides utilities for transforming Javascript
2. The @babel/traverse API isn't semver-stable (1.0.0), so neither are plugins
3. Babel can't do codemods: it always destroys formatting
4. Babel interop with other tools is through source files not trees

I have attempted to address these issues by creating a new core architecture to replace @babel/traverse and ESTree. One of my goals is to create a complete new interoperability layer that will unify the core architecture of Babel, Prettier and ESLint.

Since I'm suggesting a truly massive breaking change, I'm well aware that I would have to be offering a similarly huge expansion in capability and ease of use: people would have to want (badly) to migrate across such a breaking change. To do this I envision a complete new role for the combined tools: they would become your IDE! That is to say, your IDE's source of truth for the code on screen would actually be the exact same immutable tree structure used by Prettier and Babel. This code pattern is currently most well known for holding up the entire internet: it's the DOM.

So the proposal is this: give Babel a DOM layer, and design it so well that it's suitable to be shared between several tools and may even eventually be suitable for standardization by TC39.

Because my design was inspired by Babel for its inception, I chose the name BABLR as a mashup of Babel and ANTLR. To me it stands for Babel Ass-Backwards for Language Recognition, in honor of it's rather different control flow strategy: an inversion of Babel's. I've been working on this full time for two years now, and have been building the underlying OSS libraries like iter-tools (and the accompanying streaming regex engine) for even longer.

[nicolo-ribaudo]

This proposal is very ambitious -- from your description it sounds like a complete rewrite of Babel. There have already been other attempts for tools that aim at doing "everything" (formatting, linting, transpilation), so I recommend taking a look at them if you haven't yet: https://biomejs.dev/, https://deno.com/.

Given the complexity that this fundamental rewrite would bring, and that it would basically "reset" the project maturity/stability, I'm not convinced that this should be done as part of Babel and not as a separate project. There have already been multiple attempts at rewriting Babel (https://swc.rs/, https://sucrase.io/), and some of them have been moderately successful.

That said, I also have some more specific questions:

the exact same immutable tree structure used by Prettier and Babel. This code pattern is currently most well known for holding up the entire internet: it's the DOM.

The DOM is a (mutable) tree that can be accessed and modified with various utilities. It only conveys information about the code meaning, and not about how it's styled/formatted. Isn't the tree that @babel/traverse provides like a DOM already?

design it so well that it's suitable to be shared between several tools and may even eventually be suitable for standardization by TC39.

Note that TC39 already tried to standardize a tree format to represent code (https://github.com/tc39/proposal-binary-ast), but there has been a strong pushback to the idea of standardizing a different representation for JS.

Babel can't do codemods: it always destroys formatting

Yeah I agree that this is a pain point. There is actually a way to use Babel for codemods (https://github.com/nicolo-ribaudo/codemods), but we should make it more official/accessible.

The @babel/traverse API isn't semver-stable (1.0.0), so neither are plugins

What do you mean by this? @babel/traverse follows semver -- if we accidentally broke something in a non-major please open an issue.

[conartist6]

First, thank you so much for taking the time to put together this feedback! I appreciate it a lot.

Given the complexity that this fundamental rewrite would bring, and that it would basically "reset" the project maturity/stability, I'm not convinced that this should be done as part of Babel and not as a separate project.

Your point about maturity is quite valid, and indeed that is one of the primary reasons that I have been pursuing my work in this new BABLR organization. That said, the mission of the BABLR organization is more constrained than that of Babel. For example: BABLR is not going to have first-party ownership over JS transpilation rules. It would only want to provide a standard environment in which they're performant and easy to write. This differs from the aims of a project like Biome which aims to take over all Babel's current responsibilities.

My goal is to be able to offer you a version of BABLR that has already gained a hard-fought level of maturity (it's been rewritten roughly five times now), having started two years ago as a design based on the most mature ideas I could find.

Isn't the tree that @babel/traverse provides like a DOM already?

@babel/traverse is as close to a DOM for code as anyone has currently managed to make, and the results prove it: many independently-maintained plugins work together in a single environment to accomplish tasks more complicated than any individual could define alone. The biggest problem is that it isn't universal, it's more like we have several DOMs for code than just one. If I want to ship a codemod, I can't be sure that the person I ship it to won't be using Biome or ESLint or SWC for code transformations. The web DOM shows us that it's possible to have binary-incompatible implementations of trees and still end up with an ecosystem of compatible tools (browsers).

The way things are now the only truly interoperable format is parseable text, and making that work is a perf nightmare. To apply a transformation to text, you have to parse it, alter it, print it, and then every other tool that cared about that code has to parse the printed output again. The dream is to be able to do something as simple as prettyFormat(transpile(generateCode())).

What do you mean by this? @babel/traverse follows semver -- if we accidentally broke something in a non-major please open an issue.

When I say that traverse isn't following semver, what I really mean is that it doesn't document a public API, which is a (soft) requirement for semver according to the semver spec. I also believe that it does not adhere closely enough to the single responsibility principle for semver's taste, having in it functionality like Javascript scope analysis and variable renaming. My traversal package by comparison has no language-specific semantics defined in it, ensuring the API will continue to be stable even if the language people use it to transform changes.

Another design note here is that the complete surface area of the traverse package includes the path object, and it's often unclear which parts of that object are public and which are private (docs could say, but don't). In my code I use a thin facade over an internal path implementation, which is to say that I make it impossible for plugins to alter the behavior of the core by tampering with internals, which in turn allows me to make much stronger guarantees about the overall behavior of the system, particularly when the behavior of the system is defined by a mixture of trusted and untrusted code.