✨ Feature request — expose metadata about supported node.js APIs

[IgorMinar]

The use case

As we improve our Node.js compat story, we'd like to dynamically assemble the Node.js compat layer using a mixture of APIs provided via the runtime as well as via polyfills injected at build time (by wrangler or custom build tooling).

For APIs that are already offered by the runtime, the runtime implementation should be used. Since the runtime node.js coverage is expected to grow a little over time, we'd like the build tooling to know what APIs are provided by a given version of runtime and a given compatibility date.

By being able to consider the compatibility date and corresponding Node.js API coverage at build time, we'll be able to fine-time the build-time polyfill to absolute minimum, and also avoid situations where users could build and successfully run an app locally using a runtime that is on npm already but not yet fully rolled out to prod.

Additionally we'll also be able to introduce new Node APIs into the runtime without it becoming a change with unexpected side effects developers.

The ask

We'd like the workerd npm package to contain a metadata file in a format that is easy to parse in javascript (ideally json), that contains information about our node.js coverage enabled using the --nodejs_compat flag.

The metadata file should group supported node APIs by compat date when they were enabled. The APIs should be listed at the symbol-level granularity.

An example format could look as follows:

nodejs-compat.json

{ 
  "2023-05-20": ["buffer#Blob", "assert#AssertionError", "assert#deepEqual", "assert/strict#deepEqual"],
  "2024-03-03": ["buffer#Buffer", "path#join"],
}

In theory we might want to have a more generic metadata file (e.g. runtime-compat.json instead of nodejs-compat.json) that lists information about all flags and compat dates which then downstream tooling like wrangler or miniflare could use. I don't have strong opinions about that, as long as such change doesn't expand this scope of this request too much and doesn't result in long delays.

Maintenance considerations

Ideally this file is autogenerated rather than manually maintained as part of the bazel build before the package is published.

[IgorMinar]

// fyi: @petebacondarwin - filing the issue as we discussed.

[mrbbot]

Hey! 👋 Is this something you could derive from workerd runtime-type-information? #1959 includes changes to generate the subset of @types/node supported by workerd using the import("workerd:rtti").default.exportTypes(compatDate: string, compatFlags: string[]): ArrayBuffer function:

https://github.com/cloudflare/workerd/blob/bcoll/types-as-a-service/types/src/worker/index.mjs#L54-L67

This function gives you a Cap'n Proto encoded representation of runtime APIs that can be parsed in JavaScript (and maybe converted to a more friendly JSON format ahead-of-time). Lots of the nodejs_compat APIs are written in TypeScript. For these, the TypeScript declaration file is included in the Cap'n Proto message, which could be parsed with the TypeScript compiler API ahead-of-time to extract supported features (assuming only supported APIs were declared and not stubbed out).

[IgorMinar]

thanks for the pointers Brendan! this API could indeed be used to collect the info and produce the metadata file as requested above.

I do prefer to have a metadata file as part of the npm package so that we don't need to invoke workerd and compute this info every time we run a build.

[jasnell]

The format should also somehow account for the fact that certain APIs might be exposed purely opt-in without a default on compatibility date. For instance, none of the node.js apis are available unless the nodejs_compat flag is explicitly set to on, regardless of configured compat date. As we move forward, we might use either approach (on-or-after-date OR only-with-flag).

{
  "dates": { 
    "2023-05-20": ["buffer#Blob", "assert#AssertionError", "assert#deepEqual", "assert/strict#deepEqual"],
    "2024-03-03": ["buffer#Buffer", "path#join"]
  },
  "flags": {
    "foo": ["other#thing"]
  }
}

[IgorMinar]

@jasnell this metadata file assumes that nodejs_compat flag is on, the purpose of the compat date is to be able to tell when a certain symbol was added to the runtime. Using the compat date in the project the build system can then know which polyfills need to be injected vs which will be provided by the runtime. And as we add a few things to the runtime, the behavior of the application won't change unless the developer changes the compat date, at which point some previously built-time-injected polyfills would be swapped with runtime polyfills.

[jasnell]

Yes, but in the future we might only introduce new node.js APIs behind new opt-in-only flags, those are the ones I'm concerned about. It is also possible to turn on certain apis individually, for instance, there is now a compat flag that makes AsyncLocalStorage available without enabling the rest of nodejs_compat

[IgorMinar]

If we introduce node.js apis behind an api-specific opt-in flag, then we can decide on a compat date when this API-specific flag will be on by default when nodejs_compat flag is on. In other words, introduction of new node APIs would follow the same process as introduction of other runtime APIs except that runtime APIs are enabled via compat dates, while node apis will be enabled via a combination of compat date and nodejs_compat flag.

So for the purposes of this metadata list, we only need to know when the API gets enabled along with nodejs_compat flag.

[jasnell]

The point is that not all compat flags have default on dates. It's entirely possible that we can add a new API with a flag that never has a default on date. We certainly can set a convention that in the typical case new node.js api additions should have a default-on date, but I don't know if it's a good idea to completely rule it out.