Properly supporting semver, plus -DRAFT releases of packages

[tschneidereit]

In the Wasmtime meeting today, we discussed some challenges around semver support for component model packages.

Specifically, we identified two issues:

1. Right now Wasmtime's semver support is overly permissive, in that it makes symbols available in all semver-compatible versions, not just starting with the version they were introduced in.
2. How do we support preview/draft releases of upcoming semver-compatible versions, including treating them as semver-compatible with the last stable version? (Assuming the draft version isn't semver breaking.)

---

During the meeting, I proposed an approach that'd involve maintaining a side-table of symbols to add to the linker under aliases, where for each minor version we'd add all current contents of the side table to the linker aliased for that version, and additionally add all symbols introduced by that version to the table (and the linker.)

I think we could abstract that away by introducing methods on the linker along the lines of

• start_semver_range
• add_to_semver_range (which would invoke add_to_linker)
• end_semver_range

As a concrete example, say we want to support versions 0.2.0, 0.2.1-DRAFT, 0.2.1, and 0.2.2 of a package. (Setting aside the question of whether we'd ever want to support a -DRAFT version in parallel to the respective stable version or any higher minor versions.) This approach would have us follow these steps:

1. Call start_semver_range
2. Call add_to_semver_range for each symbol in 0.2.0
3. Call add_to_linker (❗) for symbol in 0.2.1-DRAFT
4. Call add_to_semver_range for each symbol in 0.2.1
5. Call add_to_semver_range for each symbol in 0.2.2

I'm not particularly familiar with the workings of Wasmtime's linker, so this might not work all that well in practice—but maybe it does?

[alexcrichton]

At a high level one of the goals I'd like to have with our WASI support in Wasmtime is that we're not required to keep an ever-expanding list of copies of WASI APIs, e.g. 0.2.0, 0.2.1, 0.2.2, .... If we take this as a goal, however, I don't think that we can solve problem (1) listed above because WIT itself doesn't retain a version number of when a symbol was first defined. Personally I don't think this matters too much in the sense that you would have to really go out of your way to create a component that imports 0.2.1 APIs at an 0.2.0 version and there's not necessarily a big consequence.

Additionally with adding a symbol to each semver range, if we take the above goal as one to pursue, I'm not sure how we would do that. I'm not sure how we could enumerate all the semver versions that a symbol was known at.

Otherwise though in terms of how the Linker type works you can think of it as a glorified component instance. You get to define exports and each export can itself be an instance with its own exports, so you end up with a tree-like structure where you defined all the names in the tree and the values at the leaves. So today add_to_linker is basically building the part of the tree with WASI names, for example.

The semver compat stuff at this point is currently all built where for each component instance in the tree there's a table in that instance of "if you're looking for something semver compatible with 0.2.X then go load import 0.2.3" (or whatever the max registered version is). That's what gives rise to the behavior of if you import timezones at 0.2.0 then you'll get a "hit" on timezones at 0.2.1.

I explain the Linker bits because one issue with the per-symbol versioning is that it's not quite the layer where versioning happens. At the component layer what's happening is that a component is importing a versioned instance from the host, but then in the instance there are no versions. For example wasi:cli/environment@0.2.0 has a version but get-environment does not have a version. This means that there's really a concept of per-symbol versioning but instead you can think of a version as applying to an entire subtree (e.g. an instance). With that if a function were added to an interface then given the above scheme I think it gets a bit tricky.

---

Speaking again at a high level in terms of requirements, I think another requirement we'd like to have is to avoid the need to have lots of "implement this one WASI proposal in terms of this other one". For example in Wasmtime we don't want to manually implement all of 0.2.0 in terms of 0.2.1.

Now in terms of trying to propose a solution instead of just critiquing them: the best I can think of is to maintain the last stable WASI release and the next draft we're supporting. This would then look like:

• The wasmtime-wasi crate would only have generated bindings for the latest draft that Wasmtime supports, for example 0.2.1-draft or whatever rc/date combo is currently being used.
• A new macro is added to generate an add_to_linker for the last stable release, e.g. 0.2.0.
  • This new macro only generates add_to_linker, no types.
  • This new macro defines names at the 0.2.0 version but uses the 0.2.1 traits to implement the types. If the 0.2.0 traits are actually semver-compatible this'll compile just fine.
• In the CLI by default we call the 0.2.0-based add_to_linker. If a CLI flag is passed then the 0.2.1-draft-based version would be called as well. (note that this is in addition to, not as a replacement of)

Then as part of the ecosystem we'd have an optional feature perhaps on the wasi crate for enabling 0.2.1-draft (or maybe a prerelease of the crate itself? unsure.). That would generate bindings alongside the existing 0.2.0 ones perhaps in a submodule or similar (depends on the publication process of the crate itself I think).

[alexcrichton]

I'll note though that my proposed idea does not solve (1), it leaves it as an open problem to be solved in the future, if at all

[alexcrichton]

I'm going to close this as I think it was addressed by #7994 and the remaining bits are covered by #8395