Idea for supporting experimentation and providing early access to features

[joshlf]

We frequently encounter two problems:

• We want to design a feature but not make it available until it's fully baked; currently, this means that we spend a lot of time with features in PRs and try to make them perfect before we merge to main
• Users often want features which we know in broad strokes how to support, but haven't landed because we haven't finalized the design (e.g., bytes-to-reference-type conversion, which has been requested by both Ti50 and musli-zerocopy)

Proposal: We adopt Rust-style support for "unstable" features. We would either create one Cargo feature per feature, or alternatively create a single "unstable" or "experimental" Cargo feature behind which all such features are gated. There is a much lower quality and stability bar for introducing a new feature which is gated in this way. Once a feature has baked enough, we can un-gate it.

Open questions:

• What stability guarantees do we provide? Options:
  • None: we can make breaking changes even between patch releases. This is easiest for zerocopy, but requires consumers to either accept the risk of breakage or to pin to a particular version of zerocopy.
  • We use normal semver rules, but unlike truly stable features, we make no attempt to minimize breakage between semver-breaking releases. This is friendlier to consumers, but significantly constrains what we can do.
  • Hybrid approach: We introduce both types of stability guarantee. We use the "no stability promise" option for features which we don't expect users to use (ie, this is just a way of making development convenient for us, and is roughly equivalent to how we currently use #[doc(hidden)]), and we use the "normal semver rules" option for features which we expect users to use but whose exact shape we're not confident in (bytes-to-reference-type conversion is a good example of such a feature).

cc @kupiakos @jswrenn @djkoloski

[kupiakos]

My vote is split:

• 60% "normal semver rules": major version bumps aren't what's hard, the breaking changes are
• 40% hybrid approach: marking unstable components as explicitly unstable with a Cargo feature is valuable for preventing users from accidentally jumping onto an API that will be deprecated in the next month

None

I vote 1000x against this: semver and all it brings is distinctly useful. Only #[doc(hidden)] components that are explicitly marked as unstable should be able to break in a patch version - this is normal.

We use normal semver rules, but unlike truly stable features, we make no attempt to minimize breakage between semver-breaking releases. This is friendlier to consumers, but significantly constrains what we can do.

This constraint is self-determined: it is for zerocopy to determine an acceptable level of churn. It is not any more expensive to increment a breaking version than it would be to break every user in a semver-incompatible way with a breaking patch version. I see all pre-1.0 libraries as containing zero stable elements - if you want to make a breaking change, increment the major version. A major version increase doesn't inherently require churn, it's the breaking changes from migrating between them that are really at issue. If you go this route, I expect to reach 0.20.0 or so before stabilization.

Hybrid approach

This is the way to publicly experiment with breaking changes on unstable features without incrementing a major version (whether that avoidance is misguided or not). All unstable features should be guarded by an optional #[cfg(feature = "unstable")] which documents its semver incompatibility. The library should still be guarded against breaking changes with the feature disabled.

[joshlf]

Mmmk so IIUC, it'd be acceptable from Ti50's perspective to have a Cargo feature whose semantics are that we will make less of an attempt than we normally would to avoid breakage even across semver-incompatible versions? E.g., if we were to add a method like the following:

trait FromBytes {
    #[cfg(feature = "unstable")]
    fn ref_from_bytes(bytes: &[u8]) -> Option<&Self> { ... }
}

...and across semver versions, we might rename this, remove it in favor of a different API, etc, that'd work for Ti50?

[kupiakos]

we might rename this, remove it in favor of a different API, etc, that'd work for Ti50?

I've uploaded #526 for a helper design.

I would personally rather this feature in particular not be gated behind unstable, and if it is, then read_from should be as well, since it has many of the same design questions. I would rather have a change to that design require a major version bump then have to pin all of our versions and make it more onerous to receive patch-level improvements.

The hesitance to have a non-gated design for a trivial helper like this in a pre-1.0 library is, frankly, frustrating as a user, but we could work with it if you believe stakeholders would run into problems with its existence being "stabilized" as-is. I would request a clear path to remove the feature gate with specific design questions/concerns that must be resolved. Otherwise, I worry it will languish as "unstable" indefinitely.

[joshlf]

Discussed this offline with @djkoloski, and came to the following conclusions:

There are fundamentally two axes here, and they can be handled orthogonally.

Feature baking & rebase hell

We often want to design a feature or set of features that are too large to go in a single PR, and take a long time to develop. Currently, we attempt to develop in stacked PRs similar to Gerrit, but GitHub's support for these is atrocious. Even if GitHub supported stacked PRs seamlessly, we still run into rebase hell as multiple people land other changes.

What we want is to treat zerocopy like a monorepo, and develop all features together to avoid stacked PRs and rebase hell. A solution would require:

• We need to be able to make API breaking changes at any point, so we can't be tied to semver stability
• We need a way of keeping track of what's left before a feature can be "released" as stable
• We need to make sure that external users either can't rely on the feature or can only do so if they're aware of the semver stability hazard

There's a large design space here, with questions like:

• Should we use Cargo features or raw --cfgs?
• Do we just have a single Cargo feature/cfg or one per feature?
• How do we ensure users can't enable a feature without being aware of the stability hazard?
• How do we keep track of what's left before a feature can be stabilized, and how do we ensure we never stabilize without completing this checklist?

This deserves its own separate discussion thread, which I'll open at some point.

This roughly describes Rust's nightly feature concept - we need something similar for zerocopy.

Features subject to rename & redesign

Some features are fine as-is, but we expect we may rename them or remove them in favor of differently-shaped APIs at some point in the future. David convinced me that, for these, it's sufficient to just use normal semver rules. If we want to be extra nice to users, we can try to provide smooth upgrade paths. E.g., if we rename a method, we can leave the old method but mark it as #[deprecated]. For APIs which are not merely renamed but replaced by a different way of accomplishing the same task, the deprecation message can provide a URL to a page that provides more detailed migration instructions. David's key insight is that users don't mind breaking changes - they mind breaking changes without an obvious upgrade path.