Multiple versions of Runtime API within single host

[tdimitrov]

We discussed a way to have staging functions inside the runtime api of Polkadot. The full discussion is here but I'll resume it with a few sentences. The purpose is to push functionality which we want to deploy on a real test network but not to confuse it with stable api functions. What we did was to prefix the function names with a Staging prefix, add implementation to the desired runtimes (e.g. the test ones - Westend and Rococo) and use unimplemented!() for the rest.

This approach sort of worked. However we ran into a limitation. Adding staging functions breaks the versioning because we don't bump the version after we add a staging function. Due to this misuse we lose the possibility to check if the Runtime contains the staging code that we need on the node. This means that we can't push changes to the test network and handle old runtimes gracefully as for example we can do with stable (versioned) runtime api. We need a way to overcome this.

One option is to bump the version with each new call we add/change. This will solve the version check problem but I suspect it will be a pain to maintain because the versions will increase rapidly.

Another option which we like (by saying we - I mean the Parachains team as a Substrate user) is to be able to toggle the staging functions on and off 'somehow' and have separate version for them so that it doesn't interfere with the stable one too much. I understand though that this deviates a lot from the current philosophy of the runtime api.

So can we talk about what's the best way to give the staging runtime api an actual version which is independent from the stable one and have the Runtime report it when the a staging (feature) flag is set?

@bkchr @rphmeier

[eskimor]

So in a way we actually want two runtimes to be built .... One stable one and the staging one with an incremented version number. Stable will be the default (native) and we can upgrade test networks to the staging one via sudo and see whether it works. Or what procedure do we have in mind for actually testing the upgrade?

[bkchr]

This approach sort of worked. However we ran into a limitation. Adding staging functions breaks the versioning because we don't bump the version after we add a staging function. Due to this misuse we lose the possibility to check if the Runtime contains the staging code that we need on the node. This means that we can't push changes to the test network and handle old runtimes gracefully as for example we can do with stable (versioned) runtime api. We need a way to overcome this.

We could add an error variant that tells you if a runtime api function was actually found. Then you could match and act on this?

[rphmeier]

No, I think we really need to tell the version. And even better if we can swap the versions used by the runtime based on storage (I'd like governance to be able to enable new runtime API versions). If we can't do that, then we need to build another versioning mechanism on top of the runtime API in order to tell when the "protocol API" version has changed.

[bkchr]

This is something completely different, but I like the idea. So, you would enable governance to "enable" certain runtime apis and they would return "unavailable" (or whatever) when they are not enabled?

[rphmeier]

Yes, exactly, something like that. Would you suggest implementing this as a new RuntimeApiError::Unactivated(Version) or something along those lines?

We'd want the runtime to invoke different functions based on the runtime API version that is 'activated' and also accommodate any differences in function signatures or function names between runtime API versions.

Do you think adding extra runtime conditionals which have to read a storage entry to determine the activated version of the runtime API would be a significant overhead?

The reason I'd really want this is so we can have governance explicitly vote to enable new features that are triggered by runtime API versions. Then e.g. for asynchronous backing or SASSAFRAS we can have the node logic support both and trigger it with a governance motion.

[bkchr]

Do you think adding extra runtime conditionals which have to read a storage entry to determine the activated version of the runtime API would be a significant overhead?

I don't think so. It would be just one read and it is offchain anyway.

[tdimitrov]

We could add an error variant that tells you if a runtime api function was actually found. Then you could match and act on this?

I think this will work. You won't have exact versions but you'll know that your runtime is 'not recent enough'.

@eskimor @rphmeier what do you think?

[eskimor]

We can already do that. The thing is if you are checking for the presence of the function, that is not quite the same as checking a version.

In particular: If the function was not yet added, (the case we are discussing) we just want to handle it by simply behaving in the old way. What could also happen, is that the function gets removed at a later point in time - in that case we would at least want an error message, so we will detect early in debugging that a needed function was accidentally removed.

That's actually the reason we bother about versioning at all, otherwise we could always just check whether the function is there or not and versioning becomes irrelevant (to the node).

[tdimitrov]

We should find a way to make the staging api more like a first class citizen and less relying on assumptions. To resume what we need:

• Ability to enable and disable the staging code (or ability to enable it just for specific runtimes).
• Versioning of the staging code.
• Ability to promote staging code to stable. This should be doable with the current versioning macros in substrate.

@bkchr is this doable in general and do you see any issue introducing such concept in substrate?

[rphmeier]

So in short: we want the node to be able to declare both the staging runtime API and the 'current' runtime API. The staging runtime API should have its own version so it can be detected at runtime. We should not have 2 types for this, because then you have to write a lot of dispatch logic based on the version and add a lot of trait bounds. We should have the ability to maintain multiple staging versions at once

[bkchr]

add a lot of trait bounds

This is something that will be removed in the future anyway, because this will not work anymore with the native runtime being removed.

We should have the ability to maintain multiple staging versions at once

Multiple staging versions of what? Do we speak here about the same method in multiple versions? Or do you mean different functions, but each of this function has its own version attached?