Expand range of configurations tested on CI ("latest" and "defaults/lowerbound")

[kelset]

Opening this conversation to collect some ideas about a problem we've hit a couple times recently in the 0.72 release cycle but that has been around since forever: there have been a couple of occasions in which changes have been introduced (in main) that would only be compatible with Xcode 14.3, forcing developers to update their toolchain if they wanted to use these new patch releases.

Namely:

• facebook/react-native@8b1bf05 this one never made it into a patch release 'cause we realised in time that would force consumers to upgrade to Xcode 14.3
• facebook/react-native@a4a0655; we didn't catch this in time and it created effectively a breaking change in a patch release (0.72.2) -> After updating from 0.72.1 to 0.72.2 I can't make a build (Xcode 14.2) facebook/react-native#38294 that we had to revert via 0.72.3.

In these specific scenarios, why is this considered a blocker if Xcode 14.3.1 is the latest stable release of the tool? Two data points:

• some CIs like GitHub Actions do still default to Xcode 14.2 in their images
• Apple's policy is that you need to use Xcode 14.1 or higher

So introducing those changes is an arbitrary breaking change that would disrupt VALID developers' workflows and product pipelines - which is not good.

In terms of compatibility tables, on the Android side of the world the AGP version creates the lowerbound of the Android Studio version that is supported, as described here developer.android.com/studio/releases#api-level-support (which makes it easier to track what tools will work with what version) and the Google Play requirements set the lower boundary for target API, for example right now from August 31st is going to be "New apps must target Android 13 (API level 33) or higher; except for Wear OS apps, which must target Android 11 (API level 30)".

It would be great if we could figure out a way to honour the lowerbound accepted configurations without necessarily forcing React Native itself to stick to these "older" versions. Or, to say in another way: how can we ensure that situations like facebook/react-native#38294 don't go undetected and get flagged only when developers hit the problem (which then force a rapid response to address)?

Me and @fortmarek have been discussing this and we were wondering if a good way to create a first check would be to expand the CI to test both what is being currently used ("latest") AND this lowerbound set up ("defaults" or "lowerbound"). We do have something in place for testing Node's current LTS and the previous (nodelts and nodeprevlts) and Ruby (with versions 2.7.7 and 3.2.0).

For example, if we had in place a similar thing for Xcode's CI config, those changes breaking in versions lower than 14.3 would have been detected.

These "compatibility checks" would need to be verified right before every branch cut - so that there is a clear signal of what a version of RN is compatible/expected to be compatible with (and so not to make the versions go stale/forgotten).

Thoughts? Opinions? Leave them below!

[liamjones]

I haven't really got anything to add beyond saying, from a purely selfish point of view as an RN user, this sounds like a good idea to me.

From past experience with RN and native iOS development, forced Xcode upgrades for developers can be problematic as;

1. They are sometimes tied to a macOS upgrade as well. That can then result in other unforeseen issues completely unrelated to Xcode. Especially if the new OS version hasn't gone through many patches to get the bugs out yet.
2. They sometimes dictate a Swift upgrade too, requiring various alterations to code due to language changes (though I've not really hit issues with this in recent years while working on RN projects).
3. Upgrading Xcode also implicitly upgrades the resulting iOS/etc SDKs that are used for builds. https://xcodereleases.com/ lists the Xcode/SDK version associations. Sometimes, rarely from my experience, this breaks things. For example, we are currently forced to build our RN projects on Xcode 14.2 or earlier as, as soon as we move to the iOS 16.4 (20E5212f) SDK or higher we lose all inline HTML styling from WKWebViews when using UIPrintPageRenderer.

So the longer RN can support Xcode versions that are still valid for generating builds for the App Store the better, in my opinion. 😄

[cortinico]

So the longer RN can support Xcode versions that are still valid for generating builds for the App Store the better, in my opinion. 😄

This is a valid point, but clashes with the reality of facts that we do have limited resources (in terms on CI time and human time), and we can't test every possible combination.

The same problem applies to Android, Java, Kotlin, Gradle, Node versions and so on.

[cortinico]

It would be great if we could figure out a way to honour the lowerbound accepted configurations without necessarily forcing React Native itself to stick to these "older" versions

Yup I think we should define rangers for each of the tool we support and have a CI job that tests lowerbound + default.

[kelset]

I think we should also think about which tool we need to support; for example, I don't think there's much value in checking both Node LTS and the previous LTS.

[cortinico]

Yup for Node, we probably need just the latest LTS + the current stable (so in this case 18 and 20)

[lunaleaps]

Yea I'm all for adding another configuration with the lower-bound Xcode tested. Would this have caught this issue though, was it just a 14.2 issue? Is it too costly to test all minor versions in between for building?

[cipolleschi]

I tried to change the executor for a couple of jobs to Xcode 14.1, in order to have a lowerbound check: facebook/react-native#39602

It turns out, the executor which Xcode 14.1 comes with node 16... and we need node 18 for React Native.
I'm looking into how to install a new version of node on CircleCI