Backport OCI Bundle watcher

[ricardomaraschini]

Is your feature request related to a problem? Please describe.

On versions v1.28 and v1.29 K0s does not load new OCI bundles without a process restart. This feature has been delivered on v1.30. We currently have clusters running on these "old" versions and we would like to have this back ported if possible.

Describe the solution you would like

The back port involves a small conflict resolution on a file and I have prepared two PRs. I wish they were taken into consideration:

• [Backport release-1.28] feat: implement watcher for oci bundles #4428
• [Backport release-1.29] feat: implement watcher for oci bundles #4429

I have tested and the OCI bundle loader works as in v1.30.

Describe alternatives you've considered

On an airgap environment after updating the image bundle we considered restarting k0s because (autopilot only does it when the k0s version changes).

Additional context

Besides the small conflict I have found when back porting this I also had to bump github.com/fsnotify/fsnotify from v1.6.0 to v1.7.0 on the release-1.28 branch (the former version does not support buffered channels).

[twz123]

Copying over my answer in #4321 over here:

Actually, I'm a pretty conservative guy when it comes to backports. I wouldn't want to port it back so much because it's a pretty substantial change, adding new behavior, not fixing a broken one. Shipping your on k0s builds with the backport applied wouldn't be an option for you in this case? (Just asking 🙃)

[chris-sanders]

We can carry a forked build, but it's something I try to avoid, especially since version 1.29 still has a long support window. Our preference is always to contribute to the project rather than maintain a fork.

Regarding this being considered a new feature rather than a bug: I see it differently. If we could inform users that "this feature is available starting in 1.30," I would be content. However, my understanding is that an advertised and documented feature (airgap image updates) doesn't work without this change. It seems the intention was for users to use autopilot to update images, but there is no way to do that successfully without manual intervention.

This is precisely why we need to maintain a fork if this change isn't backported. We rely on this documented capability and found that it doesn't fully work, hence the need for the update. There is a possibility that other users, like us, are relying on this feature based on the documentation and would also consider this a bug.

[chris-sanders]

@jnummelin we can carry a fork or we can help land this, we're getting to a point that we'll need to make a call. Have y'all come to a decision how we should proceed?

[jnummelin]

@chris-sanders @ricardomaraschini Well, this could be argued both way between a bug and new feature. 😅

After careful consideration and weighing in the risks and benefits, we’ve decided to go ahead with the backports. So those will get into the next patch releases on maintained branches. What tipped the scale is that this is community contribution and it’s very important for us to keep the community happy. Our general stance is still conservative backporting things that are more towards features.

[chris-sanders]

@jnummelin that's fantastic and we absolutely won't make a habit of back porting. Similarly, if there are any issues with these back port versions related to this change we're happy to look at them as well. Please don't hesitate to ask for help if this does cause an issue down the line.

[twz123]

@chris-sanders A follow-up related to the OCI bundle watcher is #4433. Perhaps you find this important enough to start specifying and implementing this?

[twz123]

The backports are merged.