Switch the container image from Fedora 39 -> RH UBI 9

[eskultety]

This PR switches the existing Fedora 39 container image to the RH UBI 9. A couple of tweaks were needed along the way to be able to perform the switch. This can be (doesn't necessarily need to be) perceived as a prerequisite to ultimately building the image in a multi-stage fashion.

Discussion: #401

Maintainers will complete the following section

• Commit messages are descriptive enough
• Code coverage from testing does not decrease and new code is covered
• Docs updated (if applicable)
• Docs links in the code are still valid (if docs were updated)

Note: if the contribution is external (not from an organization member), the CI
pipeline will not run automatically. After verifying that the CI is safe to run:

• approve GitHub Actions workflows by clicking a button
• approve the Red Hat Trusted App Pipeline container build by commenting /ok-to-test
  (as is the standard for Pipelines as Code)

[eskultety]

since v1:

• silence Hadolint DL3007: Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag as we actually always want to test against the latest image and get early feedback when anything changes rather than rely on an old image tag

EDIT: Is there a way to silence hadolint via CLI rather than a commentary in the Containerfile as that would be the preferred solution.

[eskultety]

Since v2:

• rebase on top of main
• reword commit 1

[eskultety]

ping. I still need to have a look at the integration test failure but I'd like to get first round of reviews on the general idea.

[eskultety]

ping^2

[taylormadore]

I believe that yarn v4 will require node >= 18. How do we want to handle that?

[eskultety]

I believe that yarn v4 will require node >= 18. How do we want to handle that?

With multi-stage, which would come next, that would give us all the flexibility we need, both in terms of node an Go.

[eskultety]

I found one fairly annoying downside to using UBI - for some weird reason it doesn't ship with createrepo_c which the base RHEL does ship with. We'd get it installed via pip to /usr/local/bin/ but with #544 that would change because what is proposed there is to install cachi2 inside a virtual env and copy that one to the final image. I'm mentioning it here rather than in #544 is that although we can symlink createrepo_c from the virtual env, but I'm open to a broader discussion here if it turns out that this might turn into a maintenance nightmare with respect to UBI's limited set of packages and instead we might consider something like AlmaLinux or RockyLinux instead (afterall, both are forks of RHEL).

[eskultety]

I found one fairly annoying downside to using UBI - for some weird reason it doesn't ship with createrepo_c which the base RHEL does ship with. We'd get it installed via pip to /usr/local/bin/ but with #544 that would change because what is proposed there is to install cachi2 inside a virtual env and copy that one to the final image. I'm mentioning it here rather than in #544 is that although we can symlink createrepo_c from the virtual env, but I'm open to a broader discussion here if it turns out that this might turn into a maintenance nightmare with respect to UBI's limited set of packages and instead we might consider something like AlmaLinux or RockyLinux instead (afterall, both are forks of RHEL).

For example if I try Rocky Linux's take on UBI then createrepo is present and unlike AlmaLinux Rocky still claims to be 100% bug-for-bug compatible with RHEL, i.e. a RHEL fork.

[root@aea9a6132ee5 /]# dnf search createrepo
Rocky Linux 9 - BaseOS                                                    1.6 MB/s | 2.2 MB     00:01    
Rocky Linux 9 - AppStream                                                 3.1 MB/s | 7.9 MB     00:02    
Rocky Linux 9 - Extras                                                     38 kB/s |  15 kB     00:00    
=================================== Name & Summary Matched: createrepo ===================================
python3-createrepo_c.x86_64 : Python 3 bindings for the createrepo_c library
======================================== Name Matched: createrepo ========================================
createrepo_c.x86_64 : Creates a common metadata repository
createrepo_c-libs.i686 : Library for repodata manipulation
createrepo_c-libs.x86_64 : Library for repodata manipulation
[root@aea9a6132ee5 /]# 

Any thoughts/preferences?

[ben-alkov]

for some weird reason it doesn't ship with createrepo_c

You've hit the main issue with using UBIs as anything other than a base runtime - they're very stripped down.

It sounds like Rocky is the way to go, except at that point, are we really better off than just sticking with Fed?

[eskultety]

for some weird reason it doesn't ship with createrepo_c

You've hit the main issue with using UBIs as anything other than a base runtime - they're very stripped down.

Uhm, not entirely true I guess. See this has nothing to do with runtime, but repo setup. The Rocky flavour I tested with was also marked as UBI, except without carefully constructed limitations of the base repositories. Without disclosing too much detail, let's just say that the same repository on base RHEL contains the package while the RH UBI counterpart doesn't. And it makes sense to keep RH's UBI licensing and intended usage intact I guess, but what determines the package selection that ends up in those repositories is a mystery to me. I'll just close my thought with a simple: my bad for actually not checking the package catalog on the proposed RH UBI which would tell me immediately what I could get inside and what I couldn't.

It sounds like Rocky is the way to go, except at that point, are we really better off than just sticking with Fed?

Well, pretty much anything is really better than Fedora for a production use (i.e. other than CI to uncover potential problems early) since it won't suffer from the 13 months EOL period.
EDIT: though I wonder if there are any surprising deps coming that wouldn't be available even on distros like Alma or Rocky, I guess only time will tell.