Tracking issue for manylinux_2_28 image

[mattip]

Due to issue #1012, the manylinux_2_24 image proposed as the first PEP600 compliant image did not take off. I think we should offer an image based on manylinux_2_28.

The big question is what base image to use. We can choose between C8S (centos:stream8), AlmaLinux, RockyLinux, or UBI. Personally I would prefer an image based entirely on open source software. There is a thread about this on discuss.python.org

base image	EOL ¹	gcc-toolset-11	aarch64	ppc64le	s390x	bug fix availability
CentOS Stream 8	2024-05-31 ²	yes	yes	yes	planned ³	1st
UBI 8	2029-05-31	NO	yes	yes	yes	2nd
AlmaLinux 8	2029-05-31	yes	yes	yes	soon ⁴	3rd
RockyLinux 8	2029-05-31	yes	yes	planned ⁵	planned ⁵	3rd

^{¹ including Maintenance Support, https://access.redhat.com/support/policy/updates/errata/
² https://wiki.centos.org/About/Product
³ #1282 (comment)
⁴ https://almalinux.discourse.group/t/plans-for-s390x/875
⁵ after RockyLinux 9 is out, https://forums.rockylinux.org/t/rocky-linux-8-4-available-now/3015/48}

The checklist is short, since wheel and pip, which both use packaging, and warehouse do not require additional PRs.

---

Publisher-side Support:

• Provide a manylinux2_28 docker environment, feature: add manylinux_2_28 image #1277 (almalinux based PoC)
• New release of auditwheel (might not be needed to get things started, c.f. test: add manylinux_2_28 tests auditwheel#369)
  • Find BETA testers to check wether changes are required immediately, especially C++ projects have a chance to get a failure on repair, knowing how much they're affected will help to prioritize work on auditwheel.
• Upload the new images to quay.io

Documentation:

• ???

Additional projects to check for support for pernnial wheels (left over from gh-542, perhaps these are already done?

• Add an option to target manylinux_2_28 in cibuildwheel feature: add manylinux_2_28 support cibuildwheel#1026
• Add an option to target manylinux_2_28 in multibuild
• Add an option to target manylinux_2_28 in maturin
• Distinguish between images in dockcross
• Develop list of further relevant helper utilities and update them

[tiran]

We can choose between AlmaLinux, RockyLinux, or UBI.

Is there any particular reason why C8S is missing from the list? centos:stream8 is an option, too. After all AlmaLinux and RockyLinux are getting RPM updates from C8S.

Personally I would prefer an image based entirely on open source software.

All images are entirely based on OSS and freely redistributable. (If you want to be really strict, then you have to exclude all three distributions, because they may contain proprietary firmware blobs.)

[mattip]

Is there any particular reason why C8S

No, I added C8S to the issue description.

All images are entirely based on OSS and freely redistributable

Cool, thanks for the clarification, sorry for any unintentional dispersion that I cast on any project.

[mattip]

@tiran, @kpfleming, @mayeut, @h-vetinari, do you know canonical links to docker images for glibc2_28 linux distributions? Any thoughts as to which distro we should use? I have really no idea how they all relate to each other, and would like to avoid any kind of minefield in our mutual goal to get to a stable base image.

[kpfleming]

We don't need to be concerned about firmware blobs, since only container images are under discussion here :-)

Also, I know it's being pedantic, but the 'container' world consists of more than just Docker in modern times. You may want to consider just referring to the images as "OCI images".

Most of the relation discussion has happened in the Discourse thread. All four image flavors you've mentioned originate from the same source packages (CentOS 8 Stream), and each packager builds their own binary packages.

The UBI8 image is available here. I don't have links for the other images.

[h-vetinari]

do you know canonical links to docker images for glibc2_28 linux distributions?

Assuming we're not talking about canonical-the-company (😉), there's https://hub.docker.com/_/almalinux and https://hub.docker.com/_/rockylinux. @mayeut seems to have experimented with almalinux already.

I don't have a strong preference regarding which distro (I'm guessing one deciding factor might be availability of architecture support like PPC, or perceived viability of the community; here the RH-sponsored flavours obviously have a big advantage...), though I don't understand one aspect about C8S that might be pertinent. Previously, updates flowed from RHEL to CentOS. With stream, this is supposed to be inverted, at least starting with RHEL9.

What I don't know if C8S now lives up- or downstream of RHEL, which - as @tiran and @kpfleming explained - probably matters quite little because the alternatives seems to be using the same sources (though given the history of COS8, I wouldn't be surprised if it's still downstream and just picked up the new "stream" name?).

Where this becomes relevant is if C8S now still is downstream of RHEL (as before), but won't be anymore as of RHEL9 - then having the same name ("centos stream") will imply we should just bump the version 8->9 at some point in the future, when in fact we'd be getting a very different thing. Is what I'm saying understandable, @tiran @kpfleming? Or completely off in some way?

[kpfleming]

If you've got a bit of time, one of our RH colleagues presented a session at FOSDEM 2022 last weekend which shows how the branches and releases relate to each other.

CentOS Stream 8 and CentOS Stream 9 will always be separate, the 'stream' name does not mean that C8S users will just magically get converted to C9S when it is released. It's not a 'rolling' distribution, if that's what you are concerned about.

[h-vetinari]

It's not a 'rolling' distribution, if that's what you are concerned about.

What I'm concerned about is C8S not being a rolling distribution (and thus a viable candidate here), but sharing the name with C9S, which is a rolling distribution (or at least, that's how I understood it so far; I don't necessarily mean "bleeding edge", but by being ahead of RHEL and possibly diverging, it flips the nature of what used to be CentOS on its head).

I would want to avoid a situation where - if we were to choose C8S now - this IMO misleading naming is then used to argue that a future manylinux image should be based on C9S.

[kpfleming]

Understood. I think you can avoid those issues entirely then by just choosing UBI instead of CentOS Stream :)

[bookwar]

What I'm concerned about is C8S not being a rolling distribution (and thus a viable candidate here), but sharing the name with C9S, which is a rolling distribution (or at least, that's how I understood it so far; I don't necessarily mean "bleeding edge", but by being ahead of RHEL and possibly diverging, it flips the nature of what used to be CentOS on its head).

I think there is still some misunderstanding.

CentOS Stream 8 and CentOS Stream 9 share name because they also share the concept. They are two branches of the same thing. So CentOS Stream 9 doesn't "roll" any more than CentOS Stream 8.

CentOS Stream 8 is the representation of RHEL 8 Nightly, CentOS Stream 9 is the representation of RHEL 9 Nightly.

The thing which is different between 8 and 9 currently is that you don't see the merge of the sources happening in 8, you only see the result of it. While in 9 you also see how exactly RHEL developer merges the code for RHEL=CentOS Stream, and how the change passes the RHEL verification stages.

The outcome is still the same: package lands in CentOS Stream 9 after it lands in RHEL 9 Nightly.

[h-vetinari]

OK, thanks for the explanation @bookwar. Can you comment if RHEL9 and C9S will be ABI-locked, or will C9S be allowed to break away and ahead?

[h-vetinari]

(I appreciate the insights; I'd be curious to know what Alma/Rocky will do once C9S rolls around, because the switch to stream created enough FUD that not one, but two projects got launched to fill the perceived void - if RHEL9 and C9S remain fully ABI-compatible, then I don't see much of a functional difference to the old way)

[duritong]

They will stay as ABI compatbile as RHEL minor versions did in the past, since CentOS Stream Y just represents RHEL Y.(N+1) where N is the last minor release of RHEL that got GA. Now when it comes to ABI compatbibility I would refer to RHEL's promise of ABI compatibility within minor releases, since this is essential what CentOS Stream Y will be.

CentOS previously picked up N after it became GA in RHEL and thus could have been lacking (sometimes for months!), now it is the other way around.

[bookwar]

OK, thanks for the explanation @bookwar. Can you comment if RHEL9 and C9S will be ABI-locked, or will C9S be allowed to break away and ahead?

CentOS Stream 9 becomes locked on RHEL 9 ABI compatibility right after RHEL 9 GA.

For pre-release development which is happening now we have the freedom of still changing and adjusting things. Though as RHEL 9 Beta has already been released and we are very close to GA dates there is not going to be any major changes. But after release rules become strict.

[h-vetinari]

OK, thanks for all that - happy to retract my concerns about centos 8 stream in the context of this discussion.

CentOS Stream Y just represents RHEL Y.(N+1) where N is the last minor release of RHEL that got GA

That's the key part IMO. Already previously, the minor version was just that - not very important. Important was (and remains) one cohesive "epoch" represented by the major number + all the ABI stability etc. To be honest, this is a TIL-moment for me, and now I understand way less what the value proposition of alma & rocky is supposed to be.

[tiran]

The official C8S OCI images is hosted on Quay as quay.io/centos/centos:stream8. The image provides developer toolsets like gcc-toolset-11-gcc-11.2.1-1.2.el8_5 out of the box.

[carlwgeorge]

To be honest, this is a TIL-moment for me, and now I understand way less what the value proposition of alma & rocky is supposed to be.

I'll point out that CentOS Stream is the only distribution in the Enterprise Linux family (RHEL and RHEL-like distros) that allows direct contributions to the operating system. Reporting a bug in a RHEL clone like Alma or Rocky will most likely be closed as "reproducible on RHEL, not a bug for us". Their stated goal is to be bug-for-bug compatible with RHEL. Reporting a bug to RHEL will either result in waiting for RHEL maintainers to fix it (in CentOS Stream first) or being guided to contribute the fix to CentOS Stream, which flows into the next RHEL minor release. If the PyPA values being able to contribute to the platform to help shape it to their needs, CentOS Stream is the logical choice.

[mayeut]

Let's add a small comparative table of all the proposed base images in the issue description. I'll try to sum-up and update with all the remarks as they arise.

[mayeut]

I added the comparison table.

EDITED 2022-02-13 19:42 UTC

My 2 cents for how I interpret this table for now:

• CentOS Stream 8:

  • pros:
    • This is the one where bug fix/enhancements will first appear and allows contributions as noted in Tracking issue for manylinux_2_28 image #1282 (comment)
  • cons:
    • It lacks s390x for now
    • It won't receive updates after 2024-05-31 so there will need to be a switch at this point.

  It might be a good option until EOL. I'm not clear about perceived viability of the community yet given the bunch of questions/answers we got in this thread.

• UBI 8:

  • pros:
    • Perceived viability of the community
    • All architectures availables
  • cons:
    • gcc-toolset-11 not available
    • most devel packages are not available.
    • only provides a subset of the packages the other ones provide

  The availability of gcc-toolset-11 (and probably later ones) and most devel packages is a showstopper.
  As was raised in the discourse thread, it's meant to distribute runtime apps. This also appears in UBI8 FAQ#35 screenshot:

  The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities.

  Even if we get the packages needed to build the manylinux images, it will most likely end-up as an issue for downstream projects

• AlmaLinux 8:

  • pros:
    • Provides all packages found in RHEL8, bug-for-bug compatible with RHEL
  • cons:
    • It lacks ppc64le & s390x for now
    • Perceived viability of the community

  This is the one I based my PoC on because, as mentioned earlier in this post, CentOS 8 stream will require a switch in 2024, UBI 8 lacks gcc-toolset-11 and support for ppc64le seems to be just a couple weeks away.

• RockyLinux 8:

  • pros:
    • Provides all packages found in RHEL8, bug-for-bug compatible with RHEL
  • cons:
    • It lacks ppc64le & s390x for now
    • Perceived viability of the community

  Similar to AlmaLinux 8 except support for ppc64le is not as advanced (or so it seems).

[mattip]

I would be happy to test an image on NumPy's CI when available

[mayeut]

@mattip,
PoC image for x86_64 based on AlmaLinux is available docker pull mayeut/manylinux:manylinux_2_28_x86_64.
I will try to get aarch64 & ppc64le PoC images available somehow as well.

[mayeut]

PoC images are now available for testing for x86_64, aarch64 & ppc64le (this last one uses a BETA image from AlmaLinux).
I guess whatever the base image we end up choosing, given they all should be compatible for this use-case, it's worth starting getting a bit of feedback (mostly, I'd be interested in : "does auditwheel repair works for your project").

Images repo:

• quay.io/pypa/manylinux_2_28_poc_x86_64:poc
• quay.io/pypa/manylinux_2_28_poc_ppc64le:poc
• quay.io/pypa/manylinux_2_28_poc_aarch64:poc

[mayeut]

@messense, I guess it would make sense to add https://github.com/PyO3/maturin the list of tools in the description ?

[carlwgeorge]

CentOS Stream 8:
It lacks s390x as do other distros except UBI for now. There are no plans to add it (?) whereas other distros have some.

We plan to add s390x this year as we work towards migrating C8S from the legacy koji instance to the new koji instance, which is part of our plan to align the C8S and C9S workflows. I'm not sure if this has been discussed publicly anywhere yet, but if you're familiar with koji you can see pieces of this being put into place in the new instance, such as tags and build targets.

CentOS Stream 8:
It won't receive updates after 2024-05-31 so there will need to be a switch at this point.

I understand this is not appealing, but for whatever it's worth, at that point CentOS Stream 8 and RHEL8 will be basically equivalent as RHEL 8 moves into maintenance support phase. Assuming that UBI8 has all the necessary packages available by then, it should be a seamless transition.

UBI 8:
gcc-toolset-11 not available
only provides a subset of the packages the other ones provide

Per the UBI FAQ #35, gcc-toolset-11 and anything else required can be requested by opening a bugzilla. I can't make any promises, but if it would make UBI the preferred choice for this, I suspect it would be viewed quite favorably. cc: @fatherlinux

[carlwgeorge]

I'm curious, would CentOS Stream 9 be an option here? It's default gcc is 11 (no need to enable an alternative toolset), it has all four architectures discussed here (aarch64, ppc64le, s390x, and x86_64), and it's maintained until approximately 2027-05-31 (depends on the exact date of the end of RHEL 9 full support phase).

[mattip]

would CentOS Stream 9 be an option here

What glibc does it pin?

[mayeut]

We plan to add s390x this year

Good to know, I posted both on centos-devel mailing list & https://discussion.fedoraproject.org/t/any-plans-to-support-s390x-on-centos-stream-8/36857/4, I now have the answer, thanks.

but if it would make UBI the preferred choice for this

I guess it would
EDIT: Just tested if anything else would be missing:
Error: Unable to find a match: bison gcc-toolset-11-binutils gcc-toolset-11-gcc gcc-toolset-11-gcc-c++ gcc-toolset-11-gcc-gfortran readline-devel tk-devel gdbm-devel libpcap-devel libidn-devel uuid-devel libXext-devel libXrender-devel mesa-libGL-devel libICE-devel libSM-devel
As was said on the discourse thread, UBI8 is probably meant to redistribute runtime containers. As such, there's a bunch of devel packages missing (in addition to gcc-toolset-11).
Even if we get those packages in for the manylinux image, a project downstream will probably end-up missing foo-devel and report here where we'd ask them to follow UBI FAQ#35.
It does not seem sustainable.

I'm curious, would CentOS Stream 9 be an option here?

No, manylinux_2_28 stands for glibc 2.28 and assumes a bit more about a bunch of other versions, mostly GCC, GLIBCXX symbol versions that gets linked in the binary. In this regard, using gcc-toolset-11 on CentOS Stream 8 is different than using the default gcc 11 on CentOS Stream 9.
It might be a viable option for a future manylinux_2_34 though.

[EwoutH]

Took keep momentum, would it be time to start talking about the next manylinux version? Would that be manylinux_2_34, using glibc 2.34, based on a version 9 BaseOS? (maybe AlmaLinux 9?)

[tiran]

Took keep momentum, would it be time to start talking about the next manylinux version? Would that be manylinux_2_34, using glibc 2.34, based on a version 9 BaseOS? (maybe AlmaLinux 9?)

It's a bit premature to start with manylinux_2_34. The wheel would not work on latest Debian Stable or Ubuntu 20.04. Their glibc versions are lower than 2.34. By the way there is a realistic chance that Red Hat will provide support to create a manylinux container based on RHEL UBI.

[EwoutH]

The wheel would not work on latest Debian Stable or Ubuntu 20.04. Their glibc versions are lower than 2.34.

For those OS there is the brand new manylinux_2_28 image right? Currently multiple popular releases like Fedora 35 and 36 and Ubuntu 21.10 and 22.04 LTS already support glibc 2.34+. Also many rolling releases like Gentoo, Manjaro Stable and openSUSE Tumbleweed support it.

The whole philosophy behind PEP 600 was that it could speedup new manylinux version:

And second, every time we move manylinux forward to a newer range of supported platforms, or add support for a new architecture, we have to go through a fairly elaborate process: writing a new PEP, updating the PyPI and pip codebases to recognize the new tag, waiting for the new pip to percolate to users, etc. None of this happens on Windows/macOS; it’s only a tax on Linux maintainers. This slows deployment of new manylinux versions, and consumes part of our community’s limited PEP review bandwidth, thus slowing progress of the Python packaging ecosystem as a whole. This is especially problematic for less-popular architectures, who have less volunteer resources to overcome these barriers.

Also consider that packages can choose which manylinux images they want to build. A popular distribution that cares about performance can build a lot, while smaller packages can choose a few that they see fit.

Especially for Ubuntu 22.04 LTS a new manylinux image could be very useful. That could either be glibc 2.34 or 2.35, but I think that considering we need a base image 2.34 might be easier.

[mayeut]

It's a bit premature to start with manylinux_2_34. The wheel would not work on latest Debian Stable or Ubuntu 20.04. Their glibc versions are lower than 2.34.

Agreed

By the way there is a realistic chance that Red Hat will provide support to create a manylinux container based on RHEL UBI.

Good to know !

Also consider that packages can choose which manylinux images they want to build. A popular distribution that cares about performance can build a lot, while smaller packages can choose a few that they see fit.

You probably won't get better performance since you'll probably end up using the same toolchain thanks to RH effort to have gcc-toolset-* on RHEL8. You'll mostly get higher complexity with no added benefits.

[carlwgeorge]

By the way there is a realistic chance that Red Hat will provide support to create a manylinux container based on RHEL UBI.

I think for manylinux_2_34, UBI 9 is already a great option. It has glibc 2.34 and gcc 11 (as the default, no need for scl enable).

[mayeut]

quay.io/pypa/manylinux_2_28_ppc64le this repository is empty.

there are issues with travis-ci on ppc64le at the moment, all ppc64le images are impacted, not only manylinux_2_28_ppc64le.

ppc64le images have been published.

cc: @messense

[bryanculver]

I noticed when spinning up a manylinux_2_28 container that there is no longer a "default" Python executable (not found in PATH).

Obviously for building it's expected to go to the specific version in /opt/python/ but what (if there is) is the preferred/recommended way to add Python into the PATH, say if we have a helper script that's written in Python?

[duburcqa]

Here is what I'm doing personally:

PYTHON_VERSION="cp39"
pythonLocation=$(find /opt/python -maxdepth 1 -name "${PYTHON_VERSION}-*" -print -quit)
export PATH="${pythonLocation}/bin:${PATH}"

[h-vetinari]

Regarding the tracker on top, cibuildwheel support was merged and released recently, same for maturin already a while ago, and multibuild doesn't seem to require (non-doc) changes?

[h-vetinari]

dockcross now has 2_28 as well: dockcross/dockcross#712

[h-vetinari]

@mayeut: UBI 8:

• pros:
  • [...]
• cons:
  • gcc-toolset-11 not available
  • most devel packages are not available.
  • only provides a subset of the packages the other ones provide

@carlwgeorge: Per the UBI FAQ #35, gcc-toolset-11 and anything else required can be requested by opening a bugzilla. I can't make any promises, but if it would make UBI the preferred choice for this, I suspect it would be viewed quite favorably. cc: @fatherlinux

@fatherlinux: I'll look into that veraion of gcc-toolkit :-)

Sorry for necroposting, but just wanted to check how the gcc-toolkit-on-UBI situation looks like¹. This is less relevant now for UBI 8 (where we've gone with Almalinux already anyway), but it would be relevant for UBI 9 some time down the road.

Footnotes

1. I had a look through the RHEL docs but couldn't find anything about the gcc-toolkit/devtoolset version in UBI. ↩

[carlwgeorge]

UBI 9 currently has GCC 11.3 as the default, and an optional SCL for GCC 12.2.

[root@b4af68881a12 ~]# dnf repoquery --quiet --nvr gcc gcc-toolset-12-gcc
gcc-11.3.1-4.3.el9
gcc-toolset-12-gcc-12.2.1-7.4.el9

UBI 8 currently has GCC 8.5 as the default, and an optional SCL for GCC 12.1.

[root@9655f383e722 ~]# dnf repoquery --quiet --nvr gcc gcc-toolset-12-gcc
gcc-8.5.0-16.el8_7
gcc-toolset-12-gcc-12.1.1-3.4.el8_7

[h-vetinari]

Ah, that's great news, thanks! At the time this issue was discussed last year, the gcc-toolset didn't yet exist on rhubi, AFAIR, very happy to hear this got added in the meantime!

That IMO makes rhubi 9 a very good candidate for 2_34, and rhubi 8 a good fallback for 2_28 in case anything with the alma project were to go so spectacularly wrong that we'd have to switch.

[h-vetinari]

[that makes] rhubi 8 a good fallback for 2_28 in case anything with the alma project were to go so spectacularly wrong that we'd have to switch.

You know, I really didn't intend to jinx this, but it seems that RHEL is putting the screws on the derivative distros... https://almalinux.org/blog/impact-of-rhel-changes/

If Alma is forced to change its model (looks possible ATM), we should start looking into rhubi I guess...

[snnn]

ubi8 only has gcc 8 and gcc 12. It doesn't have gcc 11 or 10 or 9. And CUDA 11.x isn't compatible with gcc 12.

[h-vetinari]

The good thing is that it's looking like Alma will be able to continue to provide support beyond CentOS' EOL, only they have given up bug-for-bug compatibility (but will still stay ABI-compatible, which is what counts). So it looks like there won't be any action necessary for manylinux_2_28.

[snnn]

From security point of view, I have many concerns on UBI. The table on the top isn't accurate. Though it says you can get security updates for UBI8 till 2029-05-31, the real situation is more complicated. First, we need to get clear what UBI is. UBI docker images are customized RHELs. For example, if you need to use python, you can get a UBI docker image with python from Red Hat, and you need to keep in mind that:

1. RHEL is not free. If you did not pay, you cannot get access their security updates.
2. Only Red Hat can make UBI images. You can build your images based on these UBI images, but you should not add new RPM packages from Red Hat's repos.

The images you get from Red Hat are well maintained in the sense that if the corresponding RHEL has published a security update for a software that is contained in a UBI image (that was made by Red Hat), the security update will be included in an updated UBI image within 6 weeks. However, you cannot get the updated RPM package from Red Hat for free. If you run "dnf repolist" command in a UBI image, you will see things like:

repo id                                       repo name
ubi-8-appstream-rpms                          Red Hat Universal Base Image 8 (RPMs) - AppStream
ubi-8-baseos-rpms                             Red Hat Universal Base Image 8 (RPMs) - BaseOS
ubi-8-codeready-builder-rpms                  Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder

Unlike RHEL, it doesn't have any "update" repos.
If you query the package version of the gmp package in a UBI8 image, you will see:

# rpm -qa |grep gmp
gmp-6.1.2-11.el8_8.1.x86_64

The version contains the necessary updates for Security Advisory RHSA-2024:1102, which is good. However, you cannot find the updated RPM package from the three repos above. It means anything installed by youself with "dnf install" command will not get enough security updates. This manylinux repo uses the command a lot. That's my concern. To remediate the problem, we need to ask Red Hat to build and publish the manylinux images, which means we need to let them take over this project. I will be more concerned if we take that approach.

Therefore, from security point of view, UBI is not a viable solution here.

[carlwgeorge]

The table on the top isn't accurate. Though it says you can get security updates for UBI8 till 2029-05-31, the real situation is more complicated.

The table is accurate. UBI8 has the same EOL date as RHEL8 (2029-05-31).

First, we need to get clear what UBI is. UBI docker images are customized RHELs.

It's not "customized RHEL", it's a subset of RHEL. The packages in UBI repos are the exact same as the corresponding packages in RHEL. When a RHEL package is updated, it goes out the the RHEL repos, and if it's on the list of UBI packages, it goes out to the UBI repos at the same time.

RHEL is not free. If you did not pay, you cannot get access their security updates.

The subset of RHEL packages that are included in UBI are free and redistributable. That's the whole point of UBI. That set of packages get all the same security updates that RHEL itself gets.

You can build your images based on these UBI images, but you should not add new RPM packages from Red Hat's repos.

You can absolutely add packages from the UBI repos to create custom UBI-based images. What you can't do is add a package from a RHEL subscription (i.e. outside the UBI content set) to a UBI image and redistribute that.

The images you get from Red Hat are well maintained in the sense that if the corresponding RHEL has published a security update for a software that is contained in a UBI image (that was made by Red Hat), the security update will be included in an updated UBI image within 6 weeks. However, you cannot get the updated RPM package from Red Hat for free.

Yes, you can. All of the packages in the UBI images, as well as additional packages, are in the UBI repos and you get updates to those packages for free.

If you run "dnf repolist" command in a UBI image, you will see things like:

repo id                                       repo name
ubi-8-appstream-rpms                          Red Hat Universal Base Image 8 (RPMs) - AppStream
ubi-8-baseos-rpms                             Red Hat Universal Base Image 8 (RPMs) - BaseOS
ubi-8-codeready-builder-rpms                  Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder

Unlike RHEL, it doesn't have any "update" repos.

RHEL doesn't have "update" repos either.

• RHEL 8 repos:
  • rhel-8-for-x86_64-baseos-rpms
  • rhel-8-for-x86_64-appstream-rpms
  • codeready-builder-for-rhel-8-x86_64-rpms
• UBI 8 repos:
  • ubi-8-baseos-rpms
  • ubi-8-appstream-rpms
  • ubi-8-codeready-builder-rpms
• RHEL 9 repos:
  • rhel-9-for-x86_64-baseos-rpms
  • rhel-9-for-x86_64-appstream-rpms
  • codeready-builder-for-rhel-9-x86_64-rpms
• UBI 9 repos:
  • ubi-9-baseos-rpms
  • ubi-9-appstream-rpms
  • ubi-9-codeready-builder

If you query the package version of the gmp package in a UBI8 image, you will see:

# rpm -qa |grep gmp
gmp-6.1.2-11.el8_8.1.x86_64

The version contains the necessary updates for Security Advisory RHSA-2024:1102, which is good. However, you cannot find the updated RPM package from the three repos above.

This one is actually a bit strange, but not for the reason you are suggesting. To explain, allow me to reference the RHEL 8 Planning Guide diagram. You can see that RHEL 8 has minor versions, and some of those minor versions have lifecycle extensions. The current versions are 8.9, 8.8 EUS, 8.6 EUS, 8.4 SAP, and 8.2 SAP. What may not be obvious in this chart is that CentOS Stream 8 is effectively the 8.10 version, which hasn't been released as RHEL 8.10 yet. When a bug is being fixed, the RHEL maintainer chooses how many branches (minor versions) to apply the fix to. Some bugs are only fixed in CentOS Stream, later showing up in the next RHEL minor version. Some bugs are fixed in CentOS Stream and the current RHEL minor version. Some bugs are fixed in other combinations of next, current, EUS, and SAP versions. In this case, the CVE in question has been fixed in these versions of RHEL 8:

• CentOS Stream 8 as gmp-6.1.2-11.el8 (queuing it up for 8.10)
• RHEL 8.8 EUS as gmp-6.1.2-11.el8_8.1
• RHEL 8.6 EUS as gmp-6.1.2-11.el8_6.1

Notably, this has not been fixed in RHEL 8.9 (the current version), which is why an update for it is not available in the UBI 8 repos. However, somehow gmp-6.1.2-11.el8_8.1 snuck into the UBI 8 image, which is how you saw it with rpm -qa. This is not something being held back in UBI per se, it's something that the maintainer chose to fix in 8.6 EUS, 8.8 EUS, and the upcoming 8.10 only, and UBI 8 appears to have accidentally gotten it baked into the image while it isn't available in the UBI 8 or RHEL 8.9 repos. When you run a UBI 8 image on a subscribed RHEL system, its repos transition to the full RHEL content set, and even in that scenario it doesn't yet see an update to gmp that fixes that CVE because the fix isn't in RHEL 8.9.

All that said, while this is an odd situation, it is not caused by a security update being unavailable in the UBI repos, as you suggested. It will also be resolved this month or next when RHEL 8.10 is released.

It means anything installed by youself with "dnf install" command will not get enough security updates.

That is false. Lets say you run dnf install gnutls in a UBI 8 container. Right now that gives you gnutls-3.6.16-8.el8_9.3, which is an update that was published to RHEL 8.9 and UBI 8 on 2024-04-11.

Therefore, from security point of view, UBI is not a viable solution here.

I'm happy to engage here to help answer questions about UBI, RHEL, and CentOS Stream, but I'm going to need to you avoid passing off assumptions as fact, and drawing conclusions from that. UBI is a perfectly viable solution, from a security point of view or otherwise, as long as all the necessary packages are included in that UBI content set. If there is anything missing I'm happy to help advocate inside Red Hat for it to be added.

[snnn]

Thanks for the explanation and being supportive. So the security patch for gmp landed in the image first(because of a mistake), while the security patches for expat, gnutls, tzdata and libsemanage landed in the dnf repos first(which is normal). I apologize for my mistake that I only saw a piece of the whole picture.