Complete new linuxNew based packaging

[karianna]

Initial List before we split into an Epic etc.

• Update README.md on how to run it and how it works
• Add Java 8, 11, and 16 support for Temurin Builds Debian, Red Hat, and SUSE support
• Fix RH Build failing due to GPG signing bug
• Test for version string correctness
• Add Java 8 JRE Temurin packages
• Parameterize the scripts so that version numbers aren't hard-coded - See discussion below.
• Bring in cacerts logic from Andreas repo - Andreas has kindly given permission. Eclipse Tracking Bug and CQ - code was Pull Requested in.
• For the ca-certs figure out a way to have adoptium-ca-certificates as an optional package and deal with that gracefully.
• Test the adoptium ca-certs package as a dependency of the main package
• Hook the the new build into a Jenkins pipeline? job
• Enable GPG signing for RPMs
• Implement uploading to Artifactory
• Integrate into Jenkins pipelines

[karianna]

From the original author:

If I remember correctly, the necessary infra for 11 and 16 is present. Doing packages for 8 is copy & paste and a little bit on top. Jenkins is able to build the packages. The only steps left to do: Pull them from Jenkins, manually sign the RPMs (create a new GPG key beforehand), and upload everything onto a server.

[jerboaa]

Initial metadata update: #331

Is there an email alias that can be used for RPMs/Debs? I suppose packaging@adoptium.com is wrong?

[karianna]

#332 for Updated README and an early attempt at adding a Java 8 temurin Debian and Red Hat packages

[karianna]

Initial metadata update: #331

Is there an email alias that can be used for RPMs/Debs? I suppose packaging@adoptium.com is wrong?

Good question. I suspect we aren't allowed to use the AdoptOpenJDK.net email addresses. Perhaps an EF-based mailing list?

[aahlenst]

Parameterize the scripts so that version numbers aren't hard-coded?

I've learnt the hard way over the past years that it's tough to reconcile OpenJDK's version numbering and Adopt's version numbering with the requirements of Debian and RPM packages. Remember the drama of 11.0.9.1. If you get it wrong, the packages won't show up on the user's computers (been there, done that). If you get it very wrong, you have to increment the epoch (been there, done that). Computing the version number correctly requires access to correct metadata (for example, because the .1 rebuilds aren't included in java -version). In the past, metadata was sometimes incomplete or wrong. Furthermore, extensive automated testing would be needed to ensure that every edge case is covered. You also have to consider that the packages are a separate "product" with their own version. If you fix a bug in the package, you have to bump the version. And you have to set it back once the next PSU comes out. Therefore, the lack of parameterization is a feature.

Fix RH Build failing due to GPG signing

I'm not sure what it means. If it means that you intend to sign the RPMs during the build, I'd reconsider. Signing RPMs during build requires sensitive key material on the build machines. Building takes time, so you need multiple machines. That was a problem in the past. So it's a deliberate choice not to sign the RPMs. I'd rather sign the RPMs just before uploading or even on the machine that distributes them (private GPG key is required on there anyway because the package indexes have to be signed).

I'd also strongly consider creating a keyring package for Debian (see https://wiki.debian.org/DebianRepository/UseThirdParty, Brave did this well in the past). This allows to rollover keys.

[karianna]

Fix RH Build failing due to GPG signing

I'm not sure what it means. If it means that you intend to sign the RPMs during the build, I'd reconsider. Signing RPMs during build requires sensitive key material on the build machines. The building takes time, so you need multiple machines. That was a problem in the past. So it's a deliberate choice not to sign the RPMs. I'd rather sign the RPMs just before uploading or even on the machine that distributes them (a private GPG key is required on there anyway because the package indexes have to be signed).

Something in the default Gradle tasks caused a GPG sign to be executed. I think I'd rather have that as a separate explicit step. Still digging into the code to see why it triggers (Suse and Debian don't).

I'd also strongly consider creating a keyring package for Debian (see https://wiki.debian.org/DebianRepository/UseThirdParty, Brave did this well in the past). This allows to roll over keys.

Will check with EF if they have something.

[smlambert]

Additional piece that we didn't pull into the linuxNew story before (back when the work from adoptium-packages-linux was pulled in, is the cacert piece, which can be found here: https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts - so add a step in the description of this issue to bring that in.

[jerboaa]

Initial metadata update: #331
Is there an email alias that can be used for RPMs/Debs? I suppose packaging@adoptium.com is wrong?

Good question. I suspect we aren't allowed to use the AdoptOpenJDK.net email addresses. Perhaps an EF-based mailing list?

@tellison suggested temurin-dev@eclipse.org which seems fine. It's just an email address for users to report bugs/issues.

[karianna]

temurin-dev@eclipse.org

I've globally replaced that in my PR

[aahlenst]

Dockerfiles for RPM-based distros need an update across the board because the old sks-keyservers.net pool was shut down.

RUN gpg --batch --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7

needs to be replaced with

RUN  gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7

[aahlenst]

Does anybody have any logs that shed some light on "Fix RH Build failing due to GPG signing?" I just ran an RPM build and it worked fine.

[aahlenst]

Would be very helpful if someone could hook it up with GitHub Actions. A workflow exists but wasn't copied over. Although it's x64 only, that alone catches some issues and helps verifying PRs.

[karianna]

RUN gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7

I swapped this out but get the error:

#9 [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu
#9 sha256:3ca4c0b2d52e7d651b6e73a9d43b94d10b5f37545dfc6f2cbb752ccc2ac64eea
#9 0.232 gpg: Signature made Thu Apr 16 06:40:32 2020 UTC
#9 0.232 gpg:                using RSA key B42F6819007F00F88E364FD4036A9C25BF357DD4
#9 0.232 gpg: Can't check signature: No public key
#9 ERROR: executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2
------
 > [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu:
------
executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2

On a side note, should I also swap the key server for the Suse script?

[karianna]

@aahlenst - Extra question. I understand you completed the ca-certs work in your repo (https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts). Should I just be looking to bring across all of https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts/ca-certificates and patch the scripts to refer to those as what I think you've done in your repo?

[karianna]

RUN gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7

I swapped this out but get the error:

#9 [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu
#9 sha256:3ca4c0b2d52e7d651b6e73a9d43b94d10b5f37545dfc6f2cbb752ccc2ac64eea
#9 0.232 gpg: Signature made Thu Apr 16 06:40:32 2020 UTC
#9 0.232 gpg:                using RSA key B42F6819007F00F88E364FD4036A9C25BF357DD4
#9 0.232 gpg: Can't check signature: No public key
#9 ERROR: executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2
------
 > [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu:
------
executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2

On a side note, should I also swap the key server for the Suse script?

I've resolved this, gosu and tini needed separate keys.

[karianna]

@aahlenst - Extra question. I understand you completed the ca-certs work in your repo (https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts). Should I just be looking to bring across all of https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts/ca-certificates and patch the scripts to refer to those as what I think you've done in your repo?

@aahlenst - Stewart and I were a little confused over this. I think the current build bundles the cacerts with the JDK itself, so is having a separate adoptium-ca-cacerts linux package actually needed?

[aahlenst]

I hope you have some time 😁

Having the keystore as part of the JDK (= included in the package) is a subpar experience.

• Adding certificates to the keystore requires to change the keystore. However, because the keystore is bundled with the package, it gets updated with every package update. So you have to re-add the certificate to the keystore after each package update. Good luck with getting that right. One possible way out would be to mark the keystore as config file in the package's metadata. Henceforth, you'd be asked whether you want to replace the keystore or not. That's an all or nothing operation. Most people won't see the prompt anyway, because few people update their systems interactively. Because the default option is to keep the modified keystore, they wouldn't ever get new CA certificates.
• The OS keystore and the Temurin keystore are different. So some tools can connect to a website while Temurin cannot and vice versa. This is very confusing (we had many reports in the past). This was somewhat mitigated by shipping Mozilla's list of CA certificates. You can configure this integration yourself, but why bother when you can just apt-get or yum install whatever is included with your distribution?

So, integrating with the tooling of the OS to centrally manage the keystore and have a consistent list of certificates in the OS and JDK is the right thing to do.

On RPM based distributions, this can be achieved rather easily. The ca-certificates packages of RHEL and friends automatically generate a suitable keystore for the JDK. You just have to symlink it.

On Debian derivatives, this is more complicated. There exists a package that can generate a keystore for the JDK. Unfortunately, it has a cyclic dependency on the JDK because the program to generate the keystore is written in Java. Some script in that package acts as a tie breaker, but it can only do that for JDK it knows. And it knows only JDKs shipped as part of Debian. We tried it in the past and failed. So, I decided to roll our own without the cyclic dependency. To be clear, it does not contain a cacerts file. But it generates one and integrates with Debian's facilities to manage CA certificates. You add a CA certificate to Debian and the cacerts will automatically be regenerated. Same happens if you remove one. As a result, the package is very small and even architecture-independent. In can be used by any JDK (not only Temurin). It just needs to be installed once.

I think that provides enough value for a little price and brings Temurin's packages on equal footing as the JDKs shipped by the distros themselves. But Temurin even has the AQA/TCK stamps of approval.

[karianna]

Thinking out loud. I'm not 100% convinced this is what users will expect. Up until now Java users have kept the O/S keystore and Java keystore separate and I'd hypothesize for many users that's a good thing (they may not want their O/S to have all of the allowable certs by Java and vice versa).

By making this change now, we'd be changing some pretty fundamental default behaviour.

Perhaps offering to do this as an optional step would be OK (if I can figure out how to do that).

[karianna]

For Eclipse Signing: see https://wiki.eclipse.org/IT_Infrastructure_Doc#What_about_GPG_signing.3F
looks like they have a maven target mvn gpg:sign-and-deploy-file we’ll need to find out what the cbi url is to send artefacts
e.g https://cbi.eclipse.org/jarsigner/sign but for gpg

George is asking the EF on the details here.

[aahlenst]

We got requests to integrate with the OS facilities. And now would be a good time to fix all the weirdness. We'd also follow every (?) distro JDK package out there. RHEL, SUSE, Debian, Alpine, all do this. It was inaptness that we didn't have it from the beginning.

If you want to make this opt-in (Debian, don't know about RHEL from the top of my head):

1. Rename cacerts to bundled_cacerts or something similar.
2. Claim a directory/file in /etc/default. Define a variable like BUNDLED_CACERTS=Yes.
3. Include tooling in the package to symlink cacerts to either bundled_cacerts or the file in the cacerts package depending on the value of BUNDLED_CACERTS.
4. Invoke that tooling during postinst.

This would have to be part of the Temurin JDK package. If you included it in the cacerts package, the cacerts package would have to know about all versions. Doable, but maybe not desirable.

Should I just be looking to bring across all of https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts/ca-certificates and patch the scripts to refer to those as what I think you've done in your repo?

You might succeed in cherry-picking the commit or use git patch apply to bring it over.

[aahlenst]

For Eclipse Signing: see https://wiki.eclipse.org/IT_Infrastructure_Doc#What_about_GPG_signing.3F
looks like they have a maven target mvn gpg:sign-and-deploy-file we’ll need to find out what the cbi url is to send artefacts
e.g https://cbi.eclipse.org/jarsigner/sign but for gpg

George is asking the EF on the details here.

That's what you have to do: https://access.redhat.com/articles/3359321. RPMs only. Debian packages should not be signed, but the package index instead.

[sxa]

We got requests to integrate with the OS facilities.

@aahlenst Makes sense but any idea why we wanted to have our own CA package instead of using the CA package supplied with each distro? e.g. ca-certificates on RHEL (Provides /etc/pki/ca-trust/extracted/java/cacerts) or ca-certificates-java (Provides /etc/ssl/certs/java/cacerts) on Ubuntu

[karianna]

OK, John and I were looking at this locally and I see what Debian does now (basically loops through the O/S certs and adds them via keytool for Java, thereby keeping them in sync - there's an update command as well). So I think it makes sense as @aahlenst suggests. Now to figure out the coding to make that a reality :-).

[karianna]

It looks like Debian already solves this with the ca-certificates-java package. I'll try setting that as a dependency.

Now, the ca-certificates-java may not have the richness of the Adopt set that we pull from Mozilla, in which case I'll go back to the fuller solution that @aahlenst proposed. Sorry for the slowness here folks, I'm learning as I go :-)

Hmmm, maybe not #105 :-)

OK fine, I'm wrong - lets do it the Andreas way - this may take me some time to code up, but will try to get it done before EOW

[aahlenst]

@karianna @sxa I know it's hard to keep track of everything I wrote. But #330 (comment) explains everything. Why ca-certificates-java doesn't work and that our cacerts package is supposed to be Debian only because RHEL has solved this differently (and better). Please re-read. I also welcome suggestions to improve my writing.

Also note that most Linux distributions effectively use Mozilla's list of certificates.

[sxa]

Thanks, and apologies that I hadn't read through that comment previously

[karianna]

Likewise - thanks for the detail - I'll give this a go tomorrow and update the PR as appropriate. This is the last step before signing to solve. Both RPM and Deb 8/11 packages have been manually tested and compared against the previous AdoptOpenJDK packages and are the same - so that's good.

[karianna]

@aahlenst - I'd like to fork and clone your repo into Adoptium that creates the adoptium-ca-certificates.deb file. Would that be OK?

[aahlenst]

@karianna Understood. I'm happy to arrange a full transfer. Or you can create a new repo and just push the old contents into it.

[maksymgendin]

@evgenkisel
This worked for me on RHEL 8 for example:

cat <<'EOF' > /etc/yum.repos.d/adoptium.repo
[Adoptium]
name=Adoptium
baseurl=https://adoptium.jfrog.io/artifactory/rpm/centos/$releasever/$basearch
enabled=1
gpgcheck=1
gpgkey=https://adoptium.jfrog.io/artifactory/api/security/keypair/default-gpg-key/public
EOF

yum install temurin-17-jdk

(and yeah, the CentOS 7 repository has recently disappeared 😮)

[theofficialgman]

are there plans on adding jre only packages to this repo? or will it always be the full jdk only

[wwuck]

When will the APT installation instructions be added to https://adoptium.net/installation.html#packages? I can already see it on the blog at https://blog.adoptium.net/2021/12/eclipse-temurin-linux-installers-available/.

[gdams]

@wwuck i’ll be adding the instructions to the website shortly

[juozaspo]

Ubuntu 21.10 deb version is missing from repository. As @Pokegamer129 has a problem setting up a repository on his system, I found out that by troubleshouting his process as described in comments of betacraftuk/betacraft-launcher#114 and I'm not sure if package for 21.04 might be compatible.
Edit: The mentioned user has solved the problem, it appears versions for hirsute works fine on impish so far. The user didn't report a problem yet while using this java package version.

[StrangeNoises]

can confirm the above. Following the instructions to the letter from the blogpost on 21.10 (impish indri) fails because there is no release for that version. Changing the source file to use hirsute does work. There are some minor (non-showstopper) issues with using update-java-alternatives but at this point we probably should be raising new issues for them if we think they're important enough.

[sxa]

at this point we probably should be raising new issues for them

Yep raising subsequent issues for that would be the right thing to do.

Ubuntu 21.10 deb version is missing from repository

We've got some discussions at PMC level about how to support newer (and "legacy") distributions so we'll hope to sort that out shortly :-)

[StrangeNoises]

well, matching which distro versions are actually supported by their own vendors seems a good start. 😀 21.04 is going to EOL very shortly; 21.10 is the current non-LTS release, although I don't think it was when JDK 17.0.1 came out 😜

But honestly assuming it's the same actual binary build being installed in each case - as opposed to, say, compiled against the extant library versions in that exact distro version - then it could probably be simplified by not making the distro version a part of the repo url.

Or in Ubuntu's sake (and its own downstream derivatives), basing it off the upstream debian version instead, as there are fewer of those and they last longer. via /etc/debian_version.

I imagine you might have similar issues with Fedora, where releases are also frequent and short-lived, vs. RHEL and what's stepped into CentOS's shoes.

[sxa]

well, matching which distro versions are actually supported by their own vendors seems a good start.

Absolutely and I will be attempting to ensure we have that as a minimum ;-)

[pho3nixx]

Now that there are JRE builds back (yay!), are there plans to provide them as Debian packages via the repository, too? Right now, only JDK packages are available there.

[maksymgendin]

I have also a question about crypto-policies: Is it planned to autoconfigure the installed Eclipse Temurin JDK to use the system-wide configured crypto-policies?

[drm00]

The blog post mentions linux package support for SLES 12, but I cannot find any folder with sles packages in the jfrog repository. Is support for SLES still planned? And will the installation documantion be put on the adoptium.net page?

[jerboaa]

The blog post mentions linux package support for SLES 12, but I cannot find any folder with sles packages in the jfrog repository. Is support for SLES still planned? And will the installation documantion be put on the adoptium.net page?

https://blog.adoptium.net/2021/12/eclipse-temurin-linux-installers-available/

[sxa]

I have also a question about crypto-policies: Is it planned to autoconfigure the installed Eclipse Temurin JDK to use the system-wide configured crypto-policies?

We do not have that in the plan at the moment.

[maksymgendin]

I have also a question about crypto-policies: Is it planned to autoconfigure the installed Eclipse Temurin JDK to use the system-wide configured crypto-policies?

We do not have that in the plan at the moment.

Thanks, @sxa.

The Red Hat distribution of OpenJDK (java-11-openjdk package) on RHEL is automatically using the system-wide crypto-policies as far as I know. Probably they configure this behavior in a similar way like you configure the usage of the system certificate store here.

If it's currently not on the plan - how can I request a feature for this?

[drm00]

The blog post mentions linux package support for SLES 12, but I cannot find any folder with sles packages in the jfrog repository. Is support for SLES still planned? And will the installation documantion be put on the adoptium.net page?

https://blog.adoptium.net/2021/12/eclipse-temurin-linux-installers-available/

@jerboaa Yes, that's the blog post I was referring to. It mentions SLES:

openSUSE/SLES Instructions

1. Import the RPM repo making sure to change the openSUSE version if you are not using OpenSUSE 15.2. RPM’s are also available for SLES 12 and 15. To check the full list of versions supported take a look at https://packages.adoptium.net/ui/repos/tree/General/rpm.

But there is no folder for SLES in the jfrog repo:

Am I missing something obvious?

[karianna]

See #426

[drm00]

@karianna Great, thanks!

[drm00]

@karianna The jfrog-repo lists sles 12 and 15 now, which is great! Thanks 👍 Currently, there is only temurin 17.0.2 available - will packages for jdk 11 be provided as well?

[karianna]

Yes they should be arriving AFAIK

[karianna]

@gdams - LMK when the installers are out and I think we can close this one and raise specific tickets for anything extra we want to add

[karianna]

Closing this issue now that the docs are up on the website.

[sfc-gh-pbennes]

Thanks everyone.