E2E - Builder tests are constantly failing due to "Out of date version of GraalVM detected: native-image 21.3.0.0-Final"

[tadayosi]

https://github.com/apache/camel-k/actions/runs/3479827694/jobs/5818851170

Error: p.go:274: > {"level":"error","ts":1668593878.5730004,"logger":"camel-k.maven.build","msg":"\t[error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.IllegalStateException: Out of date version of GraalVM detected: native-image 21.3.0.0-Final Mandrel Distribution (Java Version 11.0.13+8). Quarkus currently supports 22.3.0. Please upgrade GraalVM to this version.","stacktrace":"github.com/apache/camel-k/pkg/util/log.Logger.Error\n\tgithub.com/apache/camel-k/pkg/util/log/log.go:75\ngithub.com/apache/camel-k/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/pkg/util/maven/maven_log.go:83\ngithub.com/apache/camel-k/pkg/util/maven.mavenLogHandler\n\tgithub.com/apache/camel-k/pkg/util/maven/maven_log.go:54\ngithub.com/apache/camel-k/pkg/util.scan\n\tgithub.com/apache/camel-k/pkg/util/command.go:77\ngithub.com/apache/camel-k/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/pkg/util/command.go:56\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\

Does anyone have ideas how to fix it?

[squakez]

If we run against the latest runtime snapshot, then it should be working correctly... https://github.com/apache/camel-k-runtime/blob/main/pom.xml#L47

[tadayosi]

Ah, I see. I think the native tests are failing for the same reason. Let's wait for the latest snapshot to be available.

[squakez]

Ah, I see. I think the native tests are failing for the same reason. Let's wait for the latest snapshot to be available.

In theory it should be already running against the correct latest snapshot. That version should be public since at least last week. We should verify if it is really taking the latest one.

[tobiasoort]

Possible issue: the 22.3.X-version of ubi-quarkus-mandrel doesn't get shipped in an arm64 flavour. This might block #3820. (Edit: Another possibility is I don't understand the architecture of this docker/jvm/quarkus/graalvm project and i'm just talking nonsense. :))

[tadayosi]

Thanks to @tobiasoort. Yeah probably this line is the cause:
https://github.com/apache/camel-k/blob/main/build/Dockerfile#L16

FROM quay.io/quarkus/ubi-quarkus-mandrel:21.3.0.0-Final-java11

But then the problem is that I cannot find ubi-quarkus-mandrel image of 22.3.0 for Java 11. Does anyone know why?
https://quay.io/repository/quarkus/ubi-quarkus-mandrel?tab=tags

[tadayosi]

OK, I found this:
https://github.com/graalvm/mandrel/releases/tag/mandrel-22.3.0.1-Final

Announcement

Starting with the 22.3, Mandrel releases will no longer include builds based on OpenJDK 11. Please use the OpenJDK 17 based builds instead.

Isn't it then problematic to upgrade to ubi-quarkus-native-image:22.3.0-java11 for camel-k-runtime? I'm not sure if we can use ubi-quarkus-mandrel:22.3.0.1-Final-java17 now. Maybe we should downgrade to ubi-quarkus-native-image:22.2.0-java11.

@oscerd @squakez What do you think?

[oscerd]

We probably need to do some tests, but I don't think it will work. It's not completely clear why they decided to drop OpenJDK 11 build support.

[tadayosi]

@oscerd So do you say downgrading to ubi-quarkus-native-image:22.2.0-java11 for camel-k-runtime also won't work?

[oscerd]

I'm not sure if it's feasible to follow this path in the long term. For the next release 1.11.0 with camel-quarkus 2.14.0, it should be fine, because we still have https://github.com/apache/camel-quarkus/blob/2.14.0/pom.xml#L98, but I really don't know what will happen in the long term.

[oscerd]

btw I started the work for releasing camel-k-runtime for 1.11.0

apache/camel-k-runtime#927

[tadayosi]

That's great. But I think this issue is a blocker for 1.11.0 as it means integration native builds don't work in general.

[oscerd]

Yes, but I didn't arrive at testing native integration since it was already failing at catalog testing. Let's try to downgrade to 22.2.0 for the moment on both camel-k and camel-k-runtime, but we need to find something to deal with dropping of OpenJDK11 build support.

[squakez]

I guess that, as soon as Camel Quarkus switch to the new 22.3 it will need to drop Java 11 support as well, is that assumption right? In that case the Camel K runtime should do the same and the operator as well, at least until we find a proper way to decouple the operator from the runtime.

[tadayosi]

FYI, upgrading the base image to ubi-quarkus-mandrel:22.2.0.0-Final-java11 seems to solve the issue #3825. So it can be solved in the short term.

Could it be that Camel Quarkus stays at GraalVM 22.2.0 because it needs to keep support for Java 11? Then it's when Camel Quarkus drops Java 11 support that it switches to GraalVM 22.3.
https://github.com/apache/camel-quarkus/blob/2.14.0/pom.xml#L98

[squakez]

Well... it seems they are already taking that path in 2.13: https://github.com/apache/camel-quarkus/blob/2.13.x/pom.xml#L99 - they are dropping support of Java 11 only in Native mode.

[oscerd]

It's not really a good idea to have two different java support one in native and one in JVM.

[squakez]

I think anyhow we must rethink how we depends on the runtime. I'm drawing some ideas and I'll submit them shortly to discuss in detail.

[tobiasoort]

Does anybody know why ubi-quarkus-mandrel:22.0.0.x gets shipped both in arm64 and amd64 but 22.2.0.0.x only comes in amd64 flavour?

@tadayosi this means that taking 22.2 as a fix here blocks the multi-architecture development. It might be that they started with multi-arch images (so dropping the -arch-extension) but I can't really find that out in the manifest inspector in quay. nope, it's amd64 only, just checked the manifest.

edit2: just raised graalvm/mandrel#459 over there.

[tobiasoort]

We probably need to do some tests, but I don't think it will work. It's not completely clear why they decided to drop OpenJDK 11 build support.

graalvm/mandrel#423

"Java-17 works fine from what we can tell". If you disagree I think adding to that discussion might be helpful.

[tadayosi]

@tobiasoort Sorry but upgrade to ubi-quarkus-mandrel:22.2.0.0-Final-java11 was inevitable to make CI green.

This issue seems to be the relevant one:
quarkusio/quarkus-images#210
So if I'm not mistaken, according to the issue should we use https://quay.io/repository/quarkus/ubi-quarkus-mandrel-builder-image?tab=tags instead?

[tadayosi]

I'm closing this one as at least the E2E failure is resolved.