Camel Quarkus AvroRpcTest requires quarkus.test.flat-class-path=true on Quarkus 2.0.0.Alpha3 #17463

ppalaga · 2021-05-25T21:37:07Z

I do not think that AvroRpcTest had an issue with Quarkus 2.0.0.Alpha1 and Quarkus 2.0.0.Alpha2. This issue seems to be new in Alpha3.

Note that we have one more Avro test called AvroTest that requred quarkus.test.flat-class-path=true on Alpha1 but was fixed in Alpha2 and also works on Alpha3.

I did not have much time to investigate deeper. Here are the steps to reproduce and the symptoms:

git clone git@github.com:apache/camel-quarkus.git
cd camel-quarkus
git revert 406058c468609835f9f0ac91a5c0b76e4f52a392
mvnd clean install -Dquickly

cd integration-tests/avro-rpc
mvn clean test

Expected: the test should pass
Actual:

[INFO] Running org.apache.camel.quarkus.component.avro.rpc.it.AvroRpcHttpTest
2021-05-25 23:34:44,818 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 46769
2021-05-25 23:34:44,983 INFO  [org.ecl.jet.uti.log] (pool-4-thread-1) Logging initialized @3947ms to org.eclipse.jetty.util.log.Slf4jLog
2021-05-25 23:34:45,027 INFO  [org.ecl.jet.ser.Server] (pool-4-thread-1) jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 11.0.11+9
2021-05-25 23:34:45,046 INFO  [org.ecl.jet.ser.han.ContextHandler] (pool-4-thread-1) Started o.e.j.s.ServletContextHandler@11f3c55a{/,null,AVAILABLE}
2021-05-25 23:34:45,060 INFO  [org.ecl.jet.ser.AbstractConnector] (pool-4-thread-1) Started ServerConnector@4f9cfa4b{HTTP/1.1, (http/1.1)}{0.0.0.0:46769}
2021-05-25 23:34:45,060 INFO  [org.ecl.jet.ser.Server] (pool-4-thread-1) Started @4024ms
2021-05-25 23:34:45,060 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 39393
2021-05-25 23:34:45,143 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 46517
2021-05-25 23:34:45,147 INFO  [org.ecl.jet.ser.Server] (pool-4-thread-1) jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 11.0.11+9
2021-05-25 23:34:45,149 INFO  [org.ecl.jet.ser.han.ContextHandler] (pool-4-thread-1) Started o.e.j.s.ServletContextHandler@3f9b9fb8{/,null,AVAILABLE}
2021-05-25 23:34:45,153 INFO  [org.ecl.jet.ser.AbstractConnector] (pool-4-thread-1) Started ServerConnector@37e41a6b{HTTP/1.1, (http/1.1)}{0.0.0.0:46517}
2021-05-25 23:34:45,153 INFO  [org.ecl.jet.ser.Server] (pool-4-thread-1) Started @4117ms
2021-05-25 23:34:45,153 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 44945
2021-05-25 23:34:45,155 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 40351
2021-05-25 23:34:45,155 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 41299
2021-05-25 23:34:45,155 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 43347
2021-05-25 23:34:45,155 INFO  [org.apa.cam.qua.tes.AvailablePortFinder] (pool-4-thread-1) getNextAvailable() -> 44889
2021-05-25 23:34:45,598 INFO  [org.apa.cam.qua.cor.CamelBootstrapRecorder] (main) Bootstrap runtime: org.apache.camel.quarkus.main.CamelMainRuntime
2021-05-25 23:34:45,739 INFO  [org.ecl.jet.ser.Server] (main) jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 11.0.11+9
2021-05-25 23:34:45,742 INFO  [org.ecl.jet.ser.han.ContextHandler] (main) Started o.e.j.s.ServletContextHandler@58cebe60{/,null,AVAILABLE}
2021-05-25 23:34:45,749 INFO  [org.ecl.jet.ser.AbstractConnector] (main) Started ServerConnector@7773141c{HTTP/1.1, (http/1.1)}{0.0.0.0:40351}
2021-05-25 23:34:45,749 INFO  [org.ecl.jet.ser.Server] (main) Started @4713ms
2021-05-25 23:34:45,754 INFO  [org.ecl.jet.ser.Server] (main) jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 11.0.11+9
2021-05-25 23:34:45,755 INFO  [org.ecl.jet.ser.han.ContextHandler] (main) Started o.e.j.s.ServletContextHandler@58468259{/,null,AVAILABLE}
2021-05-25 23:34:45,759 INFO  [org.ecl.jet.ser.AbstractConnector] (main) Started ServerConnector@6a635f56{HTTP/1.1, (http/1.1)}{0.0.0.0:43347}
2021-05-25 23:34:45,760 INFO  [org.ecl.jet.ser.Server] (main) Started @4724ms
2021-05-25 23:34:45,764 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main) Routes startup summary (total:4 started:4)
2021-05-25 23:34:45,764 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main)     Started route1 (http:localhost:40351/setTestPojo)
2021-05-25 23:34:45,765 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main)     Started route2 (netty:localhost:41299/setTestPojo)
2021-05-25 23:34:45,765 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main)     Started route3 (http:localhost:43347/put)
2021-05-25 23:34:45,765 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main)     Started route4 (netty:localhost:44889/put)
2021-05-25 23:34:45,765 INFO  [org.apa.cam.imp.eng.AbstractCamelContext] (main) Apache Camel 3.10.0 (camel-1) started in 83ms (build:0ms init:49ms start:34ms)
2021-05-25 23:34:45,828 INFO  [io.quarkus] (main) Quarkus 2.0.0.Alpha3 on JVM started in 2.302s. Listening on: http://localhost:35417
2021-05-25 23:34:45,828 INFO  [io.quarkus] (main) Profile test activated. 
2021-05-25 23:34:45,828 INFO  [io.quarkus] (main) Installed features: [camel-avro-rpc, camel-core, camel-support-common, cdi, resteasy, smallrye-context-propagation]
2021-05-25 23:34:47,245 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /avro-rpc/specificConsumerGet?protocol=http failed, error id: 7223670b-2e2b-4bee-bfe2-56ae5f8780f3-1: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException
        at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
        at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
        at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
        at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
        at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: java.lang.NullPointerException
        at org.apache.camel.quarkus.component.avro.rpc.it.AvroRpcResource.specificConsumerGet(AvroRpcResource.java:142)
        at org.apache.camel.quarkus.component.avro.rpc.it.AvroRpcResource_ClientProxy.specificConsumerGet(AvroRpcResource_ClientProxy.zig:246)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
        at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:646)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:510)
        at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:460)
        at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:462)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:420)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:394)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:69)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
        ... 14 more

The text was updated successfully, but these errors were encountered:

stuartwdouglas · 2021-06-02T06:02:08Z

The TCCL appears to be wrong, Jetty is using the base runtime CL rather than the runtime CL. I can't really see where Jetty is configured though.

ppalaga · 2021-06-02T09:09:42Z

Thanks for the pointer, @stuartwdouglas . I'll try to dig deeper.

JiriOndrusek · 2021-06-10T12:38:28Z

@ppalaga @stuartwdouglas
Classloader does not seem to be an issue here. As you can see in avro, there is a fallback to correct classloader (https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/util/ClassUtils.java#L74)

I tried some validations in debug mode and here are my results:

Problem is caused by the fact, that classloader can not find class org.apache.camel.quarkus.component.avro.rpc.it.specific.generated.Key
I have the correct classloader: QuarkusClassLoader:Quarkus Base Runtime ClassLoader: TEST
If I try to execute getClassLoader().getPackage("org.apache.camel.quarkus.component.avro.rpc.it.reflection"), correct package is returned.
If I try to execute getClassLoader().getPackage("org.apache.camel.quarkus.component.avro.rpc.it.specific"), null is returned.
I verified that both packages are present in target/classes.
(I also tried to copy required classes to the new package, but result is the same)

I see only one difference between specific and reflection packages. There is subpackage 'specific.generated', which is generated by maven plugin. But I tried it also without generation, and result is the same.

Do you have an idea what could be wrong or what should I verify?

EDITED:
I found, that target class is probably banned: https://github.com/quarkusio/quarkus/blob/main/independent-projects/bootstrap/core/src/main/java/io/quarkus/bootstrap/classloading/QuarkusClassLoader.java#L421

JiriOndrusek · 2021-06-11T07:46:50Z

~~There is a fact, which could help:~~

~~Error is happening in avro thread, which has classloader QuarkusClassLoader:Quarkus Base Runtime ClassLoader - which bans classes from project~~
If I try to call the same code from JUnit test thread Class.forName("org.apache.camel.quarkus.component.avro.rpc.it.specific.generated.Key", true, Tread.currentThread().getContextClassLoader()) it uses QuarkusClassLoader:Quarkus Runtime ClassLoader: TEST, which works correctly.

JiriOndrusek · 2021-06-11T08:47:26Z

Problem is caused by https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L821
(I verified it locally)

Both previous comments could be ignored.

JiriOndrusek · 2021-06-11T09:08:32Z

Issue is reported in jetty: jetty/jetty.project#6399

manovotn · 2022-11-22T15:38:50Z

@JiriOndrusek @ppalaga is this issue still relevant? Linked GH issues are now resolved.

JiriOndrusek · 2022-11-22T15:52:51Z

I will check the issue tomorrow and let you know, whether it can be simulated with the current version of Quarkus. From the first (fast) glance at related tickets, there can not be a fix on jetty side. There are only some hints, how it could be solved on quarkus side - jetty/jetty.project#6399 (comment).

JiriOndrusek · 2022-11-24T15:14:51Z

I'm sorry for the delay. Issue is already fixed in commit (for issue)

manovotn · 2022-11-24T15:43:20Z

I'm sorry for the delay. Issue is already fixed in commit (for issue)

No problem, I was just checking if we can close this.
Thanks for looking into it!

ppalaga added the kind/bug Something isn't working label May 25, 2021

quarkus-bot bot added the triage/needs-triage label May 25, 2021

maxandersen added the area/camel label Jun 1, 2021

ppalaga mentioned this issue Jun 10, 2021

Class loader issues in AvroRpcTest with Quarkus 2.0.0.Alpha3 apache/camel-quarkus#2651

Closed

manovotn closed this as completed Nov 24, 2022

quarkus-bot bot removed the triage/needs-triage label Nov 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Camel Quarkus AvroRpcTest requires quarkus.test.flat-class-path=true on Quarkus 2.0.0.Alpha3 #17463

Camel Quarkus AvroRpcTest requires quarkus.test.flat-class-path=true on Quarkus 2.0.0.Alpha3 #17463

ppalaga commented May 25, 2021

stuartwdouglas commented Jun 2, 2021

ppalaga commented Jun 2, 2021

JiriOndrusek commented Jun 10, 2021 •

edited

JiriOndrusek commented Jun 11, 2021 •

edited

JiriOndrusek commented Jun 11, 2021

JiriOndrusek commented Jun 11, 2021

manovotn commented Nov 22, 2022

JiriOndrusek commented Nov 22, 2022

JiriOndrusek commented Nov 24, 2022

manovotn commented Nov 24, 2022

Camel Quarkus AvroRpcTest requires quarkus.test.flat-class-path=true on Quarkus 2.0.0.Alpha3 #17463

Camel Quarkus AvroRpcTest requires quarkus.test.flat-class-path=true on Quarkus 2.0.0.Alpha3 #17463

Comments

ppalaga commented May 25, 2021

stuartwdouglas commented Jun 2, 2021

ppalaga commented Jun 2, 2021

JiriOndrusek commented Jun 10, 2021 • edited

JiriOndrusek commented Jun 11, 2021 • edited

JiriOndrusek commented Jun 11, 2021

JiriOndrusek commented Jun 11, 2021

manovotn commented Nov 22, 2022

JiriOndrusek commented Nov 22, 2022

JiriOndrusek commented Nov 24, 2022

manovotn commented Nov 24, 2022

JiriOndrusek commented Jun 10, 2021 •

edited

JiriOndrusek commented Jun 11, 2021 •

edited