Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GraalVM: Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64 #429

Open
patpatpat123 opened this issue May 9, 2023 · 19 comments

Comments

@patpatpat123
Copy link

patpatpat123 commented May 9, 2023

Hello team,

This is my first issue in this repo.
Is not anything, big thanks for this very cool project.

I would like to reach out in order to report an issue:
My project is a GraalVM native image project, a simple Kafka consumer.

In my dependencies, I do have the most recent Snappy:

        <dependency>
            <groupId>org.xerial.snappy</groupId>
            <artifactId>snappy-java</artifactId>
            <version>1.1.9.1</version>
        </dependency>

On my machine, Ubuntu, I do have Snappy installed.

root@dlp:~# apt -y install snapd
root@dlp:~# systemctl  enable snapd.service snapd.socket
root@dlp:~# source /etc/profile.d/apps-bin-path.sh

Yet, when running the application, I am facing this 100% reproducible issue:

Caused by: org.apache.kafka.common.KafkaException: Received exception when fetching the next record from my-topic-1. If needed, please seek past the record to continue consumption.
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1676)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1900(Fetcher.java:1497)
	at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:717)
	at org.apache.kafka.clients.consumer.internals.Fetcher.collectFetch(Fetcher.java:683)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1314)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1243)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
	at reactor.kafka.receiver.internals.ConsumerEventLoop$PollEvent.run(ConsumerEventLoop.java:357)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base@17.0.7/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base@17.0.7/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base@17.0.7/java.lang.Thread.run(Thread.java:833)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
Caused by: org.apache.kafka.common.KafkaException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
	at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:46)
	at org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:94)
	at org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:276)
	at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:280)
	at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:364)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1619)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1656)
	... 16 common frames omitted
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:343)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:177)
	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:155)
	at org.xerial.snappy.Snappy.init(Snappy.java:70)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
	at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:145)
	at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:99)
	at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:59)
	at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:44)
	... 22 common frames omitted

Could you please help on this issue?

@sgammon
Copy link

sgammon commented May 12, 2023

i am encountering this too -- the native library loads fine on jvm, just not native image

@xerial xerial added the graalvm label May 16, 2023
@xerial
Copy link
Owner

xerial commented May 16, 2023

SnappyLoader tries to copy a native lib inside https://github.com/xerial/snappy-java/tree/master/src/main/resources/org/xerial/snappy/native using the code at
https://github.com/xerial/snappy-java/blob/v1.1.9.1/src/main/java/org/xerial/snappy/SnappyLoader.java#L361

It seems the behavior of finding resources from the classpath is different between GraalVM and JVM.

@xerial xerial changed the title Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64 GraalVM: Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64 May 23, 2023
@patpatpat123
Copy link
Author

Just checking a month after, anyone can help one this issue please?

@shautvast
Copy link

Think this may be of help @patpatpat123
finding resources from the classpath IS different between GraalVM and JVM
see https://www.graalvm.org/latest/reference-manual/native-image/dynamic-features/Resources/
you'd have to to supply the native lib to native-image option -H:IncludeResources

but I also see that instead of throwing Exception, snappy could fallback to -Djava.library.path which I don't think it does now

@xerial
Copy link
Owner

xerial commented Jun 19, 2023

FYI: Currently, I have no idea how I can fix this issue mainly because I'm not a user of GraalVM. So I will not work on this unless there will be a PR. The problem looks like just finding resource files (.so files) from the classpath (or inside the jar file) in GraalVM.

@patpatpat123
Copy link
Author

I tried the resource, adding it when running the graalVM native image, and unfortunately, the issue still persists.

I guess I can just cross my fingers for when the fix will be available..

@patpatpat123
Copy link
Author

@shautvast

Thank you for your input.

Just to avoid misunderstanding, I should include the -H: IncludeResources=abc at build time or at run time?

Also, what should be the value- H: IncludeResources**=abc** ?

Thank you

@shautvast
Copy link

shautvast commented Jun 21, 2023

@patpatpat123 I just verified that this is actually quite easy to do at build-time:
just include -H:IncludeResources=".*/libsnappyjava.dylib" in the native-image call
and make sure that the native lib is the correct one for your platform (and of course provide the snappy jar in the classpath for native-image)

so the complete call would look like this
native-image -cp [your_classes]:[PATH]/snappy-java-1.1.9.1.jar -H:IncludeResources=".*/native-lib" MainClass

inspect the snappy-jar to find the native libs

@xerial this is not something that can be fixed/changed from the library side

@xerial
Copy link
Owner

xerial commented Jul 15, 2023

It seems sqlite-jdbc, which is using the same mechanism for loading native library at runtime, can support GraalVM https://github.com/xerial/sqlite-jdbc#graalvm-native-image-support

Probably we can borrow the same technique

@patpatpat123
Copy link
Author

Hey team, I tried this:

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
    <configuration>
        <buildArgs>
            <buildArg>-H:IncludeResources=".*/native-lib"=${project.build.directory}</buildArg>
        </buildArgs>
    </configuration>
</plugin>

Based on the same example from sqlite. However, it is still not working, still same issue.

May I ask if I did something wrong in my command?
I would like to leverage org.graalvm.buildtools native-maven-plugin if possible

@liuxgo
Copy link

liuxgo commented Sep 25, 2023

@patpatpat123
Write resource-config.json like this, I only need binary file for Windows.

{
  "bundles": [],
  "resources": {
    "includes": [
      {
        "condition": {
          "typeReachable": "org.xerial.snappy.Snappy"
        },
        "pattern": "org/xerial/snappy/native/Windows/[a-z_0-9]+/snappyjava.dll"
      }
    ]
  }
}

Add build args in pom.xml like this: <buildArg>-H:ResourceConfigurationFiles=native-image/resource-config.json</buildArg>

@pavelorehov
Copy link

pavelorehov commented Oct 3, 2023

I have same issue on Linux and could NOT resolve it by adding resource to native image like below.
Also not sure how to verify that indeed libsnappyjava.so is part of native image, may be I'm doing something wrong.

Caused by: org.apache.kafka.common.KafkaException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64

<plugin>
	<groupId>org.graalvm.buildtools</groupId>
	<artifactId>native-maven-plugin</artifactId>
	<configuration>
		<buildArgs>
			<arg>-H:ResourceConfigurationFiles=${project.basedir}/src/main/resources/META-INF/native-image/resource-config.json</arg>
		</buildArgs>							
	<requiredVersion>22.3</requiredVersion>
	</configuration>
	...
</plugin>

{
  "bundles": [],
  "resources": {
    "includes": [
      {    
        "pattern": "/org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so"
      }
    ]
  }
}

The only workaround I found is to copy the libsnappyjava.so to same folder with executable and pass parameters, but that is pretty dirty fix I guess.
-Dorg.xerial.snappy.use.systemlib=true -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Dorg.xerial.snappy.tempdir=/tmp

@liuxgo
Copy link

liuxgo commented Oct 3, 2023

@pavelorehov Maybe you can try to modify resource-config.json like below:

  1. add contidion
  2. delete first “/” of pattern
      "condition": {
          "typeReachable": "org.xerial.snappy.Snappy"
        },
        "pattern": "org/xerial/snappy/native/Windows/[a-z_0-9]+/snappyjava.dll"

And you can add native maven build args <buildArg>-H:Log=registerResource:3</buildArg>, it will show register resource or not.

  [thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Windows/x86/snappyjava.dll
  [thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Windows/x86_64/snappyjava.dll

@pavelorehov
Copy link

pavelorehov commented Oct 5, 2023

Thanks a lot @liuxgo it works fine with fixes you suggested.

<plugin>
	<groupId>org.graalvm.buildtools</groupId>
	<artifactId>native-maven-plugin</artifactId>
	<configuration>
		<buildArgs>
			<arg>-H:ResourceConfigurationFiles=${project.basedir}/src/main/resources/META-INF/native-image/resource-config.json</arg>
			<arg>-H:Log=registerResource:3</arg>
		</buildArgs>							
	<requiredVersion>22.3</requiredVersion>
	</configuration>
	...
</plugin>

{
  "bundles": [],
  "resources": {
    "includes": [
      {  
        "condition": {
                    "typeReachable": "org.xerial.snappy.Snappy"
                }, 
        "pattern": "org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so"
      }
    ]
  }
}

Build log:

[thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so

@pavelorehov
Copy link

Would be great if snappy will contribute the fix to below repo so that consumers would only pick relevant version and rest will happen automatically.

https://github.com/oracle/graalvm-reachability-metadata/tree/master

@thimmwork
Copy link

Not sure if this is a different issue, but using the system property
-Dorg.xerial.snappy.use.systemlib=true
and running the binary in a docker image based on alpine that has
RUN apt update && apt install -y libsnappy-java
installed, results in a

java.util.concurrent.CompletionException: java.lang.UnsatisfiedLinkError: No snappyjava in java.library.path
at java.base@21.0.2/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base@21.0.2/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base@21.0.2/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
at java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
at java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:833)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:211)
Caused by: java.lang.UnsatisfiedLinkError: No snappyjava in java.library.path
at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibraryRelative(NativeLibrarySupport.java:136)
at java.base@21.0.2/java.lang.ClassLoader.loadLibrary(ClassLoader.java:106)
at java.base@21.0.2/java.lang.Runtime.loadLibrary0(Runtime.java:916)
at java.base@21.0.2/java.lang.System.loadLibrary(System.java:2063)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:185)
at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:157)
at org.xerial.snappy.Snappy.init(Snappy.java:70)
at org.xerial.snappy.Snappy.(Snappy.java:47)
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:469)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:185)
at org.apache.kafka.common.utils.ChunkedBytesStream.fill(ChunkedBytesStream.java:149)
at org.apache.kafka.common.utils.ChunkedBytesStream.read(ChunkedBytesStream.java:121)
at org.apache.kafka.common.utils.ByteUtils.readUnsignedVarint(ByteUtils.java:203)
at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:256)
at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:279)
at org.apache.kafka.common.record.DefaultRecordBatch$2.doReadRecord(DefaultRecordBatch.java:290)
at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.readNext(DefaultRecordBatch.java:634)
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:598)
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:567)
at org.apache.kafka.clients.consumer.internals.CompletedFetch.nextFetchedRecord(CompletedFetch.java:209)
at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:254)
at org.apache.kafka.clients.consumer.internals.AbstractFetch.fetchRecords(AbstractFetch.java:340)
at org.apache.kafka.clients.consumer.internals.AbstractFetch.collectFetch(AbstractFetch.java:306)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1262)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1186)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1649)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1624)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1421)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1313)
at java.base@21.0.2/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
... 4 common frames omitted

and I'm wondering if the line should actually be ClassLoader.loadLibrary("snappy-java") (with a dash) to correctly find the library. Does this name differ for different package managers?

@tonivade
Copy link

@thimmwork I think you can use the property org.xerial.snappy.lib.name to overwrite the default value.

@thimmwork
Copy link

Unfortunately, you cannot. When setting org.xerial.snappy.use.systemlib=true, SnappyLoader.findNativeLibrary() will return null, and its caller SnappyLoader.loadNativeLibrary() will perform a plain System.loadLibrary("snappyjava"); - no variables, no means to override behavior.

org.xerial.snappy.lib.name will only be used for loading a resource, and only if both org.xerial.snappy.use.systemlib and org.xerial.snappy.disable.bundled.libs remain false (default).

@KafkaProServerless
Copy link

I think the original author found a real issue here.
While @liuxgo 's workaround is indeed working it requires to maintain an external resource-config.json just for GraalVM.
Our team has some 61 microservice using this snappy java.
We are now in the mist of converting them into native image, and we have to maintain some 61 resource folders, with some 61 configuration files.

Would it be possible for the authors of this repo to provide a fix to remediate that?
Would it be possible to reach out and work with https://github.com/oracle/graalvm-reachability-metadata/tree/master to fix this issue?

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants