Can't install google-protobuf (3.25.1) on JRuby 9.4.5.0 on aarch64

[dentarg]

From docker run -it --rm jruby:9.4.5.0 bash (on Apple silicon)

$ docker run -it --rm -v $(pwd):/app -w /app jruby:9.4.5.0 bash
Unable to find image 'jruby:9.4.5.0' locally
9.4.5.0: Pulling from library/jruby
a5319f8e5f3f: Pull complete
7b69f35fd155: Pull complete
972efbb53556: Pull complete
655643420165: Pull complete
293ddc00a8b8: Pull complete
724459cf70a6: Pull complete
4607861d3bec: Pull complete
78ae414d88a8: Pull complete
511ff1b72590: Pull complete
35e7a63dd310: Pull complete
Digest: sha256:6cd5b08649f2f3ab07c0d200725cbde6fa41f6b746fe01fbca6b8cc34865ae41
Status: Downloaded newer image for jruby:9.4.5.0

root@7ac1c6d8979a:/app# gem install google-protobuf
Fetching google-protobuf-3.25.1-java.gem
Fetching ffi-compiler-1.0.1.gem
Successfully installed ffi-compiler-1.0.1
Building native extensions. This could take a while...
ERROR:  Error installing google-protobuf:
	ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/google-protobuf-3.25.1-java/ext/google/protobuf_c
/opt/jruby/bin/jruby -I/opt/jruby/lib/ruby/stdlib -rrubygems /opt/jruby/lib/ruby/gems/shared/gems/rake-13.1.0/exe/rake RUBYARCHDIR\=/usr/local/bundle/extensions/universal-java-1.8/3.1.0/google-protobuf-3.25.1-java RUBYLIBDIR\=/usr/local/bundle/extensions/universal-java-1.8/3.1.0/google-protobuf-3.25.1-java
mkdir -p aarch64-linux/third_party/utf8_range
cc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -std=gnu99 -O3 -Wall -Wsign-compare -Wno-declaration-after-statement -fvisibility=hidden -fPIC -Ithird_party/utf8_range -DNDEBUG=1 -DUPB_BUILD_API=1   -o aarch64-linux/third_party/utf8_range/range2-sse.o -c ./third_party/utf8_range/range2-sse.c
rake aborted!
Command failed with status (127): [cc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -std=gnu99 -O3 -Wall -Wsign-compare -Wno-declaration-after-statement -fvisibility=hidden -fPIC -Ithird_party/utf8_range -DNDEBUG=1 -DUPB_BUILD_API=1   -o aarch64-linux/third_party/utf8_range/range2-sse.o -c ./third_party/utf8_range/range2-sse.c]
/usr/local/bundle/gems/ffi-compiler-1.0.1/lib/ffi-compiler/compile_task.rb:133:in `block in define_task!'
org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
Tasks: TOP => default => ffi-protobuf:default => aarch64-linux/libprotobuf_c_ffi.so => aarch64-linux/third_party/utf8_range/range2-sse.o
(See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in /usr/local/bundle/gems/google-protobuf-3.25.1-java for inspection.
Results logged to /usr/local/bundle/extensions/universal-java-1.8/3.1.0/google-protobuf-3.25.1-java/gem_make.out

root@7ac1c6d8979a:/app# cat -n /usr/local/bundle/extensions/universal-java-1.8/3.1.0/google-protobuf-3.25.1-java/gem_make.out
     1	current directory: /usr/local/bundle/gems/google-protobuf-3.25.1-java/ext/google/protobuf_c
     2	/opt/jruby/bin/jruby -I/opt/jruby/lib/ruby/stdlib -rrubygems /opt/jruby/lib/ruby/gems/shared/gems/rake-13.1.0/exe/rake RUBYARCHDIR\=/usr/local/bundle/extensions/universal-java-1.8/3.1.0/google-protobuf-3.25.1-java RUBYLIBDIR\=/usr/local/bundle/extensions/universal-java-1.8/3.1.0/google-protobuf-3.25.1-java
     3	mkdir -p aarch64-linux/third_party/utf8_range
     4	cc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -std=gnu99 -O3 -Wall -Wsign-compare -Wno-declaration-after-statement -fvisibility=hidden -fPIC -Ithird_party/utf8_range -DNDEBUG=1 -DUPB_BUILD_API=1   -o aarch64-linux/third_party/utf8_range/range2-sse.o -c ./third_party/utf8_range/range2-sse.c
     5	rake aborted!
     6	Command failed with status (127): [cc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -std=gnu99 -O3 -Wall -Wsign-compare -Wno-declaration-after-statement -fvisibility=hidden -fPIC -Ithird_party/utf8_range -DNDEBUG=1 -DUPB_BUILD_API=1   -o aarch64-linux/third_party/utf8_range/range2-sse.o -c ./third_party/utf8_range/range2-sse.c]
     7	/usr/local/bundle/gems/ffi-compiler-1.0.1/lib/ffi-compiler/compile_task.rb:133:in `block in define_task!'
     8	org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
     9	org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
    10	org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
    11	org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
    12	Tasks: TOP => default => ffi-protobuf:default => aarch64-linux/libprotobuf_c_ffi.so => aarch64-linux/third_party/utf8_range/range2-sse.o
    13	(See full trace by running task with --trace)
    14
    15	rake failed, exit code 1

root@7ac1c6d8979a:/app# ruby -v
jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 25.392-b08 on 1.8.0_392-b08 +jit [aarch64-linux]
root@7ac1c6d8979a:/app# uname -a
Linux 7ac1c6d8979a 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Not sure what goes wrong here, maybe the issue is with https://github.com/jruby/docker-jruby and not JRuby in general. I'll let you experts decide on that.

[headius]

@dentarg The problem here is that google-protobuf only has a C extension at this point. I believe there was work planned to make it use FFI, or to add a JVM/JRuby version, but I have no idea whether that work has gone forward. Have you checked their bug tracker to see if anyone else is trying to use JRuby?

[ahorek]

@headius google-protobuf has been supported on JRuby (via ffi) since 2021 :) but you still need a working compiler to build C extensions for it.

@dentarg based on your log - cc command failed with status (127) which means you don't have a compiler (gcc / clang) installed in your docker container. Something like https://github.com/alexandreelise/install-gcc/blob/master/Dockerfile should help.

[dentarg]

I'm using https://github.com/jruby/docker-jruby so maybe this issue should be transfered to that repo?

[ahorek]

yes, but it's questionable if devkit tools should be included by default

because JRuby itself (and most gems) don't need them, only gems with C extensions do

using precompiled binaries for each platform is an alternative option, but google-protobuf requires both java and c extensions, so they would have to bundle binaries for all platforms in a single gem (platform=java). There's no way to solve this combination cleanly with the current version of rubygems & rake compiler.

[jimtng]

yes, but it's questionable if devkit tools should be included by default

Wouldn't it make it so much easier to install a lot more gems that need to be compiled?

[ahorek]

@jimtng definitely, but there are downsides.
including gcc makes images bigger, it's not a small dependency
some people may not like to include any build tools on production (for security)

anyway, we should at least improve the error message, because Command failed with status (127) isn't very helpful.

it should be something like This gem requires C extensions, but devkit isn't available, please add these (...) libraries to your bundle.

[headius]

google-protobuf has been supported on JRuby (via ffi) since 2021

@ahorek Thank you, I knew about the FFI work but I lost track of it. Glad to hear it is working!

all platforms in a single gem (platform=java)

Yeah I wish we had a better option here. RubyGems needs better support for conditional dependencies based on OS, architecture, and Ruby engine.

Wouldn't it make it so much easier to install a lot more gems that need to be compiled?

@jimtng There are very few such gems, but it is a growing problem. The more we push FFI for integrating native code, the more we see gems like sassc and google-protobuf that expect to be able to build that native code before binding it.

We have had other requests to add build tools to the JRuby docker images, and while we're not opposed to it we'd probably do so as a separate image, to avoid existing users suddenly getting a larger image with tools they don't need or want. It's a trade-off we're still actively debating.

Of course it's also pretty easy for you to just add some commands and install the necessary tools on your end while spinning up the container, right?

[chadlwilson]

Only just stumbled across this but there's some discussion and background at protocolbuffers/protobuf#14611 with more specific summary of their plans for jruby/java at protocolbuffers/protobuf#14611 (comment)

My problem was mainly with Windows because toolchain is more of a pain there and we already had toolchains installed for Linux runners for other purposes, but the wider summary of where they are going on Ruby with ffi/ffi-compiler may be useful.