Backport non gnu libc linux support from RubyGems #4488

lloeki · 2021-03-24T15:35:03Z

What was the end-user or developer problem that led to this PR?

See #3174, #4082, #4425.

What is your fix for the problem, implemented in this PR?

Implement the non-ideal but pragmatic behaviour we agreed upon:

"the previous state of things" i.e nil version in gems is a wildcard (i.e a linux gem it can be picked by Rubies on linux and linux-musl)
with the exception that if there's a linux-musl variant, it should always be preferred on Alpine (i.e a linux-musl Ruby)
and a mirror constraint that on a linux Ruby it should never pick linux-musl gems

Make sure the following tasks are checked

Describe the problem / feature
Write tests for features and bug fixes
Write code to solve the problem
Make sure you follow the current code style and write meaningful commit messages without tags

lloeki · 2021-03-24T15:41:11Z

So, this is WIP: previous work + messy SAVEPOINT commit that I'll clean up later, but I needed to push some base to discuss upon.

I did it TDD/BDD by adding rubygems and bundler tests that make sense WRT the behaviour, and made them pass.

There's still some strange behaviour that I hacked in/out, so I'm not entirely sure I did not create breakage at this stage on other platforms. I also saw some strange things I can't seem to wrap my head around: most likely I'm missing something but they could be legacy code, or bugs.

I'll continue working on this as I figure things out.

CONTRIBUTING.md

bundler/doc/development/SETUP.md

bundler/lib/bundler/gem_helpers.rb

bundler/lib/bundler/match_platform.rb

lib/rubygems/platform.rb

lloeki · 2021-03-24T16:13:55Z

test/rubygems/test_gem_platform.rb

+
+    # musl and explicit gnu should differ
+    # refute(x86_linux_gnu === x86_linux_musl, 'linux-gnu =~ linux-musl')
+    # refute(x86_linux_musl === x86_linux_gnu, 'linux-musl =~ linux-gnu')


It might be worth thinking about introducing linux-gnu someday, for those good willed maintainers that know their binary is definitely not musl compatible but with no linux-musl published (yet, or ever).

I did introduce checks for this but I think we may be getting ahead of ourselves, I'm not fully sure whether we should include it with this patch or not.

lloeki · 2021-03-24T16:17:11Z

test/rubygems/test_gem_platform.rb

+    refute(x86_linux_musl === x86_linux, 'linux-musl =~ linux')
+
+    # other libc are not glibc compatible
+    # refute(x86_linux === x86_linux_uclibc, 'linux =~ linux-uclibc')


The case of these non-musl, non-gnu libc that are known not to be compatible at all with glibc binaries (so zero chance of linux working) remains unsolved here. Anecdotal I know, but one day musl was anecdotal as well.

I fixed this one too! Never seen these in the wild, but if they're not compatible it seems good to make sure we never match "standard" linux.

deivid-rodriguez · 2021-03-26T21:36:22Z

@lloeki I left a few comments, hopefully they can help 🙏.

nightpool · 2021-05-26T14:24:27Z

After trying to upgrade to mini_racer 0.4.0 and libv8-node this weekend, I found that this issue has broken my Heroku deployment:

remote:        Fetching libv8-node 15.14.0.1 (x86_64-linux-musl)
remote:        Installing libv8-node 15.14.0.1 (x86_64-linux-musl)
....
remote:        Bundle completed (183.45s)
remote:        Cleaning up the bundler cache.
remote:        Could not find libv8-node-15.14.0.1 in any of the sources
remote: -----> Writing config/database.yml to read from DATABASE_URL
remote: -----> Detecting rake tasks
remote:        Could not detect rake tasks
remote:        ensure you can run `$ bundle exec rake -P` against your app
remote:        and using the production group of your Gemfile.
remote:        /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:91:in `block in materialize': Could not find libv8-node-15.14.0.1 in any of the sources (Bundler::GemNotFound)
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `map!'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `materialize'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/definition.rb:170:in `specs'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/definition.rb:237:in `specs_for'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/definition.rb:226:in `requested_specs'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:108:in `block in definition_method'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:20:in `setup'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler.rb:107:in `setup'
remote:        	from /tmp/build_3933b252/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/setup.rb:20:in `<top (required)>'
remote:        	from /tmp/build_3933b252/vendor/ruby-2.5.8/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
remote:        	from /tmp/build_3933b252/vendor/ruby-2.5.8/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
remote:        	from /tmp/build_3933b252/vendor/bundle/bin/rake:27:in `<main>'

At least, I'm assuming that this is the same issue? Is there any known workaround?

lloeki · 2021-05-26T14:50:29Z

Might be, it fetches linux-musl when (I assume) Heroku is glibc-based?

I really have to find some time to get back on this and finish the PR (should happen soon)

nightpool · 2021-05-26T14:57:18Z

Thanks @lloeki. I'm not sure why it would be giving that specific error though, considering that there aren't any issues at build time. I also tried pushing the precompiled x86_64 gem to my github rubygems server (here) and got a very similar error:

remote:        Fetching libv8-node 15.14.0.1
remote:        Installing libv8-node 15.14.0.1
[...]
remote:        Bundle completed (182.41s)
remote:        Cleaning up the bundler cache.
remote:        Could not find libv8-node-15.14.0.1 in any of the sources
remote: -----> Writing config/database.yml to read from DATABASE_URL
remote: -----> Detecting rake tasks
remote:        Could not detect rake tasks
remote:        ensure you can run `$ bundle exec rake -P` against your app
remote:        and using the production group of your Gemfile.
remote:        /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:91:in `block in materialize': Could not find libv8-node-15.14.0.1 in any of the sources (Bundler::GemNotFound)
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `map!'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `materialize'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/definition.rb:170:in `specs'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/definition.rb:237:in `specs_for'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/definition.rb:226:in `requested_specs'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:108:in `block in definition_method'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:20:in `setup'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler.rb:107:in `setup'
remote:        	from /tmp/build_4900502f/vendor/bundle/ruby/2.5.0/gems/bundler-1.17.3/lib/bundler/setup.rb:20:in `<top (required)>'
remote:        	from /tmp/build_4900502f/vendor/ruby-2.5.8/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
remote:        	from /tmp/build_4900502f/vendor/ruby-2.5.8/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
remote:        	from /tmp/build_4900502f/vendor/bundle/bin/rake:27:in `<main>'

lloeki · 2021-10-13T22:56:43Z

(should happen soon)

Hah, joke's on me. No time for now but I'm not forgetting about this.

lloeki · 2021-11-10T12:08:40Z

I'm going to try to redo this based on the rubygems 3.0 + bundler 1.17 patch I produced, which seems to produce good results. Of course only the specs will tell if it fully works.

wwahammy · 2022-01-18T16:15:16Z

@lloeki thanks for all the work so far on this! Is there anything I can do to help get this across the finish line? I'm certainly not familiar with the bundler source but I'm happy to help in any way I can.

virusman · 2022-03-10T06:05:23Z

@lloeki Any hope for this to get merged?

lloeki · 2022-03-10T10:02:19Z

@virusman not as is.

FYI I'm rethinking how it should work and what the expectations of the platform.version component should be so that it doesn't break anything.

Do you have a need for a specific gem outside of mini_racer? The more information I have about how people use multiple libc, the better.

vbabenkoru · 2022-03-11T04:58:59Z

Yeah I need this for mini_racer… The changes in rubygems and bundler broke it, and I can’t even deploy the current version of my app without downgrading rubygems and bundler because of that. No workarounds I’ve seen have worked for bundler.

J0WI · 2022-03-17T13:54:15Z

Do you have a need for a specific gem outside of mini_racer?

nokogiri: Avoid glibc dependency docker-library/redmine#230
wkhtmltopdf: doesn't work on Alpine zakird/wkhtmltopdf_binary_gem#53

ronaldtse · 2022-03-26T03:01:45Z

We're also encountering this issue with 4 other gems:

Failure when loaded in Alpine Linux with musl-dev lutaml/expressir#106
Failed to compile due to unsupported platform aarch64-alpine-linux-musl (on Apple M1 with docker) metanorma/emf2svg-ruby#17
Build aarch64-alpine-linux-musl for Alpine Linux on arm64 fontist/extract_ttc#7
Build aarch64-alpine-linux-musl for Alpine Linux on arm64 fontist/ffi-libarchive-binary#11

On Alpine with Ruby 3.1 (3.1-alpine at https://hub.docker.com/_/ruby) we are seeing the installation of a libc gem (x86_64-linux) but it fails on inclusion due to musl being unable to load a libc compiled library.

The only two options we have as gem maintainers are:

The x86_64-linux platform gem works natively on musl. As @lloeki says it can be impractical.
We publish platform gems for x86_64-linux-musl so that on Alpine the x86_64-linux-musl is always used.

cc: @maxirmx @alexeymorozov @CAMOBAP

hsbt · 2022-03-26T03:30:04Z

At first, the current ruby language is not supported alpine linux and musl officially. The ruby core team does not testing them with https://rubyci.org/

ronaldtse · 2022-03-26T07:13:48Z

Thanks @hsbt for the clarification! It was confusing whether it is supported or not given that the Ruby docker images include one built on Alpine... I guess it's a good time to investigate.

hsbt · 2022-03-26T07:32:52Z

I understood alpine image was published by docker team. But the ruby core team didn't work together it. we have our official docker image only https://github.com/ruby/ruby-docker-images/pkgs/container/ruby .

This means the native extension built by musl may not works well with the ruby language. I highly recommend to use debian based image or official docker image published by the ruby core team.

Co-authored-by: Loic Nageleisen <loic.nageleisen@gmail.com>

deivid-rodriguez · 2022-08-22T20:55:52Z

I merged the RubyGems part of this separately at #5852, and this PR now only backports the feature to Bundler so that this works consistently across RubyGems version. I only backported all the way back to RubyGems 3.1.0.pre.1 because trying to backport further required further changes and was a mess.

ronaldtse · 2022-08-23T00:35:41Z

Thank you @deivid-rodriguez for #5852 ! We look forward to seeing this merged too and getting to work with non-glibc platforms!

deivid-rodriguez · 2022-08-23T05:29:44Z

Thank @lloeki, he did all the heavy lifting! :)

lloeki · 2022-08-23T07:22:58Z

You're welcome, there were so many users impacted by this I could don't let that one slide :)

ronaldtse · 2022-08-23T07:27:09Z

Of course a huge thank you to @lloeki ! Looking forward to publishing platform gems for musl...

Backport non gnu libc linux support from RubyGems (cherry picked from commit 95114c8)

ronaldtse · 2022-08-24T14:09:00Z

Thank you for merging! Now we're going to start working!

deivid-rodriguez · 2022-08-24T19:42:56Z

Unfortunately, this PR created again a regression in musl platforms 😬. I'll try hard to fix it to hopefully avoid having to revert the feature.

Recent updates of the Ruby Gem dependency management utility, 'bundler', introduced a breakage to the 'jekyll-theme-chirper' Ruby Gem starting from version v2.3.21: rubygems/rubygems#4488 ``` 2022-12-03T02:23:00.509Z [WARNING]: Could not find gem 'jekyll-theme-chirpy (~> 5.2, >= 5.2.1)' with platform 'x86_64-linux' in rubygems repository https://rubygems.org/ or installed locally. The source contains the following gems matching 'jekyll-theme-chirpy (~> 5.2, >= 5.2.1)': * jekyll-theme-chirpy-5.2.1 * jekyll-theme-chirpy-5.3.0 * jekyll-theme-chirpy-5.3.1 * jekyll-theme-chirpy-5.3.2   ``` The Ruby Gem dependency management utility, 'bundler', has the same functionality as its aliased variant 'bundle` (ref: rubygems/bundler@2894378). To simplify the AWS Amplify CI/CD pipeline this commit leverages the 'bundle' utility preinstalled in the 'Amazon Linux:2' build image (Docker container). The 'Amazon Linux:2' build image contains 'bundle' version 2.0.1. This version was released almost 4 years ago: rubygems/bundler@d7ad219. The latest version of 'bundler' currently available is 2.3.26, however

Recent updates of the Ruby Gem dependency management utility, 'bundler', introduced a breakage to the 'jekyll-theme-chirper' Ruby Gem starting from version v2.3.21: rubygems/rubygems#4488 ``` 2022-12-03T02:23:00.509Z [WARNING]: Could not find gem 'jekyll-theme-chirpy (~> 5.2, >= 5.2.1)' with platform 'x86_64-linux' in rubygems repository https://rubygems.org/ or installed locally. The source contains the following gems matching 'jekyll-theme-chirpy (~> 5.2, >= 5.2.1)': * jekyll-theme-chirpy-5.2.1 * jekyll-theme-chirpy-5.3.0 * jekyll-theme-chirpy-5.3.1 * jekyll-theme-chirpy-5.3.2   ``` This commit is intended to serve as a stop-gap solution until all existing blog content has been migrated to AWS Amplify.

Recent updates of the Ruby Gem dependency management utility, 'bundler', introduced a breakage to the 'jekyll-theme-chirpy' Ruby Gem starting from version v2.3.21: rubygems/rubygems#4488 ``` 2022-12-03T02:23:00.509Z [WARNING]: Could not find gem 'jekyll-theme-chirpy (~> 5.2, >= 5.2.1)' with platform 'x86_64-linux' in rubygems repository https://rubygems.org/ or installed locally. The source contains the following gems matching 'jekyll-theme-chirpy (~> 5.2, >= 5.2.1)': * jekyll-theme-chirpy-5.2.1 * jekyll-theme-chirpy-5.3.0 * jekyll-theme-chirpy-5.3.1 * jekyll-theme-chirpy-5.3.2   ``` This commit is intended to serve as a stop-gap solution until all existing blog content has been migrated to AWS Amplify.

bundlerbot added Bundler RubyGems labels Mar 24, 2021