Bundler 2.2.5 still doesn't fallback to source gem when platform specific gem is unusable

[casperisfine]

Ref: rubygems/bundler#7522

From what I understood, bundler 2.2.x was supposed to fallback to compiling gems if the precompiled binaries can't be used.

But from what I experienced it still doesn't work:

Using bundler 2.2.5

google-protobuf-3.14.0-x86_64-linux requires ruby version < 2.8.dev, >= 2.3,
which is incompatible with the current version, ruby 3.0.0p0

[deivid-rodriguez]

Yes, definitely. This should work.

I suspect bundler resolution is sometimes falling back to the old gem dependency API which didn't have required_ruby_version information so even if the resolver is now correct, it's lacking some information.

I don't know why this is happening, but we have received two reports about this in the last 2 days, hence my suspicion. It could be a server side issue too, I'll investigate. Can you post some logs that I can see, ideally in verbose mode so we can see the network requests being made.

I suspect this is not reproducible reliably, correct?

[casperisfine]

Can you post some logs that I can see, ideally in verbose mode so we can see the network requests being made.

Our gemfile is huge and contains lots of private gems. I'll try to better reproduce in isolation and provide logs.

I suspect this is not reproducible reliably, correct?

Yes. I'm trying to pinpoint it exactly, but it isn't quite trivial. I'll try to come back with a repro script.

[casperisfine]

Ok, I think I figured it out:

source 'https://rubygems.org'

gem 'grpc'

Ruby 2.7.2: bundle install, generates the following Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    google-protobuf (3.14.0-universal-darwin)
    googleapis-common-protos-types (1.0.5)
      google-protobuf (~> 3.11)
    grpc (1.34.0-universal-darwin)
      google-protobuf (~> 3.13)
      googleapis-common-protos-types (~> 1.0)

PLATFORMS
  x86_64-darwin-19

DEPENDENCIES
  grpc

BUNDLED WITH
   2.2.5

Then chruby 3.0.0:

$ bundle install
$ bundle
Fetching gem metadata from https://rubygems.org/.........
google-protobuf-3.14.0-universal-darwin requires ruby version < 2.8.dev, >= 2.3, which is incompatible with the current version, ruby 3.0.0p0

$ diff Gemfile.lock Gemfile.lock.2.7 
4c4
<     google-protobuf (3.14.0)
---
>     google-protobuf (3.14.0-universal-darwin)
7c7
<     grpc (1.34.0)
---
>     grpc (1.34.0-universal-darwin)

So the problem seem to be that our Gemfile.lock was generated with Ruby 2.7, so it recorded the precompiled gem as usable. Then when we switch to 3.0 (because we aren't fully migrated yet) it tries to use the same version.

I'll see if I can find a workaround for us, but this seems like a bug to me. It's not uncommon to use the same Gemfile.lock concurrently with different ruby versions.

[deivid-rodriguez]

Oh, I see... But, how can we solve that?

In this case two different ruby versions have two different resolutions. I think if you want to reuse the same lockfile for both rubies you could use #4049, making the compromise of giving up on platform specific variants for this gem.

I'm open to suggestions.

[casperisfine]

There might be more to it though, because in our app I don't see the compiled gem being listed in Gemfile.lock:

$ bundle --version
Bundler version 2.2.5
$ bundle
Resolving dependencies....................
Fetching source index from https://rubygems.org/
google-protobuf-3.14.0-universal-darwin requires ruby version >= 2.3, < 2.8.dev, which is incompatible with the current version, ruby 3.0.0p0
$ grep google-protobuf Gemfile.lock 
      google-protobuf
      google-protobuf
    google-protobuf (3.14.0)
      google-protobuf (~> 3.11)
      google-protobuf (~> 3.13)
      google-protobuf (>= 3.4.1.1, < 4)
  google-protobuf

how can we solve that?

I'm afraid I'm not familiar enough with bundler internal to offer much help on this.

Couldn't bundler always list the source gem even when it select the platform specific one? Or list all variants?

I haven't dug much yet, but that Gemfile.lock I posted above looks fishy to me. It list OSX specific stuff, but then we run our CI on linux. I've also seem dependabot PRs (generated on linux) include linux specific gem name but no osx or source ones.

[casperisfine]

you could use #4049, making the compromise of giving up on platform specific variants for this gem.

I've wished for this feature for a while. It wouldn't be perfect, but at least it would unblock me I think.

[deivid-rodriguez]

Can you try bundle lock --add-platform ruby. That should also keep generic variants in the lockfile and I think should fix things.

[casperisfine]

That should also keep generic variants in the lockfile

It did:

diff --git a/Gemfile.lock b/Gemfile.lock
index 38d4965..5240cee 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -2,13 +2,18 @@ GEM
   remote: https://rubygems.org/
   specs:
     google-protobuf (3.14.0)
+    google-protobuf (3.14.0-universal-darwin)
     googleapis-common-protos-types (1.0.5)
       google-protobuf (~> 3.11)
     grpc (1.34.0)
       google-protobuf (~> 3.13)
       googleapis-common-protos-types (~> 1.0)
+    grpc (1.34.0-universal-darwin)
+      google-protobuf (~> 3.13)
+      googleapis-common-protos-types (~> 1.0)
 
 PLATFORMS
+  ruby
   x86_64-darwin-19

I think should fix things.

I'm afraid not:

$ bundle install
Fetching gem metadata from https://rubygems.org/.........
google-protobuf-3.14.0-universal-darwin requires ruby version < 2.8.dev, >= 2.3, which is incompatible with the current version, ruby 3.0.0p0

[casperisfine]

More logs:

$ bundle install --verbose
Running `bundle install --verbose` with bundler 2.2.5
Found no changes, using resolution from the lockfile
The definition is missing ["google-protobuf-3.14.0-universal-darwin", "grpc-1.34.0-universal-darwin"]
HTTP GET https://index.rubygems.org/versions
HTTP 206 Partial Content https://index.rubygems.org/versions
Fetching gem metadata from https://rubygems.org/
Looking up gems ["google-protobuf", "googleapis-common-protos-types", "grpc"]
Looking up gems ["faraday", "googleauth", "jwt", "logging", "minitest", "multi_json", "signet", "xray"]
Looking up gems ["flexmock", "lockfile", "rake", "little-plugger", "addressable", "rack", "multipart-post", "ruby2_keywords", "faraday-net_http", "memoist", "os", "json", "httpadapter", "extlib"]
Looking up gems ["rspec", "public_suffix", "english", "hoe"]
Looking up gems ["rubyforge", "RubyInline", "gemcutter", "rspec-core", "rspec-expectations", "rspec-mocks"]
Looking up gems ["ZenTest", "rspec-support", "diff-lcs", "json_pure", "net-scp"]
Looking up gems ["spruz", "net-ssh"]
Looking up gems ["needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium"]
Looking up gems ["ffi"]
google-protobuf-3.14.0-universal-darwin requires ruby version < 2.8.dev, >= 2.3, which is incompatible with the current version, ruby 3.0.0p0
Bundler::InstallError: google-protobuf-3.14.0-universal-darwin requires ruby version < 2.8.dev, >= 2.3, which is incompatible with the current version, ruby 3.0.0p0

[deivid-rodriguez]

That's unexpected, I would expect bundle lock --add-platform ruby to add only generic variants to the lockfile. That's a bug I'd say, I'll look into it.

Sorry for the issues you're having, a correct multiplatform resolution turned out to be much harder than I was expecting 🙏.

[casperisfine]

No worries, I mostly blame grpc and google-protobuf to be honest :p, they're a huge pain to work with.

And lots of thanks for trying to make this work, I'll try to figure a workaround in the meantime. I used to have a shitty monkey patch to force these gems be source gems, but it no longer works on 2.2.x, I'll have to figure out something:

if RUBY_VERSION >= '3.0'
  module BundlerHack
    def __materialize__
      if name == 'google-protobuf'
        Bundler.settings.temporary(force_ruby_platform: true) do
          super
        end
      else
        super
      end
    end
  end
  Bundler::LazySpecification.prepend(BundlerHack)
end

[casperisfine]

So I don't quite understand how or why, but but looking at the diffs better I saw that 2.7 had PLATFORMS ruby and if I were to re-bundle from scratch 3.0 would have PLATFORMS x86_64-darwin-19.

So I manually edited the Gemfile.lock such as:

diff --git a/Gemfile.lock b/Gemfile.lock
index 377e56dfba68..8bbea0be4a27 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -2700,6 +2700,7 @@ GEM
 
 PLATFORMS
   ruby
+  x86_64-darwin-19
 

And it now install properly on 3.0. That's really puzzling to me though.

[rromanchuk]

@casperisfine i just manually added + x86_64-darwin-19 to my lockfile and it finally worked. I didn't even get as far as figuring out it was from grpc. Luckily your google-protobuf got indexed and you just saved me many hours. Any luck on figuring out the "why" yet?

[casperisfine]

You mean why bundler doesn't fallback to source? No not yet.

[deivid-rodriguez]

I'll try to get this fixed this week.

[deivid-rodriguez]

Ok, so there's a few related issues here.

• One is sharing a lockfile between different rubies. That's something that was never really supported, nothing has changed there.

Try for example running bundle _2.1.4_ install under ruby 2.7

source 'https://rubygems.org'

gem 'sprockets'

And then switch to ruby 2.4 and run bundle _2.1.4_ install with the generated lockfile. You'll get:

Fetching gem metadata from https://rubygems.org/..
sprockets-4.0.2 requires ruby version >= 2.5.0, which is incompatible with the current version, ruby 2.4.10p364

• The second one is a change in bundler 2.2 where platform specific resolution happens at resolution time, not at installation time (this is required to make platform resolution correct and safe in general). In order to achieve that, bundler needs to record the specific platform it resolved against (x86_64-darwin-19) instead of the generic platform (RUBY) in the lockfile. This change actually made the first issue above appear because whereas google-protobuf-3.14.0 supports ruby 3.0, google-protobuf-3.14.0-universal-darwin does not. So since bundler now only uses information from the lockfile and doesn't do any platform resolution at installation time, it tries to install the platform specific gem as recorded in the lockfile (using a different ruby), but that's not supported by the running ruby.

I think you can solve this by removing the specific platform from the lockfile, and locking only for ruby. So, bundle lock --remove-platform x86_64-darwin-19. Once you only have RUBY in there, bundler won't try to add the specific platform anymore.

As something we can improve on our side, I think we can check whether all locked specs are compatible with the running ruby version, and re-resolve if not. So you would get lockfile changes when switching rubies, but bundler will install fine.

And finally, as I third issue I'm completely puzzled about, I have no idea how bundle lock --add-platform ruby could generated lockfile changes as explained in #4282 (comment). Could you post a reproducible example of that so that I can look into it?

[casperisfine]

• One is sharing a lockfile between different rubies. That's something that was never really supported, nothing has changed there.

Ok TIL. This had worked for us until now, it was very convenient for Ruby upgrades. We'll figure out a new strategy then.

Could you post a reproducible example of that so that I can look into it?

It's pretty much what I posted above:

Start with this Gemfile:

source 'https://rubygems.org'

gem 'grpc'

using ruby 2.7.2:

$ ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-darwin19]
$ bundle --version
Bundler version 2.2.5
$ bundle
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Using bundler 2.2.5
Using google-protobuf 3.14.0 (universal-darwin)
Using googleapis-common-protos-types 1.0.5
Using grpc 1.34.0 (universal-darwin)
Bundle complete! 1 Gemfile dependency, 4 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
$ cat Gemfile.lock 
GEM
  remote: https://rubygems.org/
  specs:
    google-protobuf (3.14.0-universal-darwin)
    googleapis-common-protos-types (1.0.5)
      google-protobuf (~> 3.11)
    grpc (1.34.0-universal-darwin)
      google-protobuf (~> 3.13)
      googleapis-common-protos-types (~> 1.0)

PLATFORMS
  x86_64-darwin-19

DEPENDENCIES
  grpc

BUNDLED WITH
   2.2.5
$ bundle lock --add-platform ruby
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Writing lockfile to /private/tmp/bundler-repro/Gemfile.lock
$ cat Gemfile.lock 
GEM
  remote: https://rubygems.org/
  specs:
    google-protobuf (3.14.0)
    google-protobuf (3.14.0-universal-darwin)
    googleapis-common-protos-types (1.0.5)
      google-protobuf (~> 3.11)
    grpc (1.34.0)
      google-protobuf (~> 3.13)
      googleapis-common-protos-types (~> 1.0)
    grpc (1.34.0-universal-darwin)
      google-protobuf (~> 3.13)
      googleapis-common-protos-types (~> 1.0)

PLATFORMS
  ruby
  x86_64-darwin-19

DEPENDENCIES
  grpc

BUNDLED WITH
   2.2.5

[deivid-rodriguez]

Ok TIL. This had worked for us until now, it was very convenient for Ruby upgrades. We'll figure out a new strategy then.

I'd like to support re-resolving if the current ruby can't satisfy the lockfile, but it's tricky because we don't currently record required_ruby_version constraints in the lockfile, so I think this would need a new section in the lockfile with those requirements, so that bundler can figure out whether it needs to re-resolve because the lockfile doesn't satisfy the running ruby.

Regarding the issue, what you posted now makes sense and it's expected. I think maybe there's was just some typos in #4282 (comment) and that caused the confusion? The diff you posted there makes it look like bundle lock --add-platform ruby added darwin specific variants to the lockfile, which doesn't make any sense.

[casperisfine]

Bringing this thread back as I'm hitting the same nokogiri issue than in sparklemotion/nokogiri#2185

Gemfile:

source 'https://rubygems.org'

gem 'nokogiri'

Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    mini_portile2 (2.5.0)
    nokogiri (1.11.1)
      mini_portile2 (~> 2.5.0)
      racc (~> 1.4)
    racc (1.5.2)

PLATFORMS
  ruby

DEPENDENCIES
  nokogiri

BUNDLED WITH
   2.2.13

trying to bundle with 3.1.0-dev:

Fetching gem metadata from https://rubygems.org/.......
nokogiri-1.11.1-x86_64-darwin requires ruby version < 3.1.dev, >= 2.5, which is incompatible with the current version, ruby 3.1.0p-1

Why is bundler trying to use the precompiled gem even though it's not in the Gemfile.lock?

I understand that you said bundling across versions isn't officially supported, but in this specific case I really fail to see why it wouldn't work (even if it's by accident)

I think this would need a new section in the lockfile with those requirements, so that bundler can figure out whether it needs to re-resolve because the lockfile doesn't satisfy the running ruby.

I'm not super familiar with the bundler internals, but can't we instead always list all the alternative gems in the Gemfile.lock, and then when you bundle install it can first try the platform specific version, and if this one is incompatible, then try the ruby platform?

That would be enough to solve 99% of the cross version bundling pains. Because we're trying to run our test suite against Ruby-head, but if we have to nuke our Gemfile.lock it's both a big pain and kind of defeat the purpose, as if the gems are not the same the value of the test result is much lower.

[deivid-rodriguez]

Hi.

Why is bundler trying to use the precompiled gem even though it's not in the Gemfile.lock?

This is for backwards compatibility with old lockfiles. If the specific platform is not there, we assume it's a lockfile not generated with recent versions (which lock the specific platform instead of "RUBY", and save platform specific variants inside the lockfile). And if it's an old lockfile, we keep the old logic of resolving platforms at install time instead of respecting what's in the lockfile.

So, my plan to fix this would be:

• Enhance the old logic to not try to install variants that don't match the current ruby and fallback to the ruby platform in that case.

• For new lockfiles, include ruby & rubygems requirements together with dependencies, so that we can do the right thing in general. I do have a plan to introduce this in a backwards compatible manner by creating new lockfile sections but keeping also the old ones so that they can still be used by old bundler versions. Old bundler versions just ignore unknown lockfile sections, so I think it will work fine. The lockfile could look something like this:

GEMv2
  remote: https://rubygems.org/
  specs:
    nokogiri (1.11.1-x86_64-darwin)
      racc (~> 1.4)
      ruby:  (>= 2.5, < 3.1.dev)
    racc (1.5.2)

PLATFORMS
  x86_64-darwin

DEPENDENCIES
  nokogiri

BUNDLED WITH
   2.3.0.dev

But then if you use it with ruby 3.1, it would detect that it needs to re-resolve, and would generate a lockifle like the following and use the "RUBY" variant:

GEMv2
  remote: https://rubygems.org/
  specs:
    mini_portile2 (2.5.0)
    nokogiri (1.11.1-x86_64-darwin)
      racc (~> 1.4)
      ruby:  (>= 2.5, < 3.1.dev)
    nokogiri (1.11.1)
      mini_portile2 (~> 2.5.0)
      racc (~> 1.4)
      ruby:  (>= 2.5)
    racc (1.5.2)

PLATFORMS
  x86_64-darwin

DEPENDENCIES
  nokogiri

BUNDLED WITH
   2.3.0.dev

I'll try to implement the first bullet point for now to get you unblocked.

[casperisfine]

. If the specific platform is not there, we assume it's a lockfile not generated with recent versions

Oh! That explains it. I thought I was going crazy for a minute.

I'll try to implement the first bullet point for now to get you unblocked.

Thank you so much!

[deivid-rodriguez]

@casperisfine #4449 should fix your issue. I will make that issue close this ticket and then re-ticktet the remaining case (fix the same issue when specific platforms are being locked in the lockfile).

[casperisfine]

❤️