Platform-specific gem ignored and Gemfile.lock unexpectedly changed by bundle install after upgrading to Ruby 3.3.0

[owst]

Describe the problem as clearly as you can

When upgrading from Ruby 3.2.3 to 3.3.0, a bundle install on platform x86_64-linux seems to ignore the platform-specific versions, installs the gem with native extensions, and modifies the Gemfile.lock as follows:

--- Gemfile.lock-3.2.3	2024-03-05 17:16:44
+++ Gemfile.lock	2024-03-05 17:18:53
@@ -4,3 +4,3 @@
-    sqlite3 (1.6.7-arm64-darwin)
-    sqlite3 (1.6.7-x86_64-darwin)
-    sqlite3 (1.6.7-x86_64-linux)
+    mini_portile2 (2.8.5)
+    sqlite3 (1.6.7)
+      mini_portile2 (~> 2.8.0)

Did you try upgrading rubygems & bundler?

Yes, issue reproduced with 3.5.6/2.5.6

Post steps to reproduce the problem

With this setup script in setup_lockfile.sh on the host machine:

set -eux

gem update --system 3.5.6

bundle init
bundle lock --add-platform arm64-darwin
bundle lock --add-platform x86_64-darwin
bundle lock --remove-platform ruby

echo "gem 'sqlite3', '= 1.6.7'" >> Gemfile
bundle install

Run the script inside a Ruby 3.2.3 container:

docker run --rm --volume $(pwd):/repro --workdir /repro ruby:3.2.3-slim bash setup_lockfile.sh

Which will generate this Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    sqlite3 (1.6.7-arm64-darwin)
    sqlite3 (1.6.7-x86_64-darwin)
    sqlite3 (1.6.7-x86_64-linux)

PLATFORMS
  arm64-darwin
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  sqlite3 (= 1.6.7)

BUNDLED WITH
   2.5.6

With this script in demonstrate_issue.sh on the host machine (in the same directory as the Gemfile{,.lock} as generated by the 3.2.3 container in the previous step):

set -eux

apt-get update
apt-get install -y build-essential pkg-config

gem update --system 3.5.6

bundle install

Run the script inside a Ruby 3.3.0 container:

docker run --rm --volume $(pwd):/repro --workdir /repro ruby:3.3.0-slim bash demonstrate_issue.sh

...then (after a delay to install the extension) the following Gemfile.lock is generated:

GEM
  remote: https://rubygems.org/
  specs:
    mini_portile2 (2.8.5)
    sqlite3 (1.6.7)
      mini_portile2 (~> 2.8.0)

PLATFORMS
  arm64-darwin
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  sqlite3 (= 1.6.7)

BUNDLED WITH
   2.5.6

Which command did you run?

bundle install after upgrading Ruby to 3.3.0.

What were you expecting to happen?

That sqlite3 (1.6.7-x86_64-linux) would be installed (thus not requiring build-essential and pkg-config) and the Gemfile.lock to not be modified.

What actually happened?

The platform-specific gem in the lockfile was ignored (with no output to say why) and the lockfile was modified to remove the other platform-specific versions.

If not included with the output of your command, run bundle env and paste the output below

In the Ruby 3.2.3 container:

Bundler       2.5.6
  Platforms   ruby, x86_64-linux
Ruby          3.2.3p157 (2024-01-18 revision 52bb2ac0a6971d0391efa2275f7a66bff319087c) [x86_64-linux]
  Full Path   /usr/local/bin/ruby
  Config Dir  /usr/local/etc
RubyGems      3.5.6
  Gem Home    /usr/local/bundle
  Gem Path    /root/.local/share/gem/ruby/3.2.0:/usr/local/lib/ruby/gems/3.2.0:/usr/local/bundle
  User Home   /root
  User Path   /root/.local/share/gem/ruby/3.2.0
  Bin Dir     /usr/local/bundle/bin
Tools
  Git         not installed
  RVM         not installed
  rbenv       not installed
  chruby      not installed

In the Ruby 3.3.0 container:

Bundler       2.5.6
  Platforms   ruby, x86_64-linux
Ruby          3.3.0p0 (2023-12-25 revision 5124f9ac7513eb590c37717337c430cb93caa151) [x86_64-linux]
  Full Path   /usr/local/bin/ruby
  Config Dir  /usr/local/etc
RubyGems      3.5.6
  Gem Home    /usr/local/bundle
  Gem Path    /root/.local/share/gem/ruby/3.3.0:/usr/local/lib/ruby/gems/3.3.0:/usr/local/bundle
  User Home   /root
  User Path   /root/.local/share/gem/ruby/3.3.0
  Bin Dir     /usr/local/bundle/bin
Tools
  Git         not installed
  RVM         not installed
  rbenv       not installed
  chruby      not installed

[owst]

After further investigation, I can see the (likely) cause - 1.6.7-x86_64-linux has REQUIRED RUBY VERSION: >= 2.7, < 3.3.DEV so is incompatible with Ruby 3.3. It looks like the first platform-specific version of sqlite3 to support Ruby 3.3 is 1.7.0-x86_64-linux and indeed, if we use 1.7.0 then there is no native extension build or lockfile diff on Ruby 3.3.

Perhaps Bundler should at least warn that it can't install the locked gem version (and give a reason why).

I'm sure there's much more to it than I've considered, but I think I'd rather have bundler fail to install rather than "silently" change the lock file like this (with the corresponding increase in bundle install times due to compiling the native extension).

As a further consideration, before I stripped down to a minimal reproducing example, our Gemfile had sqlite3 pinned as ~> 1.6.1 (that had been locked to 1.6.7) - in that case the diff when moving to Ruby 3.3 is:

--- Gemfile.lock-3.2.3	2024-03-06 13:24:28
+++ Gemfile.lock	2024-03-06 13:28:04
@@ -4,3 +4,3 @@
-    sqlite3 (1.6.7-arm64-darwin)
-    sqlite3 (1.6.7-x86_64-darwin)
-    sqlite3 (1.6.7-x86_64-linux)
+    mini_portile2 (2.8.5)
+    sqlite3 (1.6.9)`
+      mini_portile2 (~> 2.8.0)

I'm surprised that the version has changed to 1.6.9 (though I suspect Bundler has just re-resolved the constraint ~> 1.6.1) - I think I would expect Bundler to use 1.6.7, but as above I think I'd rather a warning or error to prevent this silent change altogether

[deivid-rodriguez]

Yes, this is indeed expected. In verbose mode there's a bit more information about what's going on under the hood. Bundler should inform that the locked gems are not valid for the current platform (maybe "platform" should read "ruby" in this case), and that's why it will try to re-resolve gems. Maybe we could print this debug output by default.

[owst]

Maybe we could print this debug output by default.

Thanks @deivid-rodriguez - this sounds sensible. I'm not sure about debug output in general, but certainly some warning that the requested gem(s) can't be installed and so a re-resolution was happening would have probably put me on the right path

[deivid-rodriguez]

Yeah, agree on not printing debug output in general, but this particular piece of information probably deserves more visibility. I'll try to improve default output when this kind of edge case happens 👍.