Bundler :"Could not find [gem] in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache #9051

bensheldon · 2024-02-13T23:38:32Z

Is there an existing issue for this?

I have searched the existing issues

Package ecosystem

Bundler

Package manager version

No response

Language version

Ruby 3.2+

Manifest location and content before the Dependabot update

No response

dependabot.yml content

No response

Updated dependency

Any gem with require_ruby_version > 3.1.3
In our case, this is vernier which has spec.required_ruby_version = ">= 3.2.1"

What you expected to see, versus what you actually saw

This error seems to happen when both of the following conditions are met:

The Gemfile contains a gem whose gemspec declares a required_ruby_version that is greater than Dependabot's Ruby version (currently Ruby v3.1.3)
The project has used bundler package to vendor the .gem files into vendor/cache

When both of these conditions happen, Dependabot will fail to update with Bundler::GemNotFound: Could not find [gem] in any of the sources. Here is an example:

{
    "error":"Could not find vernier-0.4.0 in any of the sources",
    "error_class":"Bundler::GemNotFound",
    "trace":[
        "/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler/spec_set.rb:149:in `block in materialized_for_all_platforms'",
        "/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler/spec_set.rb:145:in `map'",
        "/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler/spec_set.rb:145:in `materialized_for_all_platforms'",
        "/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler/runtime.rb:112:in `cache'",
        "/opt/bundler/v2/lib/functions/lockfile_updater.rb:81:in `block in cache_vendored_gems'",
        "/usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler/settings.rb:158:in `temporary'",
        "/opt/bundler/v2/lib/functions/lockfile_updater.rb:79:in `cache_vendored_gems'",
        "/opt/bundler/v2/lib/functions/lockfile_updater.rb:49:in `generate_lockfile'",
        "/opt/bundler/v2/lib/functions/lockfile_updater.rb:24:in `run'",
        "/opt/bundler/v2/lib/functions.rb:37:in `update_lockfile'",
        "/opt/bundler/v2/run.rb:33:in `<main>'"
    ]
}

This error was generated using dependabot dry run on a stripped down project: https://github.com/bensheldon/dep-resolution-experiment

Below is the full command log/stacktrace:

Dependabot command log

[dependabot-core-dev] ~ $ DEBUG_HELPERS=1 LOCAL_GITHUB_ACCESS_TOKEN="XXX" bin/dry-run.rb bundler bensheldon/dep-resolution-experiment
=> cloning into /home/dependabot/tmp/bensheldon/dep-resolution-experiment
🎈 Ecosystem Versions log: {:package_managers=>{"bundler"=>"2"}}
=> parsing dependency files
{"BUNDLE_PATH"=>"/home/dependabot/tmp/20240209-97-ysmxor/.bundle", "GEM_HOME"=>"/opt/bundler/v2/.bundle"}
ruby /opt/bundler/v2/run.rb
{"function":"parsed_gemfile","args":{"gemfile_name":"Gemfile","lockfile_name":"Gemfile.lock","dir":"/home/dependabot/tmp/bensheldon/dep-resolution-experiment"}}
=> updating 2 dependencies: activesupport, vernier

=== activesupport (7.1.0)
 => checking for updates 1/2
{"BUNDLE_PATH"=>"/home/dependabot/tmp/20240209-97-ysmxor/.bundle", "GEM_HOME"=>"/opt/bundler/v2/.bundle"}
ruby /opt/bundler/v2/run.rb
{"function":"dependency_source_type","args":{"dir":"/home/dependabot/tmp/20240209-97-ysmxor/dependabot_20240209-97-f0er82","gemfile_name":"Gemfile","dependency_name":"activesupport","credentials":[{"type":"git_source","host":"github.com","username":"x-access-token","password":"XXX"}]}}
🌍 --> GET https://rubygems.org/api/v1/versions/activesupport.json
🌍 <-- 200 https://rubygems.org/api/v1/versions/activesupport.json
 => latest available version is 7.1.3
{"BUNDLE_PATH"=>"/home/dependabot/tmp/20240209-97-ysmxor/.bundle", "GEM_HOME"=>"/opt/bundler/v2/.bundle"}
ruby /opt/bundler/v2/run.rb
{"function":"resolve_version","args":{"dependency_name":"activesupport","dependency_requirements":[{"requirement":">= 0","groups":["default"],"source":null,"file":"Gemfile"}],"gemfile_name":"Gemfile","lockfile_name":"Gemfile.lock","dir":"/home/dependabot/tmp/bensheldon/dep-resolution-experiment","credentials":[{"type":"git_source","host":"github.com","username":"x-access-token","password":"XXX"}]}}
/home/dependabot/common/lib/dependabot/shared_helpers.rb:190:in `run_helper_subprocess': Illformed requirement ["system"] (Dependabot::SharedHelpers::HelperSubprocessFailed)
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:169:in `bind_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:169:in `validate_call_skip_block_type'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:111:in `block in create_validator_slow_skip_block_type'
	from /home/dependabot/bundler/lib/dependabot/bundler/native_helpers.rb:64:in `block in run_bundler_subprocess'
	from /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler.rb:386:in `block in with_original_env'
	from /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler.rb:658:in `with_env'
	from /usr/local/lib/ruby/gems/3.1.0/gems/bundler-2.5.3/lib/bundler.rb:386:in `with_original_env'
	from /home/dependabot/bundler/lib/dependabot/bundler/native_helpers.rb:60:in `run_bundler_subprocess'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:169:in `bind_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:169:in `validate_call_skip_block_type'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:111:in `block in create_validator_slow_skip_block_type'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker/version_resolver.rb:90:in `block (2 levels) in fetch_latest_resolvable_version_details'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker/shared_bundler_helpers.rb:56:in `block in in_a_native_bundler_context'
	from /home/dependabot/common/lib/dependabot/shared_helpers.rb:58:in `block in in_a_temporary_repo_directory'
	from /home/dependabot/common/lib/dependabot/shared_helpers.rb:58:in `chdir'
	from /home/dependabot/common/lib/dependabot/shared_helpers.rb:58:in `in_a_temporary_repo_directory'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `bind_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `validate_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:193:in `block in create_validator_slow'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker/shared_bundler_helpers.rb:52:in `in_a_native_bundler_context'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker/version_resolver.rb:89:in `block in fetch_latest_resolvable_version_details'
	from /home/dependabot/common/lib/dependabot/shared_helpers.rb:266:in `with_git_configured'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `bind_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:272:in `validate_call'
	from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11193/lib/types/private/methods/call_validation.rb:193:in `block in create_validator_slow'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker/version_resolver.rb:85:in `fetch_latest_resolvable_version_details'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker/version_resolver.rb:47:in `latest_resolvable_version_details'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker.rb:205:in `latest_resolvable_version_details'
	from /home/dependabot/bundler/lib/dependabot/bundler/update_checker.rb:28:in `latest_resolvable_version'
	from bin/dry-run.rb:649:in `block in <main>'
	from bin/dry-run.rb:611:in `each'
	from bin/dry-run.rb:611:in `<main>'

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

No response

Smallest manifest that reproduces the issue

https://github.com/bensheldon/dep-resolution-experiment

The text was updated successfully, but these errors were encountered:

jurre · 2024-02-14T09:56:12Z

It's curious to me that having the files committed to vendor/cache makes a difference. The obvious solution of course is for us to upgrade our ruby version, we are currently blocked on bundler 1 support for that but also we could run into the same scenario with some gem that declares it wants < 3.2. It looks like we are able to circumvent the resolution checks without vendoring so it suggests that it might be possible to do it with a vendored cache as well

composerinteralia · 2024-02-23T18:56:29Z

It's curious to me that having the files committed to vendor/cache makes a difference.

This looks like a difference:

dependabot-core/bundler/helpers/v2/lib/functions/lockfile_updater.rb

Line 49 in 4229759

cache_vendored_gems(definition) if Bundler.app_cache.exist?

composerinteralia · 2024-02-23T20:08:45Z

Might not be related, but I wonder if we also need to add Ruby 3.3 to

dependabot-core/bundler/helpers/v2/monkey_patches/definition_ruby_version_patch.rb

Line 29 in 4229759

%w(2.5.3 2.6.10 2.7.7 3.0.5 3.2.1).each do |version|

jurre · 2024-03-05T10:31:42Z

Might not be related, but I wonder if we also need to add Ruby 3.3 to

dependabot-core/bundler/helpers/v2/monkey_patches/definition_ruby_version_patch.rb

Line 29 in 4229759

%w(2.5.3 2.6.10 2.7.7 3.0.5 3.2.1).each do |version|

Yeah I thought this might be related by I tried it and just adding 3.3.0 didn't do the trick. I spent a few minutes debugging and I do think there's a way that we can get bundler to resolve this, but I need to carve out some more time to get to the bottom of it. I'll try to find that time soon, would love to see this resolved

composerinteralia · 2024-03-05T14:41:11Z

@bensheldon and I paired on this last week and adding 3.3 there didn't seem to be enough (although we probably want that regardless). We had some initial luck with patching https://github.com/rubygems/rubygems/blob/62a21b44e3af5dcde95e4f1ff7ed8133b6b77772/bundler/lib/bundler/match_metadata.rb#L9-L11 to return true though (we saw evidence of it working at one point while we were hacking around, but didn't quite get to working code).

jurre · 2024-05-08T16:48:58Z

So, while the underlying issue isn't entirely fixed and might happen again when our Ruby become out of date, but Dependabot is now on Ruby 3.3.1 since #9597, so that should improve things a bit for now at least.

kroehre · 2024-05-22T13:23:03Z

@jurre We're having the opposite issue, dependabot has been failing for us for a month now (presumably since #9597) on a gem that has require_ruby_version < 3.3.0. Is there a way for us to configure the ruby version to fix this?

bensheldon added the T: bug 🐞 Something isn't working label Feb 13, 2024

bensheldon changed the title ~~Bundler :"Could not find GEM in any of the sources" when gem required_ruby_version > v3.1.4 and using vendor/cache~~ Bundler :"Could not find GEM in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache Feb 13, 2024

bensheldon changed the title ~~Bundler :"Could not find GEM in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache~~ Bundler :"Could not find [gem] in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache Feb 13, 2024

bensheldon mentioned this issue Mar 5, 2024

Temporarily downgrade Vernier required_ruby_version bensheldon/vernier#1

Draft

abdulapopoola added the L: ruby:bundler RubyGems via bundler label Mar 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundler :"Could not find [gem] in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache #9051

Bundler :"Could not find [gem] in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache #9051

bensheldon commented Feb 13, 2024

jurre commented Feb 14, 2024

composerinteralia commented Feb 23, 2024 •

edited

composerinteralia commented Feb 23, 2024

jurre commented Mar 5, 2024 •

edited

composerinteralia commented Mar 5, 2024

jurre commented May 8, 2024

kroehre commented May 22, 2024

Bundler :"Could not find [gem] in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache #9051

Bundler :"Could not find [gem] in any of the sources" when gem required_ruby_version > 3.1.3 and using vendor/cache #9051

Comments

bensheldon commented Feb 13, 2024

Is there an existing issue for this?

Package ecosystem

Package manager version

Language version

Manifest location and content before the Dependabot update

dependabot.yml content

Updated dependency

What you expected to see, versus what you actually saw

Native package manager behavior

Images of the diff or a link to the PR, issue, or logs

Smallest manifest that reproduces the issue

jurre commented Feb 14, 2024

composerinteralia commented Feb 23, 2024 • edited

composerinteralia commented Feb 23, 2024

jurre commented Mar 5, 2024 • edited

composerinteralia commented Mar 5, 2024

jurre commented May 8, 2024

kroehre commented May 22, 2024

composerinteralia commented Feb 23, 2024 •

edited

jurre commented Mar 5, 2024 •

edited