Platform-specific Gems not found in vendor/cache #3328

stevenharman · 2019-03-28T16:37:01Z

We have a dependency on google-protobuf, which has platform-specific gems for linux, macOS, and ruby. We also bundle package our gems so we can deploy w/o needing to hit RubyGems while building our images. We've attempted to configure bundler to cache all specified platform-specific gems so they're also updated via a bundle update.

The initial bundle install works as expected - installs the gem and updates ./vendor/cache. But subsequent bundle install, bundle package, and other commands fail with Could not find <GEM> in any of the sources. Interestingly, subsequent bundle update works as expected.

Minimal set of steps to reproduce

mkdir bundle_platform_error && cd bundle_platform_error

bundle init
bundle package

At this point we have an empty ./vendor/cache and pretty much bare Gemfile.lock. Next we configure Bundler to cache all platform gems:

bundle config --local specific_platform true && \
  bundle config --local cache_all_platforms true

bundle lock --add-platform x86_64-linux
bundle lock --add-platform x86_64-darwin-18

cat Gemfile.lock
#=> GEM
#=>   specs:
#=> 
#=> PLATFORMS
#=>   ruby
#=>   x86_64-darwin-18
#=>   x86_64-linux
#=> 
#=> DEPENDENCIES
#=> 
#=> BUNDLED WITH
#=>    2.0.1

Then, to make sure things work as expected, we add a non-platform specific gem:

echo 'gem "dumb_delegator"' >> Gemfile

bundle
#=> Fetching gem metadata from https://rubygems.org/.
#=> Resolving dependencies...
#=> Using bundler 2.0.1
#=> Using dumb_delegator 0.8.0
#=> Updating files in vendor/cache
#=>   * dumb_delegator-0.8.0.gem
#=> Bundle complete! 1 Gemfile dependency, 2 gems now installed.
#=> Use `bundle info [gemname]` to see where a bundled gem is installed.

bundle
#=> Using bundler 2.0.1
#=> Using dumb_delegator 0.8.0
#=> Updating files in vendor/cache
#=> Bundle complete! 1 Gemfile dependency, 2 gems now installed.
#=> Use `bundle info [gemname]` to see where a bundled gem is installed.

All works well. The initial bundle install installed and cached the gem. A subsequent bundle install used the existing install gems and updated the cache. Now we add a platform-specific gem:

echo 'gem "google-protobuf"' >> Gemfile

bundle
#=> Fetching gem metadata from https://rubygems.org/.
#=> Resolving dependencies...
#=> Using bundler 2.0.1
#=> Using dumb_delegator 0.8.0
#=> Using google-protobuf 3.7.1 (universal-darwin)
#=> Updating files in vendor/cache
#=> Fetching google-protobuf 3.7.1
#=>   * google-protobuf-3.7.1.gem
#=>   * google-protobuf-3.7.1-universal-darwin.gem
#=> Fetching google-protobuf 3.7.1 (x86_64-linux)
#=>   * google-protobuf-3.7.1-x86_64-linux.gem
#=> Bundle complete! 2 Gemfile dependencies, 3 gems now installed.
#=> Use `bundle info [gemname]` to see where a bundled gem is installed.

cat Gemfile.lock
#=> GEM
#=>   remote: https://rubygems.org/
#=>   specs:
#=>     dumb_delegator (0.8.0)
#=>     google-protobuf (3.7.1)
#=>     google-protobuf (3.7.1-universal-darwin)
#=>     google-protobuf (3.7.1-x86_64-linux)
#=> 
#=> PLATFORMS
#=>   ruby
#=>   x86_64-darwin-18
#=>   x86_64-linux
#=> 
#=> DEPENDENCIES
#=>   dumb_delegator
#=>   google-protobuf
#=> 
#=> BUNDLED WITH
#=>    2.0.1

That worked. All three versions of the gem were cached, and our specific platform (macOS) was installed and used. Now we'll bundle install again and we expect a noop.

bundle
#=> Using bundler 2.0.1
#=> Using dumb_delegator 0.8.0
#=> Using google-protobuf 3.7.1 (universal-darwin)
#=> Updating files in vendor/cache
#=> Could not find google-protobuf-3.7.1 in any of the sources

It errors (with a return code of 7), warning Could not find <GEM> in any of the sources. However, we can verify the gems are in the ./vendor/cache:

tree vendor/cache
#=> vendor/cache
#=> ├── dumb_delegator-0.8.0.gem
#=> ├── google-protobuf-3.7.1-universal-darwin.gem
#=> ├── google-protobuf-3.7.1-x86_64-linux.gem
#=> └── google-protobuf-3.7.1.gem
#=> 
#=> 0 directories, 4 files

Interestingly a bundle update at this point does work, and exits happily:

bundle update
#=> Fetching gem metadata from https://rubygems.org/.
#=> Resolving dependencies...
#=> Using bundler 2.0.1
#=> Using dumb_delegator 0.8.0
#=> Using google-protobuf 3.7.1 (universal-darwin)
#=> Updating files in vendor/cache
#=> Bundle updated!

But subsequent command, like bundle install and bundle package (at least), fail:

bundle
#=> Using bundler 2.0.1
#=> Using dumb_delegator 0.8.0
#=> Using google-protobuf 3.7.1 (universal-darwin)
#=> Updating files in vendor/cache
#=> Could not find google-protobuf-3.7.1 in any of the sources

bundle package
#=> Using bundler 2.0.1
#=> Using dumb_delegator 0.8.0
#=> Using google-protobuf 3.7.1 (universal-darwin)
#=> Updating files in vendor/cache
#=> Could not find google-protobuf-3.7.1 in any of the sources

Next steps?

I'm not sure where to take it from here. I doubt this is expected behavior, but would like to confirm.

Environment

Bundler       2.0.1
  Platforms   ruby, x86_64-darwin-18
Ruby          2.6.2p47 (2019-03-13 revision 67232) [x86_64-darwin18]
  Full Path   /Users/sharman/.rbenv/versions/2.6.2/bin/ruby
  Config Dir  /Users/sharman/.rbenv/versions/2.6.2/etc
RubyGems      3.0.3
  Gem Home    /Users/sharman/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0
  Gem Path    /Users/sharman/.gem/ruby/2.6.0:/Users/sharman/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0
  User Path   /Users/sharman/.gem/ruby/2.6.0
  Bin Dir     /Users/sharman/.rbenv/versions/2.6.2/bin
Tools
  Git         2.21.0
  RVM         not installed
  rbenv       rbenv 1.1.1
  chruby      not installed

Bundler Build Metadata

Built At          2019-01-04
Git SHA           d7ad2192f
Released Version  true

Bundler settings

console
  Set for the current user (/Users/sharman/.bundle/config): "pry"
gem.mit
  Set for the current user (/Users/sharman/.bundle/config): true
gem.test
  Set for the current user (/Users/sharman/.bundle/config): "rspec"
gem.coc
  Set for the current user (/Users/sharman/.bundle/config): false
specific_platform
  Set for your local app (/Users/sharman/code/bundle_platform_error/.bundle/config): true
cache_all_platforms
  Set for your local app (/Users/sharman/code/bundle_platform_error/.bundle/config): true

Possibly Related Issues?

I've seen several other Issues reported, all seemingly around the area of package-specific gems. Perhaps related? Perhaps not. They are:

The text was updated successfully, but these errors were encountered:

jochenseeber · 2019-06-07T10:47:03Z

It seems this is caused by the version specific to the development platform being installed on the local machine, and the generic version as well as all other platform specific versions not being installed.

bundle package then cannot resolve the Gem since it only tries the installed Gems. I'm not sure on what to do here. A quick fix would be to resolve all Gems remotely when running bundle package.

If someone points me in the right direction, I'm willing to create a patch.

bugg2844 · 2019-11-12T17:46:16Z

I posted a workaround here, in case it helps.

deivid-rodriguez · 2020-10-20T12:53:13Z

Could you try #4022 and verify if it fixes this issue?

jdelStrother · 2020-10-23T14:00:48Z

I ran into this with a git-gem dependency that had native extensions. bundle package --all would generate the vendor/cache/foo-123456 directory, and bundle check would pass, but it didn't generate vendor/cache/extensions so subsequent attempts to use the gem from the cache would fail.

I manually ran bundle package --all from #4022, looks like it fixes the issue for us, FWIW.

deivid-rodriguez · 2020-10-23T14:11:54Z

That's great to hear @jdelStrother, I'm closing this then!

stevenharman · 2020-10-23T15:52:51Z

Perhaps I'm not correctly installing Rubygems locally, but I think I'm still seeing the issue. Assuming the steps in the original description (above) have been followed to get to a failing state, then:

pull latest rubygems/rubygems source
Open the rubygems/rubygems/lib/rubygems.rb file and adjust the Gem::VERSION constant to "3.2.0.rc.29".
- Note: I updated the version to be sure I was running the code just pulled down via git.
Install the latest version locally:

$ ruby setup.rb                                                                                                                                                                                                                                                                                                                 [11:43:12]
  Successfully built RubyGem
  Name: bundler
  Version: 2.2.0.rc.2
  File: bundler-2.2.0.rc.2.gem
Generating ctags for bundler-2.2.0.rc.2
ctags: Warning: cannot open source file "../../../../.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/bundler-2.2.0.rc.2/lib" : No such file or directory
Bundler 2.2.0.rc.2 installed
RubyGems 3.2.0.rc.29 installed
Regenerating binstubs
Regenerating plugins
Parsing documentation for rubygems-3.2.0.rc.29
Installing ri documentation for rubygems-3.2.0.rc.29

# ... [snip]

RubyGems installed the following executables:
	/Users/sharman/.rbenv/versions/2.7.1/bin/gem
	/Users/sharman/.rbenv/versions/2.7.1/bin/bundle

Switch back to bundle_platform_error dir and verify we're using the Rubygems we just installed:

$ cd bundle_platform_error

$ rehash

$ gem --version
3.2.0.rc.29

$ bundle
Using bundler 2.2.0.rc.2
Using dumb_delegator 0.8.0
Using google-protobuf 3.7.1 (universal-darwin)
Updating files in vendor/cache
Could not find google-protobuf-3.7.1-x86_64-linux in any of the sources

Am I testing this incorrectly? If not, could we re-open this issue?

Thank you.

deivid-rodriguez · 2020-10-26T11:45:29Z

@stevenharman Yes, your testing seems correct to me, and you're the original reporter whereas the issue that led me to close this was only partially related to yours, so it makes total sense to reopen 👍. I'll try to give this a closer look.

deivid-rodriguez · 2020-10-28T15:09:03Z

@stevenharman I created #4042 which I believe should fix this. Could you verify it?

stevenharman · 2020-10-28T17:06:16Z

#4042 which I believe should fix this. Could you verify it?

@deivid-rodriguez ✅ Confirmed - this issue looks to be fixed with the changes in #4042. I'll close this issue once that PR is merged. Thank you! ❤️

deivid-rodriguez · 2020-10-28T17:39:04Z

Great!! Thanks for double checking!

hsbt transferred this issue from rubygems/bundler Mar 14, 2020

bronzdoc added the status: triage label Mar 14, 2020

colby-swandale added the Bundler label Mar 14, 2020

deivid-rodriguez added status: feedback required and removed status: triage labels Oct 20, 2020

deivid-rodriguez closed this as completed Oct 23, 2020

deivid-rodriguez reopened this Oct 26, 2020

deivid-rodriguez removed the status: feedback required label Oct 28, 2020

deivid-rodriguez mentioned this issue Oct 28, 2020

Fix issue with cache_all_platforms and specific_platform configured #4042

Merged

4 tasks

deivid-rodriguez closed this as completed in #4042 Oct 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Platform-specific Gems not found in vendor/cache #3328

Platform-specific Gems not found in vendor/cache #3328

stevenharman commented Mar 28, 2019

jochenseeber commented Jun 7, 2019

bugg2844 commented Nov 12, 2019

deivid-rodriguez commented Oct 20, 2020

jdelStrother commented Oct 23, 2020

deivid-rodriguez commented Oct 23, 2020

stevenharman commented Oct 23, 2020

deivid-rodriguez commented Oct 26, 2020

deivid-rodriguez commented Oct 28, 2020

stevenharman commented Oct 28, 2020 •

edited

deivid-rodriguez commented Oct 28, 2020

Platform-specific Gems not found in vendor/cache #3328

Platform-specific Gems not found in vendor/cache #3328

Comments

stevenharman commented Mar 28, 2019

Minimal set of steps to reproduce

Next steps?

Environment

Bundler Build Metadata

Bundler settings

Possibly Related Issues?

jochenseeber commented Jun 7, 2019

bugg2844 commented Nov 12, 2019

deivid-rodriguez commented Oct 20, 2020

jdelStrother commented Oct 23, 2020

deivid-rodriguez commented Oct 23, 2020

stevenharman commented Oct 23, 2020

deivid-rodriguez commented Oct 26, 2020

deivid-rodriguez commented Oct 28, 2020

stevenharman commented Oct 28, 2020 • edited

deivid-rodriguez commented Oct 28, 2020

stevenharman commented Oct 28, 2020 •

edited