rake-compiler v1.1.2 has broken something in Nokogiri's native gem builds

[flavorjones]

rake-compiler v1.1.2 has broken something in Nokogiri's native gem builds.

An example failure is at https://github.com/sparklemotion/nokogiri/runs/4458677557?check_suite_focus=true, which is running in the larskanis/rake-compiler-dock-mri-x86_64-darwin:1.1.0 image.

You can reproduce this by cloning sparklemotion/nokogiri and running bundle exec rake gem:x86_64-darwin (though please make sure you're using rake-compiler 1.1.2 or 1.1.3, since I'm locking to 1.1.1 in sparklemotion/nokogiri#2383).

...
/usr/local/rvm/rubies/ruby-2.5.9/bin/ruby -I. -r.rake-compiler-siteconf.rb ../../../../ext/nokogiri/extconf.rb  --enable-cross-build
Could not find racc-1.6.0 in any of the sources
Run `bundle install` to install missing gems.
*** ../../../../ext/nokogiri/extconf.rb failed ***

I've bisected the problem to f48d669 but have been unable to understand what the root cause of this, other than noting that removing the siteconf file allows the build to proceed.

[flavorjones]

I'll also note that only the darwin build seems to be affected, as linux and windows builds seem to be fine. 🧐

[kou]

Umm... #191 has many problems...

BTW, is https://github.com/sparklemotion/nokogiri/runs/4458677557?check_suite_focus=true#step:5:212 related to this? It seems that it's a Bundler or Racc related problem.

/usr/local/rvm/gems/ruby-2.5.8/gems/bundler-2.2.3/lib/bundler/spec_set.rb:87:in `block in materialize': Could not find racc-1.6.0 in any of the sources (Bundler::GemNotFound)

[flavorjones]

BTW, is https://github.com/sparklemotion/nokogiri/runs/4458677557?check_suite_focus=true#step:5:212 related to this? It seems that it's a Bundler or Racc related problem.

Yes, this is the problem that I'm reporting here. Something in the siteconf file causes the racc gem to not be found, and I could not figure out what exactly is causing it. Environment variables all look right (GEM_HOME, GEM_PATH) and the gem is in fact installed; and this problem does not happen on linux or windows cross-compilation.

[flavorjones]

I've reproduced this issue in a much simpler gem, which is the "precompiled" gem in https://github.com/flavorjones/ruby-c-extensions-explained

See flavorjones/ruby-c-extensions-explained#6 for the PR that introduces "racc" as a dependency and demonstrates this exact failure, but only with Darwin (and not Linux or Windows):

https://github.com/flavorjones/ruby-c-extensions-explained/runs/4483674590?check_suite_focus=true

/usr/local/rvm/rubies/ruby-2.5.8/bin/ruby -I. -r.rake-compiler-siteconf.rb ../../../../ext/precompiled/extconf.rb --enable-cross-build
/usr/local/rvm/gems/ruby-2.5.8/gems/bundler-2.2.3/lib/bundler/spec_set.rb:87:in `block in materialize': Could not find racc-1.6.0 in any of the sources (Bundler::GemNotFound)

Note that without "racc" everything passes: https://github.com/flavorjones/ruby-c-extensions-explained/actions/runs/1563475900

[kou]

Could you try 1.1.4?

[konsolebox]

In my gem, rake-compiler installs extension (.so) to lib/ instead of lib/gem-name/ (extension source is in ext/gem-name/) during test (which has :compile as dependency). It works fine with 1.1.1 and 1.1.2 but broken with 1.1.3. I'm yet to examine this more thoroughly but this might be related.

Edit: This also wasn't solved in 1.1.4 by the way, and this issues occurred both in local test and in AppVeyor. Also, in AppVeyor's test while it was still using 1.1.4, I had to specify -rdevkit to make it detect the 'make' executable, but I no longer had to when I restricted rake-compiler's version to 1.1.1 in gemspec. Not sure if the issue was only related to AppVeyor's environment or just a change in rake-compiler's behavior.

[flavorjones]

(Deleted my previous comment)

[flavorjones]

@kou Thank you! After cleaning and clobbering the project directory, I'm able to build successfully with rake-compiler v1.1.4. 👏 👏 👏