racc default gem bits are not being included in the distribution

[marcandre]

See report rubocop/rubocop-ast#132

See also #6416

[headius]

Thank you for the new issue. I will try to reproduce locally.

[headius]

I can reproduce locally but it really doesn't seem like something we are doing. I suspect bundler but will try to dig into this and figure out what is happening.

For the record I get this output from running the same bundle exec rake spec that rubocop-ast's CI runs:

$ bundle exec rake spec
You have requested:
  racc >= 0

The bundle currently has racc locked at 1.5.2.
Try running `bundle update racc`

If you are updating multiple gems in your Gemfile at once,
try passing them all to `bundle update`
Run `bundle install` to install missing gems.

The racc gem appears in the Gemfile.lock and seems to be fine:

$ cat Gemfile.lock
PATH
  remote: .
  specs:
    rubocop-ast (1.4.1)
      parser (>= 2.7.1.5)

GEM
  remote: https://rubygems.org/
  specs:
    ast (2.4.2)
    bump (0.10.0)
    diff-lcs (1.4.4)
    docile (1.3.5)
    json (2.5.1-java)
    oedipus_lex (2.5.2)
    parallel (1.20.1)
    parser (3.0.0.0)
      ast (~> 2.4.1)
    racc (1.5.2-java)
    rainbow (3.0.0)
...

Attempting to update it, as the error suggests, changes nothing:

$ bundle update racc
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies..........
Using rake 13.0.3
Using ast 2.4.2
Using bump 0.10.0
Using diff-lcs 1.4.4
Using docile 1.3.5
Using bundler 2.2.0
Using json 2.5.1 (java)
Using oedipus_lex 2.5.2
Using parallel 1.20.1
Using parser 3.0.0.0
Using rainbow 3.0.0
Using regexp_parser 2.1.1
Using racc 1.5.2 (java)
Using rexml 3.2.4
Using rspec-support 3.10.2
Using rubocop-ast 1.4.1 from source at `.`
Using ruby-progressbar 1.11.0
Using unicode-display_width 2.0.0
Using simplecov-html 0.10.2
Using rspec-expectations 3.10.1
Using rspec-mocks 3.10.2
Using rspec-core 3.10.1
Using rspec 3.10.0
Using simplecov 0.17.1
Using rubocop 1.11.0
Using rubocop-performance 1.10.1
Using rubocop-rspec 2.2.0
Bundler attempted to update racc but its version stayed the same
Bundle updated!

So this is a mystery. 🤔

[headius]

Running bundle exec racc activates racc fine:

[] ~/projects/rubocop-ast $ bundle exec rake spec
You have requested:
  racc >= 0

The bundle currently has racc locked at 1.5.2.
Try running `bundle update racc`

If you are updating multiple gems in your Gemfile at once,
try passing them all to `bundle update`
Run `bundle install` to install missing gems.

[] ~/projects/rubocop-ast $ bundle exec racc
no input

This may be specific to the build and how it activates racc.

[headius]

This looks like it may be a Bundler bug. I just managed to get it to work, but using bundler 2.2.11 rather than 2.2.

@deivid-rodriguez Does this sound like anything recently fixed in Bundler?

This might also explain why I was not able to reproduce locally until now.

[headius]

After initially being able to reproduce this, I am now unable. 😖

...
Using rubocop 1.11.0
Using rubocop-performance 1.10.1
Using rubocop-rspec 2.2.0
Bundle complete! 11 Gemfile dependencies, 27 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

[] ~/projects/rubocop-ast $ bundle exec rake spec
Generating lib/rubocop/ast/node_pattern/lexer.rex.rb from lib/rubocop/ast/node_pattern/lexer.rex
bundle exec racc -l -v -o lib/rubocop/ast/node_pattern/parser.racc.rb lib/rubocop/ast/node_pattern/parser.y
/Users/headius/projects/jruby-9.2/bin/jruby ...

This is with Bundler 2.2.0. I cleaned the rubocop-ast repo and rebundled and reran and it still works!

[headius]

I believe this is a packaging issue rather than anything broken in JRuby itself. It seems that not all of the racc bits are getting included in the distribution, leading to issues loading the racc executable or activating the gem itself.

[] /tmp/jruby-9.2.16.0 $ ls lib/ruby/gems/shared/gems/
did_you_mean-1.2.1     jar-dependencies-0.4.0 net-telnet-0.1.1       rake-12.3.2            test-unit-3.2.8        xmlrpc-0.3.0
fileutils-1.1.0        minitest-5.10.3        power_assert-1.1.3     rdoc-6.1.2             webrick-1.6.1

[] /tmp/jruby-9.2.16.0 $ ls lib/ruby/gems/shared/specifications/default/
cmath-1.0.0.gemspec               ipaddr-1.2.0.gemspec              jruby-readline-1.3.7-java.gemspec racc-1.5.2-java.gemspec           scanf-1.0.0.gemspec
csv-1.0.0.gemspec                 jar-dependencies-0.4.0.gemspec    json-2.5.1-java.gemspec           rake-ant-1.0.4.gemspec            webrick-1.6.1.gemspec
fileutils-1.1.0.gemspec           jruby-openssl-0.10.5-java.gemspec psych-3.2.0-java.gemspec          rdoc-6.1.2.gemspec

[] /tmp/jruby-9.2.16.0 $ ls bin
ast            gem            irb            jgem           jirb           jirb_swing     jruby          jruby.bat      jruby.exe      jrubyc         jrubyd.bat     rake           rdoc           ri             testrb
ast.bat        gem.bat        irb.bat        jgem.bat       jirb.bat       jirb_swing.bat jruby.bash     jruby.dll      jruby.sh       jrubyc.bat     jrubyw.exe     rake.bat       rdoc.bat       ri.bat         testrb.bat

We see here that the default gemspec is in place, but the executable racc has been lost. That leads to the following error when attempting to run bundled exec rake spec or even bundle exec racc:

 $ bundle exec rake spec
bundle exec racc -l -v -o lib/rubocop/ast/node_pattern/parser.racc.rb lib/rubocop/ast/node_pattern/parser.y
bundler: command not found: racc
Install missing gem executables with `bundle install`
...

Installing the gem manually leads to the next issue, because it still tries to activate the default gem and that executable still is not present where default gems install executables:

[] ~/projects/rubocop-ast $ gem install racc
Fetching racc-1.5.2-java.gem
Successfully installed racc-1.5.2-java
Parsing documentation for racc-1.5.2-java
Installing ri documentation for racc-1.5.2-java
Done installing documentation for racc after 3 seconds
1 gem installed

[] ~/projects/rubocop-ast $ bundle exec rake spec
bundle exec racc -l -v -o lib/rubocop/ast/node_pattern/parser.racc.rb lib/rubocop/ast/node_pattern/parser.y
LoadError: no such file to load -- /Users/headius/.rvm/rubies/jruby-9.2.16.0/lib/ruby/gems/shared/specifications/default/bin/racc
    load at org/jruby/RubyKernel.java:1009
  <main> at /Users/headius/.rvm/gems/jruby-9.2.16.0/bin/racc:23

When run against a HEAD build of JRuby, of course the executable is there, but in a different location (the layout of default gems has changed, perhaps?):

$ ls lib/ruby/gems/shared/gems/racc-1.5.2-java/bin/
racc

At a minimum we need to figure out why the default gem bits are not getting included in the distribution. I will try to come up with a workaround.

[deivid-rodriguez]

Hi!

I wasn't able to repro the exact error the Github Actions job is running into, but I did repro the same errors @headius got. I suspect the specific error depends on whether bundle exec ends up using Kernel.exec or Kernel.load and whether there's a racc executable in path or not, but the root cause is the same: jruby is not shipping an executable for the default gem racc.

[headius]

I will work on the dist today. We will make a small change for 9.2.x to include everything that is supposed to be there, and the larger change on master for 9.3 will be to switch to a blacklist of sources we do not want in the dist, rather than a whitelist that will get out of date.

[headius]

The following diff shows the files in a freshly cloned and build jruby-9.2 branch and the dist archive:

Only in jruby: .git
Only in jruby: .gitattributes
Only in jruby: .github
Only in jruby: .gitignore
Only in jruby: .mvn
Only in jruby: .travis-maven-settings.xml
Only in jruby: .travis.yml
Only in jruby: BUILDING.md
Only in jruby: CODE_OF_CONDUCT.md
Only in jruby: Gemfile
Only in jruby: README.md
Only in jruby: Rakefile
Only in jruby: VERSION
Only in jruby: antlib
Only in jruby: azure-pipelines.yml
Only in jruby: bench
Only in jruby/bin: install_doc
Only in jruby/bin: install_doc.bat
Only in jruby/bin: ruby
Only in jruby: core
Only in jruby: default.build.properties
Only in jruby: findbugs.xml
Only in jruby: install
Only in jruby: ivy
Files jruby/lib/jruby.jar and jruby-9.2.17.0/lib/jruby.jar differ
Only in jruby/lib: pom.rb
Only in jruby/lib: pom.xml
Only in jruby/lib/ruby/gems/shared: bin
Only in jruby/lib/ruby/gems/shared: build_info
Only in jruby/lib/ruby/gems/shared: doc
Only in jruby/lib/ruby/gems/shared: extensions
Only in jruby/lib/ruby/gems/shared/gems/did_you_mean-1.2.1: .gitignore
Only in jruby/lib/ruby/gems/shared/gems/net-telnet-0.1.1: .gitignore
Only in jruby/lib/ruby/gems/shared/gems/power_assert-1.1.3: .gitignore
Only in jruby/lib/ruby/gems/shared/gems: psych-3.2.0-java
Only in jruby/lib/ruby/gems/shared/gems: racc-1.5.2-java
Only in jruby/lib/ruby/gems/shared/gems/rake-12.3.2: .gitignore
Only in jruby/lib/ruby/gems/shared/gems/xmlrpc-0.3.0: .gitignore
Only in jruby/lib/ruby/stdlib/irb/lc: .document
Only in jruby/lib/ruby/stdlib/net: .document
Files jruby/lib/ruby/stdlib/org/yaml/snakeyaml/maven-metadata-local.xml and jruby-9.2.17.0/lib/ruby/stdlib/org/yaml/snakeyaml/maven-metadata-local.xml differ
Only in jruby/lib/ruby/stdlib/rdoc/generator/template/darkfish: .document
Only in jruby/lib/ruby/stdlib/rdoc/generator/template/json_index: .document
Only in jruby/lib: target
Only in jruby: lifecycle-mapping-metadata.xml
Only in jruby: maven
Only in jruby: mvnw
Only in jruby: mvnw.cmd
Only in jruby: pom.rb
Only in jruby: pom.xml
Only in jruby: rakelib
Only in jruby/samples/jnlp: .gitignore
Only in jruby: spec
Only in jruby: test
Only in jruby/tool: compiler2.rb
Only in jruby/tool: concurrent-ruby-travis.sh
Only in jruby/tool: experimental
Only in jruby/tool: generate_parser
Only in jruby/tool: generate_ripper
Only in jruby/tool: git_bisect_run_general.sh
Only in jruby/tool: git_bisect_run_mspec.sh
Only in jruby/tool: jruby.keystore
Only in jruby/tool/nailgun: configure.ac
Only in jruby/tool: optimize_parser.rb
Only in jruby/tool: package_docs.rb
Only in jruby/tool: patch_parser.rb
Only in jruby/tool: sequel-travis.sh
Only in jruby/tool: signature.rb
Only in jruby/tool: svm-populators.rb
Only in jruby/tool: sync_ruby
Only in jruby/tool: sync_ruby_files.rb
Only in jruby/tool: travis-script.sh
Only in jruby/tool: yarvinsns.rb

Interestingly, the bin/racc executable does not seem to get installed for either the HEAD build or the dist archive, so there's something else wrong with our default gem logic. However... if the gems were not culled during dist you would still be able to bundle exec racc, and that is one of the root causes of this issue.

[headius]

So this turned out to be a bigger mess than expected.

Our build installs the bundled and default gems while processing lib/pom.rb, using the Gem::Installer API. This works ok for the bundled gems, which install directly into the JRuby home, but the default gems get installed as normal into a temporary gem location. We perform an extracopy the appropriate files to stdlib and other places. Unfortunately this copying did not do everything it needed to do:

• lib files go to stdlib (was ok)
• executables go to lib/ruby/gems/shared/specifications/default/bin (were going to lib/ruby/gens/shared/bin and could not be found)
• bin stubs installed in bin/ dir (no stubs generated)
• platform gems did not install bin scripts at all (which is why racc was left completely without an executable)

In addition, when packaging the project only a whitelisted set of bin stubs were included in the distribution, which would exclude racc anyway.

I have pushed a fix_dist_gems branch I will turn into a PR, but I have some results:

• A clean clone + build has all the files in the right places and can run rubocop-ast's bundle exec rake spec successfully.
• A packaged distribution works as well.
• The missing bin scripts are now in the dist tarball, as shown by this diff:

diff -r -q jruby-9.2.16.0/ jruby-9.2.17.0/
Only in jruby-9.2.17.0/bin: install_doc
Only in jruby-9.2.17.0/bin: install_doc.bat
Only in jruby-9.2.17.0/bin: lock_jars
Only in jruby-9.2.17.0/bin: lock_jars.bat
Only in jruby-9.2.17.0/bin: racc
Only in jruby-9.2.17.0/bin: racc.bat
Only in jruby-9.2.17.0/bin: ruby
Files jruby-9.2.16.0/lib/jruby.jar and jruby-9.2.17.0/lib/jruby.jar differ
Only in jruby-9.2.16.0/lib/ruby/gems/shared/gems: fileutils-1.1.0
Only in jruby-9.2.16.0/lib/ruby/gems/shared/gems: jar-dependencies-0.4.0
Only in jruby-9.2.16.0/lib/ruby/gems/shared/gems: rdoc-6.1.2
Only in jruby-9.2.16.0/lib/ruby/gems/shared/gems: webrick-1.6.1
Only in jruby-9.2.17.0/lib/ruby/gems/shared/specifications/default: bin
Files jruby-9.2.16.0/lib/ruby/stdlib/org/yaml/snakeyaml/1.26/snakeyaml-1.26.jar and jruby-9.2.17.0/lib/ruby/stdlib/org/yaml/snakeyaml/1.26/snakeyaml-1.26.jar differ
Files jruby-9.2.16.0/lib/ruby/stdlib/org/yaml/snakeyaml/maven-metadata-local.xml and jruby-9.2.17.0/lib/ruby/stdlib/org/yaml/snakeyaml/maven-metadata-local.xml differ

The only questionable diff here is that some gems that appeared in the director of bundled gems are no longer there... but these four gems are default gems, not bundled gems, so we should not see them show up. They show up here because the old logic for copying binscripts for default gems only worked on non-platform gems, and then did not install the files in the right place anyway. The contents of those four gems show there's nothing lost:

[] ~/projects/jruby-9.2/tmp $ find jruby-9.2.16.0/lib/ruby/gems/shared/gems/fileutils-1.1.0/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/fileutils-1.1.0/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/fileutils-1.1.0//bin
jruby-9.2.16.0/lib/ruby/gems/shared/gems/fileutils-1.1.0//bin/setup
jruby-9.2.16.0/lib/ruby/gems/shared/gems/fileutils-1.1.0//bin/console

[] ~/projects/jruby-9.2/tmp $ find jruby-9.2.16.0/lib/ruby/gems/shared/gems/jar-dependencies-0.4.0/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/jar-dependencies-0.4.0/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/jar-dependencies-0.4.0//bin
jruby-9.2.16.0/lib/ruby/gems/shared/gems/jar-dependencies-0.4.0//bin/lock_jars

[] ~/projects/jruby-9.2/tmp $ find jruby-9.2.16.0/lib/ruby/gems/shared/gems/rdoc-6.1.2/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/rdoc-6.1.2/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/rdoc-6.1.2//exe
jruby-9.2.16.0/lib/ruby/gems/shared/gems/rdoc-6.1.2//exe/ri
jruby-9.2.16.0/lib/ruby/gems/shared/gems/rdoc-6.1.2//exe/rdoc

[] ~/projects/jruby-9.2/tmp $ find jruby-9.2.16.0/lib/ruby/gems/shared/gems/webrick-1.6.1/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/webrick-1.6.1/
jruby-9.2.16.0/lib/ruby/gems/shared/gems/webrick-1.6.1//bin
jruby-9.2.16.0/lib/ruby/gems/shared/gems/webrick-1.6.1//bin/setup
jruby-9.2.16.0/lib/ruby/gems/shared/gems/webrick-1.6.1//bin/console

(The setup and console scripts are generated by bundler (I think?) for installing the library directly into a distribution and starting up an IRB console with it loaded. They should not be included in JRuby.)

Finally, the listing of bin files for the faux 9.2.17.0 shows that all the appropriate files are present with their bat file equivalents:

$ ls -1 jruby-9.2.17.0/bin/
ast
ast.bat
gem
gem.bat
install_doc
install_doc.bat
irb
irb.bat
jgem
jgem.bat
jirb
jirb.bat
jirb_swing
jirb_swing.bat
jruby
jruby.bash
jruby.bat
jruby.dll
jruby.exe
jruby.sh
jrubyc
jrubyc.bat
jrubyd.bat
jrubyw.exe
lock_jars
lock_jars.bat
racc
racc.bat
rake
rake.bat
rdoc
rdoc.bat
ri
ri.bat
ruby
testrb
testrb.bat

We will want to reexamine and greatly simplify this process on master, but I believe the diff is fairly minimal and produces the correct file layout.

@marcandre Your original issue was about jruby-head, which should be fixed without a release once we merge this to master. The GHA ruby-install builds are based on snapshots of our normal distribution tarball from master HEAD, which was also broken in the same way as the release distributions from 9.2.x.

[headius]

Fixed by #6609.

[marcandre]

Glad this is fixed 💪
I'll double check when the changes are picked up in the next jruby-dev-builder release.

[marcandre]

Just to confirm this fixes our build on JRuby-head 🎉 , and one day on jruby too. Thanks!

[headius]

@marcandre Thank you! We will do a 9.2.17 release in a couple weeks.