Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feedback and final to-do items for precompiled native gems on Linux and OSX #2075

Closed
flavorjones opened this issue Sep 8, 2020 · 31 comments
Closed
Labels
packaging/native-gem release-blocker Blocking a milestone release
Milestone

Comments

@flavorjones
Copy link
Member

flavorjones commented Sep 8, 2020

This issue exists to capture feedback and action items about the precompiled native gems for Linux and OSX that we're shipping in the Nokogiri v1.11 release candidates (prereleases).

(RC4 is out! https://github.com/sparklemotion/nokogiri/releases/tag/v1.11.0.rc4)

How do I try this out?

We're only shipping native gems for the following platforms (run bundle platform or ruby -e 'puts Gem::Platform.local.to_s' to identify your system):

  • New platforms as of v1.11 release candidates:
    • OSX/Darwin: x86_64-darwin and arm64-darwin
    • Linux: x86-linux and x86_64-linux including musl platforms like alpine
  • Previously existing native gems (note these have already been available for many years):
    • Windows: x86-mingw32 and x64-mingw32
    • JRuby: java

If you're on Linux or OSX, try updating your Gemfile with:

gem "nokogiri", "~> 1.11.0.rc4"`

If you're having trouble with bundler trying to install a native gem and it failing, you may want to set the specific_platform bundler config as described in bundler install should not fail when a native platform gem's required_ruby_version is not met, and a suitable ruby platform gem is available · Issue #4012 · rubygems/rubygems:

bundle config set --global specific_platform true

Please let us know how it works for you in the comments!

Known Issues

Providing Feedback

If you love this idea in a vague general sense, please give this comment a ❤️.

If you tried native gems on Linux or OSX and they worked just fine, please give this comment a 👍.

If you're having problems installing or using Nokogiri's precompiled native gems, please comment on this issue with the following information:

  • output from ruby -v
  • output from gem -v
  • output from gem env
  • if you're using bundler,
    • output from bundle version
    • output from bundle config (take care to redact any personal info or credentials)
  • output from gem install or bundle install, including the command you ran

Why is a precompiled native gem useful?

Installing a precompiled native gem, installation takes about 1 second, and avoids the challenging bits of how to set up the compiler toolchain on your development system. This will reduce the support burden for Team Nokogiri and allow us to move a bit faster on bugfixes and features.

For comparison, compiling and installing the gem (and its vendored dependencies) from source, when only using a single core, my laptop takes 52 seconds. When using all 8 cores, this is cut down to 32 seconds.

Why would I not want to use a precompiled native gem?

I can imagine some folks might have trust issues; if this is you, please let us know in a comment here (or in #2013) what we could do to increase that trust (e.g., I can imagine providing a chain of custody including public build logs with cryptographic hashes of artifacts).

Anybody on a linux system old enough to not have glibc >= 2.17 will need to install (via compilation) from the ruby platform gem.

If you have other reasons for not wanting to use a precompiled native gem, let us know in the comments.

How can I avoid using a precompiled native gem?

If you want to avoid using a precompiled gem in your project, do one of the following:

  • If you're not using bundler, then run gem install nokogiri -v 1.11.0.rc4 --platform=ruby
  • If you are using bundler, then run bundle config set force_ruby_platform true and you'll never use a precompiled native gem

Shout-outs

Thanks to @larskanis for all the time he spends maintaining the gems rake-compiler and rake-compiler-dock, and for being the one who wrote the first PR for Linux native gem support.

Thanks to the GRPC team who already ships native gems (see https://github.com/grpc/grpc). They blazed the trail and provided inspiration and a how-to manual for some edge cases.

Thanks to @tenderlove who paired with me on the nasty OSX bits that I was struggling with.

Resources

@odlp
Copy link

odlp commented Sep 8, 2020

Unfortunately my system seems to be building the native extensions still:

time gem install --prerelease nokogiri
Fetching nokogiri-1.11.0.rc3.gem
Fetching mini_portile2-2.5.0.gem
Successfully installed mini_portile2-2.5.0
Building native extensions. This could take a while...
Successfully installed nokogiri-1.11.0.rc3
2 gems installed
gem install --prerelease nokogiri  46.41s user 34.10s system 93% cpu 1:26.53 total

I'm on macOS Catalina (10.15). My Ruby / Rubygems version (via Rbenv):

ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-darwin18]
gem -v
3.1.4

I noticed on Rubygems.org that RC3 only has darwin-19 build, yet my Ruby version states darwin-18. Is this a Catalina / BigSur mismatch?

@flavorjones
Copy link
Member Author

flavorjones commented Sep 8, 2020

@odlp - Thanks for giving it a try! I'm not building darwin18 gems at this point, because I don't have a system available to do so. Only darwin19 is supported right now.

(But I have an old Mac Mini sitting around, maybe I'll use that for darwin18!)

@odlp
Copy link

odlp commented Sep 8, 2020

@flavorjones thanks for clarifying!

Thanks for all this work on precompiled Nokogiri, it's an awesome improvement. Happy thought: I wonder how many tons of CO2 this'll save per year 😃

@gsiener
Copy link

gsiener commented Sep 8, 2020

macOS Catalina (10.15.6)
ruby-2.6.5 [ x86_64 ]

gsiener$ time gem install --prerelease nokogiri
Successfully installed nokogiri-1.11.0.rc3-x86_64-darwin-19
1 gem installed

real	0m2.835s
user	0m1.159s
sys	0m0.651s

@jrochkind
Copy link
Contributor

On MacOS 10.15.6, ruby 2.6.6

Installs fine. Ran tests in one app and two gems I work on, tests green in all of them.

@thbar
Copy link
Contributor

thbar commented Sep 8, 2020

Quick feedback:

  • On one ruby 2.5.8p224 (2020-03-31 revision 67882) [x86_64-darwin18] Rails app, there is no regression. It is darwin18 as pointed above, so compiling like before, but I still wanted to point out the absence of regressions 😄
  • On one JRuby app (tested on both jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.232-b09 on 1.8.0_232-b09 +jit [darwin-x86_64] and jruby 9.2.5.0 (2.5.0) 2018-12-06 6d5a228 OpenJDK 64-Bit Server VM 25.232-b09 on 1.8.0_232-b09 +jit [darwin-x86_64]), I see many test failures as soon as I upgrade from Nokogiri 1.10.10 to any of the 1.11.0.rc[1/2/3] releases. Something appears to have been botched there.

I will investigate more on the JRuby failures.

Thanks for your work on accelerating the installation, much appreciated!

@flavorjones
Copy link
Member Author

@thbar Thanks for testing this. If you need to, please open a new GH issue for the jruby problems -- I don't think they're going to be related to the native gem work for Linux and OSX. I'd like to know more about what you're seeing, though!

@captn3m0
Copy link

captn3m0 commented Sep 8, 2020

Works perfectly on linux x86_64 (click for logs)
time gem install --prerelease nokogiri
Fetching nokogiri-1.11.0.rc3-x86_64-linux.gem
Nokogiri is built with the packaged libraries: libxml2-2.9.10, libxslt-1.1.34, zlib-1.2.11, libiconv-1.15.
Successfully installed nokogiri-1.11.0.rc3-x86_64-linux
Parsing documentation for nokogiri-1.11.0.rc3-x86_64-linux
Installing ri documentation for nokogiri-1.11.0.rc3-x86_64-linux
Done installing documentation for nokogiri after 1 seconds
1 gem installed

real 5.469  user 1.821  sys 0.218 pcpu 37.28

nokogiri -v
# Nokogiri (1.11.0.rc3)
    ---
    warnings: []
    nokogiri: 1.11.0.rc3
    ruby:
      version: 2.7.1
      platform: x86_64-linux
      gem_platform: x86_64-linux
      description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34

@twk3
Copy link

twk3 commented Sep 8, 2020

Its not happy on centos6, ruby 2.6

 $ ldd /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so
/usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so: /lib64/libz.so.1: version `ZLIB_1.2.3.3' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so)
/usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so)
/usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so)
	linux-vdso.so.1 =>  (0x00007ffc61dd8000)
	libruby.so.2.5 => not found
	libm.so.6 => /lib64/libm.so.6 (0x00007f64d7b0f000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f64d78f9000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f64d76dc000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f64d7348000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f64d81c5000)

I have the newer zlib in /usr/local/lib

@flavorjones
Copy link
Member Author

@twk3 Interesting! Thank you for reporting this, I'll take a look.

@flavorjones
Copy link
Member Author

@twk3 How did you compile/build/install that version of Ruby? Looks like yum install ruby on centos6 installs Ruby 1.8.7 (!). I'd like to reproduce what you're seeing.

@twk3
Copy link

twk3 commented Sep 8, 2020

@flavorjones compiled from source, can be seen here: https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/blob/5523332b2cd3a14b4f4a4aa2b215465499ea3617/docker/Dockerfile_centos_6#L85

That environment can be pulled using docker at: registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_6:0.0.72

@flavorjones
Copy link
Member Author

@twk3 I've created #2081 to investigate the Centos6 issue you reported above. Thanks again!

@flavorjones
Copy link
Member Author

@odlp and others running Darwin 18 or earlier: please jump over to #2079 where I've got a gem file I need tested. Thanks!

@bruno-
Copy link

bruno- commented Sep 27, 2020

Any chance you can release a 1.11 rc version that works with ruby 3.0 preview?

I'm getting this when I try to use 1.11.0.rc3 on a project with ruby 3.0.0-preview1:

nokogiri-1.11.0.rc3-x86_64-darwin-19 requires ruby version < 2.8.dev, >= 2.4, which is incompatible with the current version, ruby 3.0.0p-1

@flavorjones
Copy link
Member Author

@bruno- I'll look into it this week.

@flavorjones
Copy link
Member Author

Any chance you can release a 1.11 rc version that works with ruby 3.0 preview?

I'm getting this when I try to use 1.11.0.rc3 on a project with ruby 3.0.0-preview1:

nokogiri-1.11.0.rc3-x86_64-darwin-19 requires ruby version < 2.8.dev, >= 2.4, which is incompatible with the current version, ruby 3.0.0p-1

@bruno- Please note that I've opened an upstream Bundler issue (rubygems/rubygems#4012) for this behavior. And I'm going to do my best to cut a native gem that's compatible with Ruby 3.0 in the next release candidate.

This was referenced Mar 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
packaging/native-gem release-blocker Blocking a milestone release
Projects
None yet
Development

No branches or pull requests