WIP: Cross compile for x86_64-darwin and arm64-darwin per rake-compiler-dock #2142

larskanis · 2020-12-22T08:33:54Z

This requires latest rake-compiler-dock from master branch.

What problem is this PR intended to solve?

Fat binary gems for MacOS are currently built on MacOS native. This complicates the build and release cycle. It is desirable to build all binary gems per cross compiler.

Related to #2079

Have you included adequate test coverage?

No CI, but I tested the resulting x86_64-darwin gem on MacOS-10.14.4 with both the system ruby-2.3.7 (after extending the required_ruby_version of nokogiri) and ruby-2.7.2 installed per rvm. The Mac is running on https://github.com/foxlet/macOS-Simple-KVM .

I'm unable to execute the arm64-darwin on any computer. I just did some static inspection of the built nokogiri.bundle file.

Does this change affect the behavior of either the C or the Java implementations?

C only.

codeclimate · 2020-12-22T08:38:22Z

ext/nokogiri/extconf.rb

@@ -590,6 +590,14 @@ def configure
            cflags = concat_flags(ENV["CFLAGS"], "-fPIC", "-g")
            execute "configure", ["env", "CHOST=#{host}", "CFLAGS=#{cflags}", "./configure", "--static", configure_prefix]
          end
+
+          def compile


Method compile has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

codeclimate · 2020-12-22T08:38:23Z

Code Climate has analyzed commit 713e723 and detected 1 issue on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1

The test coverage on the diff in this pull request is 100.0% (80% is the threshold).

This pull request will bring the total coverage in the repository to 94.4%.

View more on Code Climate.

flavorjones · 2020-12-23T04:45:30Z

@larskanis This is exciting! Thanks for doing this work. I'll spend some time tomorrow trying it out, and I'll ask some questions inline.

larskanis · 2020-12-23T09:25:36Z

I played a bit with universal libraries to bundle x86_64+arm64 code into one binary, but to no avail. IMHO these universal builds require a lot of additional work compared to a clear separation of CPU architectures.

flavorjones · 2020-12-23T22:00:30Z

I'm kicking the tires today and everything looks good, including the darwin x86_64 gem. I'd like to ship an rc4 in the next day or so so we can get feedback on the arm64 build in particular in time for a v1.11.0 final release by year's end.

@larskanis Are you planning to release rake-compiler-dock v1.1.0 soon so I can merge this?

larskanis · 2020-12-23T22:08:19Z

Are you planning to release rake-compiler-dock v1.1.0 soon so I can merge this?

Yes, I'm planning to release rake-compiler-dock-1.1.0 soon after ruby-3.0 and before end of the year.

flavorjones · 2020-12-23T22:30:56Z

I've pushed a few related commits. I'll likely keep rebasing this to master HEAD so that I can cut and test release candidates. Sorry for the inconvenience.

larskanis · 2020-12-28T22:22:05Z

IMHO rake-compiler-dock-1.1.0 is ready to be released. I switched to manylinux for the linux cross builds as sugested here. I'm not fully convinced about this change, but I think the advantages outweigh the disadvantages. Is this OK for you?

I would like to add the CrossRuby class of Nokogiri to rake-compiler-dock (in a somewhat more generic way). It helps to build and verify other gems similar to nokogiri. Not in 1.1.0, but in a later release.

flavorjones · 2020-12-28T23:14:53Z

@larskanis Thanks for asking - I think that's fine, I've built those images locally and am trying the builds out tonight.

If you'd like to re-use the CrossRuby class, I'd like to suggest changing the file format to be something more easily parseable (like YAML, for instance) because I end up reading it in a few different places (e.g., where I generate the CI pipelines). Maybe we can open up an issue for future discussion? I'm happy to help.

flavorjones · 2020-12-29T17:07:18Z

I used a rebased version of this branch to ship v1.11.0.rc4 just now, and things look like they're going OK! See #2149 (comment)

edit: also using rake-compiler-dock 9c6e35e and container images built at that commit

larskanis · 2020-12-29T17:38:55Z

I released rake-compiler-dock-1.1.0 and all the images some hours ago. I think this PR can be merged now.

flavorjones · 2020-12-30T18:07:23Z

OK, I've rebased onto master and fixed some of my commits; waiting for this to go green and I'll ship it.

flavorjones · 2020-12-30T19:25:25Z

@larskanis it looks like the move to centos may have introduced an issue running on musl-based systems, see cruby-native-gem-test-32bit #25.1 - Concourse. I need to do a bit more digging to be sure and reproduce locally.

larskanis · 2020-12-30T21:40:26Z

it looks like the move to centos may have introduced an issue running on musl-based systems

Oh, that's not good. And it reminds me that I should have done this musl test locally before release. It's too late for today, but I can check this tomorrow.

This requires latest rake-compiler-dock from master branch.

advapi32.dll is a standard Windows system DLL.

It is now based on manylinux2014 for increased platform compatibility.

because we were seeing these errors when building with 1.6: > error: Source option 6 is no longer supported. Use 7 or later. > error: Target option 6 is no longer supported. Use 7 or later. I think this is because JRuby has been updated in the rake-compiler-dock container since 1.0.0.

also delete unneeded script to set up darwin rake-compiler env

e.g., `rake concourse:task`

This will attempt to override the RbConfig flags that would ask for share libraries to be stripped.

flavorjones · 2020-12-31T04:41:05Z

@larskanis I've pushed an additional commit that makes extconf.rb support a --prevent-strip argument so that I could get see what's going on in the debugger.

After running the test suite about a dozen times and examining the stack walkback for each, the crashes are happening while libxml2 is trying to call either isnan or isinf (from xpath.c functions xmlXPathIsNaN or xmlXPathIsInf).

Looking at the shared object generated in the centos container, we can see these end up being method calls:

bash-5.0# nm /usr/local/bundle/gems/nokogiri-1.11.0.rc4-x86_64-linux/lib/nokogiri/3.0/nokogiri.so | egrep "isnan|isinf"
                 U __isinf@@GLIBC_2.2.5
                 U __isnan@@GLIBC_2.2.5

but in a shared object generated in a debian-based container, there are no such symbols, undefined or other. I think this is because it's defined as a macro on musl and debian?

I feel like we could work around this if necessary by providing local versions, but I'm curious if you can see an easier way to resolve this issue.

This patch is a result of rake-compiler-dock using centos 7 (manylinux2014) to cross-compile. Centos, for reasons I have not been able to discern, implements `isnan` and `isinf` as a function and not as a macro. Debian knows how to resolve that function at dynamic-link time (despite using a macro at compile time), but musl-based systems (like alpine) do not. Running `nm` on nokogiri.so created on such a centos system shows: ``` U __isinf@@GLIBC_2.2.5 U __isnan@@GLIBC_2.2.5 ``` (see sparklemotion#2142 for more info) This patch avoids using glibc's `isnan` and `isinf` calls, instead using libxml2's fallback implementation. There's history here, see libxml2 commit 8813f39: commit 8813f39 Author: Nick Wellnhofer <wellnhofer@aevum.de> Date: 2017-09-21 00:11:26 +0200 Simplify XPath NaN, inf and -0 handling Use C99 macros NAN, INFINITY, isnan, isinf. If they're not available: - Assume that (0.0 / 0.0) generates a NaN and !(x == x) tests for NaN. - Use C89's HUGE_VAL for INFINITY. Remove manual handling of NaN, infinity and negative zero in functions xmlXPathValueFlipSign and xmlXPathDivValues. Remove xmlXPathGetSign. All the tests for negative zero can be replaced with a test for negative or positive zero. Simplify xmlXPathRoundFunction. Remove Trio dependency. This should work on IEEE 754 compliant implementations even if the C99 macros aren't available, but will likely break some ancient platforms. If problems arise, my plan is to port the relevant trionan.c solution to xpath.c. Note that non-compliant implementations are impossible to fully support, anyway, since XPath requires IEEE 754. This patch would be unnecessary if any of the following was true: * centos implements these as macros, and doesn't generate an unresolved symbol for either in the shared library * we had a way to ensure `__isinf` and `__isnan` resolve on musl (e.g., we implement them locally)

flavorjones · 2020-12-31T16:35:45Z

I've pushed a commit that patches libxml2 to avoid calling isinf or isnan which will let me ship this release. I'd be happy to figure out a more permanent fix, but let's not block on it. I'll open a rake-compiler-dock issue for this.

Edit: that upstream issue is rake-compiler/rake-compiler-dock#42

flavorjones · 2020-12-31T17:48:11Z

Merged! Thanks again, @larskanis!

codeclimate bot reviewed Dec 22, 2020

View reviewed changes

larskanis changed the title ~~WIP: Cross compile for x86_64-darwin per rake-compiler-dock~~ WIP: Cross compile for x86_64-darwin and arm64-darwin per rake-compiler-dock Dec 22, 2020

flavorjones added this to the v1.11.0 milestone Dec 23, 2020

flavorjones added the packaging/native-gem label Dec 23, 2020

flavorjones force-pushed the cross-darwin branch from 772bd42 to 80010ed Compare December 23, 2020 22:29

flavorjones added the release-blocker Blocking a milestone release label Dec 23, 2020

flavorjones mentioned this pull request Dec 23, 2020

Native gems: create releases for additional darwin platforms #2079

Closed

flavorjones force-pushed the cross-darwin branch from 3a4327c to 046147b Compare December 30, 2020 18:06

larskanis and others added 8 commits December 30, 2020 20:35

Cross compile for x86_64-darwin per rake-compiler-dock

4d1fb5f

This requires latest rake-compiler-dock from master branch.

Add support for arm64-darwin platform

dff6249

Windows: Newer gcc versions with static libssp need advapi32.dll

04352f2

advapi32.dll is a standard Windows system DLL.

Adjust cross-tool for latest changes to rake-compiler-dock

94b0dd3

It is now based on manylinux2014 for increased platform compatibility.

test: update test-gem-file-contents to look for darwin bundles

0278fe3

dev: add darwin gems to build-gems; test-gem-set for batch testing

01578f5

also delete unneeded script to set up darwin rake-compiler env

test: test-gem-file-contents can be run in a temp container

3036da9

e.g., `rake concourse:task`

flavorjones force-pushed the cross-darwin branch from b827a1d to 3036da9 Compare December 31, 2020 01:36

feat: extconf.rb supports --prevent-strip

53b333f

This will attempt to override the RbConfig flags that would ask for share libraries to be stripped.

flavorjones mentioned this pull request Dec 31, 2020

manylinux2014/centos cross-compilation relying on symbols not present on musl systems rake-compiler/rake-compiler-dock#42

Closed

flavorjones merged commit 50b23c1 into sparklemotion:master Dec 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Cross compile for x86_64-darwin and arm64-darwin per rake-compiler-dock #2142

WIP: Cross compile for x86_64-darwin and arm64-darwin per rake-compiler-dock #2142

larskanis commented Dec 22, 2020 •

edited

codeclimate bot Dec 22, 2020

codeclimate bot commented Dec 22, 2020 •

edited

flavorjones commented Dec 23, 2020

larskanis commented Dec 23, 2020 •

edited

flavorjones commented Dec 23, 2020

larskanis commented Dec 23, 2020

flavorjones commented Dec 23, 2020

larskanis commented Dec 28, 2020

flavorjones commented Dec 28, 2020

flavorjones commented Dec 29, 2020 •

edited

larskanis commented Dec 29, 2020

flavorjones commented Dec 30, 2020

flavorjones commented Dec 30, 2020

larskanis commented Dec 30, 2020

flavorjones commented Dec 31, 2020

flavorjones commented Dec 31, 2020 •

edited

flavorjones commented Dec 31, 2020

WIP: Cross compile for x86_64-darwin and arm64-darwin per rake-compiler-dock #2142

WIP: Cross compile for x86_64-darwin and arm64-darwin per rake-compiler-dock #2142

Conversation

larskanis commented Dec 22, 2020 • edited

codeclimate bot Dec 22, 2020

Choose a reason for hiding this comment

codeclimate bot commented Dec 22, 2020 • edited

flavorjones commented Dec 23, 2020

larskanis commented Dec 23, 2020 • edited

flavorjones commented Dec 23, 2020

larskanis commented Dec 23, 2020

flavorjones commented Dec 23, 2020

larskanis commented Dec 28, 2020

flavorjones commented Dec 28, 2020

flavorjones commented Dec 29, 2020 • edited

larskanis commented Dec 29, 2020

flavorjones commented Dec 30, 2020

flavorjones commented Dec 30, 2020

larskanis commented Dec 30, 2020

flavorjones commented Dec 31, 2020

flavorjones commented Dec 31, 2020 • edited

flavorjones commented Dec 31, 2020

larskanis commented Dec 22, 2020 •

edited

codeclimate bot commented Dec 22, 2020 •

edited

larskanis commented Dec 23, 2020 •

edited

flavorjones commented Dec 29, 2020 •

edited

flavorjones commented Dec 31, 2020 •

edited