Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] v1.11.0 native Windows gem causes nokogumbo installation to fail #2167

Closed
flavorjones opened this issue Jan 4, 2021 · 8 comments · Fixed by #2202
Closed

[bug] v1.11.0 native Windows gem causes nokogumbo installation to fail #2167

flavorjones opened this issue Jan 4, 2021 · 8 comments · Fixed by #2202

Comments

@flavorjones
Copy link
Member

flavorjones commented Jan 4, 2021

Please describe the bug

Nokogumbo cannot build against the Nokogiri v1.11.0 native Windows gem.

It is able to build against

  • the v1.10.10 native Windows gem
  • the v1.11.0 native Linux and Darwin gems.

Help us reproduce what you're seeing

See the Nokogumbo test suite's failing windows jobs when built against v1.11.0:

MAKE='make -j2 V=1' bundle exec rake -- --with-libxml2
mkdir -p tmp/x64-mingw32/nokogumbo/2.7.2
cd tmp/x64-mingw32/nokogumbo/2.7.2
C:/hostedtoolcache/windows/Ruby/2.7.2/x64/bin/ruby.exe -I. ../../../../ext/nokogumbo/extconf.rb -- --with-libxml2
checking for whether -ID:/a/nokogumbo/nokogumbo/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.0-x64-mingw32/ext/nokogiri is accepted as CFLAGS... yes
checking for whether -ID:/a/nokogumbo/nokogumbo/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.0-x64-mingw32/ext/nokogiri/include is accepted as CFLAGS... yes
checking for whether -ID:/a/nokogumbo/nokogumbo/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.0-x64-mingw32/ext/nokogiri/include/libxml2 is accepted as CFLAGS... yes
checking for libxml/tree.h... yes
checking for nokogiri.h... yes
creating Makefile
cd -
cd tmp/x64-mingw32/nokogumbo/2.7.2
make -j2 V=1
(echo EXPORTS && echo Init_nokogumbo) > nokogumbo-x64-mingw32.def
...
x86_64-w64-mingw32-gcc -shared -o nokogumbo.so nokogumbo.o ascii.o attribute.o char_ref.o error.o foreign_attrs.o parser.o string_buffer.o string_piece.o svg_attrs.o svg_tags.o tag.o tag_lookup.o tokenizer.o token_buffer.o utf8.o util.o vector.o -L. -L/C/hostedtoolcache/windows/Ruby/2.7.2/x64/lib -L. -pipe -s -fstack-protector-strong -pipe -s -fstack-protector-strong -Wl,--enable-auto-image-base,--enable-auto-import nokogumbo-x64-mingw32.def    -lx64-msvcrt-ruby270  -lshell32 -lws2_32 -liphlpapi -limagehlp -lshlwapi  
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x34): undefined reference to `xmlFreeDoc'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x10a8): undefined reference to `xmlNewDocText'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x10c4): undefined reference to `xmlAddChild'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x10dd): undefined reference to `xmlNewDocNode'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x111e): undefined reference to `xmlAddChild'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1153): undefined reference to `xmlNewNsProp'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1191): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x11af): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x11c8): undefined reference to `xmlNewDocComment'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x120d): undefined reference to `xmlNewCDataBlock'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x123e): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1260): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x127e): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x12a0): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x12e6): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x12f6): undefined reference to `xmlSetNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x130e): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x132c): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1352): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1440): undefined reference to `htmlNewDocNoDtD'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1462): undefined reference to `xmlCreateIntSubset'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1484): undefined reference to `Nokogiri_wrap_xml_document'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x14c5): undefined reference to `htmlNewDocNoDtD'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.rdata$.refptr.mNokogiri[.refptr.mNokogiri]+0x0): undefined reference to `mNokogiri'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.rdata$.refptr.cNokogiriXmlSyntaxError[.refptr.cNokogiriXmlSyntaxError]+0x0): undefined reference to `cNokogiriXmlSyntaxError'
collect2.exe: error: ld returned 1 exit status
make: *** [Makefile:261: nokogumbo.so] Error 1

And the same suite's passing status when built against v1.10.10.

Expected behavior

Nokogumbo should be able to build identically against v1.11.0 as it does against v1.10.10.

Environment

From Nokogumbo's ci.yml config:

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu, macos, windows]
        ruby: [2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7]
        system_libraries: [true, false]
        exclude:
        - os: macos
          ruby: 2.1
        - os: macos
          ruby: 2.2
        - os: macos
          ruby: 2.3
        - os: windows
          ruby: 2.1
        - os: windows
          ruby: 2.2
        - os: windows
          ruby: 2.3
    runs-on: ${{ matrix.os }}-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: ${{ matrix.ruby }}
        bundler-cache: true
        
    - name: install html5lib tests
      run: git clone --depth 1 --branch all-error-fixes --single-branch https://github.com/stevecheckoway/html5lib-tests.git test/html5lib-tests

    - name: Run tests
      env:
        USE_SYSTEM_LIBRARIES: ${{ matrix.system_libraries }}
      shell: bash
      run: ./scripts/ci-test.sh

Additional context

Note that Linux and Darwin native gems aren't a problem.

@flavorjones
Copy link
Member Author

@larskanis I'm not sure if you have any ideas here. I probably won't have time to look into it for another day or two.

@flavorjones
Copy link
Member Author

Goal for today is to set up a Windows virtual machine so I can bisect when this started happening.

@larskanis
Copy link
Member

From first reading I would say that the linker command is missing -lnokogiri and possibly libxml and libxslt. Windows doesn't allow missing references at build time, while Linux and Macos do.

@flavorjones
Copy link
Member Author

flavorjones commented Feb 26, 2021

I think this is related to rubys/nokogumbo#165 and might be fixed by rubys/nokogumbo#166. We'll see if the Windows builds are all green and I'll come back here with more info.

@flavorjones
Copy link
Member Author

Still failing. I have a Windows dev env set up now, will investigate this weekend.

@flavorjones
Copy link
Member Author

@larskanis I think I need help with this. I can easily reproduce on a Windows dev environment running:

Output matches what's seen at https://github.com/rubys/nokogumbo/pull/163/checks?check_run_id=1990881540

I've tried adding (in nokogumbo's extconf.rb) ldflags of:

  • -LC:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/nokogiri-1.11.1-x64-mingw32/ext/nokogiri/3.0
  • and -l:nokogiri.so

and the mkmf.log output indicates that the library is found; but I still get these unresolved references.

Anything else you can think of to try?

@larskanis
Copy link
Member

No idea for now. I'll try to dig deeper tomorrow.

@flavorjones
Copy link
Member Author

OK, I think I've got this figured out. It's a combination of:

  • libxml2 and nokogiri not properly exporting symbols (note that libxslt/libexslt do this properly, though!)
  • ensuring that nokogumbo has linker flags to link against the installed nokogiri.so

The first can be solved by passing -DIN_LIBXML -ULIBXML_STATIC as cflags in the libxml2 recipe (in Nokogiri's extconf.rb) and by properly exporting things in nokogiri.h.

The second needs some more work to figure out, but for now the combination of -LC:/path/to/installation/dir/lib/nokogiri/3.0 and -l:nokogiri.so seems to be working.

flavorjones added a commit that referenced this issue Mar 7, 2021
to allow downstream gems like nokogumbo to build correctly simply by
requiring nokogiri in the extconf and running:

  append_cflags(Nokogiri::VERSION_INFO["nokogiri"]["cppflags"])
  append_ldflags(Nokogiri::VERSION_INFO["nokogiri"]["ldflags"])

this is needed because unresolved symbols aren't allowed on windows.

Related to #2167
flavorjones added a commit that referenced this issue Mar 8, 2021
…g-on-windows

fix nokogumbo linking on windows

---

**What problem is this PR intended to solve?**

Closes #2167 which relates to an issue building Nokogumbo on Windows against the precompiled libraries.

This PR does the following:
- ensures that libxml2 symbols are exported when built on windows
- ensures that nokogiri symbols are exported when built on windows
- includes `LDFLAGS` in `Nokogiri::VERSION_INFO` to allow the windows linker to resolve all symbols

**Have you included adequate test coverage?**

There's pretty good test coverage of this test case in the Nokogumbo CI suite; and I've added some light tests here in `scripts/test-gem-installation` though I'll note that we're not testing gem installation on Windows in the Nokogiri CI suite at the moment (though, see #2153).

**Does this change affect the behavior of either the C or the Java implementations?**

This should only impact downstream gems like Nokogumbo who are trying to link against the precompiled Nokogiri libraries.
This was referenced Mar 18, 2021
flavorjones added a commit that referenced this issue Apr 7, 2021
This is necessary on Windows where unresolved symbols aren't
allowed. We also limit compatibility with Nokogiri's precompiled libraries
to Nokogiri >= 1.11.2 on Windows for this reason.

Related to:
- #2145
- #2167
- #2202
flavorjones added a commit that referenced this issue Nov 16, 2022
Also modify test-nokogumbo-compatibility to skip 2.0.4 on Windows
because it's missing the LDFLAGS fix from #2167, #2202, and
nokogumbo#163.
flavorjones added a commit that referenced this issue Nov 16, 2022
Also modify test-nokogumbo-compatibility to skip 2.0.4 on Windows
because it's missing the LDFLAGS fix from #2167, #2202, and
nokogumbo#163.
flavorjones added a commit that referenced this issue Nov 17, 2022
Also modify test-nokogumbo-compatibility to skip 2.0.4 on Windows
because it's missing the LDFLAGS fix from #2167, #2202, and
nokogumbo#163.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants