Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

musllinux tests fail on s390x #472

Closed
hroncok opened this issue Oct 26, 2021 · 4 comments
Closed

musllinux tests fail on s390x #472

hroncok opened this issue Oct 26, 2021 · 4 comments
Labels
bug packaging.tags

Comments

@hroncok
Copy link
Contributor

hroncok commented Oct 26, 2021

On both 21.0 and the current main branch, the musllinux tests fail on s390x, possibly because it is Big Endian.

Steps to reproduce:

$ podman run --rm -ti docker://fedorapython/fedora-python-tox:s390x /usr/bin/bash 
[root@a3025a1b5b30 /]# git clone https://github.com/pypa/packaging/
...
[root@a3025a1b5b30 /]# cd packaging/
[root@a3025a1b5b30 packaging]# pip install nox
...
[root@a3025a1b5b30 packaging]# nox -e tests-3.9 -- tests/test_musllinux.py
nox > Running session tests-3.9
nox > Re-using existing virtual environment at .nox/tests-3-9.
nox > python -m pip install coverage<5.0.0 pretend pytest>=6.2.0 pip>=9.0.2
nox > python -m coverage run --source packaging/ -m pytest --strict-markers tests/test_musllinux.py
============================= test session starts ==============================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /packaging
collected 20 items                                                             

tests/test_musllinux.py ......FFF.....F..FFF                             [100%]

=================================== FAILURES ===================================
_____________________ test_parse_ld_musl_from_elf[x86_64] ______________________

executable = PosixPath('/packaging/tests/musllinux/musl-x86_64')
location = '/lib/ld-musl-x86_64.so.1'

    @pytest.mark.parametrize(
        "executable, location",
        [
            (BIN_GLIBC_X86_64, None),
            (BIN_MUSL_X86_64, LD_MUSL_X86_64),
            (BIN_MUSL_I386, LD_MUSL_I386),
            (BIN_MUSL_AARCH64, LD_MUSL_AARCH64),
        ],
        ids=["glibc", "x86_64", "i386", "aarch64"],
    )
    def test_parse_ld_musl_from_elf(executable, location):
        with executable.open("rb") as f:
>           assert _parse_ld_musl_from_elf(f) == location
E           AssertionError: assert None == '/lib/ld-musl-x86_64.so.1'
E            +  where None = _parse_ld_musl_from_elf(<_io.BufferedReader name='/packaging/tests/musllinux/musl-x86_64'>)

tests/test_musllinux.py:69: AssertionError
______________________ test_parse_ld_musl_from_elf[i386] _______________________

executable = PosixPath('/packaging/tests/musllinux/musl-i386')
location = '/lib/ld-musl-i386.so.1'

    @pytest.mark.parametrize(
        "executable, location",
        [
            (BIN_GLIBC_X86_64, None),
            (BIN_MUSL_X86_64, LD_MUSL_X86_64),
            (BIN_MUSL_I386, LD_MUSL_I386),
            (BIN_MUSL_AARCH64, LD_MUSL_AARCH64),
        ],
        ids=["glibc", "x86_64", "i386", "aarch64"],
    )
    def test_parse_ld_musl_from_elf(executable, location):
        with executable.open("rb") as f:
>           assert _parse_ld_musl_from_elf(f) == location
E           AssertionError: assert None == '/lib/ld-musl-i386.so.1'
E            +  where None = _parse_ld_musl_from_elf(<_io.BufferedReader name='/packaging/tests/musllinux/musl-i386'>)

tests/test_musllinux.py:69: AssertionError
_____________________ test_parse_ld_musl_from_elf[aarch64] _____________________

executable = PosixPath('/packaging/tests/musllinux/musl-aarch64')
location = '/lib/ld-musl-aarch64.so.1'

    @pytest.mark.parametrize(
        "executable, location",
        [
            (BIN_GLIBC_X86_64, None),
            (BIN_MUSL_X86_64, LD_MUSL_X86_64),
            (BIN_MUSL_I386, LD_MUSL_I386),
            (BIN_MUSL_AARCH64, LD_MUSL_AARCH64),
        ],
        ids=["glibc", "x86_64", "i386", "aarch64"],
    )
    def test_parse_ld_musl_from_elf(executable, location):
        with executable.open("rb") as f:
>           assert _parse_ld_musl_from_elf(f) == location
E           AssertionError: assert None == '/lib/ld-musl-aarch64.so.1'
E            +  where None = _parse_ld_musl_from_elf(<_io.BufferedReader name='/packaging/tests/musllinux/musl-aarch64'>)

tests/test_musllinux.py:69: AssertionError
______________ test_parse_ld_musl_from_elf_no_interpreter_section ______________

    def test_parse_ld_musl_from_elf_no_interpreter_section():
        with BIN_MUSL_X86_64.open("rb") as f:
            data = f.read()
    
        # Change all sections to *not* PT_INTERP.
        unpacked = struct.unpack("16BHHIQQQIHHH", data[:58])
        *_, e_phoff, _, _, _, e_phentsize, e_phnum = unpacked
        for i in range(e_phnum + 1):
            sb = e_phoff + e_phentsize * i
            se = sb + 56
>           section = struct.unpack("IIQQQQQQ", data[sb:se])
E           struct.error: unpack requires a buffer of 56 bytes

tests/test_musllinux.py:110: error
________________________ test_get_musl_version[x86_64] _________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x4003182d90>
executable = PosixPath('/packaging/tests/musllinux/musl-x86_64')
output = 'musl libc (x86_64)\nVersion 1.2.2\n'
version = _MuslVersion(major=1, minor=2), ld_musl = '/lib/ld-musl-x86_64.so.1'

    @pytest.mark.parametrize(
        "executable, output, version, ld_musl",
        [
            (MUSL_DIR.joinpath("does-not-exist"), "error", None, None),
            (BIN_GLIBC_X86_64, "error", None, None),
            (BIN_MUSL_X86_64, MUSL_AMD64, _MuslVersion(1, 2), LD_MUSL_X86_64),
            (BIN_MUSL_I386, MUSL_I386, _MuslVersion(1, 2), LD_MUSL_I386),
            (BIN_MUSL_AARCH64, MUSL_AARCH64, _MuslVersion(1, 1), LD_MUSL_AARCH64),
        ],
        ids=["does-not-exist", "glibc", "x86_64", "i386", "aarch64"],
    )
    def test_get_musl_version(monkeypatch, executable, output, version, ld_musl):
        def mock_run(*args, **kwargs):
            return collections.namedtuple("Proc", "stderr")(output)
    
        run_recorder = pretend.call_recorder(mock_run)
        monkeypatch.setattr(_musllinux.subprocess, "run", run_recorder)
    
>       assert _get_musl_version(str(executable)) == version
E       AssertionError: assert None == _MuslVersion(major=1, minor=2)
E        +  where None = _get_musl_version('/packaging/tests/musllinux/musl-x86_64')
E        +    where '/packaging/tests/musllinux/musl-x86_64' = str(PosixPath('/packaging/tests/musllinux/musl-x86_64'))

tests/test_musllinux.py:134: AssertionError
_________________________ test_get_musl_version[i386] __________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x400314a3a0>
executable = PosixPath('/packaging/tests/musllinux/musl-i386')
output = 'musl libc (i386)\nVersion 1.2.1\n'
version = _MuslVersion(major=1, minor=2), ld_musl = '/lib/ld-musl-i386.so.1'

    @pytest.mark.parametrize(
        "executable, output, version, ld_musl",
        [
            (MUSL_DIR.joinpath("does-not-exist"), "error", None, None),
            (BIN_GLIBC_X86_64, "error", None, None),
            (BIN_MUSL_X86_64, MUSL_AMD64, _MuslVersion(1, 2), LD_MUSL_X86_64),
            (BIN_MUSL_I386, MUSL_I386, _MuslVersion(1, 2), LD_MUSL_I386),
            (BIN_MUSL_AARCH64, MUSL_AARCH64, _MuslVersion(1, 1), LD_MUSL_AARCH64),
        ],
        ids=["does-not-exist", "glibc", "x86_64", "i386", "aarch64"],
    )
    def test_get_musl_version(monkeypatch, executable, output, version, ld_musl):
        def mock_run(*args, **kwargs):
            return collections.namedtuple("Proc", "stderr")(output)
    
        run_recorder = pretend.call_recorder(mock_run)
        monkeypatch.setattr(_musllinux.subprocess, "run", run_recorder)
    
>       assert _get_musl_version(str(executable)) == version
E       AssertionError: assert None == _MuslVersion(major=1, minor=2)
E        +  where None = _get_musl_version('/packaging/tests/musllinux/musl-i386')
E        +    where '/packaging/tests/musllinux/musl-i386' = str(PosixPath('/packaging/tests/musllinux/musl-i386'))

tests/test_musllinux.py:134: AssertionError
________________________ test_get_musl_version[aarch64] ________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x400317e130>
executable = PosixPath('/packaging/tests/musllinux/musl-aarch64')
output = 'musl libc (aarch64)\nVersion 1.1.24\n'
version = _MuslVersion(major=1, minor=1), ld_musl = '/lib/ld-musl-aarch64.so.1'

    @pytest.mark.parametrize(
        "executable, output, version, ld_musl",
        [
            (MUSL_DIR.joinpath("does-not-exist"), "error", None, None),
            (BIN_GLIBC_X86_64, "error", None, None),
            (BIN_MUSL_X86_64, MUSL_AMD64, _MuslVersion(1, 2), LD_MUSL_X86_64),
            (BIN_MUSL_I386, MUSL_I386, _MuslVersion(1, 2), LD_MUSL_I386),
            (BIN_MUSL_AARCH64, MUSL_AARCH64, _MuslVersion(1, 1), LD_MUSL_AARCH64),
        ],
        ids=["does-not-exist", "glibc", "x86_64", "i386", "aarch64"],
    )
    def test_get_musl_version(monkeypatch, executable, output, version, ld_musl):
        def mock_run(*args, **kwargs):
            return collections.namedtuple("Proc", "stderr")(output)
    
        run_recorder = pretend.call_recorder(mock_run)
        monkeypatch.setattr(_musllinux.subprocess, "run", run_recorder)
    
>       assert _get_musl_version(str(executable)) == version
E       AssertionError: assert None == _MuslVersion(major=1, minor=1)
E        +  where None = _get_musl_version('/packaging/tests/musllinux/musl-aarch64')
E        +    where '/packaging/tests/musllinux/musl-aarch64' = str(PosixPath('/packaging/tests/musllinux/musl-aarch64'))

tests/test_musllinux.py:134: AssertionError
=========================== short test summary info ============================
FAILED tests/test_musllinux.py::test_parse_ld_musl_from_elf[x86_64] - Asserti...
FAILED tests/test_musllinux.py::test_parse_ld_musl_from_elf[i386] - Assertion...
FAILED tests/test_musllinux.py::test_parse_ld_musl_from_elf[aarch64] - Assert...
FAILED tests/test_musllinux.py::test_parse_ld_musl_from_elf_no_interpreter_section
FAILED tests/test_musllinux.py::test_get_musl_version[x86_64] - AssertionErro...
FAILED tests/test_musllinux.py::test_get_musl_version[i386] - AssertionError:...
FAILED tests/test_musllinux.py::test_get_musl_version[aarch64] - AssertionErr...
========================= 7 failed, 13 passed in 0.80s =========================
nox > Command python -m coverage run --source packaging/ -m pytest --strict-markers tests/test_musllinux.py failed with exit code 1
nox > Session tests-3.9 failed.
hroncok added a commit to hroncok/packaging that referenced this issue Oct 26, 2021
@hroncok
musslinux: Always (un)pack structs as little endian
8a62bef 
Fixes problems on big endian architectures --
native order is used by default otherwise.

Fixes pypa#472
@hroncok hroncok mentioned this issue Oct 26, 2021
Closed
@hroncok
Copy link
Contributor Author

hroncok commented Oct 26, 2021

#473

@brettcannon
Copy link
Member

/cc @uranusjr

@glaubitz
Copy link

I tried this particular patch on s390x on SUSE Linux Enterprise 15-SP1, but the testsuite still fails for me.

[  333s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[  333s] 
[  333s] f = <_io.BufferedReader name='/usr/bin/python3'>
[  333s] 
[  333s]     def _parse_ld_musl_from_elf(f: IO[bytes]) -> Optional[str]:
[  333s]         """Detect musl libc location by parsing the Python executable.
[  333s]     
[  333s]         Based on: https://gist.github.com/lyssdod/f51579ae8d93c8657a5564aefc2ffbca
[  333s]         ELF header: https://refspecs.linuxfoundation.org/elf/gabi4+/ch4.eheader.html
[  333s]         """
[  333s]         f.seek(0)
[  333s]         try:
[  333s]             ident = _read_unpacked(f, "<16B")
[  333s]         except struct.error:
[  333s]             return None
[  333s]         if ident[:4] != tuple(b"\x7fELF"):  # Invalid magic, not ELF.
[  333s]             return None
[  333s]         f.seek(struct.calcsize("HHI"), 1)  # Skip file type, machine, and version.
[  333s]     
[  333s]         try:
[  333s]             # e_fmt: Format for program header.
[  333s]             # p_fmt: Format for section header.
[  333s]             # p_idx: Indexes to find p_type, p_offset, and p_filesz.
[  333s]             e_fmt, p_fmt, p_idx = {
[  333s]                 1: ("<IIIIHHH", "<IIIIIIII", (0, 1, 4)),  # 32-bit.
[  333s]                 2: ("<QQQIHHH", "<IIQQQQQQ", (0, 2, 5)),  # 64-bit.
[  333s]             }[ident[4]]
[  333s]         except KeyError:
[  333s]             return None
[  333s]         else:
[  333s]             p_get = operator.itemgetter(*p_idx)
[  333s]     
[  333s]         # Find the interpreter section and return its content.
[  333s]         try:
[  333s]             _, e_phoff, _, _, _, e_phentsize, e_phnum = _read_unpacked(f, e_fmt)
[  333s]         except struct.error:
[  333s]             return None
[  333s]         for i in range(e_phnum + 1):
[  333s] >           f.seek(e_phoff + e_phentsize * i)
[  333s] E           OSError: [Errno 22] Invalid argument
[  333s] 
[  333s] packaging/_musllinux.py:56: OSError
[  333s] _ TestManylinuxPlatform.test_linux_platforms_32_64bit_on_64bit_os[linux-x86_64-True-linux_i686] _
[  333s] 
[  333s] self = <tests.test_tags.TestManylinuxPlatform object at 0x3ff7a26c5c0>
[  333s] arch = 'linux-x86_64', is_32bit = True, expected = 'linux_i686'
[  333s] monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x3ff7a26cf98>
[  333s] 
[  333s]     @pytest.mark.parametrize(
[  333s]         "arch,is_32bit,expected",
[  333s]         [
[  333s]             ("linux-x86_64", False, "linux_x86_64"),
[  333s]             ("linux-x86_64", True, "linux_i686"),
[  333s]             ("linux-aarch64", False, "linux_aarch64"),
[  333s]             ("linux-aarch64", True, "linux_armv7l"),
[  333s]         ],
[  333s]     )
[  333s]     def test_linux_platforms_32_64bit_on_64bit_os(
[  333s]         self, arch, is_32bit, expected, monkeypatch
[  333s]     ):
[  333s]         monkeypatch.setattr(sysconfig, "get_platform", lambda: arch)
[  333s]         monkeypatch.setattr(os, "confstr", lambda x: "glibc 2.20", raising=False)
[  333s]         monkeypatch.setattr(tags._manylinux, "_is_compatible", lambda *args: False)
[  333s] >       linux_platform = list(tags._linux_platforms(is_32bit=is_32bit))[-1]
[  333s] 
[  333s] tests/test_tags.py:341: 
[  333s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

@hroncok
Copy link
Contributor Author

hroncok commented Aug 23, 2022

I can confirm this has been fixed by #538

All tests pass on both x86_64 and s390x.

Thank you!

@hroncok hroncok closed this as completed Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug packaging.tags
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

musslinux: Always (un)pack structs as little endian hroncok/packaging
3 participants
@brettcannon @glaubitz @hroncok