Broken ctypes dependency scanning on Kivy

[dogtopus]

+++ ONLY TEXT +++ DO NOT POST IMAGES +++

Description of the issue

kivy.clock contains a reference to libc.dylib which isn't ignored on Linux-based systems. Due to a recent change in Python, looking for libc via ctypes.util.find_library('libc') will cause an exception to be thrown and stops the build process. Add a try...except block around this find_library call works around the issue but there will still be a warning about it can't find libc.dylib.

Context information (for bug reports)

• Output of pyinstaller --version: 5.0.dev0 (7d77825)
• Version of Python: 3.9
• Platform: GNU/Linux
• Did you also try this on another platform? Does it work there?

No but should still work on Windows since PyInstaller excludes all *.dylib on that platform, and on OSX for obvious reasons.

• try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

• follow all the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

• start with clean installation
• use the latest development version (7d77825)
• Run your frozen program from a command window (shell) — instead of double-clicking on it
• Package your program in --onedir mode
• Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

Any kivy example program on Python 3.9 should be able to reproduce this issue.

Stacktrace / full error message

17603 INFO: Looking for ctypes DLLs
 Traceback (most recent call last):
17667 WARNING: stderr: Traceback (most recent call last):
   File "<REDACTED VENV PATH>/bin/pyinstaller", line 8, in <module>
17667 WARNING: stderr:   File "<REDACTED VENV PATH>/bin/pyinstaller", line 8, in <module>
     sys.exit(run())
17667 WARNING: stderr:     sys.exit(run())
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 114, in run
17667 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 114, in run
     run_build(pyi_config, spec_file, **vars(args))
17668 WARNING: stderr:     run_build(pyi_config, spec_file, **vars(args))
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 65, in run_build
17668 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 65, in run_build
     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
17668 WARNING: stderr:     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 725, in main
17668 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 725, in main
     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
17669 WARNING: stderr:     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 672, in build
17669 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 672, in build
     exec(code, spec_namespace)
17669 WARNING: stderr:     exec(code, spec_namespace)
   File "<REDACTED SPEC FILE>.spec", line 7, in <module>
17669 WARNING: stderr:   File "<REDACTED SPEC FILE>.spec", line 7, in <module>
     a = Analysis(['src/start.py'],
17669 WARNING: stderr:     a = Analysis(['src/start.py'],
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 242, in __init__
17669 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 242, in __init__
     self.__postinit__()
17670 WARNING: stderr:     self.__postinit__()
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
17670 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
     self.assemble()
17670 WARNING: stderr:     self.assemble()
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 438, in assemble
17670 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 438, in assemble
     ctypes_binaries = scan_code_for_ctypes(co)
17671 WARNING: stderr:     ctypes_binaries = scan_code_for_ctypes(co)
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 151, in scan_code_for_ctypes
17671 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 151, in scan_code_for_ctypes
     binaries = _resolveCtypesImports(binaries)
17671 WARNING: stderr:     binaries = _resolveCtypesImports(binaries)
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 308, in _resolveCtypesImports
17671 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 308, in _resolveCtypesImports
     cpath = find_library(os.path.splitext(cbin)[0])
17671 WARNING: stderr:     cpath = find_library(os.path.splitext(cbin)[0])
   File "/usr/lib/python3.9/ctypes/util.py", line 330, in find_library
17671 WARNING: stderr:   File "/usr/lib/python3.9/ctypes/util.py", line 330, in find_library
     _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
17672 WARNING: stderr:     _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
   File "/usr/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
17672 WARNING: stderr:   File "/usr/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
     if not _is_elf(file):
17672 WARNING: stderr:     if not _is_elf(file):
   File "/usr/lib/python3.9/ctypes/util.py", line 99, in _is_elf
17672 WARNING: stderr:   File "/usr/lib/python3.9/ctypes/util.py", line 99, in _is_elf
     with open(filename, 'br') as thefile:
17672 WARNING: stderr:     with open(filename, 'br') as thefile:
 FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'
17672 WARNING: stderr: FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'

Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs
for more about what would use to solve the issue.

[bwoodsend]

Hmm, I saw this with libc too. There's a bug in Python's ctypes itself which I've been meaning to report but I need to find a docker image it can be reproduced on because it doesn't happen on most Linuxes. It should silently return None and move on.

>>> from ctypes.util import find_library
>>> find_library("libc.so")
>>> find_library("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/brenainn/.pyenv/versions/3.9.1/lib/python3.9/ctypes/util.py", line 330, in find_library
    _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
  File "/home/brenainn/.pyenv/versions/3.9.1/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
    if not _is_elf(file):
  File "/home/brenainn/.pyenv/versions/3.9.1/lib/python3.9/ctypes/util.py", line 99, in _is_elf
    with open(filename, 'br') as thefile:
FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'

I've only been able to reproduce this on Manjaro linux. Can I have your full platform name? i.e. uname -a.

[rokm]

Hmm, on Fedora (python 3.9.2) calling find_library("libc") also returns None. And the python code involved does not seem to be patched in any way...

@bwoodsend, can you try running gcc -Wl,-t -o prog_tmp -llibc on your Majaro system and paste the console output (compilation should fail, but it's the output we're interested in)?

[bwoodsend]

brenainn@Ostrich-manjaro /e/n/pyinstaller (cross-compile-freebsd)> gcc -Wl,-t -o prog_tmp -llibc
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/Scrt1.o
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/crti.o
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/crtbeginS.o
/usr/bin/ld: cannot find -llibc
/usr/bin/ld: note to link with /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libc.a use -l:libc.a or rename it to liblibc.a
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libgcc.a
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libgcc_s.so
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libgcc_s.so.1
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libgcc.a
collect2: error: ld returned 1 exit status

[rokm]

Aha, so it's compiler-dependent. This diagnostic note:

/usr/bin/ld: note to link with /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libc.a use -l:libc.a or rename it to liblibc.a

is being matched by regex in _findLib_gcc, and yields liblibc.a.

[rokm]

I think the best way to deal with this is to simply enclose the find_library call in try/catch block, as the OP suggested. This will yield warning about library libc.dylib required via ctypes not found, but that's the current behavior on other linux distributions anyway...

We could also perform some additional checks on the library names and skip incompatible ones for particular OS (e.g., .dylib suffix is valid only on macOS, .dll suffix only on Windows; and on linux, there should be no suffix at all).

[bwoodsend]

I think the best way to deal with this is to simply enclose the find_library call in try/catch block, as the OP suggested.

Yeah, I'll do this now.

and on linux, there should be no suffix at all).

Except on Alpine. For some reason the suffix is mandatory there:

brenainn@Ostrich-manjaro /e/n/pyinstaller $ docker run -it python:3.9.0-alpine 
Python 3.9.0 (default, Nov 25 2020, 02:36:55) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.CDLL("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Error loading shared library libc: No such file or directory
>>> ctypes.CDLL("libc.so")
<CDLL 'libc.so', handle 7fc69d0a3b40 at 0x7fc69cb78a30>
>>> ctypes.CDLL("libm")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Error loading shared library libm: No such file or directory
>>> ctypes.CDLL("libm.so")
<CDLL 'libm.so', handle 7fc69d0a3b40 at 0x7fc69cb78a60>

[rokm]

Ah, right, our ctype-handling codepath is catering to both ctypes.CDLL() and find_library() behavior...