Pyinstaller can't find dynamically linked libs for Mac OS X Big Sur Beta #5107
Comments
Can you clarify what you mean here. Are you saying I also notice all those missing libraries in your build log are all (I think) system ones. Is this only applicable to macOS's system libraries or does your test program just not happen to use any others? |
|
That is true all the files are system ones. In the new beta, they moved them and made them hidden in the directory structure. All we have to do is use the command dlopen() which will check the lib for it in the cache. Here is the os.path call: |
|
Can you also check the other commands? Actively loading each dylib is not something we want to do unless we have no choice... ctypes.utils.find_library("libSystem.B.dylib")
shutil.which("libSystem.B.dylib")And just to confirm that it does work: ctypes.CDLL("libSystem.B.dylib")._name |
|
I can confirm that the first two return python/cpython#21241 addresses the issue with |
|
Well as I'm still not clear on the impact of this. Is this specific to built-in system libraries? And if so, do we need to include them? On Windows it's generally a good idea to bring system libraries with you as they often turn out not to be standard - how stable is macOS's system libraries? |
I think it is planned to fully support Big Sur in a future Python 3.8 bugfix release. This still leaves Python 3.7 and lower without the fixes.
This is specific to system libraries only and has been the default in iOS for some time. In my experience, macOS libraries are sufficiently stable so that the don't need to be included. But my experience is limited... |
|
I suppose the thing to do would be to run the test suite on a Big Sur. Would you be able to do this? |
|
Done. The output looks pretty grim but many of the failures seem to be related to not finding the system's Tk library and therefore are directly caused by this issue. For reference, I am using Python 3.8 installed with homebrew. Attached is a (large) file with the test output. |
|
Great, thanks for that. Can I ask which version of PyInstaller you used? I'm guessing you cloned the development branch? But it looks like they're all failing in the same place. Namely that |
|
I tried both the dev branch and the fix from #5013 with similar results. Still, feel free to @mention me in the PR. Actually attempting to load the libraries instead of using pyinstaller_patched_test_macOS_11.txt I know that this is something which you would rather avoid. Nevertheless, the patch goes as follows: diff --git a/PyInstaller/depend/bindepend.py b/PyInstaller/depend/bindepend.py
index 95660b5dcef0f7167bf62007422233daefb758a7..854947b364b4242f2bea89ee6cbd9cd7c382879a 100644
--- a/PyInstaller/depend/bindepend.py
+++ b/PyInstaller/depend/bindepend.py
@@ -21,6 +21,7 @@ from glob import glob
# Required for extracting eggs.
import zipfile
import collections
+from ctypes import cdll
from .. import compat
from ..compat import (is_win, is_win_10, is_unix,
@@ -692,10 +693,17 @@ def _getImports_macholib(pth):
if not os.path.isabs(run_path):
run_path = os.path.join(exec_path, run_path)
# Stop looking for lib when found in first location.
- if os.path.exists(os.path.join(run_path, lib)):
- final_lib = os.path.abspath(os.path.join(run_path, lib))
+
+ try:
+ candidate = os.path.exists(os.path.join(run_path, lib))
+ cdll.LoadLibrary(candidate)
+ except OSError:
+ pass
+ else:
+ final_lib = candidate
rslt.add(final_lib)
break
+
# Log error if no existing file found.
if not final_lib:
logger.error('Can not find path %s (needed by %s)', lib, pth)
@@ -707,11 +715,13 @@ def _getImports_macholib(pth):
# It is also replaced by 'exec_path'.
if lib.startswith('@loader_path'):
lib = lib.replace('@loader_path', '@executable_path')
+
try:
- lib = dyld_find(lib, executable_path=exec_path)
- rslt.add(lib)
- except ValueError:
+ cdll.LoadLibrary(lib)
+ except OSError:
logger.error('Can not find path %s (needed by %s)', lib, pth)
+ else:
+ rslt.add(lib)
return rslt
|
|
Ah, damn. There was a typo in the patch. It should be The correct patch: And the test results: pyinstaller_patched_test_2_macOS_11.txt Already a bit better. I will leave you to figure our the remaining failures but I'm happy to run tests for future fixes if helpful. |
|
Thanks @samschott, that's really helpful. That gives us a pretty good idea of what we need to do - although I don't feel any better about doing it. |
|
Actually one thing - surely + else:
+ rslt.add(lib)would only add the library's name rather than its path? In which case they would not end up in the final build. Although, provided macOS is stable, we shouldn't actually need them. |
|
Fair point. Finding the actual path however will be more difficult. Personally, I am using the following workaround for system libraries: def find_library(name):
paths = [
f'/System/Library/Frameworks/{name}.framework/{name}',
f'/usr/lib/lib{name}.dylib',
f'{name}.dylib',
]
for path in paths:
try:
cdll.LoadLibrary(path)
return path
except OSError:
pass
return Nonebut it's probably better to wait for the official fix of |
BoboTiG
added
@high
bug
platform:OS X
pull-request wanted
This patch works for me, thanks a lot for an early workaround 👍 |
|
I have the same issue. I'm using Mac OS Big Sur and python 3.9 with 4.1 version of pyinstaller. [4065] Error loading Python lib '/var/folders/ds/p4w3_2vd1mz4fxg7wrcbjsdr0000gn/T/_MEIAejivL/Python': dlopen: dlopen(/var/folders/ds/p4w3_2vd1mz4fxg7wrcbjsdr0000gn/T/_MEIAejivL/Python, 10): no suitable image found. How this problem can be fixed? I have already tried to sign my app, and it's working only when I choose "-D". But I need to use "-F". I can't use pyinstaller 4.0 cause it doesn't support python 3.8 or 3.9, and Big Sur doesn't support python 3.7. What to do? I need my app to work... |
|
@sergei2009op note that you probably would not want to build on Big Sur anyway if you want your users on older macs to be able to run the binary. see
|
Thank you for your answer. But I need to run my app on Big Sur in "onefile" mode without such errors. Do you know how it could be fixed? |
|
@samschott patch helped fixed a majority of my lib errors thank you! Only having a problem with libopenblas.0.dylib which I need for numpy
is the specific error. Strangely enough libopenblas.0.dylib is in the pyinstaller "binchache" folder that was created. I tried temporarily directing the path in bindepend to directly lead to it but still getting the same errors. |
|
any work in progress??meet the same problem |
|
It's not a cosmetic error for Python versions <3.9.1. I also tried with 3.7.8 and the resulting app wouldn't start. If the issue is with a Python library that PyInstaller users, we may need to wait for more minor Python updates. |
Your issue very likely has nothing to do with system libraries not being visible on Big Sur. This issue is dealing with error messages during application build, not at runtime - see the original report. |
What's the error message, then? Does it imply that the cause are hidden system libraries on Big Sur? |
Oh, you are right. My issue appears during runtime, not during build. (Build is on macOS 10.14 Mojave which is why I do not get the error message which this issue is about.) Thank you for taking the time to elaborate on your initial response. |
|
Why this issue is closed? This is still happening in Python 3.9.1 + PyInstaller 4.2 |
|
@cperezabo This issue was closed because the fix for it was merged in #5464. It will be included in the next PyInstaller release. |
|
Thanks @papr, I wasn't aware of that fix! |
|
Well.. I've been looking at the changes in that PR and it just silent the message, so maybe I'm having another issue, because since updated to Big Sur, I've not been able to build my app again, it just freezes and I have to force quit it. If I run it in CLI I have no errors so I don't know exactly how to debug it, any idea? |
|
@cperezabo Well, I suggest having a look at the other macOS Big Sur related issues, #5315 and #5491. Also, if you open a new bug report, there is a list of steps to follow in this kind of situation. I suggest following it and if the issue persists to file the issue. |
when will this new release come out? Running into this error on MacOS 10.15.7 Catalina INFO: PyInstaller: 4.2 I am using anaconda version: conda 4.8.2 [50576] Error loading Python lib '/var/folders/6z/snnb75fd7h734dd51kbdjntw0000gn/T/_MEIspzQDS/libpython3.9.dylib': dlopen: dlopen(/var/folders/6z/snnb75fd7h734dd51kbdjntw0000gn/T/_MEIspzQDS/libpython3.9.dylib, 10): no suitable image found. Did find: Update: I tried it with the developer version of pyinstaller pyinstaller 5.0.dev0 It suppressed the errors when I package using --onefile but I am still getting the same error when I run my executable [40125] Error loading Python lib '/var/folders/0r/jz_vw8vn54b85gh437lwcgdw0000gn/T/_MEIjzyz9g/libpython3.9.dylib': dlopen: dlopen(/var/folders/0r/jz_vw8vn54b85gh437lwcgdw0000gn/T/_MEIjzyz9g/libpython3.9.dylib, 10): no suitable image found. Did find:
/var/folders/0r/jz_vw8vn54b85gh437lwcgdw0000gn/T/_MEIjzyz9g/libpython3.9.dylib: code signature in (/var/folders/0r/jz_vw8vn54b85gh437lwcgdw0000gn/T/_MEIjzyz9g/libpython3.9.dylib) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed. |
|
+1 |
|
This issue is clearly still not fixed. Same errors. |
@imperialguy have you tried running the development version? You might need to rebuild the bootloaders too. |
|
Same errors here, using developer version too
|
|
Have you recompiled the bootloaders? |
Looks like x86_64 is now starting to require at least ad-hoc code signing, same as M1. This is a different issue than hidden system libraries (which is what this issue was about), and will be fixed once we implement ad-hoc re-signing as part of M1 support. |
|
The problem with missing signature on libpython might be already fixed by WIP macos11-arm64-support-cleanup branch in #5581 (but there may or may not be additional issues with invalidated signatures on other collected shared libraries, depending on how strict x86_64 actually is). |
Yes, I have. The exact error message when I run executable is this: |
Ah, you're just trying to run the partial bundle from |
My mistake, sorry guys and thank you for your support!! |
|
I'm trying to build a very basic app that uses PyQt5 on Big Sur and I have the same problem with libraries. I compiled the bootloader as indicated here: https://github.com/pyinstaller/pyinstaller. I don't think it's the app I'm building. It runs correctly if I launch it from terminal. I get errors even with the basic: The exec and the app are created. The exec opens the terminal, writes segmentation fault and closes. The app crashes immediately after I open it. |
and others
Context information (for bug reports)
pyinstaller --version:4.0.dev0Link to Release Note below.
From apple:
New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)## Description of the issue
Works on windows. This is an issue with the new beta and a change that needs to be made for the new macOS Big Sur
A minimal example program which shows the error
Stacktrace / full error message
The text was updated successfully, but these errors were encountered: