SHA1_init missing when using PyInstaller on OSX #5583

fineusing · 2021-02-24T04:22:12Z

Description of the issue

There is no problem running the source code directly with python3 under OSX 11.1
When running the executable generated by PyInstaller, I get the following error message.

File "gpsoauth/__init__.py", line 97, in perform_master_login
File "gpsoauth/google.py", line 51, in signature
File "Cryptodome/Cipher/PKCS1_OAEP.py", line 117, in encrypt
lHash = self._hashObj.new(self._label).digest()
File "Cryptodome/Hash/SHA1.py", line 158, in new
return SHA1Hash().new(data)
File "Cryptodome/Hash/SHA1.py", line 74, in __init__
result = _raw_sha1_lib.SHA1_init(state.address_of())
File "ctypes/__init__.py", line 369, in __getattr__
File "ctypes/__init__.py", line 374, in __getitem__
AttributeError: dlsym(0x7fc07e7ee6d0, SHA1_init): symbol not found

After testing, it is found that the sha1_init symbol is in the source code Cryptodome/Hash/_SHA1.abi3.so library, and the _sha1.cpython-39m-darwin.so generated by Pyinstaller does not have the sha1_init symbol.

Context information (for bug reports)

pyinstaller 5.0dev
python 3.9.1
OSX 11.1
There is no problem with the executable file generated by pyinstaller under the win platform

The text was updated successfully, but these errors were encountered:

bwoodsend · 2021-02-24T09:30:06Z

Can you please fill out the issue template.

fineusing · 2021-02-24T12:16:22Z

Can you please fill out the issue template.

ok, it has been changed

bwoodsend · 2021-02-24T14:43:28Z

Can you try using collect_dynamic_libs()? Edit you .spec file to put:

from PyInstaller.utils.hooks import collect_dynamic_libs

at the top and change binaries=[] to binaries=collect_dynamic_libs("Cryptodome"). Then rebuild using:

pyinstaller your-project-name.spec

That at least ensures the right libraries are collected.

rokm · 2021-02-24T18:00:35Z

Uf, this is a nasty subtle bug...

There is a Cryptodome/Hash/_SHA1.abi3.so library that's properly collected and which is supposed to be loaded by pycryptodomex, but we end up with handle to _sha1.cpython-38-darwin.so that's collected in the root dist folder (and comes from python itself).

The loading is done via cffi or, if that's not available, via ctypes as a fallback.

In both cases, the library is loaded by calling utility function with fully-qualified name, e.g., Crypto.Hash._SHA1. This load_pycryptodome_raw_lib helper decomposes the fully-qualified name into directory components and filename, and then tries to find the library using different extensions.

The extension suffices that are tested come from importlib.machinery.EXTENSION_SUFFIXES, and are:

['.cpython-38-darwin.so', '.abi3.so', '.so']

This means than unfrozen version first tries to load

/Users/[...]/venv/lib/python3.8/site-packages/Cryptodome/Util/../Hash/_SHA1.cpython-38-darwin.so

which fails as it does not exist. Then it tries

/Users/[...]/venv/lib/python3.8/site-packages/Cryptodome/Util/../Hash/_SHA1.abi3.so

which succeeds.

In frozen version, however, the first call with _MEIPASS/Cryptodome/Util/../Hash/_SHA1.cpython-38-darwin.so actually succeeds, but it returns handle to _MEIPASS/_SHA1.cpython-38-darwin.so.

In ctypes codepath, this happens because our loader explicitly checks if the library with requested basename exists in _MEIPASS and if it does, returns it. I.e.,

pyinstaller/PyInstaller/loader/pyiboot01_bootstrap.py

Lines 126 to 131 in 93285ec

    
           def _frozen_name(name): 
        
               if name: 
        
                   frozen_name = os.path.join(sys._MEIPASS, os.path.basename(name)) 
        
                   if os.path.exists(frozen_name) and not os.path.isdir(frozen_name): 
        
                       name = frozen_name 
        
               return name

called from

pyinstaller/PyInstaller/loader/pyiboot01_bootstrap.py

Lines 140 to 146 in 93285ec

    
           class PyInstallerCDLL(ctypes.CDLL): 
        
               def __init__(self, name, *args, **kwargs): 
        
                   name = _frozen_name(name) 
        
                   try: 
        
                       super(PyInstallerCDLL, self).__init__(name, *args, **kwargs) 
        
                   except Exception as base_error: 
        
                       raise PyInstallerImportError(name) from base_error

The same thing happens with cffi, although the mechanism is different there, but I suspect it involves this:

pyinstaller/PyInstaller/loader/pyiboot01_bootstrap.py

Lines 193 to 199 in 93285ec

    
           if sys.platform.startswith('darwin'): 
        
               try: 
        
                   from ctypes.macholib import dyld 
        
                   dyld.DEFAULT_LIBRARY_FALLBACK.insert(0, sys._MEIPASS) 
        
               except ImportError: 
        
                   # Do nothing when module 'ctypes' is not available. 
        
                   pass

So long story short, we find a library that should not be found. (Although on the other hand, I understand why also checking _MEIPASS might be needed in other scenarios).

rokm · 2021-02-24T18:06:42Z

@Robby0318, since your log indicates that you're using ctypes codepath, as a work-around you can try disabling that _frozen_name rewrite in your local PyInstaller installation (in site-packages/PyInstaller/loader/pyiboot01_bootstrap.py).

For example, try changing:

def _frozen_name(name): 
    if name: 
        [...]

to

def _frozen_name(name): 
    if name and not os.path.isabs(name):
        [...]

and rebuild your application. (Although whether this will work depends a lot on behavior of other packages that you're using...)

…ion libraries On linux and macOS, some of the built-ins are provided as extensions (e.g., _sha1.cpython-39-x86_64-linux-gnu.so) that originally reside in python3.X/lib-dynload directory. This directory is not in the ctypes' library search path, therefore running ctypes.CDLL('_sha1.cpython-39-x86_64-linux-gnu.so') in python interpreter will come up empty. In a frozen application, however, these extensions end up collected directly into _MEIPASS, which is searched by ctypes (because search behavior is explicitly overriden by the hook). Therefore, trying to load the extension's library file via ctypes.CDLL() will succeed, resultingin inconsistent behavior between unfrozen and frozen program. On macOS, this causes issues with `pycryptodomex` (pyinstaller#5583), which, due to its library search implementation, ends up with handle of `_sha1` extension module instead of its private extension/library with the same name prefix.

On macOS and linux, some of the python's built-ins have extension modules that originally reside in python3.X/lib-dynload directory. This directory is in sys.path, therefore the collected extensions have no parent directory and end up directly into _MEIPASS. This commit explicitly diverts such extensions into lib-dynload sub-directory in the _MEIPASS. In addition to decluttering the _MEIPASS on linux and macOS, this also prevents ctypes.CDLL() from picking up the extensions' shared libraries and causing inconsistent behavior between frozen and unfrozen application, which in some corner cases leads to issues with shadowing, such as pyinstaller#5583.

…ion libraries On linux and macOS, some of the built-ins are provided as extensions (e.g., _sha1.cpython-39-x86_64-linux-gnu.so) that originally reside in python3.X/lib-dynload directory. This directory is not in the ctypes' library search path, therefore running ctypes.CDLL('_sha1.cpython-39-x86_64-linux-gnu.so') in python interpreter will come up empty. In a frozen application, however, these extensions end up collected directly in the _MEIPASS, which is searched by ctypes (because search behavior is explicitly overriden by the hook). Therefore, trying to load the extension's library file via ctypes.CDLL() will succeed, resulting in inconsistent behavior between unfrozen and frozen program. On macOS, this causes issues with `pycryptodomex` (pyinstaller#5583), which, due to its library search implementation, ends up with handle of `_sha1` extension module instead of its private extension/library with the same name prefix.

On macOS and linux, some of the python's built-ins have extension modules that originally reside in python3.X/lib-dynload directory. This directory is in sys.path, therefore the collected extensions have no parent directory and end up directly in the _MEIPASS. This commit explicitly diverts such extensions into lib-dynload sub-directory in the _MEIPASS. In addition to decluttering the _MEIPASS on linux and macOS, this also prevents ctypes.CDLL() from picking up the extensions' shared libraries and causing inconsistent behavior between frozen and unfrozen application, which in some corner cases leads to issues with shadowing, such as in pyinstaller#5583.