pyinstaller fails to execute completely with error of tcl

[nbrummel]

When I run from a .spec file it does not work and throws errors. I have checked in a python shell the tk and tcl work with python. Any advice or help would be great. Here is my spec file:

# -*- mode: python ; coding: utf-8 -*-
import os

from os.path import abspath, dirname
EMERGE_DIR = dirname(abspath('.'))
block_cipher = None

added_files =   [('support/logging.yaml','.'),
                ('language_library_layer/tactile_fx/*','language_library_layer/tactile_fx'),
                ('support/system_files/system_config_default.json','support'),
                ('support/mapping_files/v0_11_transducer_mapping.json','support/mapping_files'),
                ('support/mapping_files/v0_9_transducer_mapping.json','support/mapping_files'),
                ('support/mapping_files/v0_7_transducer_mapping.json','support/mapping_files'),
                ('support/coord_files/large_circ_coords_9.json','support/coord_files'),
                ('support/coord_files/large_circ_coords_11.json','support/coord_files'),
                ('top/port_config.json','top'),
                ('tactile_engine_installer/version.txt','tactile_engine_installer')]

a = Analysis(#['application_interface_layer/tactile_engine_api.py'],
            ['top/tactile_engine.py'],
            pathex=[EMERGE_DIR + '/tactile_engine/source/code'],
            binaries=[],
            datas=added_files,
            hiddenimports=[],
            hookspath=[],
            runtime_hooks=[],
            excludes=[],
            win_no_prefer_redirects=False,
            win_private_assemblies=False,
            cipher=block_cipher,
            noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
            cipher=block_cipher)
exe = EXE(pyz,
        a.scripts,
#        a.binaries,
#        a.zipfiles,
#        a.datas,
        [],
        exclude_binaries=True,
        name='TactileEngine',
        debug=False,
        bootloader_ignore_signals=False,
        strip=False,
        upx=True,
        console=True )
coll = COLLECT(exe,
            a.binaries,
            a.zipfiles,
            a.datas,
            strip=False,
            upx=True,
            upx_exclude=[],
            name='tactile_engine')
app = BUNDLE(coll, 
                name='TactileEngine.app',
                icon='tactile_engine_installer/emerge_logo/emerge_black_m_square_48x48.icns',
                bundle_identifier=None,
                console=True)

Here is the output:

pyinstaller tactile_engine_test.spec
83 INFO: PyInstaller: 3.6
84 INFO: Python: 3.8.5
91 INFO: Platform: macOS-11.0-x86_64-i386-64bit
99 INFO: UPX is available.
101 INFO: Extending PYTHONPATH with paths
['/Users/nb/emerge/sw/tactile_engine/source/code',
 '/Users/nb/emerge/sw/tactile_engine/source/tactile_engine/source/code']
101 INFO: checking Analysis
111 INFO: Building because _python_version changed
111 INFO: Initializing module dependency graph...
115 INFO: Caching module graph hooks...
119 INFO: Analyzing base_library.zip ...
2544 INFO: Processing pre-find module path hook   distutils
2545 INFO: distutils: retargeting to non-venv dir '/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8'
5325 INFO: Caching module dependency graph...
5418 INFO: running Analysis Analysis-00.toc
5422 ERROR: Can not find path /usr/lib/libSystem.B.dylib (needed by /usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/bin/python3.8)
5425 ERROR: Can not find path /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Python)
5425 ERROR: Can not find path /usr/lib/libSystem.B.dylib (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Python)
5426 ERROR: Can not find path /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (needed by /usr/local/opt/gettext/lib/libintl.8.dylib)
5426 ERROR: Can not find path /usr/lib/libSystem.B.dylib (needed by /usr/local/opt/gettext/lib/libintl.8.dylib)
5426 ERROR: Can not find path /usr/lib/libiconv.2.dylib (needed by /usr/local/opt/gettext/lib/libintl.8.dylib)
5426 INFO: Analyzing application_interface_layer/tactile_engine_api.py
8347 INFO: Processing pre-safe import module hook   setuptools.extern.six.moves
8817 INFO: Processing pre-find module path hook   site
8817 INFO: site: retargeting to fake-dir '/usr/local/lib/python3.8/site-packages/PyInstaller/fake-modules'
10356 INFO: Processing pre-safe import module hook   six.moves
/usr/local/lib/python3.8/site-packages/pyquaternion/quaternion.py:62: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if s is 0:
/usr/local/lib/python3.8/site-packages/pyquaternion/quaternion.py:104: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if len(elements) is 1:
/usr/local/lib/python3.8/site-packages/pyquaternion/quaternion.py:113: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif s is 1:
/usr/local/lib/python3.8/site-packages/pyquaternion/quaternion.py:152: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif len(seq) is 0:
13258 INFO: Processing pre-safe import module hook   urllib3.packages.six.moves
14062 INFO: Processing module hooks...
14062 INFO: Loading module hook "hook-pkg_resources.py"...
14647 INFO: Processing pre-safe import module hook   win32com
14670 INFO: Excluding import '__main__'
14671 INFO:   Removing import of __main__ from module pkg_resources
14672 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
14673 INFO: Loading module hook "hook-lib2to3.py"...
14675 INFO: Loading module hook "hook-_tkinter.py"...
14680 ERROR: Can not find path /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so)
14680 ERROR: Can not find path /usr/lib/libSystem.B.dylib (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so)
14680 ERROR: Can not find path /System/Library/Frameworks/Tcl.framework/Versions/8.5/Tcl (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so)
14681 ERROR: Can not find path /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so)
14681 ERROR: Can not find path /usr/lib/libSystem.B.dylib (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so)
14681 ERROR: Can not find path /System/Library/Frameworks/Tcl.framework/Versions/8.5/Tcl (needed by /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_tkinter.cpython-38-darwin.so)
Traceback (most recent call last):
  File "/usr/local/bin/pyinstaller", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 681, in build
    exec(code, spec_namespace)
  File "tactile_engine_test.spec", line 19, in <module>
    a = Analysis(['application_interface_layer/tactile_engine_api.py'],
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 244, in __init__
    self.__postinit__()
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
    self.assemble()
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 422, in assemble
    self.graph.process_post_graph_hooks()
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/depend/analysis.py", line 311, in process_post_graph_hooks
    module_hook.post_graph()
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/depend/imphook.py", line 421, in post_graph
    self._process_hook_func()
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/depend/imphook.py", line 440, in _process_hook_func
    self._hook_module.hook(hook_api)
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/hook-_tkinter.py", line 251, in hook
    hook_api.add_datas(_collect_tcl_tk_files(hook_api))
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/hook-_tkinter.py", line 214, in _collect_tcl_tk_files
    tcl_root, tk_root = _find_tcl_tk(hook_api)
  File "/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/hook-_tkinter.py", line 183, in _find_tcl_tk
    ('Tcl', mapping['Tcl']),
KeyError: 'Tcl'```

[nbrummel]

I used pyinstaller tactile_engine.spec

[bwoodsend]

It should't make any difference that your running from a spec because when don't PyInstaller just generates the spec then from that spec anyway.

Presumably what is actually happening is PyInstaller thinks your project requires tkinter, it runs hook-_tkinter.py to work out what files it needs for tkinter but hook-_tkinter.py has a bug in it. If you don't actually use tkinter then you can just add 'tkinter' to excludes=[] in your spec - although we'd rather fix the hook.

@nbrummel Could you:

• Tell us how you installed Python?
• Tell us if/how you install tcl?
• If you have tcl installed, tell us where it is installed?

[nbrummel]

I'm on Mac,
I installed python with:
brew install python
did not use tcl

[nbrummel]

I placed the 'tkinter' in the excludes=[] and it worked. thank. One more question. All the errors where the libs are missing above. Do I need to add them to hidden imports?

[bwoodsend]

@nbrummel You mean this error?

  File "/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/hook-_tkinter.py", line 183, in _find_tcl_tk
    ('Tcl', mapping['Tcl']),
KeyError: 'Tcl'

If you do then no, hidden imports are for python modules, i.e anything it'd make sense to write after import in Python. I assume that the tkinter hook lacks a if not installed("tkinter"): skip this hook guard. TKinter is generally assumed to be standard lib and therefore always available.

Just as a final check - what happens if you import tkinter in Python (without PyInstaller)?

[nbrummel]

Yes I can import tkinter fine. I tried in python and ipython.
do you have any advice or a direction to point me in for this error ERROR: Can not find path /usr/lib/libSystem.B.dylib

[nbrummel]

I also tried on python 3.8 as well. same issue

[bwoodsend]

Yes I can import tkinter fine. I tried in python and ipython.

So you do must have tcl installed somewhere. Could you locate it for us?

do you have any advice or a direction to point me in for this error ERROR: Can not find path /usr/lib/libSystem.B.dylib

We can't guess the cause based on out of context single lines of error messages. You'll need to tell us how/where you got that error and what the whole message said.

[nbrummel]

So believe I have figured out the problem. I just installed the Mac osx beta 4 and have found out that apple moved some .dylib . Also, the Xcode has changed and that has turned out to mess up with PyInstaller. Do you have a build for the new osx/Xcode?

[bwoodsend]

@amifunny and/or @reritom Fancy having a stab at this one? Long story short - the tkinter hook needs to updated to track wherever the tcl dylibs have been moved to.

@nbrummel You're going to have to clarify your last comment. What does Xcode have to do with it? Unless you're using a Python shipped inside of Xcode it should be irrelevant.

[nbrummel]

I am on the new Mac OSX beta 4. Sorry for the confusion. Apple moved the libs for the new OSX Big Sur.

[nehaljwani]

Presumably what is actually happening is PyInstaller thinks your project requires tkinter, it runs hook-_tkinter.py to work out what files it needs for tkinter but hook-_tkinter.py has a bug in it.

The following might not be true for all cases, but when I was surprised to see tkinter and it's runtime dependencies popping up in the dist, I ran a pyinstaller with --log-level TRACE to find this:

...
4056 TRACE: _safe_import_hook 'tkinter' Package('test.support', '/path/to/custom/python_env/lib/python3.7/test/support/__init__.py', ['/path/to/custom/python_env/lib/python3.7/test/support']) ['Tk'] 0
4056 TRACE: _import_hook 'tkinter' Package('test.support', '/path/to/custom/python_env/lib/python3.7/test/support/__init__.py', ['/path/to/custom/python_env/lib/python3.7/test/support']) Package('test.support', '/path/to/custom/python_env/lib/python3.7/test/support/__init__.py', ['/path/to/custom/python_env/lib/python3.7/test/support']) 0
4056 TRACE: determine_parent Package('test.support', '/path/to/custom/python_env/lib/python3.7/test/support/__init__.py', ['/path/to/custom/python_env/lib/python3.7/test/support'])
4056 TRACE: determine_parent -> Package('test.support', '/path/to/custom/python_env/lib/python3.7/test/support/__init__.py', ['/path/to/custom/python_env/lib/python3.7/test/support'])
4056 TRACE: find_head_package Package('test.support', '/path/to/custom/python_env/lib/python3.7/test/support/__init__.py', ['/path/to/custom/python_env/lib/python3.7/test/support']) 'tkinter' 0
4056 TRACE: safe_import_module 'tkinter' 'tkinter' None
4056 TRACE: _find_module_path <- 'tkinter' ['/home/wani/repos/conda-forge/conda-standalone-feedstock/recipe', '/home/wani/repos/conda-forge/conda-standalone-feedstock/recipe', '/home/wani/repos/conda-forge/conda-standalone-feedstock/recipe', '/path/to/custom/python_env/bin', '/path/to/custom/python_env/lib/python37.zip', '/path/to/custom/python_env/lib/python3.7', '/path/to/custom/python_env/lib/python3.7/lib-dynload', '/path/to/custom/python_env/lib/python3.7/site-packages', '/home/wani/repos/conda-forge/conda-standalone-feedstock/recipe/pyinstaller', '/path/to/custom/python_env/lib/python3.7/site-packages/altgraph-0.17-py3.7.egg', '/path/to/custom/python_env/lib/python3.7/site-packages/macholib-1.14-py3.7.egg', '/path/to/custom/python_env/lib/python3.7/site-packages/pefile-2019.4.18-py3.7.egg', '/path/to/custom/python_env/lib/python3.7/site-packages/future-0.18.2-py3.7.egg', '/path/to/custom/python_env/lib/python3.7/site-packages/pyinstaller_hooks_contrib-2020.8-py3.7.egg', '/home/wani/repos/conda-forge/conda-standalone-feedstock/recipe', '/home/wani/repos/conda-forge/conda-standalone-feedstock/recipe', '/home/wani/repos/conda-forge/conda-standalone-feedstock/recipe']
4057 TRACE: _find_module_path -> (None, '/path/to/custom/python_env/lib/python3.7/tkinter/__init__.py', ('', '', 5))
4057 TRACE: load_module 'tkinter' None '/path/to/custom/python_env/lib/python3.7/tkinter/__init__.py'
4057 TRACE: load_package 'tkinter' '/path/to/custom/python_env/lib/python3.7/tkinter/__init__.py' []
4063 TRACE: addNode Package('tkinter',)
...

And on Linux, after excluding the 'test' module, the following objects were no longer needed:

120K    ~/.cache/pyinstaller/bincache10_py37_64bit/_testcapi.cpython-37m-x86_64-linux-gnu.so
84K     ~/.cache/pyinstaller/bincache10_py37_64bit/_tkinter.cpython-37m-x86_64-linux-gnu.so
1.3M    ~/.cache/pyinstaller/bincache10_py37_64bit/libX11.so.6
20K     ~/.cache/pyinstaller/bincache10_py37_64bit/libXau.so.6
1.4M    ~/.cache/pyinstaller/bincache10_py37_64bit/libtk8.6.so
1.8M    ~/.cache/pyinstaller/bincache10_py37_64bit/libtcl8.6.so
4.7M    total

[rokm]

Hmm, this is a bit different from other macOS Tcl/Tk issues (which we are in process of finally fixing), because it is a recent python (3.8.5) that still uses system Tcl/Tk framework - presumably this is a homebrew version (instead of the python.org build)?

At any rate, due to Big Sur hiding system libraries (see #5107), this getImports() call comes up empty, and subsequently the mapping ends up empty as well.

The cdll.LoadLibrary fix proposed in #5107 does seem to fix this issue.

[nehaljwani]

@rokm No, the output I shared is on Linux, using the conda-forge build of python 3.7

[rokm]

The issue from the original report (Tcl error with homebrew python 3.8.5 on Big Sur) should be fixed now that we do not try to collect data from system Tcl/Tk framework anymore (and are thus not affected by Big Sur hiding the system libraries). Of course, if tkinter is not used at all, it's best to exclude it.

Therefore, I'm closing this issue.

The problems stemming from Big Sur hiding the system libraries and the fact that tkinter is pulled in by applications that make no use of it both remain, but are beyond the scope of this issue.