Can't find libtorchtext.so when executing file #591

jhtong33 · 2023-05-25T09:08:06Z

Description of the issue

As title, the issue is executing the unix file, the torchtext module can't successfully import.
I also try to add the path /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib into my .zshrc, like export PATH="/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib:$PATH"

Context information (for bug reports)

Output of pyinstaller --version: 5.11.0
Version of Python: 3.11.3
Platform: Mac OS
How you installed Python: conda
Did you also try this on another platform? Does it work there? No
try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

follow all the instructions in our "If Things Go Wrong" Guide
(https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

start with clean installation
use the latest development version
Run your frozen program from a command window (shell) — instead of double-clicking on it
Package your program in --onedir mode
Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

ImportError: dlopen(/var/folders/l2/zyrmtcyj7zn13lz9xk4l96zw0000gn/T/_MEI9vRBrl/torchtext/_torchtext.so, 0x0002): Library not loaded: '@loader_path/../libtorchtext.so'
  Referenced from: '/private/var/folders/l2/zyrmtcyj7zn13lz9xk4l96zw0000gn/T/_MEI9vRBrl/torchtext/_torchtext.so'
  Reason: tried: '/private/var/folders/l2/zyrmtcyj7zn13lz9xk4l96zw0000gn/T/_MEI9vRBrl/torchtext/../libtorchtext.so' (no such file), '/usr/local/lib/libtorchtext.so' (no such file), '/usr/lib/libtorchtext.so' (no such file)

Stacktrace / full error message

Traceback (most recent call last):
  File "Classify_product_v0.py", line 16, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "torchtext/__init__.py", line 6, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 352, in exec_module
  File "torchtext/_extension.py", line 64, in <module>
  File "torchtext/_extension.py", line 61, in _init_extension
ImportError: dlopen(/var/folders/l2/zyrmtcyj7zn13lz9xk4l96zw0000gn/T/_MEI9vRBrl/torchtext/_torchtext.so, 0x0002): Library not loaded: '@loader_path/../libtorchtext.so'
  Referenced from: '/private/var/folders/l2/zyrmtcyj7zn13lz9xk4l96zw0000gn/T/_MEI9vRBrl/torchtext/_torchtext.so'
  Reason: tried: '/private/var/folders/l2/zyrmtcyj7zn13lz9xk4l96zw0000gn/T/_MEI9vRBrl/torchtext/../libtorchtext.so' (no such file), '/usr/local/lib/libtorchtext.so' (no such file), '/usr/lib/libtorchtext.so' (no such file)
[4439] Failed to execute script 'Classify_product_v0' due to unhandled exception!

Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs
for more about what would use to solve the issue.

The text was updated successfully, but these errors were encountered:

rokm · 2023-05-25T14:34:26Z

Sounds like you need to collect binaries from the torchtext package. Either by adding --collect-binaries torchtext or by writing a custom hook.

And since you're trying to use torch and macOS, you'll probably have to take a look at #375.

rokm · 2023-05-25T14:36:33Z

FWIW, I plan to implement proper official hooks for torch and torch{vision,audio,text}, once pyinstaller/pyinstaller#7619 is finished and we have symlink support in place, so we can avoid duplication issues discussed in #375.

jhtong33 · 2023-05-26T03:30:09Z

Hi @rokm,
Following the #375, I still can't successfully execute the file.
the following steps are my test:

pyinstaller --onedir <program>.py -c --add-data "Model:." --add-data "Classify_setting.txt:."
cd dist/<program>
ln -s ./torch/lib/libtorch_python.dylib .
ln -s ./torch/lib/libtorch.dylib .
ln -s ./torch/lib/libc10.dylib .
ln -s ./torch/lib/libtorch_cpu.dylib .
ln -s /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so .
8.install_name_tool -add_rpath @loader_path/.. torchtext/lib/libtorchtext.so

moreover, my warning is same as previous issue :

3157 WARNING: Cannot find path ./libtorch_cpu.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so)
3157 WARNING: Cannot find path ./libc10.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so)
3313 WARNING: Cannot find path ./libtorch.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3313 WARNING: Cannot find path ./libtorch_cpu.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3313 WARNING: Cannot find path ./libtorch_python.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3313 WARNING: Cannot find path ./libc10.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3316 WARNING: Cannot find path ./libtorch.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3316 WARNING: Cannot find path ./libtorch_cpu.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3316 WARNING: Cannot find path ./libtorchtext.so (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3316 WARNING: Cannot find path ./libc10.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3316 WARNING: Cannot find path ./libtorch_python.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
52570 INFO: Looking for eggs```

rokm · 2023-05-26T08:58:54Z

Can we see the whole run-time error message, then? Also, what versions of torch libraries are you using, and how did you install them (via conda or via pip)?

rokm · 2023-05-26T09:08:47Z

ln -s /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so .

install_name_tool -add_rpath @loader_path/.. torchtext/lib/libtorchtext.so

Don't do this. Symlinking a system-installed file into application bundle make the application bundle non-portable.

rokm · 2023-05-26T10:19:32Z

If torchtext/lib/libtorchtext.so was collected, then you probably need to just create symlink to it into top-level directory: ln -s ./torchtext/lib/libtorchtext.dylib .

If it was not collected, you might need to copy it manually or add --collect-binaries torchtext to your PyInstaller command, and then add the symlink to top-level directory.

jhtong33 · 2023-05-29T01:57:05Z

Can we see the whole run-time error message, then? Also, what versions of torch libraries are you using, and how did you install them (via conda or via pip)?

I installed torch via pip install torch torchvision

the detailed message is

451 INFO: PyInstaller: 5.11.0
451 INFO: Python: 3.11.3 (conda)
460 INFO: Platform: macOS-12.6.4-arm64-arm-64bit
461 INFO: wrote /Users/tong/Library/CloudStorage/OneDrive/02_TagTeam/Python/Classify_wNN/packaged/Classify_product_v1.spec
463 INFO: UPX is not available.
469 INFO: Extending PYTHONPATH with paths
['/Users/tong/Library/CloudStorage/OneDrive/02_TagTeam/Python/Classify_wNN/packaged']
677 INFO: Appending 'binaries' from .spec
678 INFO: Appending 'datas' from .spec
680 INFO: checking Analysis
680 INFO: Building Analysis because Analysis-00.toc is non existent
680 INFO: Initializing module dependency graph...
681 INFO: Caching module graph hooks...
687 INFO: Analyzing base_library.zip ...
2080 INFO: Loading module hook 'hook-heapq.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
2130 INFO: Loading module hook 'hook-encodings.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
3117 INFO: Loading module hook 'hook-pickle.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
4096 INFO: Caching module dependency graph...
4136 INFO: running Analysis Analysis-00.toc
4139 INFO: Analyzing /Users/tong/Library/CloudStorage/OneDrive/02_TagTeam/Python/Classify_wNN/packaged/Classify_product_v1.py
4152 INFO: Loading module hook 'hook-pandas.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
283 INFO: NumExpr defaulting to 8 threads.
4996 INFO: Loading module hook 'hook-platform.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
5025 INFO: Loading module hook 'hook-xml.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
5040 INFO: Loading module hook 'hook-numpy.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/numpy/_pyinstaller'...
5481 INFO: Loading module hook 'hook-difflib.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
5618 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
6281 INFO: Loading module hook 'hook-sysconfig.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
6654 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
7343 INFO: Loading module hook 'hook-pytz.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
7463 INFO: Loading module hook 'hook-pkg_resources.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
8599 INFO: Processing pre-safe import module hook six.moves from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
9080 INFO: Loading module hook 'hook-scipy.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
9225 INFO: Loading module hook 'hook-pycparser.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
9549 INFO: Processing pre-find module path hook distutils from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
9698 INFO: Loading module hook 'hook-distutils.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
9731 INFO: Loading module hook 'hook-distutils.util.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
9821 INFO: Loading module hook 'hook-setuptools.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
10850 INFO: Loading module hook 'hook-packaging.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
11548 INFO: Loading module hook 'hook-matplotlib.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
11759 INFO: Processing pre-safe import module hook gi from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-gi.py'.
11814 INFO: Loading module hook 'hook-PIL.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
11857 INFO: Loading module hook 'hook-PIL.Image.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
12203 INFO: Loading module hook 'hook-xml.dom.domreg.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
12562 INFO: Loading module hook 'hook-PIL.ImageFilter.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
12934 INFO: Loading module hook 'hook-jinja2.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
14671 INFO: Loading module hook 'hook-matplotlib.backends.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
15265 INFO: Loading module hook 'hook-IPython.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
16089 INFO: Loading module hook 'hook-pygments.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
17461 INFO: Loading module hook 'hook-lib2to3.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
17794 INFO: Loading module hook 'hook-wcwidth.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
18529 INFO: Loading module hook 'hook-jedi.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
18871 INFO: Loading module hook 'hook-parso.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
19662 INFO: Loading module hook 'hook-sqlite3.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
20018 INFO: Loading module hook 'hook-nbformat.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
20151 INFO: Loading module hook 'hook-jsonschema.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
20632 INFO: Processing pre-safe import module hook urllib3.packages.six.moves from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-urllib3.packages.six.moves.py'.
20808 INFO: Loading module hook 'hook-charset_normalizer.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
20911 INFO: Loading module hook 'hook-cryptography.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
22018 INFO: Loading module hook 'hook-certifi.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
22404 INFO: Loading module hook 'hook-zmq.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
23033 INFO: Loading module hook 'hook-platformdirs.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
23485 INFO: Loading module hook 'hook-scipy.linalg.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
23764 INFO: Loading module hook 'hook-scipy.sparse.csgraph.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
24959 INFO: Loading module hook 'hook-scipy.spatial.transform.rotation.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
26079 INFO: Loading module hook 'hook-pandas.io.formats.style.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
27353 INFO: Loading module hook 'hook-pandas.plotting.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
27947 INFO: Loading module hook 'hook-openpyxl.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
29515 INFO: Loading module hook 'hook-torch.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
29516 INFO: hook-torch: raising recursion limit to 5000
42652 INFO: Loading module hook 'hook-scipy.io.matlab.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
43320 INFO: Loading module hook 'hook-torchvision.ops.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
44432 INFO: Processing pre-safe import module hook win32com from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
46486 INFO: Processing module hooks...
46512 WARNING: Hidden import "charset_normalizer.md__mypyc" not found!
46781 INFO: Loading module hook 'hook-scipy.special._ufuncs.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
46782 INFO: Loading module hook 'hook-_tkinter.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
46783 INFO: checking Tree
46783 INFO: Building Tree because Tree-00.toc is non existent
46783 INFO: Building Tree Tree-00.toc
46787 INFO: checking Tree
46787 INFO: Building Tree because Tree-01.toc is non existent
46787 INFO: Building Tree Tree-01.toc
46790 INFO: checking Tree
46790 INFO: Building Tree because Tree-02.toc is non existent
46790 INFO: Building Tree Tree-02.toc
46831 INFO: Loading module hook 'hook-setuptools.msvc.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
47384 INFO: Loading module hook 'hook-PIL.SpiderImagePlugin.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
47405 INFO: Loading module hook 'hook-scipy.special._ellip_harm_2.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
47405 INFO: Loading module hook 'hook-matplotlib.backends.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
47406 INFO: Matplotlib backend selection method: automatic discovery of used backends
47459 INFO: Discovered Matplotlib backend(s) via `matplotlib.use()` call in module 'sympy.testing.runtests': ['Agg', 'Agg']
47463 INFO: The following Matplotlib backends were discovered by scanning for `matplotlib.use()` calls: ['Agg', 'Agg']. If your backend of choice is not in this list, either add a `matplotlib.use()` call to your code, or configure the backend collection via hook options (see: https://pyinstaller.org/en/stable/hooks-config.html#matplotlib-hooks).
47463 INFO: Selected matplotlib backends: ['Agg', 'Agg']
47463 INFO: Loading module hook 'hook-scipy.stats._stats.py' from '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks'...
48649 INFO: Looking for ctypes DLLs
48698 WARNING: Library user32 required via ctypes not found
48819 WARNING: Library msvcrt required via ctypes not found
48827 WARNING: Library vcruntime140_1.dll required via ctypes not found
48828 WARNING: Library vcruntime140.dll required via ctypes not found
48828 WARNING: Library msvcp140.dll required via ctypes not found
48896 WARNING: Library libnvidia-ml.so.1 required via ctypes not found
48918 WARNING: Ignoring /usr/lib64/libgomp.so.1 imported from /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torch/_inductor/codecache.py - only basenames are supported with ctypes imports!
48951 INFO: Analyzing run-time hooks ...
48963 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
48964 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgres.py'
48967 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
48968 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
48970 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_setuptools.py'
48970 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/rthooks/pyi_rth_traitlets.py'
48971 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_mplconfig.py'
48971 INFO: Including run-time hook '/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth__tkinter.py'
49247 INFO: Looking for dynamic libraries
/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/building/build_main.py:177: UserWarning: The numpy.array_api submodule is still experimental. See NEP 47.
  __import__(package)
This version of python seems to be incorrectly compiled
(internal generated filenames are not absolute).
This may make the debugger miss breakpoints.
Related bug: http://bugs.python.org/issue1666807
/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/isolated/_child.py:96: VisibleDeprecationWarning: zmq.eventloop.minitornado is deprecated in pyzmq 14.0 and will be removed.
    Install tornado itself to use zmq with the tornado IOLoop.

  while run_next_command(read_fh, write_fh):
/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/datapoints/__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
/Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/transforms/v2/__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
  warnings.warn(_BETA_TRANSFORMS_WARNING)
3285 WARNING: Cannot find path ./libtorch_cpu.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so)
3285 WARNING: Cannot find path ./libc10.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so)
3285 WARNING: Cannot find path ./libtorch.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so)
3451 WARNING: Cannot find path ./libtorch_python.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3451 WARNING: Cannot find path ./libtorch_cpu.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3451 WARNING: Cannot find path ./libc10.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3451 WARNING: Cannot find path ./libtorch.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchvision/_C.so)
3454 WARNING: Cannot find path ./libtorch_python.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3454 WARNING: Cannot find path ./libtorch_cpu.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3454 WARNING: Cannot find path ./libc10.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3454 WARNING: Cannot find path ./libtorchtext.so (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
3454 WARNING: Cannot find path ./libtorch.dylib (needed by /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/_torchtext.so)
53384 INFO: Looking for eggs
53444 INFO: Python library not among binary dependencies. Performing additional search...
53446 INFO: Using Python library /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/libpython3.11.dylib
53465 INFO: Warnings written to /Users/tong/Library/CloudStorage/Onedrive/02_TagTeam/Python/Classify_wNN/packaged/build/Classify_product_v1/warn-Classify_product_v1.txt
53604 INFO: Graph cross-reference written to /Users/tong/Library/CloudStorage/Onedrive/02_TagTeam/Python/Classify_wNN/packaged/build/Classify_product_v1/xref-Classify_product_v1.html
53818 INFO: checking PYZ
53818 INFO: Building PYZ because PYZ-00.toc is non existent
53818 INFO: Building PYZ (ZlibArchive) /Users/tong/Library/CloudStorage/OneDrive-/02_TagTeam/Python/Classify_wNN/packaged/build/Classify_product_v1/PYZ-00.pyz
56920 INFO: Building PYZ (ZlibArchive) /Users/tong/Library/CloudStorage/OneDrive-/02_TagTeam/Python/Classify_wNN/packaged/build/Classify_product_v1/PYZ-00.pyz completed successfully.
56966 INFO: EXE target arch: arm64
56966 INFO: Code signing identity: None
56966 INFO: checking PKG
56966 INFO: Building PKG because PKG-00.toc is non existent
56966 INFO: Building PKG (CArchive) Classify_product_v1.pkg
57004 INFO: Building PKG (CArchive) Classify_product_v1.pkg completed successfully.
57005 INFO: Bootloader /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/PyInstaller/bootloader/Darwin-64bit/run
57005 INFO: checking EXE
57005 INFO: Building EXE because EXE-00.toc is non existent
57005 INFO: Building EXE from EXE-00.toc
57005 INFO: Copying bootloader EXE to /Users/tong/Library/CloudStorage/OneDrive-/02_TagTeam/Python/Classify_wNN/packaged/build/Classify_product_v1/Classify_product_v1
57012 INFO: Converting EXE to target arch (arm64)
57035 INFO: Removing signature(s) from EXE
57047 INFO: Appending PKG archive to EXE
57070 INFO: Fixing EXE headers for code signing
57091 INFO: Rewriting the executable's macOS SDK version (12.1.0) to match the SDK version of the Python library (11.0.0) in order to avoid inconsistent behavior and potential UI issues in the frozen application.
57092 INFO: Re-signing the EXE
57150 INFO: Building EXE from EXE-00.toc completed successfully.
57206 INFO: checking COLLECT
57206 INFO: Building COLLECT because COLLECT-00.toc is non existent
57208 INFO: Building COLLECT COLLECT-00.toc
66634 INFO: Building COLLECT COLLECT-00.toc completed successfully.

jhtong33 · 2023-05-29T02:14:30Z

If torchtext/lib/libtorchtext.so was collected, then you probably need to just create symlink to it into top-level directory: ln -s ./torchtext/lib/libtorchtext.dylib .

If it was not collected, you might need to copy it manually or add --collect-binaries torchtext to your PyInstaller command, and then add the symlink to top-level directory.

Following the steps above,
when I import the module after executing the file, it will be killed!

my commands:

pyinstaller --onedir Classify_product_v1.py -c --add-data "Model:." --add-data "Classify_setting.txt:." --collect-binaries torchtext
cp /Users/tong/opt/anaconda3/envs/scikit_xgb/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so dist/Classify_product_v1/torchtext/lib
cd dist/Classify_product_v1
ln -s ./torchtext/lib/libtorchtext.so .

terminal message

Begining to import module ... please wait a minute 
zsh: killed     

Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

rokm · 2023-05-29T08:19:10Z

It is killed because of dylib duplication - that's why you need to also apply steps from #375.

jhtong33 · 2023-05-30T01:38:02Z

I try to copy *dylib manually and new error message is

***** Begining to import module ... please wait a minute
libc++abi: terminating with uncaught exception of type c10::Error: Type c10::intrusive_ptr<ConvPackedParamsBase<2>> could not be converted to any of the known types.
Exception raised from operator() at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/jit_type.h:1798 (most recent call first):
frame #0: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 92 (0x1447712b8 in libc10.dylib)
frame #1: c10::detail::getTypePtr_<c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > >::call()::'lambda'()::operator()() const + 300 (0x294e19ff8 in libtorch_cpu.dylib)
frame #2: c10::Type::SingletonOrSharedTypePtr<c10::Type> c10::getFakeTypePtrCopy<c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > >() + 32 (0x294e19d44 in libtorch_cpu.dylib)
frame #3: c10::detail::infer_schema::(anonymous namespace)::createArgumentVector(c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 200 (0x294616aac in libtorch_cpu.dylib)
frame #4: c10::detail::infer_schema::make_function_schema(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 96 (0x294616860 in libtorch_cpu.dylib)
frame #5: c10::detail::infer_schema::make_function_schema(c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 60 (0x294616e1c in libtorch_cpu.dylib)
frame #6: std::__1::unique_ptr<c10::FunctionSchema, std::__1::default_delete<c10::FunctionSchema> > c10::detail::inferFunctionSchemaFromFunctor<at::Tensor (*)(at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long)>() + 92 (0x294e5ad90 in libtorch_cpu.dylib)
frame #7: torch::CppFunction::CppFunction<at::Tensor (at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long)>(at::Tensor (*)(at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long), std::__1::enable_if<c10::guts::is_function_type<at::Tensor (at::Tensor, c10::intrusive_ptr<ConvPackedParamsBase<2>, c10::detail::intrusive_target_default_null_type<ConvPackedParamsBase<2> > > const&, double, long long)>::value, std::nullptr_t>::type) + 116 (0x294e5ac78 in libtorch_cpu.dylib)
frame #8: at::native::(anonymous namespace)::TORCH_LIBRARY_IMPL_init_quantized_QuantizedCPU_4(torch::Library&) + 40 (0x294e588e8 in libtorch_cpu.dylib)
frame #9: torch::detail::TorchLibraryInit::TorchLibraryInit(torch::Library::Kind, void (*)(torch::Library&), char const*, c10::optional<c10::DispatchKey>, char const*, unsigned int) + 208 (0x2942ad65c in libtorch_cpu.dylib)
frame #10: _GLOBAL__sub_I_qconv.cpp + 88 (0x294e5fd00 in libtorch_cpu.dylib)
frame #11: invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 164 (0x104f81f4c in dyld)
frame #12: invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 168 (0x104fab550 in dyld)
frame #13: invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 528 (0x104fa1ed8 in dyld)
frame #14: dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168 (0x104f6df98 in dyld)
frame #15: dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192 (0x104fa1c80 in dyld)
frame #16: dyld3::MachOAnalyzer::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, unsigned char const*, bool&) block_pointer) const + 148 (0x104faae80 in dyld)
frame #17: dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 432 (0x104fab180 in dyld)
frame #18: dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 172 (0x104f81e8c in dyld)
frame #19: dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 216 (0x104f82038 in dyld)
frame #20: dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 180 (0x104f82014 in dyld)
frame #21: dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 180 (0x104f82014 in dyld)
frame #22: dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 180 (0x104f82014 in dyld)
frame #23: dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 124 (0x104f82104 in dyld)
frame #24: dyld4::APIs::dlopen_from(char const*, int, void*) + 520 (0x104f91c64 in dyld)
<omitting python frames>

zsh: abort      

Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

jhtong33 added the triage label May 25, 2023

rokm transferred this issue from pyinstaller/pyinstaller May 25, 2023

rokm mentioned this issue May 27, 2023

building: macOS: use @rpath to rewrite binaries' dependency paths pyinstaller/pyinstaller#7664

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't find libtorchtext.so when executing file #591

Can't find libtorchtext.so when executing file #591

jhtong33 commented May 25, 2023 •

edited

rokm commented May 25, 2023

rokm commented May 25, 2023 •

edited

jhtong33 commented May 26, 2023 •

edited

rokm commented May 26, 2023

rokm commented May 26, 2023 •

edited

rokm commented May 26, 2023

jhtong33 commented May 29, 2023 •

edited

jhtong33 commented May 29, 2023 •

edited

rokm commented May 29, 2023

jhtong33 commented May 30, 2023

Can't find libtorchtext.so when executing file #591

Can't find libtorchtext.so when executing file #591

Comments

jhtong33 commented May 25, 2023 • edited

Description of the issue

Context information (for bug reports)

Make sure everything is packaged correctly

A minimal example program which shows the error

Stacktrace / full error message

rokm commented May 25, 2023

rokm commented May 25, 2023 • edited

jhtong33 commented May 26, 2023 • edited

rokm commented May 26, 2023

rokm commented May 26, 2023 • edited

rokm commented May 26, 2023

jhtong33 commented May 29, 2023 • edited

jhtong33 commented May 29, 2023 • edited

rokm commented May 29, 2023

jhtong33 commented May 30, 2023

jhtong33 commented May 25, 2023 •

edited

rokm commented May 25, 2023 •

edited

jhtong33 commented May 26, 2023 •

edited

rokm commented May 26, 2023 •

edited

jhtong33 commented May 29, 2023 •

edited

jhtong33 commented May 29, 2023 •

edited