Unable to package the latest version of pytorch1.8.0 and numpy1.19.5.

[huang229]

Previously used pytorch 1.2.0 and numpy 1.16.1. Using pyinstaller to package is very successful, and the executable file exe can run both locally and on the server. But after using pytorch1.8.0 and numpy1.19.5, the package fails, which can be divided into the following two cases: first, the .exe executable file generated by pyinstaller4.0 can only run locally, and the server fails directly without exception; second, pyinstaller4.2 cannot generate the executable file .exe.

[bwoodsend]

the server fails directly without exception;

Are you using conda?

[rokm]

first, the .exe executable file generated by pyinstaller4.0 can only run locally, and the server fails directly without exception;

If you're using anaconda environment, then .exe silently crashing is likely caused by missing libiomp5md DLL. This should be fixed by #5168 in develop.

second, pyinstaller4.2 cannot generate the executable file .exe.

What's the error message?

Please try the develop version, and if it also fails to build an .exe, copy the error message here.

[huang229]

@rokm @bwoodsend I didn't use conda. Install python3.6.7 locally, and then install pytorch1.8.0 with pip.The results are as follows：

C:\Users\huang\Desktop\skullkeypointptorch>pyinstaller -F deploy_test.py
565 INFO: PyInstaller: 4.0+19fb799a11
565 INFO: Python: 3.6.7
567 INFO: Platform: Windows-10-10.0.17763-SP0
569 INFO: wrote C:\Users\huang\Desktop\skullkeypointptorch\deploy_test.spec
576 INFO: UPX is not available.
583 INFO: Extending PYTHONPATH with paths
['C:\\Users\\huang\\Desktop\\skullkeypointptorch',
 'C:\\Users\\huang\\Desktop\\skullkeypointptorch']
613 INFO: checking Analysis
613 INFO: Building Analysis because Analysis-00.toc is non existent
614 INFO: Initializing module dependency graph...
618 INFO: Caching module graph hooks...
640 INFO: Analyzing base_library.zip ...
5384 INFO: Caching module dependency graph...
5480 INFO: running Analysis Analysis-00.toc
5515 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by D:\python-3.6\python.exe
5753 INFO: Analyzing C:\Users\huang\Desktop\skullkeypointptorch\deploy_test.py
7720 INFO: Processing pre-find module path hook distutils from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
7735 INFO: distutils: retargeting to non-venv dir 'D:\\python-3.6\\lib'
9227 INFO: Processing pre-safe import module hook setuptools.extern.six.moves from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks\\pre_safe_import_module\\hook-setuptools.extern.six.moves.py'.
9860 INFO: Processing pre-find module path hook site from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks\\pre_find_module_path\\hook-site.py'.
9868 INFO: site: retargeting to fake-dir 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\fake-modules'
18342 INFO: Processing module hooks...
18342 INFO: Loading module hook 'hook-cv2.py' from 'D:\\python-3.6\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
18355 INFO: Loading module hook 'hook-pycparser.py' from 'D:\\python-3.6\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
18356 INFO: Loading module hook 'hook-torch.py' from 'D:\\python-3.6\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
18816 INFO: Loading module hook 'hook-distutils.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
18832 INFO: Loading module hook 'hook-encodings.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
18960 INFO: Loading module hook 'hook-lib2to3.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
19043 INFO: Loading module hook 'hook-numpy.core.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
19180 INFO: Loading module hook 'hook-numpy.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
19182 INFO: Loading module hook 'hook-pkg_resources.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
19791 INFO: Processing pre-safe import module hook win32com from 'D:\\python-3.6\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\pre_safe_import_module\\hook-win32com.py'.
19922 WARNING: Hidden import "pkg_resources.py2_warn" not found!
19923 WARNING: Hidden import "pkg_resources.markers" not found!
19924 INFO: Excluding import '__main__'
19925 INFO:   Removing import of __main__ from module pkg_resources
19925 INFO: Loading module hook 'hook-scipy.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
19947 INFO: Loading module hook 'hook-setuptools.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
20669 INFO: Loading module hook 'hook-sqlite3.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
20892 INFO: Loading module hook 'hook-sysconfig.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
20894 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
20895 INFO: Loading module hook 'hook-xml.py' from 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks'...
21027 INFO: checking Tree
21028 INFO: Building Tree because Tree-00.toc is non existent
21028 INFO: Building Tree Tree-00.toc
21054 INFO: Looking for ctypes DLLs
21141 INFO: Analyzing run-time hooks ...
21147 INFO: Including run-time hook 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks\\rthooks\\pyi_rth_multiprocessing.py'
21167 INFO: Including run-time hook 'D:\\python-3.6\\lib\\site-packages\\pyinstaller-4.0+19fb799a11-py3.6.egg\\PyInstaller\\hooks\\rthooks\\pyi_rth_pkgres.py'
21190 INFO: Looking for dynamic libraries
22325 WARNING: lib not found: torch_python.dll dependency of D:\python-3.6\lib\site-packages\torch\_C.cp36-win_amd64.pyd
23164 INFO: Looking for eggs
23164 INFO: Using Python library D:\python-3.6\python36.dll
23164 INFO: Found binding redirects:
[]
23174 INFO: Warnings written to C:\Users\huang\Desktop\skullkeypointptorch\build\deploy_test\warn-deploy_test.txt
23287 INFO: Graph cross-reference written to C:\Users\huang\Desktop\skullkeypointptorch\build\deploy_test\xref-deploy_test.html
23403 INFO: checking PYZ
23403 INFO: Building PYZ because PYZ-00.toc is non existent
23404 INFO: Building PYZ (ZlibArchive) C:\Users\huang\Desktop\skullkeypointptorch\build\deploy_test\PYZ-00.pyz
24686 INFO: Building PYZ (ZlibArchive) C:\Users\huang\Desktop\skullkeypointptorch\build\deploy_test\PYZ-00.pyz completed successfully.
24713 INFO: checking PKG
24713 INFO: Building PKG because PKG-00.toc is non existent
24713 INFO: Building PKG (CArchive) PKG-00.pkg
Traceback (most recent call last):
  File "D:\python-3.6\Scripts\pyinstaller-script.py", line 11, in <module>
    load_entry_point('pyinstaller==4.0+19fb799a11', 'console_scripts', 'pyinstaller')()
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\building\build_main.py", line 720, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\building\build_main.py", line 667, in build
    exec(code, spec_namespace)
  File "C:\Users\huang\Desktop\skullkeypointptorch\deploy_test.spec", line 33, in <module>
    console=True )
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\building\api.py", line 437, in __init__
    upx_exclude=self.upx_exclude
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\building\api.py", line 200, in __init__
    self.__postinit__()
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\building\datastruct.py", line 160, in __postinit__
    self.assemble()
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\building\api.py", line 285, in assemble
    pylib_name=pylib_name)
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\archive\writers.py", line 332, in __init__
    super(CArchiveWriter, self).__init__(archive_path, logical_toc)
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\archive\writers.py", line 64, in __init__
    self._finalize()
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\archive\writers.py", line 96, in _finalize
    self.save_trailer(toc_pos)
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\archive\writers.py", line 439, in save_trailer
    tocstr = self.toc.tobinary()
  File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\archive\writers.py", line 264, in tobinary
    flag, ord(typcd), nm + pad))
struct.error: argument out of range

[bwoodsend]

Is pytorch enormous (>2GB)? How big is it if you build in one dir mode?

[rokm]

File "D:\python-3.6\lib\site-packages\pyinstaller-4.0+19fb799a11-py3.6.egg\PyInstaller\archive\writers.py", line 264, in tobinary
flag, ord(typcd), nm + pad))
struct.error: argument out of range

You are exceeding 2 GiB limit for onefile builds.

[huang229]

@rokm @bwoodsend Using the pyinstaller -D deploy_test.py packaging mode, we found that there are 6G files in total. This model can be packaged successfully, but it's not what I need. I have a closer look at the torch file in this mode. It has 5.4G, which is very huge. The previous torch 1.2 packaged torch file is less than 1G. I wonder why the new version of pytorch 1.8 has such a huge package. My intuition tells me that it seems that pyinstaller has packed everything in pytorch without analyzing the files that need to be packaged.

[rokm]

My intuition tells me that it seems that pyinstaller has packed everything in pytorch without analyzing the files that need to be packaged.

You are unfortunately correct - the pytorch hook does brute-force collect everything.

[huang229]

@rokm It's true. Whether it's pytorch1.8 or numpy1.19.5, as long as you import numpy or import torch, you will get a huge file. This is a big difference from the file size of the earlier versions of pyotch1.2 and numpy 1.16.1.

[rokm]

This is a big difference from the file size of the earlier versions of pyotch1.2 and numpy 1.16.1.

Sure, but the behavior of the hook hasn't changed - it's been collecting everything before, and it's collecting everything now. So what's changed is the size of PyTorch between the versions:

• 1.2 CPU-only: 317 MiB (with DLLs in lib accounting for 250 MiB)
• 1.2 GPU (CUDA 9.2): 1.16 GiB (with DLLs in lib accounting for 1.08 GiB)
• 1.8 CPU-only: 989 MiB (with DLLS in lib accounting for 959 MiB)
• 1.8 GPU (CUDA 11): 5.39 GiB (with DLLs in lib accounting for 5.37 GiB)

So there's nothing that can really be done about it - even if hook was smarter about collecting python part of pytorch, the huge part are actually DLLs (and large portion of those are CUDA DLLs), and those need to be collected anyway.

[huang229]

@rokm @bwoodsend Thank you very much for your help. I understand. This should be a new version of CUDA to compatible with hardware and sparse computing operations introduced many new modules, resulting in the huge CUDA module.