Windows: PyZMQ before v20 is not working anymore

[fleimgruber]

Please see #752 for context

python -m nuitka --version

1.1rc1
Commercial: None
Python: 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)]
Flavor: Unknown
Executable: C:\Users\LeimgruberF\dev\precert-poetry\.venv\Scripts\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10

The issue template said to *not* report against factory version, but I first saw this error on factory when testing for the fix in #752 (comment), also tested with Nuitka 1.0, same error.

Python installed via pyenv-win, Nuitka via Poetry from local factory branch. Poetry installs to a virtualenv.

On running the built exe, I get

Traceback (most recent call last):
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\main.py", line 17, in <module>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\precert\hil_device.py", line 10, in <module precert.hil_device>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\typhoon\api\hil\__init__.py", line 24, in <module typhoon.api.hil>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\typhoon\api\hil\stub.py", line 21, in <module typhoon.api.hil.stub>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\zmq\__init__.py", line 50, in <module zmq>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\zmq\backend\__init__.py", line 40, in <module zmq.backend>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\zmq\utils\sixcerpt.py", line 34, in reraise
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\zmq\backend\__init__.py", line 27, in <module zmq.backend>
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\zmq\backend\select.py", line 28, in select_backend
  File "C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\zmq\backend\cython\__init__.py", line 6, in <module zmq.backend.cython>
ImportError: LoadLibraryExW 'C:\Users\LEIMGR~1\dev\PRECER~3\MAIN~1.DIS\zmq\backend\cython\error.pyd' failed: The specified module could not be found.

The package in question there is "pyzmq", version 19.0.2.

python -m pip freeze

Arpeggio @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/19/8a/06/ddc9e6612816c7c1b692d912e7051c4d685959ecec8500d20448dc6199/Arpeggio-2.0.0-py2.py3-none-any.whl
atomicwrites @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/1b/09/43/f8f36d57c7cc6f6a038e649e082fa3ea231c686664e203db89b869ada7/atomicwrites-1.4.1.tar.gz
attrs @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/5b/25/94/5d165cd3190cb7dfdc98de86c28e8131fc6e19d83a504412664abffec0/attrs-21.4.0-py2.py3-none-any.whl
backports.functools-lru-cache @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/1a/04/f8/c0e432d1fa0d4e6c216b1234d97460af4830f09589be021ae209e12d31/backports.functools_lru_cache-1.6.4-py2.py3-none-any.whl
bokeh @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/09/f2/aa/9402881200f8a39ceb86adf6ca37506da8ff9f4b7a602e4a4a1e077af4/bokeh-2.4.3-py3-none-any.whl
certifi @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/b0/0d/df/4aa04d9b421fc5f1438b55d4b37516d21ef0ba40158bf7f9dfb28122ca/certifi-2022.6.15-py3-none-any.whl
charset-normalizer @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/bf/59/3f/6838dadc818c6595ae4a00c31af0da89039c1eaa370871d38072845e8d/charset_normalizer-2.1.0-py3-none-any.whl
colorama @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/28/7e/1b/4d5c7485d20628c199b003e9107afc53d57aea621842a1bc23c31c0da1/colorama-0.4.5-py2.py3-none-any.whl
commonmark @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/11/56/f6/d054064b623fab5c7e4420f60d931f49fea2dacdebe1dc991201010c84/commonmark-0.9.1-py2.py3-none-any.whl
idna @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/d5/04/80/9c17fd3240a37d12ff2ef042b0306aeb1abd2d8b95f150fd60be938352/idna-3.3-py3-none-any.whl
iniconfig @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/fa/b0/c6/10cfac68c9e6de9d2a1678366ca89fd9292b362c1760dbe758e41691cb/iniconfig-1.1.1-py2.py3-none-any.whl
Jinja2 @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/03/94/a8/08397f6fc37e01c8e8b6a33b514be99392ee894e8bff66c3a428238373/Jinja2-3.1.2-py3-none-any.whllxml @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/5d/4b/3a/ae95f8e4ea56da02ab76357adc6862caf6c4fc4fe455f62808266c87aa/lxml-4.9.1-cp39-cp39-win_amd64.whl
MarkupSafe @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/0d/39/c5/934bea1d1da42f89497a3cf6083991ea9da3f6c323d19dbb31655ae920/MarkupSafe-2.1.1-cp39-cp39-win_amd64.whl
msl-loadlib @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/46/06/ea/9ff86b84ae5500b0421ed8c63a683b1a6e2af0d38cd22f34d97531932c/msl_loadlib-0.9.0-py2.py3-none-win_amd64.whl
Nuitka @ file:///C:/Users/LeimgruberF/dev/Nuitka
numpy @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/9d/d6/73/cf1767b5e8acb38f454ea768ecb9878d2b2594db520b1a6528e28f71fa/numpy-1.23.1-cp39-cp39-win_amd64.whl
orderedset @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/71/88/fd/2d8695bb45e47eb6194b0b5eb7a765da5bdf91ef61e632ab44ab024df5/orderedset-2.0.3.tar.gz
packaging @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/47/3f/ce/b240169f7d8bef1ff24a0269b709721ce86543c2ec25e0b6adb2c2d7ac/packaging-21.3-py3-none-any.whl
pandas @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/73/c6/b7/4088fd96a04452d251429867daaa4d216b5f5f2df91467d5bc2b03a291/pandas-1.4.3-cp39-cp39-win_amd64.whl
Pillow @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/c3/8a/31/f73d842f35059cefbfaf966ab29231da26ecf0a3f946adbc8af2fe6be2/Pillow-9.2.0-cp39-cp39-win_amd64.whl
pluggy @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/81/78/ca/13f743a3628faf5a0b7f021efb45f2193acba3a13663d498f6b34bf02e/pluggy-1.0.0-py2.py3-none-any.whl
psutil @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/9c/54/8f/ba5eead2e3ffe354a5c6fc16fd33d1777621f6294ca5e383a0d70f14bb/psutil-5.9.1-cp39-cp39-win_amd64.whl
py @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/b3/5c/47/ba5a596e01a2b61fa2daa6a438252483ad8c04e6c99e5dc22eaf8a489a/py-1.11.0-py2.py3-none-any.whl
pyarrow @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/80/c0/fc/571f25ff79574a65333e07a91b4bf1a207dfcfad8843ea41ff85e3f0f6/pyarrow-8.0.0-cp39-cp39-win_amd64.whl
Pygments @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/7b/08/56/6328ccebb5deef75b77c4ac4fae65b8e5302a1c0b847754cd4c5063f9e/Pygments-2.12.0-py3-none-any.whl
pyModbusTCP @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/63/66/c9/cb882da5b7cc830efca9febb168961e94c578bec61ec0177621605639b/pyModbusTCP-0.1.10.tar.gz
pyparsing @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/c9/49/7b/434b2668226040d1097efcdddb802b28d05b02854b659ac17913a9a90c/pyparsing-3.0.9-py3-none-any.whl
pytest @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/a6/1c/db/e6f5a63ca07ec1971c4169e071b56ba077ceaf0abc2f69b803f2adcb2c/pytest-7.1.2-py3-none-any.whlpython-dateutil @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/53/f8/2a/7d63ce15df7386e9536e83413453f8aa845b47fb425f05c4ca2fb231c3/python_dateutil-2.8.2-py2.py3-none-any.whl
python-docx @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/7f/3f/b0/ca05b61dd6a8beb8bc8317700154416271ddda4db5425c92e9d780cba7/python-docx-0.8.11.tar.gz
python-dotenv @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/26/7d/fc/00dba3061e0311441e0aa6225a9b3e9bca3485a9fefe7282a4cce094c5/python_dotenv-0.20.0-py3-none-any.whl
pytz @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/fb/60/46/e704d81037c87ab74c6677ae79ab43dda85c31e3ae38f53c44486b593a/pytz-2022.1-py2.py3-none-any.whl
PyYAML @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/b9/e9/82/65a26ab81c9c52760322f10f23af03c060ce99f121aaa0422853fcef38/PyYAML-6.0-cp39-cp39-win_amd64.whl
pyzmq @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/4a/cf/c4/4e4e8e7934f239701fa22133d24bfeab614ebc45e15939b27f5006c20a/pyzmq-19.0.2-cp39-cp39-win_amd64.whl
requests @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/96/68/40/1b2019f1ca6a6d07b2a3107daab4dca1e11111be3af28a435f7cd0463a/requests-2.28.1-py3-none-any.whl
rich @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/b3/da/06/f753ae26f3cc205a3def96a552e410661396615bece9067caaeadca921/rich-12.5.1-py3-none-any.whl
six @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/08/9f/47/c16ae03035fc69eaf100ea39657a49baaeef714e25a52575710c34cd48/six-1.16.0-py2.py3-none-any.whl
textX @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/eb/cb/04/08ae6eebd273545cf9d163a84a6d18e60bcd990b27e48060b9a83889ce/textX-2.3.0-py2.py3-none-any.whl
tomli @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/73/7c/d9/9f2752fc5b05f9176c6f3adc6484be1cec75a68925b8c5fe39d6493a07/tomli-2.0.1-py3-none-any.whl
tornado @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/da/94/d9/59498339449a8bdb55d4866c51140af53c7b4eb54b11fbbeebe66c6b68/tornado-6.2-cp37-abi3-win_amd64.whl
Typhoon-HIL-API @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/79/41/ec/b2339790c721f284fed6ad5bd7cc8ab4a11f4bf93e9a083bd444f9c23b/Typhoon_HIL_API-1.19.0-py2.py3-none-any.whl
typing_extensions @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/c7/a2/3c/b8e5224a09b65b04693c613a52a41d6161affcaca091fbb647520b186c/typing_extensions-4.3.0-py3-none-any.whl
urllib3 @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/30/26/bc/2eb29ccbecde8a15ec98499d4c18d6cd2316febdce851786c0cca22bde/urllib3-1.26.11-py2.py3-none-any.whl
win-inet-pton @ file:///C:/Users/LeimgruberF/AppData/Local/pypoetry/Cache/artifacts/ae/0c/24/934002c9e790044f44e537fd0cce1925904b1e67f184121bfabac1b678/win_inet_pton-1.1.0-py2.py3-none-any.whl

Reproducing example (main.py):

import typhoon.api.hil as hil

Nuitka command used

poetry run python -m nuitka --standalone --mingw --show-scons --plugin-enable=numpy .\main.py

[kayhayen]

Thanks, I hope I didn't break the zmq plugin, this ought to be working.

[kayhayen]

So, I was using pyzmq==22.3.0 successfully just now. Probably your old version has other implicit dependencies.

[fleimgruber]

Thanks for testing, with pyzmq==22.3.0 I don't see the error. The old pyzmq version was due to another dependency (https://pypi.org/project/Typhoon-HIL-API/) that since upgraded to pyzmq>=22.3.0 as well. Solved for me.

[kayhayen]

I still suspect, that it's a regression, there was code in ZMQ plugin, that deals with this:

        if module_name == "zmq.libzmq" and isWin32Windows():
            # TODO: Very strange thing for zmq on Windows, needs the .pyd file in wrong dir too. Have
            # this done in a dedicated form somewhere.
            shutil.copyfile(
                os.path.join(dist_dir, "zmq\\libzmq.pyd"),
                os.path.join(
                    dist_dir, "libzmq" + getSharedLibrarySuffix(preferred=True)
                ),
            )

I managed to reproduce the issue, and will have a look, if that file copy would solve it. The access for plugins to the folder got removed, but of course this would still be doable. I guess it's only needed for pyzmq 19, and not supporting it anymore is a regression, therefore. I want of course software to work without up or downgrading, so to me this is relevant still. Should also be easy to migrate.

[kayhayen]

This is fixed on the factory branch, but needed new features in the Yaml config, so backporting to 1.0 is really out of the question, but I guess I will go for a quick 1.1 release anyway.

[kayhayen]

https://nuitka.net/doc/factory.html

[fleimgruber]

I will test this with 1.1 then, as the Python env with pyzmq 19.0.2 is in a conda env where I am not able to install from a local Nuitka with factory. Will wait for conda-forge to pick up Nuitka 1.1, test and report back.

[kayhayen]

Part of current develop and will be in 1.1 release.

[kayhayen]

This is part of the stable release 1.1 that I just made.