Autobahn and pyInstaller not compatible anymore

[nvxluca]

As results on Pypy, Pyinstaller is not compatible anymore with Autobahn. Is there a plan to fix this compatibility? Thanks

[oberstet]

As results on Pypy

What results?

Apparently, this issue - if it actually is one - misses any logs, versions, etc. Don't get me wrong, but currently, it's pretty useless, you might be new to open-source, so no offense;)

I recognize you've been directed here which is good, thanks @om26er! this repo should probably have an issue template asking for the usual / relevant stuff. Do you happen to have one?

[nvxluca]

No problem and thank you for your answer.
No, I don't see any issue template.
I try to provide more info below:

• Python version is 3.12.0
• Pip version is 23.3.1
• Autobahn version is 23.6.2
• When I try to bundle the project with pyinstaller, the process fails at the as below:
  import(package)
  File ...\Lib\site-packages\autobahn\xbr_init_.py", line 149, in
  from autobahn.xbr._schema import FbsSchema, FbsObject, FbsType, FbsRPCCall, FbsEnum, FbsService, FbsEnumValue,
  File ...\Lib\site-packages\autobahn\xbr_schema.py", line 45, in
  from zlmdb.flatbuffers.reflection.Schema import Schema as _Schema
  No module named 'zlmdb'60387 INFO: Extra DLL search directories (AddDllDirectory): ...

When I try to install the zlmdb package, it fails (I tried also with different versions):

Collecting lmdb>=1.4.0 (from zlmdb)
Using cached lmdb-1.4.1.tar.gz (881 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [27 lines of output]
py-lmdb: Using bundled liblmdb with py-lmdb patches; override with LMDB_FORCE_SYSTEM=1 or LMDB_PURE=1.
Traceback (most recent call last):
File "", line 92, in
ModuleNotFoundError: No module named 'patch_ng'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "...\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
      main()
    File "...\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "...\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "C:...AppData\Local\Temp\2\pip-build-env-zkoqs4x_\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\...\AppData\Local\Temp\2\pip-build-env-zkoqs4x_\overlay\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "C:\Users...AppData\Local\Temp\2\pip-build-env-zkoqs4x_\overlay\Lib\site-packages\setuptools\build_meta.py", line 480, in run_setup
      super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
    File "C:\...\AppData\Local\Temp\2\pip-build-env-zkoqs4x_\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 94, in <module>
  Exception: Building py-lmdb from source on Windows requires the "patch-ng" python module.
  [end of output]

The point now is that the patch-ng is already installed with version 1.17.4

Regarding your question, when I go to Pypi and search for autobahn (https://pypi.org/project/autobahn/) I can see the following:

When I click on the "pyinstaller failing" button to get more info I'm redirected to the following url where I have the impression that Autobahn has currently some issue with pyinstaller:
https://github.com/crossbario/autobahn-python/actions?query=workflow%3Apyinstaller

As further info, pyinstaller 3.5 was succesfully running on my project with Python 3.7.4, autobahn 20.12.3.

Please let me know if more details are needed, thanks.

[oberstet]

No, I don't see any issue template.

yes, sure, that was only a hint / question to @om26er actually whether we want to have such a template in this repo here in general (independent of your specific issue) asking users to provide infos .. which I now did manually, but I like to automate stuff! ;)

Exception: Building py-lmdb from source on Windows requires the "patch-ng" python module.

ah, right, see, couple of things now are clear

you are using windows, which sucks, and issues like these are expected from time to time, not only with autobahn, but in general with lots of OSS, as many developers have moved to sane/decent OSs .. so if you can, I'd recomment format c:\ and install ubuntu 22.04

next, zLMDB is installed with Autobahn only if you install the all or xbr flavors (https://github.com/crossbario/autobahn-python/blob/master/setup.py#L118) - it is NOT required for Autobahn WebSocket or basic WAMP at all!

finally, the issue pops up on Windows and for zLMDB and probably should be hosted in that repo https://github.com/crossbario/zlmdb/ - BUT: now that you've opened it here, let's keep it here

Please let me know if more details are needed, thanks.

no, thanks, it's only on Windows, that's all I (me personally) need to know;)

here are you options in summary:

1. scrap Windows to save time and keep your sanity, install and use Linux
2. use the Linux box/environment that comes bundled with Windows these days (I forgot the name but Windows ships with a full/crippled Linux these days)
3. wait until someone cares about windows and works around windows defects in this issue/autobahn
4. wait for microsoft to fix windows or moves to linux themself
5. only do pip install autobahn, not pip install autobahn[all] or pip install autobahn[xbr]

sorry, but I hope this helps!

[nvxluca]

Thank you, this helps a lot my understanding but it's not yet solving at the current stage.
The option to install Autobahn WebSocket is perfect for me but when I run "pip uninstall autobahn" and then "pip install autobahn" I can see the autobahn folder deleting and then recreating with the xbr folder inside. I made this test because also after reinstalling autobahn with the only "pip install autobahn" command I had (and I still have) the same error above related to zlmdb.
As further info, after ran pip uninstall everytime, I tried already also to run "pip install autobahn[]" and "pip install autobahn[twisted]" (as recommended here: https://autobahn.readthedocs.io/en/latest/installation.html#windows-installation).
Am I doing something wrong?
(On the other side, the option to use the Linux environment is not immediately feasible in my case, but I'll take into account for the next future, thanks for your clear indication).

[oberstet]

For WebSocket using Twisted for networking only, you should be good with pip install autobahn[websocket, twisted]

If you want to use WebSocket compression (optional, quite exotic), add compress

On non-Windows, and on CPython (not PyPy), add accelerate

All the dist flags are here

autobahn-python/setup.py

Line 275 in 359f868

extras_require={

---

but when I run "pip uninstall autobahn" and then "pip install autobahn" I can see the autobahn folder deleting and then recreating with the xbr folder inside.

yeah, it's a small mess, specifically on windows, on standard or non-standard pythons;)

have a look at pip install --help .. stuff like "force reinstall" and "-U" .. in extreme, hard del the folders in your python lib folder or sth .. or even better: use virtual envs, and a new virtual env every time for such things!. I do use dozens of venvs all the time. anyways, can't explain all this stuff, but here

(cpy311_7) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ which python
/home/oberstet/cpy311_7/bin/python
(cpy311_7) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ python -V
Python 3.11.1
(cpy311_7) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ ~/cpy311
cpy311/     cpy311 /    cpy311_1/   cpy311_2/   cpy311_3/   cpy311_4/   cpy311_5/   cpy311_6/   cpy311_7/   cpy311b1/   cpy311b1_1/
(cpy311_7) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ ~/cpy311/bin/python -venv ~/cpy311_8
bash: /home/oberstet/cpy311/bin/python: Datei oder Verzeichnis nicht gefunden
(cpy311_7) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ ~/cpy311/bin/python3 -m venv ~/cpy311_8
(cpy311_7) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ source ~/cpy311_8/bin/activate
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ which python
/home/oberstet/cpy311_8/bin/python
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ python -V
Python 3.11.1
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ pip install autobahn^C
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ python -c "import autobahn"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'autobahn'
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ pip install autobahn
Collecting autobahn
  Downloading autobahn-23.6.2.tar.gz (480 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 480.8/480.8 kB 4.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting txaio>=21.2.1
  Using cached txaio-23.1.1-py2.py3-none-any.whl (30 kB)
Collecting cryptography>=3.4.6
  Using cached cryptography-41.0.5-cp37-abi3-manylinux_2_28_x86_64.whl (4.4 MB)
Collecting hyperlink>=21.0.0
  Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Requirement already satisfied: setuptools in /home/oberstet/cpy311_8/lib/python3.11/site-packages (from autobahn) (65.5.0)
Collecting cffi>=1.12
  Using cached cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (464 kB)
Collecting idna>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: txaio, pycparser, idna, hyperlink, cffi, cryptography, autobahn
  DEPRECATION: autobahn is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for autobahn ... done
Successfully installed autobahn-23.6.2 cffi-1.16.0 cryptography-41.0.5 hyperlink-21.0.0 idna-3.4 pycparser-2.21 txaio-23.1.1

[notice] A new release of pip available: 22.3.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ python -c "import autobahn"
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ python -c "from autobahn._version import __version__; print(__version__)"
23.6.2
(cpy311_8) oberstet@intel-nuci7:~/scm/typedefint/sub_ghz_dual_planar_helical$ 

[nvxluca]

I created a new virtual env but I see the folder xbr always creating when using the commands you suggested.
And I still have the same issue as I installed the autobahn including the xbr option. But I've understood now what's the issue, thank you very much.

[om26er]

I don't have a template, however to keep things super simple we should create a template that asks the basic

1. version of autobahn and the platform
2. If it's a bug
   • we need the steps to reproduce the bug
   • we need the logs if its a crash
3. If it's a feature request nothing special is needed just the details about the feature

I will come up with a PR to add an issue template and we may discuss there

[alex-ong]

I also came from pyinstaller 3.5 (got a dependabot to upgrade to 5.13.2)
I assume from reading above its related to PyPy change

	Python	Pyinstaller	Autobahn[twisted]	Works?
Before	3.7	3.5	??	Y
Working	3.11	5.13.2	21.3.1	Y
Other	3.11	5.13.2	23.6.2	Y

Note that latest gives a warning about mnemonic, then when i install that, it complains about base58

My application is for windows users; unfortunately i can't make all my users install linux 💯

[oberstet]

My application is for windows users; unfortunately i can't make all my users install linux

=) yeah, I feel your pain, unfortunately. having been trapped for many years since DOS 3.3, I escaped - personally - more than a decade ago .. but there is no escape in the industry.

anyways, trying to give some hints/tips nevertheless .. on Windows, my best tip is: only use the autobahn flavor that you need ... such as

pip install autobahn[websocket, asyncio]

this will not install xbr, or lmdb or such. it will not install twisted, but work from Python stdlib libraries (asyncio) only.

autobahn[websocket] itself is pure python only - it will run on any python on any windows

if you need twisted rather than asyncio, then you must get that working on windows

usually, this isn't a problem as twisted is quite mature and people look after it also on crap platforms (windows)

now, pyinstaller, and pyinstaller on windows is a whole other level of challenge of course

I'd recommend getting a trivial non-autobahn python example with only twisted + pyinstaller on windows working with the latest cpython / twisted that you want first

if you need autobahn[xbr] or any of those flavors requiring lmdb, then I'd test minimal twisted+lmdb wrapped with pyinstaller for windows first

be aware, you can easily waste many hours fiddling around with this. sorry, DOS 3.3 is still a thing for users, and format c:\ is my usual reply which isn't what they want to hear though;)