release wheels for python 3.9

[xmatthias]

Currently, pypi does not contain wheels for python 3.9.

I know python 3.9 just came out - but it would be great if the last release (or a new release - considering the last release has been over a year ago) could be updated to include Wheels for python 3.9 to simplify installation.

[amotl]

Dear @avalentino, @tomkooij, @PyTables and @numfocus,

first things first: Thank you so much for conceiving and maintaining PyTables.

Binary wheels for Python 3.9 for all of lxml, h5py, Pandas and Numpy are already available on PyPI these days. At @earthobservations, we just have been able to upgrade to their most recent releases on Wetterdienst and it works like a charm, see earthobservations/wetterdienst#202 (comment).

For NumExpr, we also just outlined the situation at pydata/numexpr#367.

It would be awesome if you could also build and upload cp39 artefacts to PyPI to make it easier for downstream packages and their users to upgrade to Python 3.9.

With kind regards,
Andreas.

[avalentino]

OK i managed to build wheels for linux locally.
@tomkooij if you do not have plans to build wheels via pytables-wheels I could try to upload mines.

[amotl]

Dear Antonio,

thanks a bunch already. While I am just getting started to learn about the package publishing infrastructure behind the different Python packages for scientific computing, there is nothing that I could really contribute here other than pointing out what others are doing, especially in the light of switching away from Rackspace and staging packages through either AppVeyor/Travis/Anaconda or GitHub Actions these days.

In this manner, the discussion over at pydata/numexpr#367 has progressed and you might want to look into what @robbmcleod and @mattip (thank you!) have been doing / are suggesting there.

Specifically, cibuildwheel by @joerick, @YannickJadoul and contributors seems to be a nice way to build wheels these days and supports the full stack of Python versions and CI platforms.

With kind regards,
Andreas.

[avalentino]

Dear Andreas,
thank you for the pointers. I also used cibuildwheel to build wheels locally. I agree with you, it is a very nice tool.
I have just uploaded wheels for GNU/Linux.
Unfortunately I do not have too much time in this moment to setup a CI job to generate and upload wheels automatically.
If you are able to generate missing wheels please feel free to share them in some way or contact me in private and I will upload them to PyPi.
In any case if you are interested to collaborate with the PyTables project I think we can give you commit access to the repo.

[amotl]

Dear Antonio,

well, I will be happy to provide a wheel for macOS 10.13.6 using Python 3.9.0. Saying that, I am a newcomer here so I will probably have to learn from you first.

I just familiarized myself a little bit with the matter by cloning the repository, building it and running the tests on Python 3.8.6. It worked fine so far.

Now, I ran a build using python setup.py bdist_wheel within a pyenv'ed Python 3.9.0 virtualenv after switching to the v3.6.1 tag. The outcome is tables-3.6.1-cp39-cp39-macosx_10_13_x86_64.whl.

Do you believe using the internal blosc sources (see below) is fine?

With kind regards,
Andreas.

---

$ pip install cython numpy
Collecting cython
  Using cached Cython-0.29.21-py2.py3-none-any.whl (974 kB)
Collecting numpy
  Using cached numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl (15.4 MB)
Installing collected packages: cython, numpy
Successfully installed cython-0.29.21 numpy-1.19.4

---

$ python setup.py bdist_wheel
* Using Python 3.9.0 (default, Nov 17 2020, 23:27:37)
* USE_PKGCONFIG: True
* Found HDF5 headers at ``/usr/local/include``, library at ``/usr/local/lib``.
* Found LZO 2 headers at ``/usr/local/include``, library at ``/usr/local/lib``.
* Skipping detection of LZO 1 since LZO 2 has already been found.
* Found bzip2 headers at ``/usr/include``, library at ``/usr/lib``.
/var/folders/06/w9pzygdj7vx53n_0l9q_lhph0000gn/T/blosc_list_compressors2dldxok7.c:2:5: warning: implicit declaration of function 'blosc_list_compressors' is
      invalid in C99 [-Wimplicit-function-declaration]
    blosc_list_compressors();
    ^
1 warning generated.
ld: library not found for -lblosc
clang: error: linker command failed with exit code 1 (use -v to see invocation)
* Could not find blosc headers and library; using internal sources.

---

$ python -m tables.tests.test_all
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PyTables version:    3.6.1
HDF5 version:        1.12.0
NumPy version:       1.19.4
Numexpr version:     2.7.1 (not using Intel's VML/MKL)
Zlib version:        1.2.11 (in Python interpreter)
LZO version:         2.10 (Mar 01 2017)
BZIP2 version:       1.0.6 (6-Sept-2010)
Blosc version:       1.16.3 (2019-03-08)
Blosc compressors:   blosclz (1.1.0), lz4 (1.8.3), lz4hc (1.8.3), snappy (1.1.1), zlib (1.2.8), zstd (1.3.8)
Blosc filters:       shuffle, bitshuffle
Cython version:      0.29.21
Python version:      3.9.0 (default, Nov 17 2020, 23:27:37)
[Clang 10.0.0 (clang-1000.10.44.4)]
Platform:            macOS-10.13.6-x86_64-i386-64bit
Byte-ordering:       little
Detected cores:      8
Default encoding:    utf-8
Default FS encoding: utf-8
Default locale:      (None, UTF-8)

[amotl]

Hi again,

Unfortunately I do not have too much time in this moment to setup a CI job to generate and upload wheels automatically.

I am in the same situation, but maybe I can spare some time for that. At pydata/numexpr#367 (comment), @mattip shared some recommendations.

While he said that NumPy uses Travis CI, numpy/numpy#17738 shows that all Linux x86 jobs moved to GHA already and they are also planning to do this for the Windows/macOS builds.

Should PyTables go the same route and maybe already use cibuildwheel for that job?

With kind regards,
Andreas.

[mattip]

Just to clarify: my intention in the comment you linked to was to point out that NumPy still uses travis-ci for some non-x86 platforms.

[amotl]

All right, thanks for clarifying this, Matti! I believe that ARM64/AArch64 will gain more popularity in the future and I am looking forward to learn how to build packages for this by reusing what you are doing at NumPy [1]. Thanks for all your suggestions!

From looking at the NumPy repository, am I right to assume you are not using cibuildwheel yet?

[1] https://github.com/numpy/numpy/blob/master/.travis.yml

[YannickJadoul]

Not a lot of times to get into a library/build process that I don't know, but if there are questions about cibuildwheel, happy to answer them :-) (here or on Gitter, or wherever suits best)

Apart from the cibuildwheel docs, I'd suggest just starting out with the examples and seeing what goes wrong. If setup.py bdist_wheel or pip wheel . don't need too many weird dependencies or custom steps to prepare, things should be fairly smooth.

[amotl]

Thanks for your encouragement, Yannick!

As the whole build machinery of @numpy looks rather complex, I believe it might not yet be possible to easily go along with cibuildwheel. However, here with PyTables, I just had to invoke pip install cython numpy before being able to build with python setup.py bdist_wheel.

So, I will be happy to apply the cibuildwheel GHA example here if @avalentino does not have any objections.

[amotl]

Dear @mattip,

accidentally, I just discovered how SQLAlchemy runs tests and builds wheels for ARM64/AArch64 on GHA through Docker/QEMU, see sqlalchemy/sqlalchemy#5480 by @odidev.

With kind regards,
Andreas.

[amotl]

Dear @YannickJadoul and @mattip,

through tornadoweb/tornado#2923, I also just discovered the Run-On-Arch GitHub Action by @uraimo, which promises to execute commands on armv6, armv7, aarch64, s390x and ppc64le. At its core, it also uses non-x86 emulation in Docker via QEMU.

However, @bdarnell said at tornadoweb/tornado#2923 (comment) that

the sticking point is that cibuildwheel doesn't know how to build arm64 wheels anywhere but travis-ci.

Maybe this is something you could look into, Yannick?

With kind regards,
Andreas.

[avalentino]

Dear @amotl personally I think that GHA + cibuildwheel is the way to go.
Don't know if other developers have comments about this.

The key point for PyTables builds is the setup of C/C++ libraries necessary to build.
Blosc is not a problem because we have the source in the repo but:

• hdf5
• zlib
• bzip2 (optional)
• lzo2 (optional)

must be installed on the system before building wheels.

Fro linux I just used

CIBW_BEFORE_ALL_LINUX="yum install -y hdf5-devel zlib-devel bzip2-devel lzo-devel"

For osx it seems to me that pytables-wheels uses brew,
Not sure what happens for windows builds.

My recommendation is to go by steps: setting up the base PR with linux only (please mark it as [WIP]), and then adding other OSs and architectures gradually (eventually also in separate PRs).

Thanks a lot for your effort.

[amotl]

Dear Antonio,

thanks again for your suggestions. According to [1], Homebrew is also available within GHA for easily installing out of band dependencies on macOS as well as Chocolatey for Windows environments, see also [2].

With kind regards,
Andreas.

[1] https://www.edwardthomson.com/blog/github_actions_4_installing_tools.html
[2] https://github.com/numpy/numpy/blob/v1.19.4/azure-steps-windows.yml#L31-L37

[avalentino]

Very nice!
Thanks for the pointers @amotl.

[YannickJadoul]

@amotl

However, here with PyTables, I just had to invoke pip install cython numpy before being able to build with python setup.py bdist_wheel.

That should be pretty straightforward to do with CIBW_BEFORE_BUILD, indeed.

The mentioned C libraries are potentially a bit more tricky, but should still work similarly. The issue might be on how to package these into a wheel for Windows (since Windows doesn't have an auditwheel or delocate variant (yet?); related to pypa/cibuildwheel#459 as well). Any idea how that happens currently, to provide Windows wheels?

through tornadoweb/tornado#2923, I also just discovered the Run-On-Arch GitHub Action by @uraimo, which promises to execute commands on armv6, armv7, aarch64, s390x and ppc64le. At its core, it also uses non-x86 emulation in Docker via QEMU.

That's pretty cool, thanks! Maybe this is worth opening a cibuildwheel issue for (or add this to an already existing issue, like pypa/cibuildwheel#364?). It should not be all too hard to enable cibuildwheel for this, I think. However, I would (as @avalentino rightfully suggests) take this bit by bit, and start out with the basic GHA example and adapt it gradually to build PyTables wheels. Already having manylinux, macOS, and Windows wheels for Intel architecture seems to be a big step forward?

[amotl]

Dear Antonino and Yannick,

However, I would (as @avalentino rightfully suggests) take this bit by bit, and start out with the basic GHA example and adapt it gradually to build PyTables wheels. Already having manylinux, macOS, and Windows wheels for Intel architecture seems to be a big step forward?

Sure, sorry for hijacking this thread a bit. But there have been so much valuable information coming together with nice people caring about these topics (thanks!), so I believed it to be a good idea to share these findings with you within this very context.

That's pretty cool, thanks! Maybe this is worth opening a cibuildwheel issue for (or add this to an already existing issue, like pypa/cibuildwheel#364?).

Sure, will do.

Personally, I think that GHA + cibuildwheel is the way to go.

Now, I will actually have to find some spare time to work on the low hanging fruits into the direction we are aiming at ;].

With kind regards,
Andreas.

[avalentino]

FYI I have setup a basic GHA workflow for building wheels on linux. It would be nice to have also builds for windows and macosx.

[amotl]

Hi again,

based on @eumiro's #834 and @avalentino's b71ca54 (thanks!), I wanted to let you know that I am currently working on adding build targets for macOS and Windows on behalf of [1]. The outcome is promising so far [2], I hope to be able to submit a patch soon after cleaning up and documenting the workflow a bit more.

With kind regards,
Andreas.

[1] master...daq-tools:gha-macos-windows-ci
[2] https://github.com/daq-tools/PyTables/actions

[xmatthias]

Any news regarding the wheels for windows / macOS?

Not having these makes using tables (with python 3.9) on these platforms a real pain.

[tomkooij]

@amotl Ok, this is a VERY late reply. I just stumbled upon this.

I invited you (and antonio as he is the only owner with an account on pypi-test).

Sorry about not replying to this. Hope this still is usefull.

[avalentino]

Thanks @tomkooij.
You made a lot of work on wheel in the past.
Would you like to review this PR? Do you have time?

[pedrocamargo]

How is this issue coming along any help needed? Would it make sense to manually upload Chris Golhke's wheels to PyPi until this is resolved?

[joaoe]

Hi.
Just a friendly reminder that releases of python since 3.8 are not supported by pytables on mac nor windows, and you are forcing projects into technical debt by having to keep python 3.7.

[RalfG]

To people looking for an update on this issue, @xmatthias has done some great work already to get 3.9 and 3.10 wheels:

• Run CI against python 3.10 #917
• Build linux Wheels for 3.10 #918
• Enable CI on windows 3.9 and 3.10 #923
• Add windows wheels and independent test to GHA #926
• Add CI wheel build for MacOS #927

Currently, open issues are:

• Remove MacOS arm64 wheel  #929
• Build hdf5 from source when building wheels #930

[avalentino]

@xmatthias since #923 is now merged I think that this issue can be closed.
Please feel free to reopen if I missed something.

[joaoe]

@avalentino Hi. The 3.9 Windows wheels are not published at https://pypi.org/project/tables/3.6.1/#files .
There are some old 3.6, 3.7, 3.8 (October 2019) wheels but I'm not sure where those came from.
What is missing ? Thank you.

[avalentino]

The machinery to generate wheels is now fixed.
We are going to make a new release in few days.