Tables 3.8.0 #979
Tables 3.8.0 #979
Conversation
|
BTW, I have also disabled the generation of Win32 wheels in favor of Win64 only. For a rational, see this discussion. I fully agree with the reasons argued there, so I am in favor of discontinuing Win32 wheels. Sounds that good? |
OK for me |
|
Regarding the support for arm64 on OSX we had a lot of requests for it. |
It seems like the last comleted ci run on master (from 3 hours ago) did have (successful) wheels built for arm64 on macos (this run: https://github.com/PyTables/PyTables/actions/runs/3696416831/jobs/6260085086) - so i wonder if that's really the case. Was this fixed in the meantime on master? |
Yes. I think upgrading to latest Blosc sources did the trick. We have wheels for the major platforms now, so this is getting in good shape. |
I was able to install tables from the latest artifacts. But it is not working on my M1 machine. I think compiled blosc2 doesn't have arm64 binaries. Error is basically: |
I have been able to fix the issues with the Python-Blosc2 wheels, and they work fine for M1 Mac now. However, when I applied the same configuration to PyTables, I keep getting errors: You many want to test the latest wheel artifacts and see how it goes, but I don't think arm64 wheels work on any M1 Mac. However, I have tested that installing from sources (via e.g. Opinions? |
I was able to pass this error with setting HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.2_2 environment variable. I compiled from sources after that. And tests were running.
Tried latest artifacts again. Same error there. Tables don't have arm64 libraries. But they are available in blosc2 wheel as you said. So maybe if we include arm64 library in the tables wheel and pass HDF5_DIR environment to the installation we can solve this problem. |
the idea of a wheel is that there is no requirement to have other system dependencies installed. |
I meant it to be done in CI system which looks like it’s already there. |
Exactly. The thing is that the wheels for MacOSX arm64 are done exactly in the same way for both x86_64 and arm64. The fact that they work on Mac Intel, but not on Mac M1 makes me think that something is not right in the CIBuildWheel library (which, BTW, works great for python-blosc2). At any rate, in case we cannot fix that, and as said before, it is still fortunate that PyTables wheels can be built flawlessly in an Mac arm64, so this should be not a show stopper. |
|
@aemr3 would you mind testing the latest wheels? https://github.com/PyTables/PyTables/actions/runs/3712304400 blosc sources were updated to the fixed version - so i suspect / hope this does fix it for PyTables, too. |
Unfortunately it didn’t work. I have tested them 3 days ago. I have tried to fix the issues on a fork but it didn’t work too. Github ci is too slow to work actively on it. Currently I don’t have an intel mac to replicate ci build process but working on it. Once I get access to an intel mac I will try my best to fix the issues. In the meantime I think @FrancescAlted can release the version, I don’t want to block the release. Maybe we could also get help from the community in the coming days. |
|
Sorry for not getting back for a bit, I have been sick. I'm still trying to understand the exact changes and what the issue is. But I looked around a little and I see that the python I wonder, since Secondarily. without fully understanding the issue on mac yet, could we not get the arm64 and x86_64 binaries from the blosc2 package, merge them into a fat binary and then distribute that? Then the OS should correctly pick the right arch. |
|
Ok, I looked on mac and the I think I know the issue though. On Mac, cibuildwheel does not create a docker container for the build. And because now both x86 and arm builds happen in one go, rather then in 2 different CI runs, the wrong libraries are included for the arm run. There's also an issue where now the cache is shared between x86 and arm builds which potentially is part of this (or a future issue). The fix should be to make them run separately in the matrix so that the CI spins up a different vm for each build. And making sure the cache is not shared: #982 should do it. |
|
On second thought, my PR alone won't fix the issue. That's because when pip installs blosc2, from which we get the binary, pip installs it for the host arch, which is x86_64 and not arm64. So that's the binary that we repackage as well. Instead we need to do |
|
Ok, I double checked that your latest build includes x86 binaries for blosc2, not arm64. So I wonder if I am doing something incorrect by using this hack: https://github.com/PyTables/PyTables/blob/master/setup.py#L839-L848 |
|
A curious thing that I noticed is that, when building wheels for MacOSX arm64, all the required wheels (and not only the blosc2 ones) choose the x86_64 arch instead (see section "Running before_build..." in https://github.com/PyTables/PyTables/actions/runs/3740325873/jobs/6349927213). No idea why. |
Right, exactly. Pip will always install the wheels for the host arch. The host arch on github CI is x86_64 because github does not offer arm64 machines. Therefore, pip downloads and installs x86_64 wheels, from which we get the blosc2 binary and repackage into our arm64 wheel that we're trying to build. We basically have two options:
I feel like 1 is a more appropriate approach because we can just let |
|
Here's a test of how option 1 would work. I installed tables and uninstalled blosc2, and things work: (venv) me263-2:Downloads matte$ python3
Python 3.10.9 (v3.10.9:1dd9be6584, Dec 6 2022, 14:37:36) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tables
>>> tables.which_lib_version("blosc2")
(('2.6.1.dev', '$Date:: 2022-12-08 #$'), '2.6.1.dev', '$Date:: 2022-12-08 #$')Then I deleted the tables bundled (venv) me263-2:Downloads matte$ python3
Python 3.10.9 (v3.10.9:1dd9be6584, Dec 6 2022, 14:37:36) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tables
Traceback (most recent call last):
File "/Users/matte/Downloads/venv/lib/python3.10/site-packages/tables/__init__.py", line 26, in <module>
cdll.LoadLibrary(blosc2_lib)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
return self._dlltype(name)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(libblosc2.dylib, 6): image not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/matte/Downloads/venv/lib/python3.10/site-packages/tables/__init__.py", line 28, in <module>
cdll.LoadLibrary(os.path.join(current_dir, blosc2_lib))
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
return self._dlltype(name)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/matte/Downloads/venv/lib/python3.10/site-packages/tables/libblosc2.dylib, 6): image not foundSo I installed blosc2 and made sure its dylib was on the path (manually) and it works again: (venv) me263-2:Downloads matte$ pip install blosc2
Collecting blosc2
Using cached blosc2-0.6.6-cp310-cp310-macosx_10_9_x86_64.whl (3.9 MB)
Requirement already satisfied: msgpack in ./venv/lib/python3.10/site-packages (from blosc2) (1.0.4)
Installing collected packages: blosc2
Successfully installed blosc2-0.6.6
(venv) me263-2:Downloads matte$ export DYLD_FALLBACK_LIBRARY_PATH=venv/lib
(venv) me263-2:Downloads matte$ python3
Python 3.10.9 (v3.10.9:1dd9be6584, Dec 6 2022, 14:37:36) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import blosc2
>>> import tables
>>> tables.which_lib_version("blosc2")
(('2.6.1.dev', '$Date:: 2022-12-08 #$'), '2.6.1.dev', '$Date:: 2022-12-08 #$')
>>> We could set the path appropriately automatically when blosc2 is initialized if we were happy with relying on blosc2. Otherwise, I could look into option 2. |
Ok. But we need the libblosc2.dylib* distributed in the blosc2 package to compile PyTables, so unless I am missing something, we probably need to go in the 2 direction. Perhaps something like: should work? |
Hey, this looks pretty cool actually. So, as far as I understand this, tables would come with its own version of Blosc2 C library, and even if you put manually another version in the dylib path first, this continue to work? I'd buy it indeed. Please send a PR.
What you achieved for option 1 looks fine to me already. Which advantage would have option 2 then? |
It's not quite that. Tables at build time will have installed the python blosc2 package. This is what it builds against. However, we won't copy the blosc2 binary from that package and re-package it in our wheel. Instead, we only have to make sure blosc2 is installed at runtime and that we can use the same binary we built against (or a compatible with same major/minor version) to run against. This allows us to completely offload the blosc2 binary to the python blosc2 package. They correctly packaged already for arm/x86 so it wouldn't require any work from our end to maintain blosc2. Please see #983. |
As discussed in: #979
With mkl installed, setting this environment variable will trigger a "TypeError: an integer is required (got type str)" from numexpr's `set_vml_num_threads` call that comes from the last line of File.__init__, since environment variables are strings.
|
Ok. This looks good. Let's the party begin. |
After fixing several issues related with a bad location of the libblosc2 library in different platforms, I have started this PR for releasing 3.8.0.
Right now, we are able to produce wheels (see current artifacts) that are able to install and run the test suite in both Linux and Mac OSX. However, it looks like the wheels for Windows are not able to load the extensions (I suspect that I am not including libblosc2 properly here); as my access to Win machines is quite restricted, I'd appreciate help from somebody more knowledgeable.
Also, I have disabled the generation of wheels for arm64 and macosx because it was not compiling correctly. I think this is due to the fact that the current build mechanism in setup.py is not aware of Mac on arm64, and I don't have cycles enough to address this right now. As Mac OSX can emulate x86 perfectly on arm64 platforms, I consider this a minor issue, and propose to go ahead without that. But it would be great if anybody wants to dig more into this.
I am currently pointing at having a release during this week, provided that we can fix the issue with the Windows platform.
@matham @avalentino @tomkooij @eumiro