-
Notifications
You must be signed in to change notification settings - Fork 256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add cibuildwheel GHA #1283
base: master
Are you sure you want to change the base?
add cibuildwheel GHA #1283
Conversation
b228041
to
49a7d4e
Compare
Ubuntu wheels are using
macOS is passing with the libs |
What about setting the cibuildwheel image to use But, in the existing wheel generation process (https://github.com/MacPython/netcdf4-python-wheels), netCDF and its dependencies are compiled from source, so currently netCDF 4.9.2 is being bundled for the wheels on PyPI.
EDIT: I didn't read in the original description that you already suggested building the dependencies from source |
I've done something similar for my nc-complex package: CI workflow, cibuildwheel config in the I used Note that I use my fork of netcdf-C because I'm waiting for PR #2762 to fix finding HDF5 correctly from CMake. Once that's in, using the lastest |
be9c828
to
214cbb6
Compare
I'll try to focus on building hdf5 and netcdf-c over the next week There are a few options I want to try:
As a conda-forge founder, core member, and maintainer of many geo/met/ocean packages, I'm heavily biased towards 2. What do you think @ZedThree, @jswhit, and @dopplershift? * we already do something similar for the Windows packages and I want to move the current Windows builds to this, regardless of the Linux and macOS choice, to keep things in a single repository. |
I suggest using I would definitely try avoiding building HDF5 from scratch if possible, as you might find you end up building a lot more of the stack! |
Looks like their hdf5 has |
What about the approach of using conda built libs? (ocefpaf/netcdf4-win-wheels#6) If you want to enable all the features of hdf5/netcdf-c it requires a lot of dependencies to be built - don't think we want to do all that here. |
Option 2 is a "modern" version of that where the main advantage is that it uses cibuildwheel+conda-forge clibs, and keep things in this repo, no need for third party ones. I'll try that and see if it works. |
ac1a5ab
to
7579696
Compare
CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "delvewheel repair -w {dest_dir} {wheel}" | ||
CIBW_TEST_COMMAND: > | ||
python -c "import netCDF4; print(f'netCDF4 v{netCDF4.__version__}')" | ||
&& xcopy {project}\\test . /E/H |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The {project}
directory is on Drive D:
, not C:
and for some odd reason the PATH variables change if you switch drives on Windows. It is easier just to copy them to C:
.
.github/workflows/cibuildwheel.yml
Outdated
CIBW_TEST_COMMAND: > | ||
python -c "import netCDF4; print(f'netCDF4 v{netCDF4.__version__}')" | ||
&& xcopy {project}\\test . /E/H | ||
&& python -m pip install --upgrade numpy cython packaging |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can probably drop numpy, not sure why it wasn't installed with the wheel. Cython and packaging are test dependencies and we can figure out a more elegant way to handle this later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move them to pyproject.toml
?
[project.optional-dependencies]
tests = [
"Cython",
"packaging",
]
5891d1b
to
0b5a475
Compare
Windows wheels using conda package is working 🎉 I'll try the same for Linux and macOS next week. Time to clock out a bit. |
203bc1d
to
371dc0f
Compare
@jswhit everything is passing but the devil is in the details as usual. I'd love for more eyes here. We could also try to consolidate some of the many environment variables and options, netcdf-c version, etc but I prefer to leave that for another PR. |
set -ex | ||
|
||
|
||
download_and_build_netcdf() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ZedThree it turns out that building Linux with conda packages is more challenging than on Windows! Thanks for you approach, it worked just great here.
f5bddfe
to
63eae57
Compare
.github/workflows/cibuildwheel.yml
Outdated
@@ -70,7 +70,6 @@ jobs: | |||
&& sh .ci/build_deps.sh | |||
CIBW_BEFORE_BUILD_MACOS: brew install hdf5 netcdf | |||
CIBW_TEST_SKIP: "*_arm64" | |||
CIBW_TEST_REQUIRES: pytest cython |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like cibuildwheel still needs this regardless of it being declared in the pyporoject.toml. The levels of abstraction and isolation when building wheels is quite confusing. I guess we need to to a pip [test] install here. It has been to long that I used pip and I forget.
93eb57b
to
6305503
Compare
This looks great! Thank you @ocefpaf and @ZedThree. I'd like to get this merged soon so we can get a 1.6.5 release out with 3.12 wheels. The old MacPython wheel building repo is broken, and it looks like the multibuild package it relies on is no longer actively maintained. |
Just to say - for the record - we (the multibuild maintainers) think the build is broken because your multibuild is not up to date (see MacPython/netcdf4-python-wheels#23) - and we are maintaining multibuild - but no complaints from us for switching away. |
Things we need to consider before shipping these wheels (we can merge it and keep iterating on them though):
* Here we have:
This PR:
With multi-build
Edit: One can download the latest wheel build from this PR in https://github.com/ocefpaf/netcdf4-python/actions/runs/6617950814 The size of those macosx arm64 builds worries me but I cannot test them. @dopplershift do you have a M1/M2 machine? Could you test those? |
1.6.5 multi-build wheels now available |
Awesome! I'm not confident in switching to this just yet. I'll work on reducing the file size and test the macOS builds in the days. BTW, I did test both Linux and Windows they are working as expected. |
Apologies @matthew-brett - as expected, it was simply a dumb mistake on my part. I really appreciate all the work you've done on multi-build, we've used it here for a long time. |
@ocefpaf The 3.12 wheel is working fine here on my M2 in my 3.12 environment. Not sure what's up with the file size.
@jswhit Anything beyond getting things well-tested before these wheels appear on PyPI? |
@ocefpaf So while my MetPy test suite passed with no unexpected failures, I see the reason for the difference in file sizes:
Looks like quite a few libraries (like |
@dopplershift the netcdf-c library included in the multi-build wheels has support for compression plugins (blosc, zstd, lzma, etc), but the plugins themselves are not bundled. Without the plugins, I'm not sure how useful this is - otherwise I think these wheels are functionally equivalent. I've just uploaded 1.6.5 multi-build wheels to pypi, so there's no rush on this PR |
Thank for testing! And for figuring out the size difference. I guess that the system hdf5 there is probably the reason for this. I did like the small size though 😁 @jswhit I agree, no rush. I'll try to reduce the Linux wheel size, build hdf5 from source, and check what is needed for the plugins. That last one I don't really know what to do but it will be fun to figure it out. |
a25a107
to
9014863
Compare
This is just a proof of concept to see if we can build wheels here. We probably want to build out own hdf5 and netcdf-c b/c the ones from CentOS are quite old probably the source of the failures we are seeing in the tests here. See https://github.com/MacPython/netcdf4-python-wheels/blob/master/config.sh#L13-L26 for the versions used in the muiltbuild.
xref.: #1204