Numba 0.57.0rc1 Checklist

[stuartarchibald]

Numba 0.57.0rc1

• Merge to main.
  • "remaining Pull-Requests from milestone".
    • Add dict(*iterable) constructor #8334
    • Fix opcode "spelling" change since Python 3.11 in CUDA debug test. #8870
    • Remove use of compile_isolated from generator tests. #8879
    • Fix missing dependency guard on pyyaml in test_azure_config. #8880
    • Replace use of compile_isolated in test_obj_lifetime #8881
• Check Numba's version support table documentation. Update via PR if
  needed.
• Review deprecation schedule and notices. Make PRs if need be.
  • Remove obsolete deprecation notices #8864
  • Revise CUDA deprecation notices #8866
  • Any more?
• Merge change log changes.
  • Update CHANGE_LOG for 0.57.0rc1 #8869
• Create 0.57 release branch.
• Create PR against the release branch to make numba/testing/main.py
  to refer to origin/release0.57 instead of origin/main. Pin llvmlite and NumPy on release branch #8884
• Dependency version pinning on release branch
  • Pin llvmlite to >=0.40.0rc1,<0.41.0.
  • Pin NumPy if needed (1.24?)
  • Pin TBB if needed (no need; don't know what to pin as max)
  • Pin llvmlite and NumPy on release branch #8884
• Run the HEAD of the release branch through the build farm and confirm:
  • Build farm CPU testing has passed. numba_pynext_cpu_yaml_18 numba_yaml_333
  • Build farm CUDA testing has passed.
  • Build farm wheel testing has passed. numba_wheel_78
• Annotated tag 0.57.0rc1 on release branch (no v prefix).
• Build and upload conda packages on buildfarm (check "upload").
• Build wheels and sdist on the buildfarm (check "upload").
  numba_pynext_cpu_yaml_19
numba_wheel_79
numba_yaml_334
• Verify packages uploaded to Anaconda Cloud and move to numba/label/main. https://anaconda.org/numba/numba/files?version=0.57.0rc1
• Upload wheels and sdist to PyPI (upload from ci_artifacts).
• Verify wheels for all platforms arrived on PyPi. https://pypi.org/project/numba/0.57.0rc1/
• Initialize and verify ReadTheDocs build. (https://numba.readthedocs.io/en/0.57.0rc1/)
• Post announcement to discourse group and ping the release testers group
  using @RC_Testers. https://numba.discourse.group/t/ann-numba-0-57-0rc1-and-llvmlite-0-40-0rc1/1879
• Post link to Twitter.

Post Release:

• Clean up ci_artifacts by moving files to sub-directories
• Tag 0.58.0dev0 to start new development cycle on main.
• Update llvmlite dependency via PR to main, PR includes version updates
  to: (Bump llvmlite requirement to 0.41.0dev0 #8916)
  • setup.py
  • numba/__init__.py
  • docs/environment.yml
  • buildscripts/incremental/setup_conda_environment.sh
  • buildscripts/incremental/setup_conda_environment.cmd
  • buildscripts/condarecipe.local/meta.yaml
• Update release checklist template with any additional bullet points that
  may have arisen during the release (Update release checklist template #8925).
• Close milestone (and then close this release issue).

[stuartarchibald]

Update for those subscribed to this ticket: as can be seen above by the list of remaining PRs to be merged from the milestone, Numba is very close to being ready to branch from main for the 0.57 release series. Unfortunately, in testing of main last night, the build farm reported an intermittent segmentation fault (segfault) in the test suite on a few of the test jobs. As a result, creating the release branch will be temporarily on hold whilst the cause of the segfault is investigated and hopefully fixed. Once resolved this ticket will be updated. Thank you for your patience and understanding.

[stuartarchibald]

It took a while to isolate this segfault (#8841 (comment)), due to it taking 20 minutes to (intermittently) reproduce it. Issue #8873 tracks the problem and contains debugging and discussion. An update will be posted once a decision is made about this issue.

[stuartarchibald]

Update: #8873 has been debugged, understood, and is hopefully fixed in #8879.

[stuartarchibald]

Update: #8879 and #8881 were needed to fix the issue in #8873, a build and test run containing these patches was successful. The process to branch and build out the RC has started!

[OWKenobi]

thanks for fixing this!

from a noob point of view: how long from now on (rough estimate) till 0.57 will be available in the form of "pip install numba"? Also, will llvmlite be updated just with it?

[sklam]

RC1 wheels are posted. Use pip install --pre numba; or

pip install numba==0.57.0rc1

pip install llvmlite==0.40.0rc1

to get the specific versions exactly.

[sklam]

Conda packages are also available:

numba: https://anaconda.org/numba/numba/files?version=0.57.0rc1
llvmlite: https://anaconda.org/numba/llvmlite/files?version=0.40.0rc1

Installable with conda install -c numba numba=0.57.0rc1.

[slayoo]

FWIW, just reporting that pip install --pre numba gives us a valid installation on Linux and macOS with Pythons 3.7-3.11, while on Windows (Github Actions' windows-latest) it only works for Python 3.7, later versions seem to install OK, but when trying to import numba, there are the following errors:

   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\numba\__init__.py", line 69, in <module>
    from numba.core import config
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\numba\core\config.py", line 16, in <module>
    import llvmlite.binding as ll
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\llvmlite\binding\__init__.py", line 4, in <module>
    from .dylib import *
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\llvmlite\binding\dylib.py", line 3, in <module>
    from llvmlite.binding import ffi
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\llvmlite\binding\ffi.py", line 187, in <module>
    raise OSError(msg)
OSError: Could not find/load shared object file: llvmlite.dll
 Error was: Could not find module 'C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\llvmlite\binding\llvmlite.dll' (or one of its dependencies). Try using the full path with constructor syntax.

(see: https://github.com/numba-mpi/numba-mpi/actions/runs/4645911575/jobs/8222023577?pr=46)

[sklam]

Thanks for the report for the windows issue and the confirmation of the issues. I'm debugging it now.

[sklam]

The windows issue is llvmlite specific and is very likely another case of numba/llvmlite#650.
I have tested the wheels in a windows machine but it works fine. This indicates that it is likely a dependency problem. I'm trying to get a vanilla windows to test what it's missing. My current guess is that it requires the C++ redistributable package (see numba/llvmlite#650 (comment)).

[ldeluigi]

I installed the C++ redistributable but the problem still occurs. Even after reboot.

[sklam]

On a vanilla windows with python installed from python.org, I can replicate the llvmlite error. To fix it, I need to install C++ Redistributable packages for Visual Studio 2015, 2017, 2019, and 2022 and copy a zlib.dll next to the llvmlite.dll (as mentioned in numba/llvmlite#650 (comment)). I got the zlib.dll from miniconda. For conda users, this can be fixed by conda install zlib.

[Alpengreis]

@sklam

Had installed the related C++ Redistrutable already. Then I copied a zlib.dll next to llvmite.dll. It still doesn't work ...

PS: Win 10 with Python 3.11.3 ...

PPS: NOW IT WORKS, tried with another zlib.dll - thank you very much!!
I took the zlib.dll in right now current 1.2.13 version from XMedia Recode Portable (64 bit) ...

[eriknw]

I am finding that numba 0.57.0rc1 segfaults on Windows when installed with pip or conda for the following:

import numpy as np
from numba import njit

f = njit(lambda x: np.exp2(x))
f(np.int8(2))

I create the environments via

conda create -n py310pip -c conda-forge python=3.10
pip install numba==0.57.0rc1

and

conda create -n py310conda -c numba -c conda-forge python=3.10 numba

I have not found any other function for which this occurs.

I discovered this when testing python-graphblas. Here's an example traceback:

LLVM ERROR: Symbol not found: ldexpf

LLVM ERROR: Symbol not found: ldexpf
Fatal Python error: Aborted

Current thread 0x00000bb0 (most recent call first):
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\llvmlite\binding\ffi.py", line 152 in __call__
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\llvmlite\binding\executionengine.py", line 92 in finalize_object
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\codegen.py", line 1060 in wrapper
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\codegen.py", line 999 in _finalize_specific
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\codegen.py", line 797 in _finalize_final_module
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\codegen.py", line 765 in finalize
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\codegen.py", line 567 in _ensure_finalized
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\codegen.py", line 989 in get_pointer_to_function
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\ccallback.py", line 75 in compile
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\compiler_lock.py", line 35 in _acquire_compile_lock
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\numba\core\decorators.py", line 311 in wrapper
  File "D:\a\python-graphblas\python-graphblas\graphblas\core\operator\unary.py", line 235 in _build
  File "D:\a\python-graphblas\python-graphblas\graphblas\core\operator\unary.py", line 305 in register_new
  File "D:\a\python-graphblas\python-graphblas\graphblas\unary\numpy.py", line 152 in __getattr__
  File "D:\a\python-graphblas\python-graphblas\graphblas\tests\test_numpyops.py", line 83 in test_npunary
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\python.py", line 194 in pytest_pyfunc_call
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\python.py", line 1799 in runtest
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\runner.py", line 169 in pytest_runtest_call
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\runner.py", line 262 in <lambda>
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\runner.py", line 341 in from_call
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\runner.py", line 261 in call_runtest_hook
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\runner.py", line 222 in call_and_report
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\runner.py", line 133 in runtestprotocol
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\runner.py", line 114 in pytest_runtest_protocol
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\main.py", line 348 in pytest_runtestloop
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\main.py", line 323 in _main
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\main.py", line 269 in wrap_session
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\config\__init__.py", line 166 in main
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\_pytest\config\__init__.py", line 189 in console_main
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\pytest\__main__.py", line 5 in <module>
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\coverage\execfile.py", line 199 in run
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\coverage\cmdline.py", line 830 in do_run
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\coverage\cmdline.py", line 659 in command_line
  File "C:\Miniconda3\envs\graphblas\lib\site-packages\coverage\cmdline.py", line 943 in main
  File "C:\Miniconda3\envs\graphblas\Scripts\coverage-script.py", line 10 in <module>
graphblas/tests/test_numpyops.py::test_npunary 

[maxnoe]

Not sure if this is the right place, but I just wanted to drop a note that we for a relatively large project relying heavily on numba for some hot code paths tried out the rc1 on python 3.11 and our test suite passes with that, so great work!

[sklam]

@maxnoe, Thank you for testing the RC.

[oyvindln]

Also FYI Ubuntu 23.04 released with a version of numba 0.56 with changes for Python 3.11 support backported. Whether it's 100% stable I don't know.

http://changelogs.ubuntu.com/changelogs/pool/universe/n/numba/numba_0.56.4+dfsg-2/changelog

[benz0li]

@numba @stuartarchibald et al.: Thank you for all the hard work.

[stuartarchibald]

@numba @stuartarchibald et al.: Thank you for all the hard work.

@benz0li thank you for your kind words of support for the Numba folks, they are much appreciated!