Repository navigation
NumPy 1.24 support #8464
Description
Activity
Need to accommodate: numpy/numpy#22638
Need to update ufunc loop selection WRT numpy/numpy#22422 (#8538 tracks).
I've local patches for both those issues (not PR-ready, but having a quick test). Will push a branch if my local testing goes OK.
Reacted by Sebastian BergBranch is here (very simple workarounds for now!): https://github.com/gmarkall/numba/tree/np-124
Errors with my branch at present:
====================================================================== ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ] test sum with axis and dtype parameters over a whole range of dtypes ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws py_res = pyfunc(arr, axis=axis, dtype=out_dtype) File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws return a.sum(axis=axis, dtype=dtype) File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum return umr_sum(a, axis, dtype, out, keepdims, initial, where) TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation. ====================================================================== ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ] test sum with axis and dtype parameters over a whole range of dtypes ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws py_res = pyfunc(arr, axis=axis, dtype=out_dtype) File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws return a.sum(axis=axis, dtype=dtype) File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum return umr_sum(a, axis, dtype, out, keepdims, initial, where) TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation. ====================================================================== ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ] test sum with axis and dtype parameters over a whole range of dtypes ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws py_res = pyfunc(arr, axis=axis, dtype=out_dtype) File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws return a.sum(axis=axis, dtype=dtype) File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum return umr_sum(a, axis, dtype, out, keepdims, initial, where) TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation. ====================================================================== ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ] test sum with axis and dtype parameters over a whole range of dtypes ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws py_res = pyfunc(arr, axis=axis, dtype=out_dtype) File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws return a.sum(axis=axis, dtype=dtype) File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum return umr_sum(a, axis, dtype, out, keepdims, initial, where) TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation. ====================================================================== ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ] test sum with axis and dtype parameters over a whole range of dtypes ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws py_res = pyfunc(arr, axis=axis, dtype=out_dtype) File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws return a.sum(axis=axis, dtype=dtype) File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum return umr_sum(a, axis, dtype, out, keepdims, initial, where) TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation. ====================================================================== ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ] test sum with axis and dtype parameters over a whole range of dtypes ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws py_res = pyfunc(arr, axis=axis, dtype=out_dtype) File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws return a.sum(axis=axis, dtype=dtype) File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum return umr_sum(a, axis, dtype, out, keepdims, initial, where) TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation. ====================================================================== ERROR: test_hmax (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/cuda/tests/cudapy/test_intrinsics.py", line 836, in test_hmax arg1 = np.float(5.) File "/home/gmarkall/numbadev/numpy/numpy/__init__.py", line 284, in __getattr__ raise AttributeError("module {!r} has no attribute " AttributeError: module 'numpy' has no attribute 'float' ====================================================================== ERROR: test_hmin (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/cuda/tests/cudapy/test_intrinsics.py", line 848, in test_hmin arg1 = np.float(5.) File "/home/gmarkall/numbadev/numpy/numpy/__init__.py", line 284, in __getattr__ raise AttributeError("module {!r} has no attribute " AttributeError: module 'numpy' has no attribute 'float' ====================================================================== ERROR: test_comp_nest_with_dependency (numba.tests.test_comprehension.TestArrayComprehension) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_comprehension.py", line 369, in test_comp_nest_with_dependency self.check(comp_nest_with_dependency, 5) File "/home/gmarkall/numbadev/numba/numba/tests/test_comprehension.py", line 252, in check pyres = pyfunc(*args) File "/home/gmarkall/numbadev/numba/numba/tests/test_comprehension.py", line 365, in comp_nest_with_dependency l = np.array([[i * j for j in range(i+1)] for i in range(n)]) ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (5,) + inhomogeneous part. ====================================================================== ERROR: test_basic92 (numba.tests.test_stencils.TestManyStencils) Issue #3497, bool return type evaluating incorrectly. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 3110, in test_basic92 self.check_against_expected(kernel, expected, A) File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 811, in check_against_expected raise RuntimeError(str1 + str2) RuntimeError: The following implementations should not have raised an exception but did: ['parfors', 'parfors'] Errors were: parfors: Message: <class 'AttributeError'>: module 'numpy' has no attribute 'bool' parfors: Message: <class 'TypeError'>: unsupported operand type(s) for -: 'NoneType' and 'float' ====================================================================== ERROR: test_basic93 (numba.tests.test_stencils.TestManyStencils) Issue #3497, bool return type evaluating incorrectly. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 3140, in test_basic93 self.check_against_expected(kernel, expected, A, options={'cval': True}) File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 811, in check_against_expected raise RuntimeError(str1 + str2) RuntimeError: The following implementations should not have raised an exception but did: ['parfors', 'parfors'] Errors were: parfors: Message: <class 'AttributeError'>: module 'numpy' has no attribute 'bool' parfors: Message: <class 'TypeError'>: unsupported operand type(s) for -: 'NoneType' and 'float' ====================================================================== FAIL: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) test sum with axis and dtype parameters over a whole range of dtypes ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/support.py", line 855, in tearDown self.memory_leak_teardown() File "/home/gmarkall/numbadev/numba/numba/tests/support.py", line 830, in memory_leak_teardown self.assert_no_memory_leak() File "/home/gmarkall/numbadev/numba/numba/tests/support.py", line 839, in assert_no_memory_leak self.assertEqual(total_alloc, total_free) AssertionError: 424 != 423 ====================================================================== FAIL: test_hypot (numba.tests.test_mathlib.TestMathLib) ---------------------------------------------------------------------- RuntimeWarning: overflow encountered in scalar multiply During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_mathlib.py", line 518, in test_hypot self.assertRaisesRegexp(RuntimeWarning, AssertionError: "overflow encountered in .*_scalars" does not match "overflow encountered in scalar multiply" ====================================================================== FAIL: test_hypot_npm (numba.tests.test_mathlib.TestMathLib) ---------------------------------------------------------------------- RuntimeWarning: overflow encountered in scalar multiply During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/gmarkall/numbadev/numba/numba/tests/test_mathlib.py", line 523, in test_hypot_npm self.test_hypot(flags=no_pyobj_flags) File "/home/gmarkall/numbadev/numba/numba/tests/test_mathlib.py", line 518, in test_hypot self.assertRaisesRegexp(RuntimeWarning, AssertionError: "overflow encountered in .*_scalars" does not match "overflow encountered in scalar multiply" ---------------------------------------------------------------------- Ran 11529 tests in 2216.932s FAILED (failures=3, errors=11, skipped=639, expected failures=23)Nice! Seems half of them are just removed Python aliases
np.bool, etc. and the stricterdtypematching. The last two are only slightly improved error messages.The refcounting thing looks like a small NumPy bug probably, likely in some error path, but I didn't do my leak checking marathon yet.
The refcounting error pops up when a test fails, once the actual issues are resolved these generally disappear.
Reacted by Sebastian Berg(In fact they're already gone in my local branch)
Note: I edited the description to make a checklist of items to address.
numpy 1.24.1 is released now.
I look forward for the support in a new release of numba.
Happy new year !Reacted by pythonic2020 and Royinumpy 1.24.1 is released now. I look forward for the support in a new release of numba. Happy new year !
Thanks for the heads-up, I'll re-run the checks with 1.24.1. The PR for 1.24 support is mostly ready, just needs a little more work before it's ready for review.
3 remaining items
This came up for me today. Very interested :)
Is there a timeline for a numba release that supports nump 1.24?
Now that #8691 is merged, I think this can be closed.
Thanks for the reminder, closing.
Hi @stuartarchibald,
Is it possible to have numpy 1.24 support uncoupled from python 3.11?
The lack of numpy 1.24 support currently blocks us.
Reacted by Charles Vejnar, Jean-Baptiste VESLIN, Wilder Rodrigues, GuillaumePlessix, J.J. van den Berg, Glen Takahashi and Suchan LeeSame here. We are on Python 3.9 and need to move to Numpy 1.24, but Numba is blocking us.
@TNonet @wilderrodrigues I'll raise this at the Numba meetings next week (the public one is always on a Tuesday, see the discourse post for details). Rest assured that efforts are being made to make releasing Numba faster and less involved but this in itself takes time to develop.
I'd guess that for this time around the view would be that it will consume a prohibitively large amount of resources to do an immediate release of Numba with NumPy 1.24 as the impact would be a delay to Python 3.11 support (it would involve back-porting this to the Numba 0.56.x series and changing a lot of testing infrastructure as there's no NumPy 1.24 packages on Anaconda's default channel as yet).
As an aside, I'm curious as to what is in NumPy 1.24 that creates the need for an update? Or is it not so much the contents of NumPy 1.24 but rather the related package dependency chains on which your software is relying?
Same here with python 3.10
numpy < 1.24has problems with recent releases of setuptools when buiding with--no-build-isolationas we do (see here)
We have a workaround but we would like to move tonumpy>=1.24Reacted by Royi@stuartarchibald Thank you for the context.
To answer your aside, it is purely package dependency chains in my situation.
Reacted by GuillaumePlessix@TNonet @wilderrodrigues I'll raise this at the Numba meetings next week (the public one is always on a Tuesday, see the discourse post for details). Rest assured that efforts are being made to make releasing Numba faster and less involved but this in itself takes time to develop.
I'd guess that for this time around the view would be that it will consume a prohibitively large amount of resources to do an immediate release of Numba with NumPy 1.24 as the impact would be a delay to Python 3.11 support (it would involve back-porting this to the Numba 0.56.x series and changing a lot of testing infrastructure as there's no NumPy 1.24 packages on Anaconda's default channel as yet).
As an aside, I'm curious as to what is in NumPy 1.24 that creates the need for an update? Or is it not so much the contents of NumPy 1.24 but rather the related package dependency chains on which your software is relying?
For us it's all about the dependency chains. A couple of projects are now stuck because of this. For instance, we cannot update our own projects with new features developed on the dependencies because of Numba not working with Python 1.24.
Thanks for confirming @TNonet and @wilderrodrigues.
Reacted by Wilder Rodrigues- added a commit that references this issue
on May 3, 2023
This is an issue stub to track support for NumPy 1.24. Anticipated release date (guessing based on previous release cadence) will be sometime near the end of 2022.
Items to address:
np.MachArremoval: DEP: Finalize MachAr and machar deprecations numpy/numpy#22638resolve_dtypes()and_resolve_dtypes_and_context(): ENH: Exposeufunc.resolve_dtypesand strided loop access numpy/numpy#22422sum()method of arrays is no longer allowed: DEP: Expire deprecation of dtype/signature allowing instances numpy/numpy#22540np.boolnp.floatnumba.tests.test_mathlib.TestMathLib.test_hypot)