NumPy 1.24 support

This is an issue stub to track support for NumPy 1.24. Anticipated release date (guessing based on previous release cadence) will be sometime near the end of 2022.

Items to address:

np.MachAr removal: DEP: Finalize MachAr and machar deprecations numpy/numpy#22638
Additional ufunc methods resolve_dtypes() and _resolve_dtypes_and_context(): ENH: Expose ufunc.resolve_dtypes and strided loop access numpy/numpy#22422
Specifying a timedelta dtype for the sum() method of arrays is no longer allowed: DEP: Expire deprecation of dtype/signature allowing instances numpy/numpy#22540
Removal of np.bool
Removal of np.float
Some changes to exception messages we replicate (see numba.tests.test_mathlib.TestMathLib.test_hypot)

Need to accommodate: numpy/numpy#22638

Need to update ufunc loop selection WRT numpy/numpy#22422 (#8538 tracks).

I've local patches for both those issues (not PR-ready, but having a quick test). Will push a branch if my local testing goes OK.

Branch is here (very simple workarounds for now!): https://github.com/gmarkall/numba/tree/np-124

Errors with my branch at present:

======================================================================
ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ]
test sum with axis and dtype parameters over a whole range of dtypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws
    py_res = pyfunc(arr, axis=axis, dtype=out_dtype)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws
    return a.sum(axis=axis, dtype=dtype)
  File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)
TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation.

======================================================================
ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ]
test sum with axis and dtype parameters over a whole range of dtypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws
    py_res = pyfunc(arr, axis=axis, dtype=out_dtype)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws
    return a.sum(axis=axis, dtype=dtype)
  File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)
TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation.

======================================================================
ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ]
test sum with axis and dtype parameters over a whole range of dtypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws
    py_res = pyfunc(arr, axis=axis, dtype=out_dtype)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws
    return a.sum(axis=axis, dtype=dtype)
  File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)
TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation.

======================================================================
ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ]
test sum with axis and dtype parameters over a whole range of dtypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws
    py_res = pyfunc(arr, axis=axis, dtype=out_dtype)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws
    return a.sum(axis=axis, dtype=dtype)
  File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)
TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation.

======================================================================
ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ]
test sum with axis and dtype parameters over a whole range of dtypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws
    py_res = pyfunc(arr, axis=axis, dtype=out_dtype)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws
    return a.sum(axis=axis, dtype=dtype)
  File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)
TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation.

======================================================================
ERROR: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods) [Testing np.sum with timedelta64[M] input and timedelta64[M] output ]
test sum with axis and dtype parameters over a whole range of dtypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 1283, in test_sum_axis_dtype_kws
    py_res = pyfunc(arr, axis=axis, dtype=out_dtype)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_array_methods.py", line 200, in array_sum_axis_dtype_kws
    return a.sum(axis=axis, dtype=dtype)
  File "/home/gmarkall/numbadev/numpy/numpy/core/_methods.py", line 49, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)
TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation.

======================================================================
ERROR: test_hmax (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/cuda/tests/cudapy/test_intrinsics.py", line 836, in test_hmax
    arg1 = np.float(5.)
  File "/home/gmarkall/numbadev/numpy/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'float'

======================================================================
ERROR: test_hmin (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/cuda/tests/cudapy/test_intrinsics.py", line 848, in test_hmin
    arg1 = np.float(5.)
  File "/home/gmarkall/numbadev/numpy/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'float'

======================================================================
ERROR: test_comp_nest_with_dependency (numba.tests.test_comprehension.TestArrayComprehension)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_comprehension.py", line 369, in test_comp_nest_with_dependency
    self.check(comp_nest_with_dependency, 5)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_comprehension.py", line 252, in check
    pyres = pyfunc(*args)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_comprehension.py", line 365, in comp_nest_with_dependency
    l = np.array([[i * j for j in range(i+1)] for i in range(n)])
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (5,) + inhomogeneous part.

======================================================================
ERROR: test_basic92 (numba.tests.test_stencils.TestManyStencils)
Issue #3497, bool return type evaluating incorrectly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 3110, in test_basic92
    self.check_against_expected(kernel, expected, A)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 811, in check_against_expected
    raise RuntimeError(str1 + str2)
RuntimeError: The following implementations should not have raised an exception but did:
['parfors', 'parfors']
Errors were:

parfors: Message: <class 'AttributeError'>: module 'numpy' has no attribute 'bool'

parfors: Message: <class 'TypeError'>: unsupported operand type(s) for -: 'NoneType' and 'float'



======================================================================
ERROR: test_basic93 (numba.tests.test_stencils.TestManyStencils)
Issue #3497, bool return type evaluating incorrectly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 3140, in test_basic93
    self.check_against_expected(kernel, expected, A, options={'cval': True})
  File "/home/gmarkall/numbadev/numba/numba/tests/test_stencils.py", line 811, in check_against_expected
    raise RuntimeError(str1 + str2)
RuntimeError: The following implementations should not have raised an exception but did:
['parfors', 'parfors']
Errors were:

parfors: Message: <class 'AttributeError'>: module 'numpy' has no attribute 'bool'

parfors: Message: <class 'TypeError'>: unsupported operand type(s) for -: 'NoneType' and 'float'



======================================================================
FAIL: test_sum_axis_dtype_kws (numba.tests.test_array_methods.TestArrayMethods)
test sum with axis and dtype parameters over a whole range of dtypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/support.py", line 855, in tearDown
    self.memory_leak_teardown()
  File "/home/gmarkall/numbadev/numba/numba/tests/support.py", line 830, in memory_leak_teardown
    self.assert_no_memory_leak()
  File "/home/gmarkall/numbadev/numba/numba/tests/support.py", line 839, in assert_no_memory_leak
    self.assertEqual(total_alloc, total_free)
AssertionError: 424 != 423

======================================================================
FAIL: test_hypot (numba.tests.test_mathlib.TestMathLib)
----------------------------------------------------------------------
RuntimeWarning: overflow encountered in scalar multiply

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_mathlib.py", line 518, in test_hypot
    self.assertRaisesRegexp(RuntimeWarning,
AssertionError: "overflow encountered in .*_scalars" does not match "overflow encountered in scalar multiply"

======================================================================
FAIL: test_hypot_npm (numba.tests.test_mathlib.TestMathLib)
----------------------------------------------------------------------
RuntimeWarning: overflow encountered in scalar multiply

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_mathlib.py", line 523, in test_hypot_npm
    self.test_hypot(flags=no_pyobj_flags)
  File "/home/gmarkall/numbadev/numba/numba/tests/test_mathlib.py", line 518, in test_hypot
    self.assertRaisesRegexp(RuntimeWarning,
AssertionError: "overflow encountered in .*_scalars" does not match "overflow encountered in scalar multiply"

----------------------------------------------------------------------
Ran 11529 tests in 2216.932s

FAILED (failures=3, errors=11, skipped=639, expected failures=23)

Nice! Seems half of them are just removed Python aliases np.bool, etc. and the stricter dtype matching. The last two are only slightly improved error messages.

The refcounting thing looks like a small NumPy bug probably, likely in some error path, but I didn't do my leak checking marathon yet.

The refcounting error pops up when a test fails, once the actual issues are resolved these generally disappear.

(In fact they're already gone in my local branch)

Note: I edited the description to make a checklist of items to address.

See also #8620.

numpy 1.24.1 is released now.
I look forward for the support in a new release of numba.
Happy new year !

numpy 1.24.1 is released now. I look forward for the support in a new release of numba. Happy new year !

Thanks for the heads-up, I'll re-run the checks with 1.24.1. The PR for 1.24 support is mostly ready, just needs a little more work before it's ready for review.

PR for review: #8691 - this resolves all items in the checklist in the issue description above.

This came up for me today. Very interested :)

Is there a timeline for a numba release that supports nump 1.24?

Now that #8691 is merged, I think this can be closed.

Now that #8691 is merged, I think this can be closed.

Thanks for the reminder, closing.

@wlievens

Is there a timeline for a numba release that supports nump 1.24?

@wlievens perhaps subscribe to #8304 for updates as Python 3.11 work has been the governing factor for the next release of Numba.

@stuartarchibald

Hi @stuartarchibald,

Is it possible to have numpy 1.24 support uncoupled from python 3.11?

The lack of numpy 1.24 support currently blocks us.

Same here. We are on Python 3.9 and need to move to Numpy 1.24, but Numba is blocking us.

@TNonet

@TNonet @wilderrodrigues I'll raise this at the Numba meetings next week (the public one is always on a Tuesday, see the discourse post for details). Rest assured that efforts are being made to make releasing Numba faster and less involved but this in itself takes time to develop.

I'd guess that for this time around the view would be that it will consume a prohibitively large amount of resources to do an immediate release of Numba with NumPy 1.24 as the impact would be a delay to Python 3.11 support (it would involve back-porting this to the Numba 0.56.x series and changing a lot of testing infrastructure as there's no NumPy 1.24 packages on Anaconda's default channel as yet).

As an aside, I'm curious as to what is in NumPy 1.24 that creates the need for an update? Or is it not so much the contents of NumPy 1.24 but rather the related package dependency chains on which your software is relying?

Same here with python 3.10
numpy < 1.24 has problems with recent releases of setuptools when buiding with --no-build-isolation as we do (see here)
We have a workaround but we would like to move to numpy>=1.24

@stuartarchibald

@stuartarchibald Thank you for the context.

To answer your aside, it is purely package dependency chains in my situation.

@TNonet

@TNonet @wilderrodrigues I'll raise this at the Numba meetings next week (the public one is always on a Tuesday, see the discourse post for details). Rest assured that efforts are being made to make releasing Numba faster and less involved but this in itself takes time to develop.

I'd guess that for this time around the view would be that it will consume a prohibitively large amount of resources to do an immediate release of Numba with NumPy 1.24 as the impact would be a delay to Python 3.11 support (it would involve back-porting this to the Numba 0.56.x series and changing a lot of testing infrastructure as there's no NumPy 1.24 packages on Anaconda's default channel as yet).

As an aside, I'm curious as to what is in NumPy 1.24 that creates the need for an update? Or is it not so much the contents of NumPy 1.24 but rather the related package dependency chains on which your software is relying?

For us it's all about the dependency chains. A couple of projects are now stuck because of this. For instance, we cannot update our own projects with new features developed on the dependencies because of Numba not working with Python 1.24.

@TNonet

Thanks for confirming @TNonet and @wilderrodrigues.

released numpy<1.24 since numba/numba#8464 was solved, closes #233

⬆️ Upgrade `numpy` Dependency (#543)

Navigation Menu

Repository navigation

NumPy 1.24 support #8464

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

NumPy 1.24 support #8464

Description

Activity

stuartarchibald commented on Nov 21, 2022

stuartarchibald commented on Nov 21, 2022

gmarkall commented on Nov 23, 2022

gmarkall commented on Nov 23, 2022

gmarkall commented on Nov 24, 2022

seberg commented on Nov 24, 2022

gmarkall commented on Nov 24, 2022

gmarkall commented on Nov 24, 2022

gmarkall commented on Nov 24, 2022

gmarkall commented on Nov 24, 2022

thebaptiste commented on Jan 2, 2023

gmarkall commented on Jan 2, 2023

gmarkall commented on Jan 3, 2023

3 remaining items

RoelantStegmann commented on Mar 3, 2023

wlievens commented on Mar 7, 2023

gmarkall commented on Mar 7, 2023

stuartarchibald commented on Mar 8, 2023

stuartarchibald commented on Mar 8, 2023

TNonet commented on Mar 8, 2023

wilderrodrigues commented on Mar 9, 2023

stuartarchibald commented on Mar 9, 2023

thebaptiste commented on Mar 9, 2023

TNonet commented on Mar 9, 2023

wilderrodrigues commented on Mar 14, 2023

stuartarchibald commented on Mar 17, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions