Skip to content

NumPy 1.24 - #8691

Merged
sklam merged 23 commits into
numba:mainfrom
gmarkall:np-124-review
Mar 6, 2023
Merged

sklam merged 23 commits into
numba:mainfrom
gmarkall:np-124-review

Conversation

@gmarkall

@gmarkall gmarkall commented Jan 3, 2023

Copy link
Copy Markdown
Member

This PR updates Numba to support NumPy 1.24, and is ready for review. Presently CI will fail due to the lack of NumPy 1.24 packages in Anaconda, but this should be resolved in time.

Each individual commit message details the changes made and their rationale - each should be reviewable as an individual change.

See testing with two other PRs:

I believe this was written in error and should always have been float16.
This test only checked for a plain match when comparing outputs.
However, in some cases a reconstruction check can be necessary, as in
`test_linalg_svd`.
Setting an array element with a sequence is removed in NumPy 1.24.
The modified regex matches the existing message produced by NumPy <
1.24, and the new improved message in 1.24.
This always produced invalid results (though they were consistent
between Numba and NumPy) but now this fails in NumPy 1.24 with an
exception:

```
TypeError: The `dtype` and `signature` arguments to ufuncs only select
the general DType and not details such as the byte order or time unit.
You can avoid this error by using the scalar types `np.float64` or the
dtype string notation.
```

Note that the exception message is misleading, and using the dtype
string notation does not provide a workaround.
np.bool was removed in NumPy 1.24.
The API version has long since been greater than 0x7 / 0x8 for any
supported NumPy.
If an unexpected ufunc method was encountered, `init_ufunc_dispatch()`
would return an error code indicating failure without setting an
exception, leading to errors like

```
SystemError: initialization of _internal failed without raising an
exception
```

as reported in Issue numba#8615.

This commit fixes the issue by setting an appropriate exception in this
case.
NumPy 1.24 adds a new method, `resolve_dtypes()`, and a private method
`_resolve_dtypes_and_context()`. We handle these by just ignoring them
(ignoring all private methods in general) in order to provide the same
level of functionality in Numba as for NumPy 1.23.

There is further room to build new functionality on top of this:

- Providing an implementation of `resolve_dtypes()` for `DUFunc`
  objects.
- Using the `resolve_dtypes()` method in place of logic in Numba that
  implements a similar dtype resolution process.
This results in the following output from `print_azure_matrix()`:

```
NumPy | Python | Count
-----------------------
 1.21 |  3.8   |   4
 1.22 |  3.8   |   4
 1.22 |  3.9   |   1
 1.23 |  3.8   |   2
 1.23 |  3.9   |   2
 1.23 |  3.10  |   1
 1.24 |  3.10  |   3
 1.24 |  3.8   |   1
 1.24 |  3.9   |   1
```

There are 19 slices, so the aim was to have five slices for each NumPy
version (1.21, 1.22, 1.23, 1.24) except for 1.21 which has 4 slices.
@larsoner

larsoner commented Jan 3, 2023

Copy link
Copy Markdown
Contributor

Presently CI will fail due to the lack of NumPy 1.24 packages in Anaconda, but this should be resolved in time.

Just a drive-by comment/idea to potentially unblock progress here: you could use the conda-forge channel in the NumPy 1.24 builds temporarily to get 1.24.1 testing going

@gmarkall

gmarkall commented Jan 3, 2023

Copy link
Copy Markdown
Member Author

Presently CI will fail due to the lack of NumPy 1.24 packages in Anaconda, but this should be resolved in time.

Just a drive-by comment/idea to potentially unblock progress here: you could use the conda-forge channel in the NumPy 1.24 builds temporarily to get 1.24.1 testing going

Thanks for the suggestion - #8620 tests all slices against 1.24. I don't want to change the setup for this PR, because this PR is in the form I'd like reviewed and eventually merged.

@gmarkall

gmarkall commented Jan 3, 2023

Copy link
Copy Markdown
Member Author

(At the time I set up #8620, there were no conda-forge packages either because the first 1.24 RC had just been released, so it uses pip)

@tangkong tangkong mentioned this pull request Jan 3, 2023
5 of 7 tasks
DrTodd13
DrTodd13 previously approved these changes Jan 3, 2023

@DrTodd13 DrTodd13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine with the parfor related part of this PR.

@stuartarchibald stuartarchibald left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @gmarkall, the organised commits/commit messages were very helpful in review. There's a few minor suggestions/things to look at in the review but otherwise looks good. I've checked the Azure config update and it looks like it has a good spread of NumPy versions (roughly 5 builds per supported NumPy version).

Comment thread docs/source/user/installing.rst Outdated
Comment thread numba/np/ufunc/_internal.c Outdated
Comment thread numba/np/ufunc/_internal.c
Comment thread numba/tests/test_array_methods.py
Comment thread numba/tests/test_comprehension.py Outdated
@stuartarchibald stuartarchibald added 4 - Waiting on author Waiting for author to respond to review Effort - medium Medium size effort needed and removed 3 - Ready for Review labels Jan 11, 2023
@gmarkall

Copy link
Copy Markdown
Member Author

Ah, a merge from main is needed. That should kick things off anyway.

@gmarkall

Copy link
Copy Markdown
Member Author

gpuci run tests

@SomeoneSerge

Copy link
Copy Markdown

Hello! Would be great to see this PR merged:)

I'm totally unfamiliar with numba processes, so I'll just ask: do you think there's a way you could split the release into two phases, such that you wouldn't have to wait for anaconda to get up to speed? I.e. you'd first just make master compatible with the new numpy, making no promises, and keep running your own CI against the last available version. And then as soon as conda has caught up you announce official compatibility?

@sklam

sklam commented Mar 2, 2023

Copy link
Copy Markdown
Member

@SomeoneSerge, we have decided to do something like that. We are now relying on conda-forge for testing the latest dependency. Good news is that this PR has already pass tests on linux-64 in our buildfarm. Just waiting for the rest to complete to merge this PR.

@sklam

sklam commented Mar 3, 2023

Copy link
Copy Markdown
Member

This PR has passed all 64-bit platforms. 32-bit platform failures are so far buildfarm issues.

@sklam

sklam commented Mar 6, 2023

Copy link
Copy Markdown
Member

We just need to get AzureCI to use conda-forge for np1.24

@gmarkall

gmarkall commented Mar 6, 2023

Copy link
Copy Markdown
Member Author

gpuci run tests

@gmarkall

gmarkall commented Mar 6, 2023

Copy link
Copy Markdown
Member Author

gpuci run tests

@stuartarchibald stuartarchibald left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes to the public CI configuration since the last approval look as expected for use of conda-forge NumPy packages for NumPy version 1.24. Thanks for making this adjustment @gmarkall.

@sklam sklam added 5 - Ready to merge Review and testing done, is ready to merge and removed 4 - Waiting on CI Review etc done, waiting for CI to finish labels Mar 6, 2023
@sklam sklam changed the title NumPy 1.24 (PR for review) NumPy 1.24 Mar 6, 2023
@sklam

sklam commented Mar 6, 2023

Copy link
Copy Markdown
Member

Public CI failed due to timeout on windows. We can safely ignore the failure given it passed on buildfarm already.

@sklam
sklam merged commit 9ffe080 into numba:main Mar 6, 2023
@esc esc mentioned this pull request Mar 10, 2023
@gmarkall gmarkall mentioned this pull request Mar 24, 2023
2 tasks done
shaneahmed added a commit to TissueImageAnalytics/tiatoolbox that referenced this pull request May 3, 2023
- np.int, np.bool, np.float is depreciated in v1.20 in favour of np.int_, np.bool_, np.float_

Waiting on some dependencies to update:
- numba not yet compatible with numpy 1.24
  >   from numba.np.ufunc import _internal
  E   SystemError: initialization of _internal failed without raising an exception

   - numba/numba#8464
   - numba/numba#8691
   - numba/numba#8841
   - https://github.com/numba/numba/milestone/63

- [x] Waiting for next numba release with numpy 1.24 support.

---------
Co-authored-by: John Pocock <John-P@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

5 - Ready to merge Review and testing done, is ready to merge Effort - medium Medium size effort needed highpriority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants