Error on import with numpy HEAD

Reporting a bug

I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/main/CHANGE_LOG).
I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

This will be a somewhat sparse bug report, but putting here early in case anyone recognizes this. We can do more work to replicate the environment if not.

In our upstream tests, xarray tests nightly against unreleased versions of our dependencies to get ahead of any incompatible code. Occasionally it also catches bugs in upstream libraries.

Here, we're getting an error importing some items from numba.np.ufunc, including the somewhat confusing SystemError: initialization of _internal failed without raising an exception at the bottom of the stack trace.

pydata/xarray#7306

Any ideas on what might be causing this?

Thanks in advance!

@max-sixty

Thanks for the report @max-sixty. Do you know what version of Python, Numba and NumPy (and where from, e.g. conda-forge or pip or ...) were present when this a) failed and b) was last successful?

I think:

SystemError: initialization of _internal failed without raising an exception

is saying that the module failed to initialize as it returned an error code, but there was no exception raised to go with the failure result (e.g. function PyInit_<module name> returned NULL but an exception wasn't set).

@stuartarchibald

Thanks for the swift response @stuartarchibald !

Do you know what version of Python, Numba and NumPy (and where from, e.g. conda-forge or pip or ...) were present when this a) failed and b) was last successful?

Here's the full version info: https://github.com/pydata/xarray/actions/runs/3519182359/jobs/5906608476#step:8:74

Actually it seems we don't install the upstream version of numba, so this was running on 0.56.4 py310ha5257ce_0. FYI the script we use to install dependencies is at https://github.com/pydata/xarray/blob/main/ci/install-upstream-wheels.sh.

This is the first day it's failed with this error, so it is likely to be a change somewhere in the past day.

Though I realize "somewhere" isn't that helpful, and it now seems that it may not be a numba-specific error given that hasn't changed. Do you possibly have any similar tests which run on the latest branch of numpy etc? If those pass, then this is more likely to be something specific to xarray / numbagg, even though that didn't look likely at first glance.

is saying that the module failed to initialize as it returned an error code, but there was no exception raised to go with the failure result (e.g. function PyInit_<module name> returned NULL but an exception wasn't set).

Ah great, that makes much more sense.

Add numba to nightly upstream

This is the first day it's failed with this error, so it is likely to be a change somewhere in the past day.

That's not quite true, it started failing 5 days ago but there was a bug in our CI which caused that to go unnoticed. Probably doesn't change too much, though.

@stuartarchibald

Thanks for the swift response @stuartarchibald !

No problem!

Do you know what version of Python, Numba and NumPy (and where from, e.g. conda-forge or pip or ...) were present when this a) failed and b) was last successful?

Here's the full version info: https://github.com/pydata/xarray/actions/runs/3519182359/jobs/5906608476#step:8:74

Actually it seems we don't install the upstream version of numba, so this was running on 0.56.4 py310ha5257ce_0. FYI the script we use to install dependencies is at https://github.com/pydata/xarray/blob/main/ci/install-upstream-wheels.sh.

This is the first day it's failed with this error, so it is likely to be a change somewhere in the past day.

Though I realize "somewhere" isn't that helpful, and it now seems that it may not be a numba-specific error given that hasn't changed. Do you possibly have any similar tests which run on the latest branch of numpy etc? If those pass, then this is more likely to be something specific to xarray / numbagg, even though that didn't look likely at first glance.

Thanks for this information. Numba doesn't automatically test against NumPy latest/mainline, but it does test NumPy alphas/pre-releases and provides feedback for these. Once a NumPy package becomes more generally available across a number of distribution mechanisms, support for that release of NumPy is added to Numba. As a result, released versions of Numba have explicit version constraints with respect to packages such as NumPy. As it's quite common for updates to e.g. NumPy to break something in Numba, Numba tries really hard to provide predicable failure modes (e.g. won't install or RuntimeError for unsupported NumPy being found).

Do you think it'd be possible to diff the logs between last working and now? That might hint at the problem?

is saying that the module failed to initialize as it returned an error code, but there was no exception raised to go with the failure result (e.g. function PyInit_<module name> returned NULL but an exception wasn't set).

Ah great, that makes much more sense.

Great, glad that's clearer, it's quite a "low level" problem.

I just checked, and as far as I can tell the only difference are some I/O libraries (unrelated, I think) and the OS version. We use the ubuntu-latest tag to run on github actions, and apparently that changed from 20.04.5 to 22.04.1. Not sure if that should make a difference? Edit: the version of __glibc changes as well: __glibc=2.31=0 → __glibc=2.35=0

Edit: For reference, here's the logs: passing (we have other errors, but they're definitely unrelated) and failing

@keewis

Thanks for checking @keewis. I'm going to make some guesses based on Numba internals... I think the following:

That the issue is unlikely to be glibc, Python or Python C-API related as to get to the point of importing numba.np.ufunc._internal Numba would have had to have imported other C extension modules that would have similar glibc linkage and C-API use and these evidently worked "OK" as errors were not raised.
That whilst Numba guards against using unsupported NumPy versions, current NumPy nightlies are likely to be reporting that they are 1.23 series builds as IIRC 1.24 hasn't been tagged, so Numba is effectively using something that hasn't been tested.
Following some discussion at the weekly triage meeting, @gmarkall noted that there's some potential for susceptibility against changes in NumPy ufunc definitions in the init_ufunc_dispatch function in _internal.c. NumPy has recently merged a change to help with Update ufunc loop signature resolution to use NumPy #22422 #8538 which may well be the cause of the problem if 2. is also correct.

Numba relies on NumPy infrastructure for e.g. ufunc creation, and so there is a very tight coupling in this respect, between Numba and NumPy (the code is all in C and uses Python/NumPy C-API).

To debug and potentially fix the issue reported, maybe first try installing a supported released version of NumPy? If that fixes it, it narrows the problem down to what is suspected in the above.

Hope this helps.

@stuartarchibald

To debug and potentially fix the issue reported, maybe first try installing a supported released version of NumPy? If that fixes it, it narrows the problem down to what is suspected in the above.

Our tests are still very much passing on the newest released versions of libraries; so this is only an issue with the versions from HEADs. I just tried installing numpy from main — I get the same failure with numba 0.56.3 but it works with numba 0.56.4. I notice that's consistent with the declared dependencies of numba, so it seems numba is doing everything it should be...

Possibly over at xarray, we should at least ensure we're on the latest numba for this test. I did a PR to do that earlier: pydata/xarray#7311 (review). Let us know if there's some pre-compiled version of dependencies (pydata/xarray#7311 (comment)), otherwise we'll just use pip.

Feel free to close, thanks for your attention @stuartarchibald !

@max-sixty

for me numba=0.56.4 from conda-forge fails locally with numpy=1.24.0.dev0+1120.gf30af6acd, so not sure what I'm doing wrong? @max-sixty, was that with the numba head?

Edit: the install would be

pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre "numpy==1.24.0.dev0+1120.gf30af6acd" --upgrade

and pip does indeed complain that numba requires numpy<1.24 but installs it anyways (as intended in this case, because we do actually want to test with the most recent version, even if that breaks something). So while point 2 from #8615 (comment) does not apply, point 3 may very well be the cause of the problem.

Redoing the test; I also get a failure with:

both that nightly wheel command, and pip install . on numpy''s HEAD
both numba 0.56.3 and 0.56.4

So I think my prior success with 0.56.4 must have been a mistake (I notice that pip reinstalls a lower version of numpy when installing a new version of numba, so that may have been it, mea culpa)

...so I'm not sure what we should do from Xarray's POV — I guess we can try running tests without numba? But that misses some of the utility of these upstream tests.

I would anticipate that any issue, of the nature as described above, that is in 0.56.3 would also be in 0.56.4. The changes are here:

numba/CHANGE_LOG

Lines 1 to 14 in 3e96339

    
           Version 0.56.4 (3 November, 2022) 
        
           --------------------------------- 
        
           This is a bugfix release to fix a regression in the CUDA target in relation to 
        
           the ``.view()`` method on CUDA device arrays that is present when using NumPy 
        
           version 1.23.0 or later. 
        
           Pull-Requests: 
        
           * PR `#8537 <https://github.com/numba/numba/pull/8537>`_: Make ol_compatible_view accessible on all targets (`gmarkall <https://github.com/gmarkall>`_) 
        
           * PR `#8552 <https://github.com/numba/numba/pull/8552>`_: Update version support table for 0.56.4. (`stuartarchibald <https://github.com/stuartarchibald>`_) 
        
           * PR `#8553 <https://github.com/numba/numba/pull/8553>`_: Update CHANGE_LOG for 0.56.4 (`stuartarchibald <https://github.com/stuartarchibald>`_) 
        
           * PR `#8570 <https://github.com/numba/numba/pull/8570>`_: Release 0.56 branch: Fix overloads with ``target="generic"`` for CUDA (`gmarkall <https://github.com/gmarkall>`_) 
        
           * PR `#8571 <https://github.com/numba/numba/pull/8571>`_: Additional update to CHANGE_LOG for 0.56.4 (`stuartarchibald <https://github.com/stuartarchibald>`_)

but in summary, it involved adding a couple of lines to fix a regression in <array>.view()-like capabilities on the CUDA hardware target. Something that should not in any way impact the issue above as it's all in JIT code generation and specifically for CUDA users.

...so I'm not sure what we should do from Xarray's POV — I guess we can try running tests without numba? But that misses some of the utility of these upstream tests.

I'm not sure what is best for the Xarray project, however this information might help informing that decision.

Numba's releases are restricted for use against specific versions of NumPy for reasons as demonstrated in this issue. It is often the case that Numba might "work" on other versions of NumPy, but this cannot be guaranteed as they might not be binary compatible etc.
Numba main branch has no restrictions on the version of NumPy, but it also doesn't track NumPy HEAD. The folks who maintain Numba do an upgrade of the supported NumPy version, often in a single patch, as pre-release packages become available for testing. I'll raise this issue with the other maintainers at the next triage meeting, but historically the view has been:
- There are simply not the resources (human, packages, or compute) for Numba to track NumPy HEAD.
- Development of Numba is considerably easier in stable environments. There are already many variations:
  - Python versions (the bytecode from which Numba compiles changes in literally every version)
  - NumPy versions (algorithms, APIs, etc change in pretty much every version)
  - OS/Architecture (Numba supports at least 7, each with their own individual issues)

Hope this helps?

The following quick-n-dirty patch gets around the SystemError with NumPy main:

diff --git a/numba/np/ufunc/_internal.c b/numba/np/ufunc/_internal.c
index 98a643788..c83b4f403 100644
--- a/numba/np/ufunc/_internal.c
+++ b/numba/np/ufunc/_internal.c
@@ -326,10 +326,13 @@ init_ufunc_dispatch(int *numpy_uses_fastcall)
             } else if (strncmp(crnt_name, "reduceat", 9) == 0) {
                 ufunc_dispatch.ufunc_reduceat =
                     (PyCFunctionWithKeywords)crnt->ml_meth;
+            } else if (strncmp(crnt_name, "resolve_dtypes", 15) == 0) {
             } else {
                 result = -1;
             }
             break;
+        case '_':
+            break;
         default:
             result = -1; /* Unknown method */
         }

However, I have not tried much beyond that yet!

Temporarily pin numpy to 1.23.5

I was having the same problem with the librosa library (The reason was that librosa was working with different versions of scipy and matplotlib), so I uninstalled numpy, matplotlib, and scipy, installed librosa first which installed its own compatible versions of libraries above and then I installed the necessary libraries. That seems to work for me.

Having this issue with

python=3.8.15, build: h4de0772_0_cpython, Channel: conda-forge
numba=0.56.4, build: pypi_0, channel: pypi
numpy=1.23.5, build: pypi_0, channel: pypi

Thank you all for your comments/feedback on this issue. This issue seems to have diverged a little from the originally reported problem and so I'm closing this issue as resolved with some specific follow up issues to subscribe to if they are important to you:

With respect to the originally posted issue, this comment Error on import with numpy HEAD #8615 (comment) (and some discussion immediately after) explains the decisions surrounding Numba not tracking NumPy HEAD.
With respect to the originally posted issue, NumPy 1.24 #8691 is the patch for making Numba compatible with NumPy 1.24.
The NumPy 1.24 related problem that manifests as the error:
```
SystemError: initialization of _internal failed without raising an exception
```
is tracked in Numba __init__.py not catching incompatible NumPy version early enough #8718.

what version of numpy should i install?

what version of numpy should i install?

If you're using the last released version of Numba (0.56.4 at time of writing), NumPy 1.18 - 1.23 are supported.

@max-sixty

for me numba=0.56.4 from conda-forge fails locally with numpy=1.24.0.dev0+1120.gf30af6acd, so not sure what I'm doing wrong? @max-sixty, was that with the numba head?

Edit: the install would be
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre "numpy==1.24.0.dev0+1120.gf30af6acd" --upgrade
and pip does indeed complain that numba requires numpy<1.24 but installs it anyways (as intended in this case, because we do actually want to test with the most recent version, even if that breaks something). So while point 2 from #8615 (comment) does not apply, point 3 may very well be the cause of the problem.

Thanks. This help a lot. I reinstall numpy by "pip install numpy==1.23.0". Then, the error when "import numba" was solved.

Is it fixed in any version?

Limit version of numpy to <1.24

Add sourcepredict recipe (#40456)

Hello everyone, the solutions posted here did not fix the problem for me. I tried numba v0.56.4 and v0.56.3 and both of these versions did not work with either numpy 1.23.0 or 1.24.0. I installed everything with conda-forge and I am using WSL. Any help would be greatly appreciated. Thank you!

Add sourcepredict recipe (bioconda#40456)

bugfix: update py-numba to 0.57.1

Hello everyone, the solutions posted here did not fix the problem for me. I tried numba v0.56.4 and v0.56.3 and both of these versions did not work with either numpy 1.23.0 or 1.24.0. I installed everything with conda-forge and I am using WSL. Any help would be greatly appreciated. Thank you!

I fixed with conda install cudatoolkit on a cpu only machine

Navigation Menu

Repository navigation

Error on import with numpy HEAD #8615

Reporting a bug

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	Version 0.56.4 (3 November, 2022)
	---------------------------------

	This is a bugfix release to fix a regression in the CUDA target in relation to
	the ``.view()`` method on CUDA device arrays that is present when using NumPy
	version 1.23.0 or later.

	Pull-Requests:

	* PR `#8537 <https://github.com/numba/numba/pull/8537>`_: Make ol_compatible_view accessible on all targets (`gmarkall <https://github.com/gmarkall>`_)
	* PR `#8552 <https://github.com/numba/numba/pull/8552>`_: Update version support table for 0.56.4. (`stuartarchibald <https://github.com/stuartarchibald>`_)
	* PR `#8553 <https://github.com/numba/numba/pull/8553>`_: Update CHANGE_LOG for 0.56.4 (`stuartarchibald <https://github.com/stuartarchibald>`_)
	* PR `#8570 <https://github.com/numba/numba/pull/8570>`_: Release 0.56 branch: Fix overloads with ``target="generic"`` for CUDA (`gmarkall <https://github.com/gmarkall>`_)
	* PR `#8571 <https://github.com/numba/numba/pull/8571>`_: Additional update to CHANGE_LOG for 0.56.4 (`stuartarchibald <https://github.com/stuartarchibald>`_)

Error on import with numpy HEAD #8615

Description

Reporting a bug

Activity

stuartarchibald commented on Nov 22, 2022

max-sixty commented on Nov 22, 2022

keewis commented on Nov 22, 2022

stuartarchibald commented on Nov 22, 2022

keewis commented on Nov 22, 2022

stuartarchibald commented on Nov 22, 2022

max-sixty commented on Nov 22, 2022

keewis commented on Nov 22, 2022

max-sixty commented on Nov 22, 2022

stuartarchibald commented on Nov 23, 2022

stuartarchibald commented on Nov 23, 2022

gmarkall commented on Nov 23, 2022

24 remaining items

Veda-73 commented on Jan 16, 2023

ssvenoe commented on Jan 18, 2023

stuartarchibald commented on Jan 18, 2023

inikishev commented on Feb 11, 2023

gmarkall commented on Feb 14, 2023

arijin commented on Feb 17, 2023

pimiento commented on Mar 25, 2023

chiiingky commented on Apr 27, 2023

fyenne commented on Aug 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions