Error on import with numpy HEAD

Reporting a bug

I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/main/CHANGE_LOG).
I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

This will be a somewhat sparse bug report, but putting here early in case anyone recognizes this. We can do more work to replicate the environment if not.

In our upstream tests, xarray tests nightly against unreleased versions of our dependencies to get ahead of any incompatible code. Occasionally it also catches bugs in upstream libraries.

Here, we're getting an error importing some items from numba.np.ufunc, including the somewhat confusing SystemError: initialization of _internal failed without raising an exception at the bottom of the stack trace.

pydata/xarray#7306

Any ideas on what might be causing this?

Thanks in advance!

The following is also needed because the deprecated MachAr is now removed:

diff --git a/numba/np/arraymath.py b/numba/np/arraymath.py
index 7f53c6563..12edd8e38 100644
--- a/numba/np/arraymath.py
+++ b/numba/np/arraymath.py
@@ -3994,7 +3994,7 @@ def _gen_np_machar():
         return impl
 
 
-_gen_np_machar()
+# _gen_np_machar()
 
 
 def generate_xinfo(np_func, container, attr):

with this I'm able to start running the test suite - as things pass by, everything appears OK so far, but will update here if the test run completes.

I had to cancel my test run as I was working on other things, but it looked like things were generally working with those small changes: https://github.com/gmarkall/numba/tree/np-124 (YMMV, etc!)

@stuartarchibald

Thanks @stuartarchibald — that makes sense. I think we'll explore excluding anything that runs numba in these tests if this crops up again

2. There are simply not the resources (human, packages, or compute) for Numba to track NumPy HEAD.

Totally empathize! FWIW, we've found that the upstream HEAD tests reduce the maintainer burden, since we can solve problems at a slower pace — no need to rush when there's an upstream release that breaks something — and we can provide feedback on breakages where there's a mistaken breaking change. But numba is a more technically complex project than Xarray, so possibly the learning is not transferable.

@stuartarchibald

Thanks @stuartarchibald — that makes sense. I think we'll explore excluding anything that runs numba in these tests if this crops up again

@max-sixty no problem, glad it was helpful.

There are simply not the resources (human, packages, or compute) for Numba to track NumPy HEAD.

Totally empathize! FWIW, we've found that the upstream HEAD tests reduce the maintainer burden, since we can solve problems at a slower pace — no need to rush when there's an upstream release that breaks something — and we can provide feedback on breakages where there's a mistaken breaking change. But numba is a more technically complex project than Xarray, so possibly the learning is not transferable.

Thanks for sharing this. I can see that for some projects, aspects of the described approach would certainly be beneficial in terms of maintenance burden.

The Numba maintainers do track changes to dependencies and work with other projects if issues are anticipated, they also often "try out" the code base against newer versions of projects that impact Numba (cPython, LLVM, NumPy) etc to assess the level of effort needed to do the necessary updates. I think the learning from the Numba side is probably, that in developing a compiler, having a stable and predictable environment in which to develop makes it easier to isolate problems. This is in part due to problems often being very niche e.g. one single Python+NumPy+OS+CPU combination might exhibit a particular issue!

Just a heads up that I think I'm hitting this error in CIs now that NumPy 1.24 has been released:

https://github.com/mne-tools/mne-python/actions/runs/3732610574/jobs/6332287508#step:9:56

I expect others might hit it soon, too.

I expect others might hit it soon, too.

Yup, same story in pandas https://github.com/pandas-dev/pandas/actions/runs/3737292562/jobs/6342364883

________________________ ERROR collecting test session _________________________
../../../micromamba/envs/test/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
../../../micromamba/envs/test/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
    exec(co, module.__dict__)
pandas/tests/window/conftest.py:93: in <module>
    @pytest.fixture(params=[pytest.param("numba", marks=td.skip_if_no("numba")), "cython"])
pandas/util/_test_decorators.py:192: in skip_if_no
    not safe_import(package, min_version=min_version), reason=msg
pandas/util/_test_decorators.py:95: in safe_import
    mod = __import__(mod_name)
../../../micromamba/envs/test/lib/python3.8/site-packages/numba/__init__.py:43: in <module>
    from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
../../../micromamba/envs/test/lib/python3.8/site-packages/numba/np/ufunc/__init__.py:3: in <module>
    from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
../../../micromamba/envs/test/lib/python3.8/site-packages/numba/np/ufunc/decorators.py:3: in <module>
    from numba.np.ufunc import _internal
E   SystemError: initialization of _internal failed without raising an exception

We can pin numpy to 1.23.5 for now, no issue

For devs who are trying to fix their environments
The problem occurs in the latest version of numpy. Installing a lower version of numpy (<1.24) fixed the problem for us.

fix: pin numpy version

Looks like I forgot to mention in this issue, the PR / branch for NumPy 1.24 support is #8620

[FIX] Matplotlib and numba errors (#142)

Workaround for numba/numba#8615

Less restrictive workaround for numba/numba#8615

requiremens: Pin numpy < 1.24

dev-python/numba: fix Numpy dep

init_ufunc_dispatch: Handle unexpected ufunc methods gracefully

The PR for review for NumPy 1.24 support is: #8691

Having the same issue over here with an old numpy version 😕 rolling back to an old version dindt work for me

Fix numpy version to avoid numba conflict

Pin numpy to avoid numba/numba#8615

pin numpy to avoid numba/numba#8615

Temporarily pin numpy

Temporarily pin numpy to 1.23.5

I was having the same problem with the librosa library (The reason was that librosa was working with different versions of scipy and matplotlib), so I uninstalled numpy, matplotlib, and scipy, installed librosa first which installed its own compatible versions of libraries above and then I installed the necessary libraries. That seems to work for me.

Having this issue with

python=3.8.15, build: h4de0772_0_cpython, Channel: conda-forge
numba=0.56.4, build: pypi_0, channel: pypi
numpy=1.23.5, build: pypi_0, channel: pypi

Thank you all for your comments/feedback on this issue. This issue seems to have diverged a little from the originally reported problem and so I'm closing this issue as resolved with some specific follow up issues to subscribe to if they are important to you:

With respect to the originally posted issue, this comment Error on import with numpy HEAD #8615 (comment) (and some discussion immediately after) explains the decisions surrounding Numba not tracking NumPy HEAD.
With respect to the originally posted issue, NumPy 1.24 #8691 is the patch for making Numba compatible with NumPy 1.24.
The NumPy 1.24 related problem that manifests as the error:
```
SystemError: initialization of _internal failed without raising an exception
```
is tracked in Numba __init__.py not catching incompatible NumPy version early enough #8718.

what version of numpy should i install?

what version of numpy should i install?

If you're using the last released version of Numba (0.56.4 at time of writing), NumPy 1.18 - 1.23 are supported.

@max-sixty

for me numba=0.56.4 from conda-forge fails locally with numpy=1.24.0.dev0+1120.gf30af6acd, so not sure what I'm doing wrong? @max-sixty, was that with the numba head?

Edit: the install would be
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre "numpy==1.24.0.dev0+1120.gf30af6acd" --upgrade
and pip does indeed complain that numba requires numpy<1.24 but installs it anyways (as intended in this case, because we do actually want to test with the most recent version, even if that breaks something). So while point 2 from #8615 (comment) does not apply, point 3 may very well be the cause of the problem.

Thanks. This help a lot. I reinstall numpy by "pip install numpy==1.23.0". Then, the error when "import numba" was solved.

Is it fixed in any version?

Limit version of numpy to <1.24

Add sourcepredict recipe (#40456)

Hello everyone, the solutions posted here did not fix the problem for me. I tried numba v0.56.4 and v0.56.3 and both of these versions did not work with either numpy 1.23.0 or 1.24.0. I installed everything with conda-forge and I am using WSL. Any help would be greatly appreciated. Thank you!

Add sourcepredict recipe (bioconda#40456)

bugfix: update py-numba to 0.57.1

Hello everyone, the solutions posted here did not fix the problem for me. I tried numba v0.56.4 and v0.56.3 and both of these versions did not work with either numpy 1.23.0 or 1.24.0. I installed everything with conda-forge and I am using WSL. Any help would be greatly appreciated. Thank you!

I fixed with conda install cudatoolkit on a cpu only machine

Navigation Menu

Repository navigation

Error on import with numpy HEAD #8615

Reporting a bug

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error on import with numpy HEAD #8615

Description

Reporting a bug

Activity

gmarkall commented on Nov 23, 2022

gmarkall commented on Nov 23, 2022

max-sixty commented on Nov 24, 2022

stuartarchibald commented on Nov 24, 2022

larsoner commented on Dec 19, 2022

MarcoGorelli commented on Dec 20, 2022

zarifaziz commented on Dec 21, 2022

gmarkall commented on Dec 21, 2022

gmarkall commented on Jan 3, 2023

fmPeretti commented on Jan 4, 2023

Veda-73 commented on Jan 16, 2023

ssvenoe commented on Jan 18, 2023

stuartarchibald commented on Jan 18, 2023

inikishev commented on Feb 11, 2023

gmarkall commented on Feb 14, 2023

arijin commented on Feb 17, 2023

pimiento commented on Mar 25, 2023

chiiingky commented on Apr 27, 2023

fyenne commented on Aug 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions