Repository navigation
Error on import with numpy HEAD #8615
Description
Activity
Thanks for the report @max-sixty. Do you know what version of Python, Numba and NumPy (and where from, e.g.
conda-forgeorpipor ...) were present when this a) failed and b) was last successful?I think:
SystemError: initialization of _internal failed without raising an exceptionis saying that the module failed to initialize as it returned an error code, but there was no exception raised to go with the failure result (e.g. function
PyInit_<module name>returnedNULLbut an exception wasn't set).Thanks for the swift response @stuartarchibald !
Do you know what version of Python, Numba and NumPy (and where from, e.g.
conda-forgeorpipor ...) were present when this a) failed and b) was last successful?Here's the full version info: https://github.com/pydata/xarray/actions/runs/3519182359/jobs/5906608476#step:8:74
Actually it seems we don't install the upstream version of numba, so this was running on
0.56.4 py310ha5257ce_0. FYI the script we use to install dependencies is at https://github.com/pydata/xarray/blob/main/ci/install-upstream-wheels.sh.This is the first day it's failed with this error, so it is likely to be a change somewhere in the past day.
Though I realize "somewhere" isn't that helpful, and it now seems that it may not be a numba-specific error given that hasn't changed. Do you possibly have any similar tests which run on the latest branch of
numpyetc? If those pass, then this is more likely to be something specific to xarray / numbagg, even though that didn't look likely at first glance.is saying that the module failed to initialize as it returned an error code, but there was no exception raised to go with the failure result (e.g. function
PyInit_<module name>returnedNULLbut an exception wasn't set).Ah great, that makes much more sense.
- added a commit that references this issue
on Nov 22, 2022 This is the first day it's failed with this error, so it is likely to be a change somewhere in the past day.
That's not quite true, it started failing 5 days ago but there was a bug in our CI which caused that to go unnoticed. Probably doesn't change too much, though.
Thanks for the swift response @stuartarchibald !
No problem!
Do you know what version of Python, Numba and NumPy (and where from, e.g.
conda-forgeorpipor ...) were present when this a) failed and b) was last successful?Here's the full version info: https://github.com/pydata/xarray/actions/runs/3519182359/jobs/5906608476#step:8:74
Actually it seems we don't install the upstream version of numba, so this was running on
0.56.4 py310ha5257ce_0. FYI the script we use to install dependencies is at https://github.com/pydata/xarray/blob/main/ci/install-upstream-wheels.sh.This is the first day it's failed with this error, so it is likely to be a change somewhere in the past day.
Though I realize "somewhere" isn't that helpful, and it now seems that it may not be a numba-specific error given that hasn't changed. Do you possibly have any similar tests which run on the latest branch of
numpyetc? If those pass, then this is more likely to be something specific to xarray / numbagg, even though that didn't look likely at first glance.Thanks for this information. Numba doesn't automatically test against NumPy latest/mainline, but it does test NumPy alphas/pre-releases and provides feedback for these. Once a NumPy package becomes more generally available across a number of distribution mechanisms, support for that release of NumPy is added to Numba. As a result, released versions of Numba have explicit version constraints with respect to packages such as NumPy. As it's quite common for updates to e.g. NumPy to break something in Numba, Numba tries really hard to provide predicable failure modes (e.g. won't install or
RuntimeErrorfor unsupported NumPy being found).Do you think it'd be possible to
diffthe logs between last working and now? That might hint at the problem?is saying that the module failed to initialize as it returned an error code, but there was no exception raised to go with the failure result (e.g. function
PyInit_<module name>returnedNULLbut an exception wasn't set).Ah great, that makes much more sense.
Great, glad that's clearer, it's quite a "low level" problem.
I just checked, and as far as I can tell the only difference are some I/O libraries (unrelated, I think) and the OS version. We use the
ubuntu-latesttag to run on github actions, and apparently that changed from20.04.5to22.04.1. Not sure if that should make a difference? Edit: the version of__glibcchanges as well:__glibc=2.31=0→__glibc=2.35=0Edit: For reference, here's the logs: passing (we have other errors, but they're definitely unrelated) and failing
Thanks for checking @keewis. I'm going to make some guesses based on Numba internals... I think the following:
- That the issue is unlikely to be
glibc,Pythonor Python C-API related as to get to the point of importingnumba.np.ufunc._internalNumba would have had to have imported other C extension modules that would have similarglibclinkage and C-API use and these evidently worked "OK" as errors were not raised. - That whilst Numba guards against using unsupported NumPy versions, current NumPy nightlies are likely to be reporting that they are 1.23 series builds as IIRC 1.24 hasn't been tagged, so Numba is effectively using something that hasn't been tested.
- Following some discussion at the weekly triage meeting, @gmarkall noted that there's some potential for susceptibility against changes in NumPy ufunc definitions in the
init_ufunc_dispatchfunction in_internal.c. NumPy has recently merged a change to help with Update ufunc loop signature resolution to use NumPy #22422 #8538 which may well be the cause of the problem if 2. is also correct.
Numba relies on NumPy infrastructure for e.g. ufunc creation, and so there is a very tight coupling in this respect, between Numba and NumPy (the code is all in C and uses Python/NumPy C-API).
To debug and potentially fix the issue reported, maybe first try installing a supported released version of NumPy? If that fixes it, it narrows the problem down to what is suspected in the above.
Hope this helps.
- That the issue is unlikely to be
To debug and potentially fix the issue reported, maybe first try installing a supported released version of NumPy? If that fixes it, it narrows the problem down to what is suspected in the above.
Our tests are still very much passing on the newest released versions of libraries; so this is only an issue with the versions from HEADs. I just tried installing
numpyfrom main — I get the same failure with numba 0.56.3 but it works with numba 0.56.4. I notice that's consistent with the declared dependencies ofnumba, so it seemsnumbais doing everything it should be...Possibly over at xarray, we should at least ensure we're on the latest numba for this test. I did a PR to do that earlier: pydata/xarray#7311 (review). Let us know if there's some pre-compiled version of dependencies (pydata/xarray#7311 (comment)), otherwise we'll just use
pip.Feel free to close, thanks for your attention @stuartarchibald !
for me
numba=0.56.4fromconda-forgefails locally withnumpy=1.24.0.dev0+1120.gf30af6acd, so not sure what I'm doing wrong? @max-sixty, was that with thenumbahead?Edit: the install would be
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre "numpy==1.24.0.dev0+1120.gf30af6acd" --upgradeand
pipdoes indeed complain thatnumbarequiresnumpy<1.24but installs it anyways (as intended in this case, because we do actually want to test with the most recent version, even if that breaks something). So while point 2 from #8615 (comment) does not apply, point 3 may very well be the cause of the problem.Reacted by Spencer Hallyburton, Takuma Yoneda, Tim Stevens and Gabriel GilbreathReacted by Gabriel GilbreathRedoing the test; I also get a failure with:
- both that nightly wheel command, and
pip install .on numpy''s HEAD - both numba 0.56.3 and 0.56.4
So I think my prior success with 0.56.4 must have been a mistake (I notice that pip reinstalls a lower version of numpy when installing a new version of numba, so that may have been it, mea culpa)
...so I'm not sure what we should do from Xarray's POV — I guess we can try running tests without
numba? But that misses some of the utility of these upstream tests.- both that nightly wheel command, and
I would anticipate that any issue, of the nature as described above, that is in 0.56.3 would also be in 0.56.4. The changes are here:
Lines 1 to 14 in 3e96339
but in summary, it involved adding a couple of lines to fix a regression in<array>.view()-like capabilities on the CUDA hardware target. Something that should not in any way impact the issue above as it's all in JIT code generation and specifically for CUDA users.- changed the title
[-]"SystemError: initialization of _internal failed without raising an exception"[/-][+]Error on import with numpy HEAD[/+]on Nov 23, 2022 ...so I'm not sure what we should do from Xarray's POV — I guess we can try running tests without numba? But that misses some of the utility of these upstream tests.
I'm not sure what is best for the
Xarrayproject, however this information might help informing that decision.- Numba's releases are restricted for use against specific versions of NumPy for reasons as demonstrated in this issue. It is often the case that Numba might "work" on other versions of NumPy, but this cannot be guaranteed as they might not be binary compatible etc.
- Numba
mainbranch has no restrictions on the version of NumPy, but it also doesn't track NumPyHEAD. The folks who maintain Numba do an upgrade of the supported NumPy version, often in a single patch, as pre-release packages become available for testing. I'll raise this issue with the other maintainers at the next triage meeting, but historically the view has been:- There are simply not the resources (human, packages, or compute) for Numba to track NumPy
HEAD. - Development of Numba is considerably easier in stable environments. There are already many variations:
- Python versions (the bytecode from which Numba compiles changes in literally every version)
- NumPy versions (algorithms, APIs, etc change in pretty much every version)
- OS/Architecture (Numba supports at least 7, each with their own individual issues)
- There are simply not the resources (human, packages, or compute) for Numba to track NumPy
Hope this helps?
Reacted by Muhammad YasirroniThe following quick-n-dirty patch gets around the
SystemErrorwith NumPymain:diff --git a/numba/np/ufunc/_internal.c b/numba/np/ufunc/_internal.c index 98a643788..c83b4f403 100644 --- a/numba/np/ufunc/_internal.c +++ b/numba/np/ufunc/_internal.c @@ -326,10 +326,13 @@ init_ufunc_dispatch(int *numpy_uses_fastcall) } else if (strncmp(crnt_name, "reduceat", 9) == 0) { ufunc_dispatch.ufunc_reduceat = (PyCFunctionWithKeywords)crnt->ml_meth; + } else if (strncmp(crnt_name, "resolve_dtypes", 15) == 0) { } else { result = -1; } break; + case '_': + break; default: result = -1; /* Unknown method */ }
However, I have not tried much beyond that yet!
24 remaining items
- added a commit that references this issue
on Jan 13, 2023 I was having the same problem with the librosa library (The reason was that librosa was working with different versions of scipy and matplotlib), so I uninstalled numpy, matplotlib, and scipy, installed librosa first which installed its own compatible versions of libraries above and then I installed the necessary libraries. That seems to work for me.
Reacted by Mirko MiorelliHaving this issue with
python=3.8.15, build: h4de0772_0_cpython, Channel: conda-forge
numba=0.56.4, build: pypi_0, channel: pypi
numpy=1.23.5, build: pypi_0, channel: pypiReacted by David Betancourt Montellano and cdahms123Thank you all for your comments/feedback on this issue. This issue seems to have diverged a little from the originally reported problem and so I'm closing this issue as resolved with some specific follow up issues to subscribe to if they are important to you:
- With respect to the originally posted issue, this comment Error on import with numpy HEAD #8615 (comment) (and some discussion immediately after) explains the decisions surrounding Numba not tracking NumPy
HEAD. - With respect to the originally posted issue, NumPy 1.24 #8691 is the patch for making Numba compatible with NumPy 1.24.
- The NumPy 1.24 related problem that manifests as the error:
is tracked in Numba
SystemError: initialization of _internal failed without raising an exception__init__.pynot catching incompatible NumPy version early enough #8718.
Reacted by Fred Frey- With respect to the originally posted issue, this comment Error on import with numpy HEAD #8615 (comment) (and some discussion immediately after) explains the decisions surrounding Numba not tracking NumPy
what version of numpy should i install?
what version of numpy should i install?
If you're using the last released version of Numba (0.56.4 at time of writing), NumPy 1.18 - 1.23 are supported.
Reacted by David Waterworthfor me
numba=0.56.4fromconda-forgefails locally withnumpy=1.24.0.dev0+1120.gf30af6acd, so not sure what I'm doing wrong? @max-sixty, was that with thenumbahead?Edit: the install would be
pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre "numpy==1.24.0.dev0+1120.gf30af6acd" --upgradeand
pipdoes indeed complain thatnumbarequiresnumpy<1.24but installs it anyways (as intended in this case, because we do actually want to test with the most recent version, even if that breaks something). So while point 2 from #8615 (comment) does not apply, point 3 may very well be the cause of the problem.Thanks. This help a lot. I reinstall numpy by "pip install numpy==1.23.0". Then, the error when "import numba" was solved.
Reacted by Gabriel Gilbreath, Elliot Tower and Lydon ChandraIs it fixed in any version?
- added a commit that references this issue
on Apr 14, 2023 - added a commit that references this issue
on Apr 18, 2023 Hello everyone, the solutions posted here did not fix the problem for me. I tried numba v0.56.4 and v0.56.3 and both of these versions did not work with either numpy 1.23.0 or 1.24.0. I installed everything with conda-forge and I am using WSL. Any help would be greatly appreciated. Thank you!
Reacted by Isha Pathania, Nils Wildt, Nitin Madnani, GRISHANIN Vasiliy, Dmitry, Fred Frey, Julian Nubert, Somshubra Majumdar, bilzard, Jelmer Kuperus and 2 more- added a commit that references this issue
on Apr 28, 2023 - added a commit that references this issue
on Aug 7, 2023 Hello everyone, the solutions posted here did not fix the problem for me. I tried numba v0.56.4 and v0.56.3 and both of these versions did not work with either numpy 1.23.0 or 1.24.0. I installed everything with conda-forge and I am using WSL. Any help would be greatly appreciated. Thank you!
I fixed with
conda install cudatoolkiton a cpu only machine
Reporting a bug
visible in the change log (https://github.com/numba/numba/blob/main/CHANGE_LOG).
i.e. it's possible to run as 'python bug.py'.
This will be a somewhat sparse bug report, but putting here early in case anyone recognizes this. We can do more work to replicate the environment if not.
In our upstream tests, xarray tests nightly against unreleased versions of our dependencies to get ahead of any incompatible code. Occasionally it also catches bugs in upstream libraries.
Here, we're getting an error importing some items from
numba.np.ufunc, including the somewhat confusingSystemError: initialization of _internal failed without raising an exceptionat the bottom of the stack trace.pydata/xarray#7306
Any ideas on what might be causing this?
Thanks in advance!