Fix numpy2.0 incompatbility issues by sklam · Pull Request #9602 · numba/numba

sklam · 2024-06-03T16:30:38Z

…state`

sklam · 2024-06-03T16:41:40Z

BFID numba_smoketest_cpu_yaml_195

sklam · 2024-06-03T22:10:55Z

I have to rerun smoketest entirely. It turns out the buildfarm is relying on scipy1.14rc1 which is now yanked due to a OSX problem. I changed the buildfarm to use scipy1.13.1 and only reran the one failed OSX job. However, it failed with:

======================================================================
FAIL: test_linalg_matrix_power (numba.tests.test_linalg.TestLinalgMatrixPower.test_linalg_matrix_power)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ci/miniconda3/envs/testenv_435cdd13/lib/python3.12/site-packages/numba/tests/test_linalg.py", line 2440, in test_linalg_matrix_power
    check(a, pwr)
  File "/Users/ci/miniconda3/envs/testenv_435cdd13/lib/python3.12/site-packages/numba/tests/test_linalg.py", line 2428, in check
    np.testing.assert_allclose(got, expected, rtol=res, atol=res)
  File "/Users/ci/miniconda3/envs/testenv_435cdd13/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1684, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/Users/ci/miniconda3/envs/testenv_435cdd13/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/ci/miniconda3/envs/testenv_435cdd13/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 885, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=5e-15, atol=5e-15

Mismatched elements: 1 / 49 (2.04%)
Max absolute difference among violations: 5.50254287e-15
Max relative difference among violations: 9.45511966e-14
 ACTUAL: array([[ 5.648402e-01, -2.775672e-01, -6.156343e-02,  1.464872e-01,
         1.746609e-01,  5.463447e-01,  4.996644e-01],
       [-4.047352e-01, -1.517353e-01,  5.044917e-01,  5.434956e-01,...
 DESIRED: array([[ 5.648402e-01, -2.775672e-01, -6.156343e-02,  1.464872e-01,
         1.746609e-01,  5.463447e-01,  4.996644e-01],
       [-4.047352e-01, -1.517353e-01,  5.044917e-01,  5.434956e-01,...

----------------------------------------------------------------------

Now, I need to rerun all platforms to see if it is OSX specific.

stuartarchibald

Thanks for the patch. On inspection it seems like this will fix the compatibility issues discovered in testing NumPy 2.0's rc2 build. As noted in the comments, the issue in relation to matrix power needs addressing.

stuartarchibald · 2024-06-04T12:44:30Z

    dtype = np.result_type(x_dt, y_dt, np.float64)

-    if dtype == np.complex_:
+    if dtype == np.complex128:


I searched for other occurrences of np.complex_ in the code base and could not find any.

sklam · 2024-06-04T15:40:57Z

There's a consistent problem with matrix_power function when using scipy from pip:

$ python runtests.py -k test_linalg_matrix_power
*** stack smashing detected ***: terminated
Fatal Python error: Aborted

Current thread 0x00007fa498e93280 (most recent call first):
  File "/home/siu/dev/numba/numba/tests/test_linalg.py", line 2422 in check
  File "/home/siu/dev/numba/numba/tests/test_linalg.py", line 2440 in test_linalg_matrix_power
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/case.py", line 579 in _callTestMethod
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/case.py", line 623 in run
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/case.py", line 678 in __call__
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/suite.py", line 122 in run
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/suite.py", line 84 in __call__
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/runner.py", line 217 in run
  File "/home/siu/dev/numba/numba/testing/main.py", line 170 in run
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/main.py", line 274 in runTests
  File "/home/siu/dev/numba/numba/testing/main.py", line 371 in run_tests_real
  File "/home/siu/dev/numba/numba/testing/main.py", line 386 in runTests
  File "/home/siu/dev/envs/numba.py311np2/lib/python3.11/unittest/main.py", line 102 in __init__
  File "/home/siu/dev/numba/numba/testing/main.py", line 207 in __init__
  File "/home/siu/dev/numba/numba/testing/__init__.py", line 54 in run_tests
  File "/home/siu/dev/numba/numba/testing/_runtests.py", line 25 in _main
  File "/home/siu/dev/numba/numba/runtests.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 229 in run_module
  File "/home/siu/dev/numba/runtests.py", line 22 in <module>

Extension modules: mkl._mklinit, mkl._py_mkl_service, numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg, scipy._lib._ccallback_c, numba.core.typeconv._typeconv, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, numba.mviewbuf, numba.types.itertools, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.special.cython_special (total: 57)
Aborted (core dumped)

Observations so far:

FAILING on scipy 1.13.0, 1.13.1 from PIP (wheel's blas=openblas 0.3.26.dev, 0.3.27 respsectively; numpy uses MKL)
OK with scipy 1.13.0 from default channel (blas=mkl)
OK with scipy 1.13.0 from conda-forge (blas=mkl)
OK with scipy 1.13.1 from conda-forge (blas=openblas 0.3.27)
OK on scipy 1.13.1 from PIP when both numpy and scipy uses openblas!

This might be a MKL-openblas conflict or something with openmp.

sklam · 2024-06-04T15:48:43Z

Confirmed that the yanked scipy=1.14.0rc1 wheel doesn't have the stack smashing issue even with numpy+MKL

sklam · 2024-06-04T21:04:00Z

Update to future readers. The following explanation is not accurate. OOB discussions has concluded that the issue is further down the call chain, but it is indeed due to a different in ABI. The difference is that scipy wheels are built without g77 wrappers for BLAS. The issue is resolved as of SciPy 1.14.

This appears to be a scipy problem. Consider the following disassembled code for zdotu from scipy 1.13.1 wheel vs scipy 1.13.1 conda-forge package:

SCIPY 1.13.1 wheel:


(gdb) disassemble __pyx_f_5scipy_6linalg_11cython_blas_zdotu
Dump of assembler code for function __pyx_f_5scipy_6linalg_11cython_blas_zdotu:
=> 0x00007fff3f022920 <+0>:     push   %r12
   0x00007fff3f022922 <+2>:     mov    %rdi,%r10
   0x00007fff3f022925 <+5>:     mov    %r8,%r9
   0x00007fff3f022928 <+8>:     mov    %rcx,%r8
   0x00007fff3f02292b <+11>:    mov    %rdx,%rcx
   0x00007fff3f02292e <+14>:    mov    %rsi,%rdx
   0x00007fff3f022931 <+17>:    mov    %r10,%rsi
   0x00007fff3f022934 <+20>:    sub    $0x10,%rsp
   0x00007fff3f022938 <+24>:    mov    %rsp,%rdi
   0x00007fff3f02293b <+27>:    callq  0x7fff3f0451b0 <zdotuwrp_>
   0x00007fff3f022940 <+32>:    movsd  (%rsp),%xmm0
   0x00007fff3f022945 <+37>:    movsd  0x8(%rsp),%xmm1
   0x00007fff3f02294b <+43>:    add    $0x10,%rsp
   0x00007fff3f02294f <+47>:    pop    %r12
   0x00007fff3f022951 <+49>:    retq

SCIPY 1.13.1 conda-forge:

(gdb) disassemble __pyx_f_5scipy_6linalg_11cython_blas_zdotu
Dump of assembler code for function __pyx_f_5scipy_6linalg_11cython_blas_zdotu:
=> 0x00007fffde648ab0 <+0>:     sub    $0x28,%rsp
   0x00007fffde648ab4 <+4>:     mov    %r8,%r9
   0x00007fffde648ab7 <+7>:     mov    %rcx,%r8
   0x00007fffde648aba <+10>:    mov    %rdx,%rcx
   0x00007fffde648abd <+13>:    mov    %fs:0x28,%rax
   0x00007fffde648ac6 <+22>:    mov    %rax,0x18(%rsp)
   0x00007fffde648acb <+27>:    mov    %rsp,%rax
   0x00007fffde648ace <+30>:    mov    %rsi,%rdx
   0x00007fffde648ad1 <+33>:    mov    %rdi,%rsi
   0x00007fffde648ad4 <+36>:    mov    %rax,%rdi
   0x00007fffde648ad7 <+39>:    addr32 callq 0x7fffde6648c0 <zdotuwrp_>
   0x00007fffde648add <+45>:    movsd  (%rsp),%xmm0
   0x00007fffde648ae2 <+50>:    movsd  0x8(%rsp),%xmm1
   0x00007fffde648ae8 <+56>:    mov    0x18(%rsp),%rax
   0x00007fffde648aed <+61>:    sub    %fs:0x28,%rax
   0x00007fffde648af6 <+70>:    jne    0x7fffde648afd <__pyx_f_5scipy_6linalg_11cython_blas_zdotu+77>
   0x00007fffde648af8 <+72>:    add    $0x28,%rsp
   0x00007fffde648afc <+76>:    retq   
   0x00007fffde648afd <+77>:    callq  *0x2ad1d(%rip)        # 0x7fffde673820

The wheel version is allocate 0x10 stack space while the conda-forge version is allocating 0x28 stack space. I believe this can explain the stack smashing. Essentially, the wheel version is expecting a different ABI for zdotuwrp_ (caller allocate space for return type right?).

gmarkall · 2024-06-05T09:53:52Z

I can't quite align the explanation with what I can see in the SciPy code. The zdotuwrp_ function is also part of SciPy, and looks like:

void F_FUNC(zdotuwrp,ZDOTUWRP)(double_complex *ret, CBLAS_INT *n, double_complex *zx, \
            CBLAS_INT *incx, double_complex *zy, CBLAS_INT *incy){
    *ret = F_FUNC(wzdotu,WZDOTU)(n, zx, incx, zy, incy);
}

So there isn't a return value to allocate space for. The return value arrives in %xmm0 and %xmm1, and these values are copied into the stack space of the caller immediately following the call. It looks more like the two implementations have been compiled slightly differently leading to different stack usage for the function, but I can't see that there's proof of an issue here.

A couple of other anomalies I see in this explanation:

If the caller __pyx_f_5scipy_6linalg_11cython_blas_zdotu and the callee zdotuwrp_ were both built as part of SciPy, it would be a little surprising for an ABI mismatch that wasn't widely causing issues (that would have been noticed).
The value we get back from zdotu is a double complex, which would only require 16 (0x10) bytes of storage anyway.

sklam · 2024-06-05T15:06:23Z

Reading the assembly again, i think the extra stack space is the frame-pointer:

   0x00007fffde648abd <+13>:    mov    %fs:0x28,%rax
   0x00007fffde648ac6 <+22>:    mov    %rax,0x18(%rsp)

which is later read back and check:

   0x00007fffde648ae8 <+56>:    mov    0x18(%rsp),%rax
   0x00007fffde648aed <+61>:    sub    %fs:0x28,%rax
   0x00007fffde648af6 <+70>:    jne    0x7fffde648afd <__pyx_f_5scipy_6linalg_11cython_blas_zdotu+77>

Maybe this is just two builds with different compiler flags. Something about frame-pointer checking for security.

gmarkall · 2024-06-05T15:23:57Z

The conda-forge version is probably compiled with -fstack-protector: c.f. https://security.stackexchange.com/questions/158609/how-is-the-stack-protection-enforced-in-a-binary

sklam · 2024-06-05T15:24:28Z

If the caller __pyx_f_5scipy_6linalg_11cython_blas_zdotu and the callee zdotuwrp_ were both built as part of SciPy, it would be a little surprising for an ABI mismatch that wasn't widely causing issues (that would have been noticed).

Testing for these cython exports was only added in scipy 1.14.0: scipy/scipy#20422
The yanked scipy1.14.0rc1 wheel is working fine. I think it is something in 1.13 that is unnoticed.

gmarkall · 2024-06-05T16:29:33Z

~~Are you testing with NumPy 2.0? This may be the reason for the issue with 1.13: scipy/scipy#18975~~

Edit: the PR I linked to was already in 1.12, so it's not that.

conda-forge::scipy1.13.1 has a higher error on osx-64. The test is reporting difference of: Max absolute difference among violations: 6.605827e-15 Max relative difference among violations: 1.13509129e-13

sklam · 2024-06-06T18:43:52Z

new BFID: numba_smoketest_cpu_yaml_198

sklam · 2024-06-06T21:57:53Z

smoketest passed

gmarkall

This looks good to me - I see no more uses of np.complex_ or __getstate__() so I expect it to resolve the issues observed with the build on conda-forge of Numba 0.60.0rc1 with NumPy 2.0.0rc2 (conda-forge/numba-feedstock#142)

gmarkall · 2024-06-10T09:01:10Z

Having followed the discussion about the tolerance loosening in OOB conversation, and inspecting the function of the test (matrix powers where some of the powers are relatively large) I think the loosening of the tolerance seems appropriate, so I think this is OK to RTM.

Fix numpy2.0 incompatbility issues

sklam added 2 commits June 3, 2024 11:18

Fix numba#9598. replace np.complex_ with np.complex128

008489e

Fix numba#9599. replace rng.__getstate__() with `rng.bit_generator.…

6a2e571

…state`

sklam added the skip_release_notes Skip towncrier requirement label Jun 3, 2024

sklam added the Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm label Jun 3, 2024

esc added this to the 0.60.0 milestone Jun 4, 2024

stuartarchibald reviewed Jun 4, 2024

View reviewed changes

stuartarchibald mentioned this pull request Jun 4, 2024

test_linalg_matrix_power fails due to tolerance issues when built against lapack 3.12.0 #9560

Open

1 of 2 tasks

Increase toleraance

75e37e7

conda-forge::scipy1.13.1 has a higher error on osx-64. The test is reporting difference of: Max absolute difference among violations: 6.605827e-15 Max relative difference among violations: 1.13509129e-13

sklam marked this pull request as ready for review June 6, 2024 20:09

sklam added 3 - Ready for Review BuildFarm Passed For PRs that have been through the buildfarm and passed and removed Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm labels Jun 6, 2024

gmarkall approved these changes Jun 10, 2024

View reviewed changes

gmarkall added 5 - Ready to merge Review and testing done, is ready to merge and removed 3 - Ready for Review labels Jun 10, 2024

esc merged commit 14b50af into numba:main Jun 10, 2024

sklam deleted the fix/np2compat branch June 10, 2024 17:06

esc added a commit to esc/numba that referenced this pull request Jun 11, 2024

Merge pull request numba#9602 from sklam/fix/np2compat

48faddf

Fix numpy2.0 incompatbility issues

esc mentioned this pull request Jun 11, 2024

Cherry pick/rc bugfixes for 0.60.0 final #9620

Merged

Conversation

sklam commented Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sklam commented Jun 3, 2024

Uh oh!

sklam commented Jun 3, 2024

Uh oh!

stuartarchibald left a comment

Choose a reason for hiding this comment

Uh oh!

stuartarchibald Jun 4, 2024

Choose a reason for hiding this comment

Uh oh!

sklam commented Jun 4, 2024

Uh oh!

sklam commented Jun 4, 2024

Uh oh!

sklam commented Jun 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmarkall commented Jun 5, 2024

Uh oh!

sklam commented Jun 5, 2024

Uh oh!

gmarkall commented Jun 5, 2024

Uh oh!

sklam commented Jun 5, 2024

Uh oh!

gmarkall commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sklam commented Jun 6, 2024

Uh oh!

sklam commented Jun 6, 2024

Uh oh!

gmarkall left a comment

Choose a reason for hiding this comment

Uh oh!

gmarkall commented Jun 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sklam commented Jun 3, 2024 •

edited

Loading

sklam commented Jun 4, 2024 •

edited

Loading

gmarkall commented Jun 5, 2024 •

edited

Loading