Skip to content

linux-64 conda CI fails on NumPy 2.3: CUDA-sim test_fp16_intrinsics_common #10807

Closed
#10822
Closed
linux-64 conda CI fails on NumPy 2.3: CUDA-sim test_fp16_intrinsics_common#10807
#10822
Assignees
swap357
Labels
bug

Description

@swap357
swap357
opened on Aug 28, 2026

Weekly linux-64 conda builder failed only on py 3.13 + np 2.3.
https://github.com/numba/numba/actions/runs/32559494414/job/96999073327

Same workflow: py 3.13 + np 2.2 passed, py 3.14 + np 2.5 passed, all other cells passed.

5 failures on the failing job were all CUDASIM related:
numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic.test_fp16_intrinsics_common (sin, cos, log, log10, exp). The test compares per-element CUDASIM np.sin(x, dtype=np.float16) to a vectorized fn(x, dtype=np.float16).

NumPy 2.3 changed the array path (AVX512-FP16 SVML, ~1–4 ULP; numpy#30821). Reverted in 2.4.5. Default assert_allclose (rtol=1e-7) fails on 1 float16 ulp.

This job is CPU-only. CUDA tests run because the workflow sets NUMBA_ENABLE_CUDASIM=1 and numba.tests discovery always includes numba.cuda.tests.

repro-

NUMBA_ENABLE_CUDASIM=1 python -m numba.runtests -b \
  numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic.test_fp16_intrinsics_common

Needs NumPy 2.3.x on a CPU with AVX512-FP16.

Activity

  1. swap357
    added
    bug
    on Aug 28, 2026

  2. swap357 commented on Aug 28, 2026

    @swap357
    swap357
    on Aug 28, 2026
    ContributorAuthor

    This is CPU-specific behavior.
    Failures were observed on -
    Intel emerald rapids (https://github.com/numba/numba/actions/runs/32559494414/job/96999073327#step:8:719)
    previous to this, workflows with py3.13+np2.3 package testing ran with -
    AMD zen3 (https://github.com/numba/numba/actions/runs/31871841507/job/94981897785#step:8:720)
    AMD zen4 (https://github.com/numba/numba/actions/runs/31246399942/job/93783038692#step:8:720)

  3. sklam
    assigned
    swap357
    on Sep 1, 2026

  4. esc commented on Sep 8, 2026

    @esc
    esc
    on Sep 8, 2026
    Member

    Failing job: https://github.com/numba/numba/actions/runs/33952384424/job/101269798444

    Failure also observed on -
    Intel granite rapids (https://github.com/numba/numba/actions/runs/33952384424/job/101269798444#step:8:13424)

    numba -s reports CPU Name: graniterapids with avx512fp16 in CPU Features.

    Granite Rapids is Emerald Rapids' successor and also has AVX512-FP16.

    ======================================================================
FAIL: test_fp16_intrinsics_common (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic.test_fp16_intrinsics_common) (fn=<ufunc 'sin'>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numba/cuda/tests/cudapy/test_intrinsics.py", line 772, in test_fp16_intrinsics_common
    np.testing.assert_allclose(r, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 1711, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         verbose=verbose, header=header, equal_nan=equal_nan,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         strict=strict)
                         ^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 919, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 4 / 32 (12.5%)
Max absolute difference among violations: 0.0004883
Max relative difference among violations: 0.00068
 ACTUAL: array([-0.293 , -0.8447, -0.7266,  0.8687,  0.3357,  0.654 , -0.1199,
       -0.2788,  0.809 ,  0.98  , -0.823 , -0.75  , -0.5767, -0.9927,
        0.4475, -0.9897,  0.991 ,  0.316 ,  0.6543,  0.0396,  0.971 ,...
 DESIRED: array([-0.293 , -0.8447, -0.7266,  0.8687,  0.3357,  0.654 , -0.1199,
       -0.2788,  0.809 ,  0.98  , -0.8237, -0.75  , -0.5767, -0.9927,
        0.4475, -0.9897,  0.991 ,  0.316 ,  0.6543,  0.0396,  0.971 ,...

======================================================================
FAIL: test_fp16_intrinsics_common (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic.test_fp16_intrinsics_common) (fn=<ufunc 'cos'>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numba/cuda/tests/cudapy/test_intrinsics.py", line 772, in test_fp16_intrinsics_common
    np.testing.assert_allclose(r, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 1711, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         verbose=verbose, header=header, equal_nan=equal_nan,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         strict=strict)
                         ^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 919, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 2 / 32 (6.25%)
Max absolute difference among violations: 0.0004883
Max relative difference among violations: 0.00071
 ACTUAL: array([-0.956 ,  0.535 ,  0.687 , -0.4954, -0.942 , -0.757 ,  0.9927,
        0.9604,  0.588 , -0.1996, -0.5674,  0.6616, -0.817 ,  0.1198,
        0.8945, -0.1428,  0.1337,  0.9487,  0.7563, -0.999 ,  0.2391,...
 DESIRED: array([-0.956 ,  0.535 ,  0.6875, -0.4954, -0.942 , -0.757 ,  0.9927,
        0.9604,  0.588 , -0.1996, -0.5674,  0.6616, -0.817 ,  0.1198,
        0.8945, -0.1428,  0.1337,  0.9487,  0.7563, -0.9995,  0.2391,...

======================================================================
FAIL: test_fp16_intrinsics_common (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic.test_fp16_intrinsics_common) (fn=<ufunc 'log'>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numba/cuda/tests/cudapy/test_intrinsics.py", line 772, in test_fp16_intrinsics_common
    np.testing.assert_allclose(r, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 1711, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         verbose=verbose, header=header, equal_nan=equal_nan,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         strict=strict)
                         ^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 919, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 13 / 32 (40.6%)
Max absolute difference among violations: 0.007812
Max relative difference among violations: 0.0008717
 ACTUAL: array([11.04 , 10.41 ,  9.41 ,  8.555, 10.39 , 10.82 , 10.69 ,  8.96 ,
       10.86 ,  9.98 , 10.4  ,  9.93 , 10.805,  8.95 , 10.69 , 10.35 ,
       10.5  , 10.375, 10.98 , 10.91 ,  9.984, 10.73 , 10.484, 10.85 ,...
 DESIRED: array([11.05 , 10.41 ,  9.414,  8.555, 10.39 , 10.82 , 10.69 ,  8.97 ,
       10.86 ,  9.97 , 10.41 ,  9.93 , 10.805,  8.96 , 10.69 , 10.35 ,
       10.5  , 10.375, 10.984, 10.914,  9.984, 10.73 , 10.484, 10.86 ,...

======================================================================
FAIL: test_fp16_intrinsics_common (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic.test_fp16_intrinsics_common) (fn=<ufunc 'log10'>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numba/cuda/tests/cudapy/test_intrinsics.py", line 772, in test_fp16_intrinsics_common
    np.testing.assert_allclose(r, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 1711, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         verbose=verbose, header=header, equal_nan=equal_nan,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         strict=strict)
                         ^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 919, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 1 / 32 (3.12%)
Max absolute difference among violations: 0.003906
Max relative difference among violations: 0.0008197
 ACTUAL: array([4.797, 4.52 , 4.086, 3.715, 4.51 , 4.7  , 4.64 , 3.893, 4.715,
       4.332, 4.516, 4.312, 4.69 , 3.889, 4.64 , 4.496, 4.56 , 4.504,
       4.77 , 4.74 , 4.336, 4.656, 4.555, 4.715, 4.81 , 3.926, 4.3  ,
       4.574, 4.758, 4.36 , 4.695, 4.434], dtype=float16)
 DESIRED: array([4.797, 4.52 , 4.086, 3.715, 4.51 , 4.7  , 4.64 , 3.893, 4.715,
       4.332, 4.516, 4.312, 4.69 , 3.889, 4.64 , 4.496, 4.56 , 4.504,
       4.766, 4.74 , 4.336, 4.656, 4.555, 4.715, 4.81 , 3.926, 4.3  ,
       4.574, 4.758, 4.36 , 4.695, 4.434], dtype=float16)

======================================================================
FAIL: test_fp16_intrinsics_common (numba.cuda.tests.cudapy.test_intrinsics.TestCudaIntrinsic.test_fp16_intrinsics_common) (fn=<ufunc 'exp'>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numba/cuda/tests/cudapy/test_intrinsics.py", line 780, in test_fp16_intrinsics_common
    np.testing.assert_allclose(r, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/true/conda-bld/numba_1788593207150/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.13/site-packages/numpy/testing/_private/utils.py", line 1711, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         verbose=verbose, header=header, equal_nan=equal_nan,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         strict=strict)
                         ^^^^^^^^^^^^^^

  5. swap357 commented on Sep 8, 2026

    @swap357
    swap357
    on Sep 8, 2026
    ContributorAuthor

    Based on discussion in maintainers meeting, solution here is to patch the these CUDA sim tests to increase tolerance.

  6. swap357
    closed this as completedin #10822on Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions