Skip to content

test_capi.test_opt.test_call_len_known_length fails on JIT+debug builds #139393

Closed
Closed
test_capi.test_opt.test_call_len_known_length fails on JIT+debug builds#139393
Assignees
picnixz
Labels
3.15bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)testsTests in the Lib/test dirtopic-C-APItopic-JITtype-bugAn unexpected behavior, bug, or error

Description

@ashm-dev
ashm-dev
opened on Sep 28, 2025

Bug report

Bug description:

Bug Description

The test_capi test suite fails when CPython is built from the main branch with debug, optimizations (PGO), LTO, and the experimental JIT enabled.

Steps to Reproduce

  1. Clone the cpython repository.
  2. Configure the build with the following flags: 
    ./configure --with-pydebug --enable-optimizations --with-lto --enable-experimental-jit
  3. Build and run tests: 
    sudo make -j"$(nproc)" test

Expected Behavior

All tests should pass.

Actual Behavior

The test run fails with one failed test: test_capi.
The specific failing test is test_call_len_known_length within test.test_capi.test_opt.TestUopsOptimization.

Test Output

sudo_make_j_nproc_test.txt

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. ashm-dev
    added
    type-bugAn unexpected behavior, bug, or error
    on Sep 28, 2025
  2. picnixz
    changed the title [-]test_capi fails on debug build with JIT, LTO and PGO[/-] [+]`test_capi.test_opt.test_call_len_known_length` fails on JIT LTO+PGO debug builds[/+] on Sep 28, 2025
  3. picnixz
    added
    testsTests in the Lib/test dir
    topic-C-API
    on Sep 28, 2025

  4. picnixz commented on Sep 28, 2025

    @picnixz
    picnixz
    on Sep 28, 2025
    Member

    Does it actually fail with just JIT + debug? or does it fail for JIT+LTO+PGO+debug (or maybe just LTO+PGO+debug) in particular? Stated otherwise, which feature is necessary for the test to fail?

  5. ashm-dev commented on Sep 28, 2025

    @ashm-dev
    ashm-dev
    on Sep 28, 2025
    ContributorAuthor

    @picnixz I've completed testing with all requested configurations. Here is the summary of the results:

    1. --with-pydebug --enable-optimizations --with-lto --enable-experimental-jit
      (debug + LTO + PGO + JIT)

      • Result: test_capi fails.

    2. --with-pydebug --enable-experimental-jit
      (debug + JIT)

      • Result: test_capi passes, but test_gdb fails.

    3. --with-pydebug --enable-optimizations --with-lto
      (debug + LTO + PGO)

      • Result: All tests pass successfully.

    This confirms that the test_capi failure is specifically triggered by the combination of the experimental JIT and the optimizations (--enable-optimizations --with-lto).

    Neither JIT alone nor the optimizations alone (in a debug build) cause test_capi to fail.

    Logs:

    pydebug_pgo_lto_sudo_make_j_nproc_test.txt

    pydebug_jit_sudo_make_j_nproc_test.txt

  6. picnixz
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    buildThe build process and cross-build
    topic-JIT
    and removed
    buildThe build process and cross-build
    on Sep 28, 2025

  7. picnixz commented on Sep 28, 2025

    @picnixz
    picnixz
    on Sep 28, 2025
    Member

    Thank you. I don't think LTO would be responsible for that, but can you also check debug+JIT+PGO?

  8. ashm-dev commented on Sep 28, 2025

    @ashm-dev
    ashm-dev
    on Sep 28, 2025
    ContributorAuthor

    You were right! I've run the build with debug+JIT+PGO (by using ./configure --with-pydebug --enable-optimizations --enable-experimental-jit), and the test fails in the exact same way.

    pydebug_jit_pgo_sudo_make_j_nproc_test.txt

  9. ashm-dev commented on Sep 28, 2025

    @ashm-dev
    ashm-dev
    on Sep 28, 2025
    ContributorAuthor

    @picnixz I did additional testing with the debug+ LTO+JIT configuration (without PGO) to verify your assumption that LTO is not the cause.

    Results for ./configure --with-pydebug --with-lto --enable-experimental-jit:

    • test_capi also crashes with the same AssertionError error: '_CALL_LEN' not found.
    • In addition, the test_gdb tests also crash in this build (test.test_gdb.test_backtrace, test.test_gdb.test_misc, test.test_gdb.test_pretty_print).

    This clarifies the overall picture. It seems that your assumption that only PGO is the problem was not entirely correct. The error in test_capi is reproduced when combining JIT with any of the advanced optimizations (LTO or PGO).

    pydebug_jit_lto_sudo_make_j_nproc_test.txt

  10. picnixz commented on Sep 28, 2025

    @picnixz
    picnixz
    on Sep 28, 2025
    Member

    Disclaimer: I'm using clang-18 for the JIT instead of clang-19 so some attributes are not valid but it shouldn't change much I guess.

    I think I know the cause. The test assumes that the length is >= _PY_NSMALLPOSINTS. But we recently updated _PY_NSMALLPOSINTS to be 1025 instead of 256 (in 7ce25ed). To be precise, I think we hit the following check:

    PyObject *temp = PyLong_FromSsize_t(tuple_length);
if (temp == NULL) {
    goto error;
}
if (_Py_IsImmortal(temp)) {
    REPLACE_OP(this_instr, _POP_CALL_ONE_LOAD_CONST_INLINE_BORROW,
               0, (uintptr_t)temp);
}

    For a small tuple length, we update the instruction. However, I can confirm that debug + JIT (no optimizations) also trigger the issue with clang-18. Can you verify once again that you have a clean build?

  11. picnixz
    self-assigned this
    on Sep 28, 2025

  12. ashm-dev commented on Sep 28, 2025

    @ashm-dev
    ashm-dev
    on Sep 28, 2025
    ContributorAuthor

    I understand you!
    All my previous launches were on the default gcc from ubuntu
    I decided to check how the tests will behave on clang, and yes, my results are the same as yours, jit+pgo and jit are falling due to capi, and the rest are going fine.

    Versions:
    gcc

    ➜  cpython gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/12/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 12.4.0-5ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-12/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-12 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-12-eQp2fa/gcc-12-12.4.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-eQp2fa/gcc-12-12.4.0/debian/tmp-gcn/usr --enable-offload-defaulted --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.4.0 (Ubuntu 12.4.0-5ubuntu1)

    clang

    ➜  cpython clang -v
Ubuntu clang version 20.1.2 (0ubuntu1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-20/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/13
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Candidate multilib: .;@m64
Selected multilib: .;@m64

    Clang test logs:

    clang_pydebug_jit_lto_pgo_sudo_make_j_nproc_test.txt
    clang_pydebug_jit_lto_sudo_make_j_nproc_test.txt
    clang_pydebug_jit_pgo_sudo_make_j_nproc_test.txt
    clang_pydebug_jit_sudo_make_j_nproc_test.txt
    clang_pydebug_lto_pgo_sudo_make_j_nproc_test.txt

  13. picnixz
    changed the title [-]`test_capi.test_opt.test_call_len_known_length` fails on JIT LTO+PGO debug builds[/-] [+]`test_capi.test_opt.test_call_len_known_length` fails on JIT builds[/+] on Sep 28, 2025
  14. picnixz
    added
    3.15bugs and security fixes
    on Sep 28, 2025
  15. picnixz
    changed the title [-]`test_capi.test_opt.test_call_len_known_length` fails on JIT builds[/-] [+]`test_capi.test_opt.test_call_len_known_length` fails on JIT+debug builds[/+] on Sep 28, 2025
  16. picnixz
    added a commit that references this issue on Sep 28, 2025

    gh-139393: fix `_CALL_LEN` JIT tests for tuples (#139394)

    3779f2b
  17. picnixz
    closed this as completedon Sep 28, 2025

  18. picnixz commented on Sep 28, 2025

    @picnixz
    picnixz
    on Sep 28, 2025
    Member

    Thanks for the report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.15bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)testsTests in the Lib/test dirtopic-C-APItopic-JITtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions