Skip to content

PPC64LE Fedora Stable Clang 3.x fails test_gdb #104736

Open
Open
PPC64LE Fedora Stable Clang 3.x fails test_gdb#104736
Labels
testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@JelleZijlstra
JelleZijlstra
opened on May 22, 2023

Buildbot https://buildbot.python.org/all/#/builders/435 has been failing test_gdb with errors like these:

======================================================================
FAIL: test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction) [_testcapi.meth_varargs]
Verify that "py-bt" displays invocations of PyCFunction instances
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb.py", line 944, in test_pycfunction
    self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_varargs' not found in 'Breakpoint 1 (meth_varargs) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n2307\t    return Py_BuildValue("NO", _null_to_none(self), args);\n#0  meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n#1  PyModuleDef_Type ()\nBacktrace stopped: frame did not save the PC\nUnable to locate python frame\n'

It last succeeded on commit 373bca0 and first failed on fcd5fb4, suggesting that the culprit is one of these commits:

Neither looks likely to affect test_gdb though.

Linked PRs

Activity

  1. JelleZijlstra
    added
    type-bugAn unexpected behavior, bug, or error
    on May 22, 2023

  2. encukou commented on May 22, 2023

    @encukou
    encukou
    on May 22, 2023
    Member

    It looks like GDB can't extract the C stack here. I SSHd to the buildbot and tried:

    $ gdb --args ./python -c 'import _testcapi; _testcapi.meth_varargs()'
[...]
Reading symbols from ./python...

(gdb) b meth_varargs
Function "meth_varargs" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (meth_varargs) pending.

(gdb) r
[...]
Breakpoint 1, meth_varargs (self=<module at remote 0x7ffff6f77d70>, args=())
    at ./Modules/_testcapimodule.c:2307
2307	    return Py_BuildValue("NO", _null_to_none(self), args);

(gdb) bt
#0  meth_varargs (self=<module at remote 0x7ffff6f77d70>, args=()) at ./Modules/_testcapimodule.c:2307
#1  0x00007ffff7cf7db0 in PyModuleDef_Type () from /home/pviktori/cpython/libpython3.12d.so.1.0
Backtrace stopped: frame did not save the PC

    The stack looks corrupted: PyModuleDef_Type is data, not a function.
    This looks like “miscommunication” between clang and gdb (bad debug info?)

    The test seems very sensitive to platform/compiler, e.g.:

    if 'Clang' in platform.python_compiler() and sys.platform == 'darwin':
    raise unittest.SkipTest("test_gdb doesn't work correctly when python is"
                            " built with LLVM clang")

    Note that the “installed” buildbots are green. (AFAIK these test Python after make install.) But when I make install, GDB still fails to extract the backtrace on the installed Python.

    ping @vstinner, the PEP-11 ppc64le/clang maintainer

  3. encukou commented on May 22, 2023

    @encukou
    encukou
    on May 22, 2023
    Member

    See #65462 (comment):

    Unless someone has a better idea, I suggest we just disable test_gdb on OS X or, somewhat more precisely, when Python has been compiled with LLVM clang.

    Doesn't clang always use LLVM?
    The skip is now 'Clang' in platform.python_compiler() and sys.platform == 'darwin', should it just be 'Clang' in platform.python_compiler()?

  4. stratakis commented on May 22, 2023

    @stratakis
    stratakis
    on May 22, 2023
    Contributor

    The failure started after the update of Clang to 16.0.0 from 15.0.7.

  5. thesamesam commented on May 23, 2023

    @thesamesam
    thesamesam
    on May 23, 2023
    Contributor

    See #65462 (comment):

    Unless someone has a better idea, I suggest we just disable test_gdb on OS X or, somewhat more precisely, when Python has been compiled with LLVM clang.

    Doesn't clang always use LLVM? The skip is now 'Clang' in platform.python_compiler() and sys.platform == 'darwin', should it just be 'Clang' in platform.python_compiler()?

    I took it to mean "not Apple's Clang" which is distinct (has its own versioning and is essentially a fork). I might be wrong here though.

  6. vstinner commented on May 23, 2023

    @vstinner
    vstinner
    on May 23, 2023
    Member

    Maybe tets_gdb can skip a test if "frame did not save the PC" is found in the output. We already have many skips like that.

  7. carljm commented on Aug 30, 2023

    @carljm
    carljm
    on Aug 30, 2023
    Member

    It looks like multiple Clang buildbots are still failing test_gdb.

  8. vstinner commented on Sep 7, 2023

    @vstinner
    vstinner
    on Sep 7, 2023
    Member

    I closed issue #109053 as a duplicate of this issue.

  9. vstinner commented on Sep 13, 2023

    @vstinner
    vstinner
    on Sep 13, 2023
    Member

    I did tests on two ppc64le machines. In short, test_gdb only failed on ppc64le if Python is built with clang version 16.0.6 (clang -Og). Well, that's what we saw on buildbots. But I preferred to confirm manually. I built Python manually and ran test_gdb manually with -u all to no skip any test.

    • Machine 1:

      • Fedora 38
      • Linux xxx 6.2.8-200.fc37.ppc64le #1 SMP Wed Mar 22 18:34:43 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux
      • gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
      • clang version 16.0.6 (Fedora 16.0.6-2.fc38)
      • GNU gdb (GDB) Fedora Linux 13.2-3.fc38

    • Machine 2:

      • Fedora 37
      • Linux localhost.localdomain 5.8.17-200.fc32.ppc64le #1 SMP Thu Oct 29 18:00:08 UTC 2020 ppc64le GNU/Linux
      • gcc (GCC) 12.3.1 2023050 (Red Hat 12.3.1-1)
      • clang version 15.0.7 (Fedora 15.0.7-2.fc37)
      • GNU gdb (GDB) Fedora Linux 13.2-2.fc37

    GCC

    Test: ./configure --with-pydebug && make && ./python -m test test_gdb -u all -v.

    Build built with: gcc -Og.

    • Machine 1: test_gdb: SUCCESS

      • Only 1 test was skipped: test_print_after_up(): Unable to read information on python frame found in gdb output

    • Machine 2: test_gdb: SUCCESS

      • Only 1 test was skipped: test_print_after_up(): Unable to read information on python frame found in gdb output"
      • Total tests: run=46 skipped=1

    LLVM clang

    Test: ./configure --with-pydebug CC=clang && make && ./python -m test test_gdb -u all -v.

    Build built with: clang -Og.

    • Machine 1: test_gdb: FAILURE

      • Failed tests: test_pycfunction() [_testcapi.meth_varargs] and test_pycfunction [_testcapi.meth_fastcall_keywords] failed.
      • On 46 tests, 39 were skipped: (unable to read python frame information) or Unable to read information on python frame found in gdb output.
      • Total tests: run=46 failures=2 skipped=39

    • Machine 2: test_gdb: SUCCESS

      • Only 5 tests were run in fact, 41 have been skipped: (unable to read python frame information) or Unable to read information on python frame found in gdb output
      • Total tests: run=46 skipped=41

  10. vstinner commented on Sep 13, 2023

    @vstinner
    vstinner
    on Sep 13, 2023
    Member

    Reproduce _testcapi.meth_varargs() issue.

    script.py:

    import _testcapi
def foo():
    _testcapi.meth_varargs()
def bar():
    foo()
bar()

    cmds:

    file ./python
source Tools/gdb/libpython.py
set breakpoint pending yes
break meth_varargs
run script.py
where
py-bt

    gdb output on ppc64le with clang -Og:

    $ gdb -x cmds
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_varargs" not defined.
Breakpoint 1 (meth_varargs) pending.

Breakpoint 1, meth_varargs (self=<module at remote 0x7fffea3d23f0>, args=()) at ./Modules/_testcapimodule.c:2060
2060	    return Py_BuildValue("NO", _null_to_none(self), args);

#0  meth_varargs (self=<module at remote 0x7fffea3d23f0>, args=()) at ./Modules/_testcapimodule.c:2060
#1  0x0000000010625d78 in PyModuleDef_Type ()
Backtrace stopped: frame did not save the PC

Unable to locate python frame

    The important message here is Backtrace stopped: frame did not save the PC. The gdb traceback is really just 2 frames and the 2nd frame PyModuleDef_Type () is wrong :-(

    Comparison on x86-64, Python built with gcc -Og:

    vstinner@mona$ gdb -x cmds
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_varargs" not defined.
Breakpoint 1 (meth_varargs) pending.

Breakpoint 1, meth_varargs (self=self@entry=<module at remote 0x7fffea4c29f0>, args=args@entry=()) at ./Modules/_testcapimodule.c:2059
2059	{


#0  meth_varargs (self=self@entry=<module at remote 0x7fffea4c29f0>, args=args@entry=()) at ./Modules/_testcapimodule.c:2059
#1  0x00000000004f2dcc in cfunction_call (func=func@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=(), 
    kwargs=kwargs@entry=0x0) at Objects/methodobject.c:551
#2  0x00000000004ab06f in _PyObject_MakeTpCall (tstate=tstate@entry=0x980358 <_PyRuntime+508728>, 
    callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=0x7ffff7fb9128, nargs=<optimized out>, 
    keywords=keywords@entry=0x0) at Objects/call.c:242
#3  0x00000000004ab282 in _PyObject_VectorcallTstate (tstate=0x980358 <_PyRuntime+508728>, 
    callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:185
#4  0x00000000004ab2d0 in PyObject_Vectorcall (callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, 
    args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#5  0x000000000059e417 in _PyEval_EvalFrameDefault (tstate=0x980358 <_PyRuntime+508728>, frame=0x7ffff7fb90d0, throwflag=0) at Python/generated_cases.c.h:3773
(...)
#20 0x000000000041b78f in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15


Traceback (most recent call first):
  <built-in method meth_varargs of module object at remote 0x7fffea4c29f0>
  File "/home/vstinner/python/main/script.py", line 3, in foo
    _testcapi.meth_varargs()
  File "/home/vstinner/python/main/script.py", line 5, in bar
    foo()
  File "/home/vstinner/python/main/script.py", line 6, in <module>
    bar()
    • gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
    • GNU gdb (GDB) Fedora Linux 13.2-3.fc38

    On x86-64 with gcc, I have 20 frames up to main() and the 2nd frame is cfunction_call() as expected.

  11. vstinner commented on Sep 13, 2023

    @vstinner
    vstinner
    on Sep 13, 2023
    Member

    script2.py:

    import _testcapi
def foo():
    _testcapi.meth_fastcall_keywords()
def bar():
    foo()
bar()

    cmds2:

    file ./python
source Tools/gdb/libpython.py
set breakpoint pending yes
break meth_fastcall_keywords
run script2.py
where
py-bt

    gdb output on ppc64 with clang -Og:

    $ gdb -x cmds2
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_fastcall_keywords" not defined.
Breakpoint 1 (meth_fastcall_keywords) pending.

Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffea3d23f0>, args=0x7fffea410128, nargs=0, kwargs=0x0) at ./Modules/_testcapimodule.c:2107
2107	    PyObject *pyargs = _fastcall_to_tuple(args, nargs);


#0  meth_fastcall_keywords (self=<module at remote 0x7fffea3d23f0>, args=0x7fffea410128, nargs=0, kwargs=0x0) at ./Modules/_testcapimodule.c:2107
#1  0x0000000000000000 in ?? ()


Unable to locate python frame

    Here, I think that the important part is #1 0x0000000000000000 in ?? (). When gdb says that, usually it's likely that the traceback is truncated or wrong.

    For comparison, gdb on x86-64 on Python built with gcc -Og:

    vstinner@mona$ gdb -x cmds2
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_fastcall_keywords" not defined.
Breakpoint 1 (meth_fastcall_keywords) pending.

Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0)
    at ./Modules/_testcapimodule.c:2106
2106	{


#0  meth_fastcall_keywords (self=<module at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0)
    at ./Modules/_testcapimodule.c:2106
#1  0x00000000004f259a in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>, 
    args=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=0x0) at Objects/methodobject.c:441
#2  0x00000000004ab208 in _PyObject_VectorcallTstate (tstate=0x980358 <_PyRuntime+508728>, 
    callable=callable@entry=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargsf=9223372036854775808, 
    kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:187
#3  0x00000000004ab2d0 in PyObject_Vectorcall (callable=callable@entry=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>, 
    args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#4  0x000000000059e417 in _PyEval_EvalFrameDefault (tstate=0x980358 <_PyRuntime+508728>, frame=0x7ffff7fb90d0, throwflag=0) at Python/generated_cases.c.h:3773
(...)
#19 0x000000000041b78f in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15


Traceback (most recent call first):
  <built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>
  File "/home/vstinner/python/main/script2.py", line 3, in foo
    _testcapi.meth_fastcall_keywords()
  File "/home/vstinner/python/main/script2.py", line 5, in bar
    foo()
  File "/home/vstinner/python/main/script2.py", line 6, in <module>
    bar()

    Here the 19 frames are complete up to main().

    • gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
    • GNU gdb (GDB) Fedora Linux 13.2-3.fc38
  12. vstinner
    added 2 commits that reference this issue on Sep 13, 2023

    pythongh-104736: Fix test_gdb tests on ppc64le with clang

    2ea0fd8

    pythongh-104736: Fix test_gdb tests on ppc64le with clang

    4958e10

  13. vstinner commented on Sep 13, 2023

    @vstinner
    vstinner
    on Sep 13, 2023
    Member

    I wrote PR #109360 which fix test_gdb on Linux ppc64lc when Python is built with clang -Og (LLVM clang 16).

  14. vstinner
    added a commit that references this issue on Sep 13, 2023

    gh-104736: Fix test_gdb tests on ppc64le with clang (#109360)

    44d9a71

  15. 5 remaining items

  16. Yhg1s
    added a commit that references this issue on Sep 13, 2023

    [3.12] gh-104736: Fix test_gdb tests on ppc64le with clang (GH-109360) (

    c30d5df
  17. vstinner
    added a commit that references this issue on Sep 13, 2023

    pythongh-104736: Fix test_gdb tests on ppc64le with clang (python#109360

    ad00248
  18. vstinner
    added 2 commits that reference this issue on Oct 4, 2023

    pythongh-104736: Fix test_gdb tests on ppc64le with clang (python#109360

    5ead95a

    [3.12] gh-109972: Enhance test_gdb (#110026) (#110351)

    bbce8bd
  19. miss-islington
    added a commit that references this issue on Oct 4, 2023

    [3.12] pythongh-109972: Enhance test_gdb (pythonGH-110026) (pythonGH-…

    50ec4bb
  20. vstinner
    added a commit that references this issue on Oct 4, 2023

    [3.11] [3.12] gh-109972: Enhance test_gdb (GH-110026) (GH-110351) (#1…

    aa8d3db

  21. encukou commented on Feb 14, 2025

    @encukou
    encukou
    on Feb 14, 2025
    Member

    The issue reappeared on Python 3.11 with clang 19 (f41). New failure:

    =================================================================
FAIL: test_pycfunction_fastcall_keywords (test.test_gdb.test_cfunction.CFunctionTests.test_pycfunction_fastcall_keywords) [_testcapi.meth_fastcall_keywords]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb/test_cfunction.py", line 67, in check_pycfunction
    self.check(func_name, cmd)
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb/test_cfunction.py", line 26, in check
    self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_fastcall_keywords' not found in 'Breakpoint 1 (meth_fastcall_keywords) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724\n5724\t    PyObject *pyargs = _fastcall_to_tuple(args, nargs);\n#0  meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724\n#1  _PyRuntime ()\n#2  _PyObject_VectorcallTstate (tstate=, callable=<unknown at remote 0x105e0c88>, args=<optimized out>, nargsf=140737115783464, kwnames=<optimized out>) at ./Include/internal/pycore_call.h:92\n#3  _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=, throwflag=<optimized out>) at ./Include/object.h:524\n#4  _PyEval_EvalFrame (tstate=<_PyRuntime+166328>, frame=, throwflag=0) at ./Include/internal/pycore_ceval.h:73\n#5  _PyEval_Vector (tstate=<_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434\n#6  PyEval_EvalCode (co=<optimized out>, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x106728d0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7fffe9daf650>, \'_testcapi\': <module at remote 0x7fffe9bea9f0>, \'foo\': <function at remote 0x7fffe9bcf6a0>, \'bar\': <function at remote 0x7fffe9bf0ec0>}, locals=<optimized out>) at Python/ceval.c:1148\n#7  run_eval_code_obj (tstate=<_PyRuntime+166328>, co=, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x106728d0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remo
emote 0x106728d0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7fffe9daf650>, \'_testcapi\': <module at remote 0x7fffe9bea9f0>, \'foo\': <function at remote 0x7fffe9bcf6a0>, \'bar\': <function at remote 0x7fffe9bf0ec0>}, flags=) at Python/pythonrun.c:1632\n#10 PyRun_SimpleStringFlags (command="\\nimport _testcapi\\ndef foo():\\n    _testcapi.meth_fastcall_keywords()\\ndef bar():\\n    foo()\\nbar()\\n\\n", flags=) at Python/pythonrun.c:487\n#11 pymain_run_command (command=<optimized out>) at Modules/main.c:255\n#12 pymain_run_python (exitcode=<optimized out>) at Modules/main.c:596\n#13 Py_RunMain () at Modules/main.c:684\n#14 pymain_main (args=) at Modules/main.c:714\n#15 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:738\n#16 main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15\nTraceback (most recent call first):\n  File "<string>", line 4, in foo\n  File "<string>", line 6, in bar\n  File "<string>", line 7, in <module>\n'
Stdout:
  test call: _testcapi.meth_fastcall_keywords()

    that is:

    Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724
5724	    PyObject *pyargs = _fastcall_to_tuple(args, nargs);
#0  meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724
#1  _PyRuntime ()
#2  _PyObject_VectorcallTstate (tstate=, callable=<unknown at remote 0x105e0c88>, args=<optimized out>, nargsf=140737115783464, kwnames=<optimized out>) at ./Include/internal/pycore_call.h:92
#3  _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=, throwflag=<optimized out>) at ./Include/object.h:524
#4  _PyEval_EvalFrame (tstate=<_PyRuntime+166328>, frame=, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#5  _PyEval_Vector (tstate=<_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#6  PyEval_EvalCode (co=<optimized out>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x106728d0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fffe9daf650>, '_testcapi': <module at remote 0x7fffe9bea9f0>, 'foo': <function at remote 0x7fffe9bcf6a0>, 'bar': <function at remote 0x7fffe9bf0ec0>}, locals=<optimized out>) at Python/ceval.c:1148
#7  run_eval_code_obj (tstate=<_PyRuntime+166328>, co=, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x106728d0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remo
emote 0x106728d0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fffe9daf650>, '_testcapi': <module at remote 0x7fffe9bea9f0>, 'foo': <function at remote 0x7fffe9bcf6a0>, 'bar': <function at remote 0x7fffe9bf0ec0>}, flags=) at Python/pythonrun.c:1632
#10 PyRun_SimpleStringFlags (command="\nimport _testcapi\ndef foo():\n    _testcapi.meth_fastcall_keywords()\ndef bar():\n    foo()\nbar()\n\n", flags=) at Python/pythonrun.c:487
#11 pymain_run_command (command=<optimized out>) at Modules/main.c:255
#12 pymain_run_python (exitcode=<optimized out>) at Modules/main.c:596
#13 Py_RunMain () at Modules/main.c:684
#14 pymain_main (args=) at Modules/main.c:714
#15 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:738
#16 main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
Traceback (most recent call first):
  File "<string>", line 4, in foo
  File "<string>", line 6, in bar
  File "<string>", line 7, in <module>
  22. encukou
    reopened this on Feb 14, 2025
  23. picnixz
    added
    testsTests in the Lib/test dir
    on Feb 14, 2025

  24. vstinner commented on Feb 19, 2025

    @vstinner
    vstinner
    on Feb 19, 2025
    Member

    The issue reappeared on Python 3.11 with clang 19 (f41).

    @encukou: This build is a success, test_gdb didn't fail.

  25. encukou commented on Feb 20, 2025

    @encukou
    encukou
    on Feb 20, 2025
    Member

    Oops! See the next one, or https://buildbot.python.org/#/builders/967/builds/1169

  26. vstinner commented on Feb 21, 2025

    @vstinner
    vstinner
    on Feb 21, 2025
    Member

    #2 _PyObject_VectorcallTstate (tstate=, callable=<unknown at remote 0x105e0c88>, args=, nargsf=140737115783464, kwnames=) at ./Include/internal/pycore_call.h:92

    That's the classical <optimized out> problem. When the Python binary is optimized, sometimes gdb fails to read function arguments.

    I'm not sure why this problem only occurs on the 3.11 branch. Python internals likely changed in the meanwhile, and so clang optimizes the binary differently. Maybe gdb is able to read function arguments on Python 3.12+.

    The 3.11 branch no longer accepts bugfixes, so I don't think that it's worth it to investigate.

  27. encukou commented on Feb 21, 2025

    @encukou
    encukou
    on Feb 21, 2025
    Member

    Should we mark the builder as unstable, then?

  28. vstinner commented on Feb 21, 2025

    @vstinner
    vstinner
    on Feb 21, 2025
    Member

    Should we mark the builder as unstable, then?

    It sounds like a reasonable tradeoff. Is it possible to only mark a builder as unstable on a specific branch?

  29. vstinner commented on Feb 26, 2025

    @vstinner
    vstinner
    on Feb 26, 2025
    Member

    I created #130593 for the 3.11 branch to skip test_gdb if Python if built with clang -Og (default optimization level when using --with-pydebug). It's a backport from main of #130550 change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions