Repository navigation
Test suite crashes when run single process --with-pydebug enabled #121832
Description
Activity
I have confirmed the crash is occurring in the SubInterpreterTest added as part of #121602. If that test is commented out, the test suite passes without error.
FYI, I'm pretty sure gh-121636 will fix this.
The gist is: the change in gh-121602 runtime initialization makes a copy of the (mostly) unmodified
PyTypeObjectstruct for each builtin static type. During finalization we restore that copy, including thetp_version_tag. Clearly a part of that is causing things to get in an inconsistent state on iOS. The fix in gh-121636 restricts the copy/restore to only the "tp" slots (function pointers), restoring the pre-121602 status quo for the rest, includingtp_version_tag.Is there a way you could verify gh-121636 fixes the iOS build?
All that said, I'm definitely curious why this would be manifesting only on iOS?
FYI, I'm pretty sure gh-121636 will fix this.
...
Is there a way you could verify gh-121636 fixes the iOS build?The iOS buildbot should be sufficient to verify any fix; from the look of it, the current state of #121636 doesn't fix it. The test failure isn't very helpful - it's a hard crash of the emulator, which the test report doesn't show very nicely - but with the test_types SubinterpreterTest commented out, the suite completes and passes on iOS. I've verified the same tests locally, and I get the same failure on #121636.
All that said, I'm definitely curious why this would be manifesting only on iOS?
You and me both :-)
My best guess right now it's not iOS specific, but a function of how the test suite is executed on iOS. iOS is forced to run the entire test suite sequentially in a single process. This isn't (AFAIK) the default mode of operation on any other runner; so it's possible the specific sequence of test execution is causing an odd state to emerge. One of the debugging tasks I'm going to look into today is to try and reproduce this on macOS (the other is to try and find a subset of tests that causes the failure, rather than needing to get to test_types alphabetically... after 20 minutes)
Reacted by Eric SnowLooks like gh-121636 isn't sufficient. I'm going to try a different approach.
See gh-121882.
Dang it... looks like gh-121882 doesn't fix the problem either... 😢
Confirming: the test_types test doesn't fail on macOS when the test suite is executed in single process mode, so the problem isn't purely about execution order. Now trying to bisect the test suite to find a smaller subset of tests that fail.
Scratch that - it does crash on macOS - just you just have to remember to enable
--with-pydebug.@ericsnowcurrently To reproduce:
./configure --enable-universalsdk="`xcrun --show-sdk-path`" --with-universal-archs=universal2 --with-pydebug make -j8 ./python.exe -m test(Although I also had to patch
test_buildin.py::PtyTests.test_input_ttyto avoid a lockup due to #116402).Key details: Clean macOS build, with pydebug enabled; running as a single process (not multiprocess as
make testwill do), on an M1 MacBook Pro.Test failure:
... 0:17:13 load avg: 14.30 [424/478/1] test_types Assertion failed: (self->tp_version_tag != 0), function init_static_type, file typeobject.c, line 8476. Fatal Python error: Aborted Current thread 0x00000001fb35fac0 (most recent call first): <no Python frame> Extension modules: _testinternalcapi, _testcapi, _testlimitedcapi, _testbuffer, _testclinic, _testclinic_limited, _ctypes_test, xxsubtype, _testexternalinspection, _testsinglephase_basic_wrapper, _testsinglephase_basic_copy, _testsinglephase_with_reinit, _testsinglephase_with_state, _test_non_isolated, _test_shared_gil_only (total: 15) zsh: abort ./python.exe -m testNow trying to narrow down a subset of these tests that will fail.
- changed the title
[-]iOS testbed crashes when `--with-pydebug` enabled[/-][+]Test suite crashes when run single process `--with-pydebug` enabled[/+]on Jul 17, 2024 16 remaining items
@ericsnowcurrently can you make sure to revert #122150 when this is fixed?
- added a commit that references this issue
on Jul 23, 2024 - added a commit that references this issue
on Jul 24, 2024 - added 4 commits that reference this issue
on Jul 24, 2024 - added a commit that references this issue
on Jul 25, 2024
Crash report
What happened?
The
make testiostesting target crashes when compiled--with-pydebugenabled (as is done in CI).The error is a SIGABRT, raised during
test_types; however, runningtest_typesby itself isn't enough to reproduce the problem.The error is raised on L8476 of
typeobject.c:The same failure can be manufactured on macOS with M1 hardware, as long as the test suite is executed as a single process (i.e.
python -m test, not the multi-process option enabled bymake test).Full error trace on iOS:
stacktrace.txt
The same problem doesn't appear to occur when debug is not enabled.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS, iOS
Output from running 'python -VV' on the command line:
3.14.0a0; verified on 2bac2b86, but possibly present back to dc03ce7
Linked PRs