Repository navigation
test_json case test_highly_nested_objects_decoding fails under PGO on Clang 20 #140125
Description
Activity
I cannot reproduce this issue on x86-64 Linux with clang 20. I used these commands:
./configure --enable-optimizations CC=clang LD=clang time make ./python -m test -v test_jsonMy clang version:
clang version 20.1.8 (Fedora 20.1.8-4.fc42).Thanks for trying to reproduce!
Just to be clear, my failure occurred while running the profile task, e.g., here, rather than during the test run after a successful build — so I'd expect it to fail during
make.We're specifically using clang 20.1.4 over in python-build-standalone. I tried to reproduce this on HEAD (2ca3c85) on my macOS machine with LLVM clang 20.1.3 from homebrew and could not — the build succeeded. I suspect there may be an interaction with another facet of our build system.
I've opened astral-sh/python-build-standalone#822 to reproduce in our CI matrix and determine if this is specific to aarch64 macOS (I encountered this locally) or occurs on other platforms.
astral-sh/python-build-standalone#822 confirms that this is only an issue on aarch64 and x86-64 macOS.
I tried to reproduce in cpython with build flags that more closely resemble python-build-standalone's
export CFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -Werror=unguarded-availability-new' export CPPFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -Werror=unguarded-availability-new' export CFLAGS_JIT='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -Werror=unguarded-availability-new' export LDFLAGS='-arch arm64 -mmacosx-version-min=11.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -LModules/_hacl -Wl,-headerpad,40' ./configure --build=aarch64-apple-darwin --host=aarch64-apple-darwin --prefix=/install --with-system-expat --with-system-libmpdec --without-ensurepip --with-tail-call-interp --enable-shared --with-mimalloc --enable-optimizations --enable-experimental-jit=yes-off ac_cv_lib_intl_textdomain=no ac_cv_func_ptsname_r=no ac_cv_func_explicit_bzero=no --with-dbmliborder=ndbmwhich requires the following patch to allow LLVM 20 in the JIT (we are currently using this in 3.14 without problem)
diff --git a/Tools/jit/_llvm.py b/Tools/jit/_llvm.py index bc3b50ffe61..c90abad38da 100644 --- a/Tools/jit/_llvm.py +++ b/Tools/jit/_llvm.py @@ -11,7 +11,7 @@ import _targets -_LLVM_VERSION = "19" +_LLVM_VERSION = "20" _EXTERNALS_LLVM_TAG = "llvm-19.1.7.0" _P = typing.ParamSpec("_P")
Unfortunately, this failed with a segfault
./Programs/_freeze_module getpath ./Modules/getpath.py Python/frozen_modules/getpath.h make[2]: *** [Python/frozen_modules/getpath.h] Segmentation fault: 11 make[2]: *** Deleting file `Python/frozen_modules/getpath.h' make[1]: *** [profile-gen-stamp] Error 2 make: *** [profile-run-stamp] Error 2If I use
--enable-experimental-jit=noinstead, the build succeeds. Unfortunately, that means our configure flags are not the sole cause of thetest_jsonfailure. I suspect the segfault is a JIT interaction with LLVM 20. I'm not sure why I get this segfault in cpython but not in python-build-standalone. It's worth noting my clang versions do not match exactly.However, the
test_jsonfailure cannot be caused by the JIT because it is occurring on our x86-64 macOS and free-threaded builds, which have the JIT disabled.I cannot reproduce the segfault after running
make clean, so I think we can disregard that for now.Looking at the test case... the failure is at
cpython/Lib/test/test_json/test_recursion.py
Lines 74 to 79 in 77eade3
which is checking that there are not segfaults when parsing a deep object.
The recursion limit should be set to 20,000 (+ current depth)
cpython/Lib/test/support/__init__.py
Lines 2398 to 2409 in 41712c4
The object is an order of magnitude deeper, so we should be hitting the recursion limit.
Specifically, we're failing in the
cjsontest, so it's just the C-accelerated JSON implementation that has different behaviorcpython/Lib/test/test_json/__init__.py
Lines 23 to 29 in 872eafd
Applying the following patch to revert 7016044 from #139232 (cc @rokm and @markshannon) resolves the issue
diff --git a/Python/ceval.c b/Python/ceval.c index f48f412fab8..3bf1c098003 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -448,12 +448,6 @@ hardware_stack_limits(uintptr_t *top, uintptr_t *base) ULONG guarantee = 0; SetThreadStackGuarantee(&guarantee); *base = (uintptr_t)low + guarantee; -#elif defined(__APPLE__) - pthread_t this_thread = pthread_self(); - void *stack_addr = pthread_get_stackaddr_np(this_thread); // top of the stack - size_t stack_size = pthread_get_stacksize_np(this_thread); - *top = (uintptr_t)stack_addr; - *base = ((uintptr_t)stack_addr) - stack_size; #else /// XXX musl supports HAVE_PTHRED_GETATTR_NP, but the resulting stack size /// (on alpine at least) is much smaller than expected and imposes undue limits
You can see the CI pass in astral-sh/python-build-standalone#823
@zanieb Do you perhaps have a problematic build with PGO+LTO available for testing? I imagine you would need to disable the cjson test that triggers segfault to get past the profiling stage for PGO...
But it would be interesting to see if a) such a build also causes segfault in my test case from #139231, b) if it does, what the stack depth increments were between recursions. Could be that LTO causes a problem similar to #137573, and the cost of (the last) recursion step ends up exceeding the margin.
Indeed! I've disabled this test in order to release 3.15.0a1 in python-build-standalone so
uv python install 3.15.0a1will get you what you've requested. I'm not sure if the failure will occur when run outside of the profiling step, I haven't checked yet.edit: I tried the questionable invocation
PYTHONPATH=Lib uvx python@3.15 -m test test_jsonand the tests passed
edit: I tried the questionable invocation
Can you try the example from #139231? (The recorded stack depth increments between recursion steps there are what is most interesting - i.e., whether they are suspiciously large).
Although if the
test_jsonpasses, it may be that the final PGO+LTO build is OK, and it is the intermediate (presumably just LTO-enabled) build that is problematic.❯ uvx python@3.13 test.py | tail Recursion level: 4990, stack pointer: 0x16F3272B0, depth: 4451.33 kB Recursion level: 4991, stack pointer: 0x16F326F20, depth: 4452.22 kB Recursion level: 4992, stack pointer: 0x16F326B90, depth: 4453.11 kB Recursion level: 4993, stack pointer: 0x16F326800, depth: 4454.00 kB Recursion level: 4994, stack pointer: 0x16F326470, depth: 4454.89 kB Recursion level: 4995, stack pointer: 0x16F3260E0, depth: 4455.78 kB Recursion level: 4996, stack pointer: 0x16F325D50, depth: 4456.67 kB Recursion level: 4997, stack pointer: 0x16F3259C0, depth: 4457.56 kB Recursion level: 4998, stack pointer: 0x16F325630, depth: 4458.45 kB Recursion Limit ok (reached level 4999) ❯ uvx python@3.15 test.py | tail Recursion level: 10007, stack pointer: 0x16C9D19F0, depth: 8137.52 kB Recursion level: 10008, stack pointer: 0x16C9D16B0, depth: 8138.33 kB Recursion level: 10009, stack pointer: 0x16C9D1370, depth: 8139.14 kB Recursion level: 10010, stack pointer: 0x16C9D1030, depth: 8139.95 kB Recursion level: 10011, stack pointer: 0x16C9D0CF0, depth: 8140.77 kB Recursion level: 10012, stack pointer: 0x16C9D09B0, depth: 8141.58 kB Recursion level: 10013, stack pointer: 0x16C9D0670, depth: 8142.39 kB Recursion level: 10014, stack pointer: 0x16C9D0330, depth: 8143.20 kB Recursion level: 10015, stack pointer: 0x16C9CFFF0, depth: 8144.02 kB Recursion Limit ok (reached level 10016)(note I compiled the
stackpointer.dylibwith gcc as suggested there)It's also plausible that the instrumentation from PGO is causing the problem.
Reacted by Rok MandeljcThose increments are in line what I would expect (aside from the fact that stack size is 8 MB and not 16 MB - I assume you are overriding the setting in your builds)?
It's also plausible that the instrumentation from PGO is causing the problem.
That's true; the instrumentation is likely to result in increased stack consumption per recursion. It would be interesting to see how much (i.e., if it exceeds the value of the margin). Do you have an easy way to run
test.pywith the intermediate (instrumented) build?Do you have an easy way to run test.py with the intermediate (instrumented) build?
I don't unfortunately. I presume it's doable via a patch but it doesn't sound trivial.
It looks like this affects 3.14.1 as well now.
Have you tried increasing the depth from 200k to, say, 500k? Is it much slower?
If that doesn't fail, it implies that PGO is thinking that deep recursion is a hot path, and maybe turning the calls into tailcalls.We don't need the test to raise a recursion error, just not crash. So we could remove the assertion.
The test for the recursion error is just to check that the test is hitting the stack limit.@markshannon that's an interesting idea. It looks like that resolves the issue astral-sh/python-build-standalone#901
- added a commit that references this issue
on Dec 8, 2025 - added a commit that references this issue
on Dec 8, 2025
Building 3.15.0a1 on aarch64 macOS with Clang 20 in python-build-standalone failed during the PGO test run in
test_highly_nested_objects_decoding:This failure is specific to macOS, and also occurs on x86-64. I haven't reproduced it outside of
python-build-standaloneyet. Jump to #140125 (comment) for the root cause.Linked PRs
test_jsonfrom 200k to 500k #142226test_jsonfrom 200k to 500k (GH-142226) #142416