Repository navigation
test_embed fails if Python is built with LTO and LLVM clang on macOS #110313
Description
Bug report
Bug description:
# Next, run the profile task to generate the profile information.
LLVM_PROFILE_FILE="code-%p.profclangr" ./python.exe -m test --pgo --timeout=
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 3.49 Run 44 tests sequentially
0:00:00 load avg: 3.49 [ 1/44] test_array
0:00:01 load avg: 3.49 [ 2/44] test_base64
0:00:01 load avg: 3.49 [ 3/44] test_binascii -- test_base64 failed (env changed)
0:00:01 load avg: 3.49 [ 4/44] test_binop
0:00:01 load avg: 3.49 [ 5/44] test_bisect
0:00:01 load avg: 3.49 [ 6/44] test_bytes
0:00:05 load avg: 3.37 [ 7/44] test_bz2 -- test_bytes failed (env changed)
0:00:07 load avg: 3.37 [ 8/44] test_cmath
0:00:07 load avg: 3.34 [ 9/44] test_codecs
0:00:09 load avg: 3.34 [10/44] test_collections
0:00:10 load avg: 3.34 [11/44] test_complex
0:00:10 load avg: 3.34 [12/44] test_dataclasses
0:00:11 load avg: 3.34 [13/44] test_datetime
0:00:16 load avg: 3.39 [14/44] test_decimal
-------------------------------------------------------- NOTICE --------------------------------------------------------
test_decimal may generate "malloc can't allocate region"
warnings on macOS systems. This behavior is known. Do not
report a bug unless tests are also failing.
See https://github.com/python/cpython/issues/85100
------------------------------------------------------------------------------------------------------------------------
./Modules/_decimal/libmpdec/context.c:57: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=842105263157895168) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=842105263157895168) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=421052631578947584) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(67804,0x7fffa28f43c0) malloc: *** mach_vm_map(size=421052631578947584) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
0:00:22 load avg: 3.28 [15/44] test_difflib
0:00:23 load avg: 3.10 [16/44] test_embed
test test_embed failed
0:00:28 load avg: 3.17 [17/44] test_float -- test_embed failed (41 failures)
0:00:28 load avg: 3.17 [18/44] test_fstring
0:00:31 load avg: 3.17 [19/44] test_functools
./Modules/_decimal/libmpdec/context.c:57: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time
0:00:31 load avg: 3.17 [20/44] test_generators
0:00:32 load avg: 3.17 [21/44] test_hashlib
0:00:33 load avg: 3.08 [22/44] test_heapq
0:00:34 load avg: 3.08 [23/44] test_int
0:00:34 load avg: 3.08 [24/44] test_itertools
0:00:38 load avg: 2.99 [25/44] test_json
0:00:41 load avg: 2.99 [26/44] test_long -- test_json failed (env changed)
0:00:44 load avg: 2.83 [27/44] test_lzma
0:00:45 load avg: 2.83 [28/44] test_math
0:00:48 load avg: 2.84 [29/44] test_memoryview
0:00:49 load avg: 2.84 [30/44] test_operator
0:00:49 load avg: 2.84 [31/44] test_ordered_dict
0:00:51 load avg: 2.84 [32/44] test_patma
0:00:51 load avg: 2.84 [33/44] test_pickle
0:00:58 load avg: 2.63 [34/44] test_pprint
0:00:58 load avg: 2.63 [35/44] test_re
0:01:00 load avg: 2.63 [36/44] test_set -- test_re failed (env changed)
0:01:05 load avg: 2.50 [37/44] test_sqlite3
0:01:06 load avg: 2.50 [38/44] test_statistics -- test_sqlite3 failed (env changed)
0:01:08 load avg: 2.38 [39/44] test_str
0:01:11 load avg: 2.38 [40/44] test_struct -- test_str failed (env changed)
0:01:12 load avg: 2.38 [41/44] test_tabnanny -- test_struct failed (env changed)
0:01:13 load avg: 2.51 [42/44] test_time -- test_tabnanny failed (env changed)
0:01:15 load avg: 2.51 [43/44] test_xml_etree
0:01:16 load avg: 2.51 [44/44] test_xml_etree_c
Total duration: 1 min 18 sec
Total tests: run=8,985 failures=41 skipped=192
Total test files: run=44/44 failed=1 env_changed=8
Result: FAILURE
make: *** [Makefile:800: profile-run-stamp] Error 2
This is on macOS 10.12.6, building with LLVM / Clang 17.0.1. The system otherwise has Python 3.11.6 installed.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
debohman commentedon Oct 10, 2023
I see now that
test_embedusesPrograms/_testembedto execute. I don't know how embedding works, but clearly something has broken in3.13. It works properly in3.12.0built with the same toolchain.Under
3.13:Under
3.12.0:vstinner commentedon Oct 10, 2023
I don't understand this test_embed error neither:
In
Include/object.h,PyBaseObject_Typevariable is exported by:Log:
vstinner commentedon Oct 10, 2023
Python/dynload_shlib.cadds a lead underscore:vstinner commentedon Oct 10, 2023
cc @sobolevn @corona10: Can you reproduce these issues?
debohman commentedon Oct 10, 2023
I think it is a problem with embedding on my platform in
3.13, which I assume enables executing Python code from inside a a C program. As I have demonstrated, it works fine in3.12.0with the same exact toolchain.debohman commentedon Oct 10, 2023
I suspect that the problem occurs when Python code being executed in an embedded environment tries to import a runtime loaded module. That is why much of
test_embedfails.vstinner commentedon Oct 10, 2023
In the Python API, the variable is called
PyExc_ValueErrorwithout the leading underscore.vstinner commentedon Oct 10, 2023
Do you reproduce the issue with a more classic command?
CC=clang CXX=clang++ ./configurethen:
make && ./python.exe -m test -v test_embeddebohman commentedon Oct 10, 2023
That is because macOS uses the traditional C symbol naming, which places an underscore before C symbols.
vstinner commentedon Oct 10, 2023
Aha, I can reproduce some issues on Linux: 8 tests failed with "env changed".
Example:
Each subprocess spawned by tests create a
code-<pid>.profclangrfile.debohman commentedon Oct 10, 2023
Okay, here is what I have determined so far:
Programs/_testembedhas no symbols in3.13! That is why the runtime loading is failing. I am trying to figure out what is different in the build environment forPrograms/_testembedbetween3.12and3.13.pythongh-110313: regrtest: don't remove files created by PGO build
vstinner commentedon Oct 10, 2023
Tests spawning subprocesses create many
code-<pid>.profclangrfiles in the current directory. It's unclear to me if they should be used to train the PGO build or not.PR #110654 avoids the warning, but
.profclangrcreated by subprocesses are still removed, since regrtest starts by creating a temporary directory and changes the current working directory for this one.vstinner commentedon Oct 10, 2023
If you want to to bisect the test_embed regression, you can use
./configure PROFILE_TASK="-m test --pgo test_embed"command to only run test_embed. Oh. But before, failures of the profile task were ignored by|| true:debohman commentedon Oct 11, 2023
I found the problem. The offending commit is 3e3a7da. After reverting that and rebuilding from scratch:
Details
vstinner commentedon Oct 11, 2023
Can you reproduce the issue without PGO, just with LTO? In short, does the following command reproduce your issue?
On Fedora 38 with clang 16 (
clang version 16.0.6 (Fedora 16.0.6-3.fc38)), I get:The NOLTO value comes from
configure.ac:[-]PGO: test_embed fails which causes python build to fail[/-][+]test_embed fails if Python is built with LTO=fulll and LLVM clang 17[/+][-]test_embed fails if Python is built with LTO=fulll and LLVM clang 17[/-][+]test_embed fails if Python is built with LTO=full and LLVM clang 17[/+]vstinner commentedon Oct 11, 2023
@mdboom:
What is your macOS version? What is your LLVM clang version?
vstinner commentedon Oct 11, 2023
Great bisection, that's very helpful, thanks a lot!
debohman commentedon Oct 11, 2023
I was testing with LTO and without PGO, now I am rebuilding with LTO and PGO.
vstinner commentedon Oct 11, 2023
Ok, so the issue only comes from LTO=full.
debohman commentedon Oct 11, 2023
It seems to be the embedding that is somehow tripped up by this.
debohman commentedon Oct 11, 2023
I don't think we have established that LLVM 17 is the problem. To do that we would need to try building with LLVM 16.
debohman commentedon Oct 11, 2023
Building with PGO and LTO are fine now.
vstinner commentedon Oct 11, 2023
Test on Fedora Rawhide with LLVM clang 17:
It works as expected. By the way, PGO+LTO build is tested on Fedora Rawhide on multiple architectures:
vstinner commentedon Oct 11, 2023
Would you mind to elaborate "now"? Before you wrote that it didn't work. What changed? Are you still testing the latest main branch?
You're doing tests on macOS 10.12.6 with Clang 17.0.1, right?
[-]test_embed fails if Python is built with LTO=full and LLVM clang 17[/-][+]test_embed fails if Python is built with LTO=full and LLVM clang 17 on macOS[/+]vstinner commentedon Oct 11, 2023
I added "on macOS" to the issue title :-)
debohman commentedon Oct 11, 2023
Sorry, with 3e3a7da reverted.
vstinner commentedon Oct 11, 2023
If you can run your test with LLVM 16 on macOS, that would be nice yes, since I fail to reproduce the issue on Linux with LLVM 17.
debohman commentedon Oct 11, 2023
Yes, I will try that, but it will have to be later. It is past dinnertime now.
debohman commentedon Oct 11, 2023
The issue exists when building with LLVM 16, and backing out commit 3e3a7da resolves it.
vstinner commentedon Oct 11, 2023
@corona10: Should we just give up on the "NOLTO" idea, building some object files with LTO (full) or then attempt to use a different LTO mode (thin) to link them? Nor only revert 3e3a7da but also remove NOLTO variables.
corona10 commentedon Oct 11, 2023
@debohman Are you using a custom compiler toolchain rather than using the Apple basic toolchain?
corona10 commentedon Oct 11, 2023
I am okay with reverting the PR, IIUC it was just intended to speed up the build, but I need to know what effect will occur if drop the "NOLTO" idea.
corona10 commentedon Oct 11, 2023
If we drop the "NOLTO" idea, it will occur #96761 again. Please let me know If I misunderstand.
corona10 commentedon Oct 11, 2023
Or do you intend to revert #29859 too?
debohman commentedon Oct 11, 2023
The stock linker is being used, the rest of the toolchain is the newer llvm / clang.
mdboom commentedon Oct 11, 2023
It's macOS 13.6
I'm building CPython as follows:
mdboom commentedon Oct 11, 2023
I can confirm that backing out 3e3a7da also resolves things for me.
pythongh-110313: Revert "pythongh-90108: Disable LTO on _freeze_modul…
vstinner commentedon Oct 11, 2023
I wrote PR #110720 to revert my "NoLTO" change which introduced the bug.
I don't understand exactly why/how Programs/_testembed is miscompiled on macOS using LLVM 16 or 17, but it's bad: we should not miscompile Python. My change was just to optimize "build Python" command faster (run
make), it should not miscompile Python.It would be nice if someone can dig into the issue, but I don't have the bandwidth for that. I cannot reproduce the issue on Linux, and I don't have access to any macOS machine.
gh-110313: Revert "gh-90108: Disable LTO on _freeze_module and _teste… (
[-]test_embed fails if Python is built with LTO=full and LLVM clang 17 on macOS[/-][+]test_embed fails if Python is built with LTO and LLVM clang on macOS[/+]debohman commentedon Oct 11, 2023
If I may hazard a guess as to why this is failing, we seem to be using objects built with LTO and linking them to create
Programs/_testembedwithout LTO. Perhaps if you wish to to build_testembedwithout LTO, then you need a set of object files built without any LTO. Note that everything works fine when building Python without any LTO.In the case which fails, the generated
_testembedcontains no symbols, so when it tries to runtime load a module, it fails because it cannot resolve the undefined symbols in the module.vstinner commentedon Oct 12, 2023
I suppose that the best way to build with LTO only once instead of 3 times (freeze, testembed, libpython) is to use
./configure --enable-shared. So most object files are only linked once into libpython, and then libpython is reused in freeze and testembed.If I recall correctly, GCC implementation of LTO is very different and so the NOLTO hack to reduce build time when enable-shared is not used still works. Maybe we just use -flto=full in NOLTO on macOS (when cland is used).
debohman commentedon Nov 6, 2023
Shall we close this issue? I have successfully built
3.13.0a1and currentmainwith both PGO and LTO.debohman commentedon Nov 6, 2023
I am also using the latest Clang
17.0.4.vstinner commentedon Nov 6, 2023
Ok, I close the issue.
pythongh-110313: Revert "pythongh-90108: Disable LTO on _freeze_modul…
Abhaygarg656 commentedon Oct 14, 2024