Repository navigation
PEP-739 build-details generation is incorrect when cross-building #136267
Description
Bug report
Bug description:
When doing cross-builds, the PEP-739 build-details.json gets generated using inconsistent data, some of which based on the build python, and some from the target platform python.
See for example the following gist:
https://gist.github.com/itamaro/8f23ebec2e142fd7480241b76c16c4e0
The gist contains the build-details.json generated for a 3.15 GIL-disabled build, using a 3.15 GIL-enabled build as the build-python.
Concretely:
cd ~/pybuild/python/main-default
./configure --with-pydebug
make -j
cd ~/pybuild/python/main-nogil
./configure --with-pydebug --disable-gil
make -j
# now use the GIL-enabled python to regenerate build-details.json for the nogil build
_PYTHON_PROJECT_BASE=. _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_td_darwin_darwin _PYTHON_SYSCONFIGDATA_PATH=build/lib.macosx-14.5-arm64-3.15 ~/pybuild/python/main-default/python.exe -E Tools/build/generate-build-details.py /tmp/build-details.jsonSome examples from the gist:
Incorrect - abi.flags is ["d"] instead of ["t", "d"]:
"abi": {
"flags": [
"d"
]
},Correct - libpython.static has the correct SOABI, including t:
"libpython": {
"static": "/usr/local/lib/libpython3.15td.a"
},Incorrect - suffixes.extensions includes extensions for the GIL-enabled ABI:
"extensions": [
".cpython-315d-darwin.so",
".cpython-315-darwin.so",
".abi3.so",
".so"
]Notes
I used a "cross-build" on the same MacOS host here, with GIL enabled vs GIL disabled, because it was the easiest for a full end-to-end repro on a single machine.
The issue is more general though, as I originally observed it when generating build-info.json files for a MacOS-target runtime (both GIL enabled and disabled) using a Linux-GIL-enabled build-python. That setup is more involved, but the mixing is also more apparent, with some fields using "darwin" and other fields using "linux-gnu-x86_64" as the multiarch string.
CPython versions tested on:
CPython main branch, 3.15, 3.14
Operating systems tested on:
macOS, Linux
Activity
Zheaoli commentedon Jul 4, 2025
I think this issue is caused by that we are mixing up the
targetandbuildinfo. I made a PoC patch forabi.flagsandsuffixes.extensions. I'm not sure this is acceptableZheaoli@f7bfb6c
itamaro commentedon Jul 13, 2025
cc @FFY00 more generally, should build-details be generated strictly from sysconfigdata, so if sysconfigdata is accurate then build-details would also be accurate?
sys.implementationandbuild-details.json#135785 as a duplicate of this issueAA-Turner commentedon Sep 16, 2025
Cross posting @sobolevn's comment from #135785 (closed as duplicate):
FFY00 commentedon Oct 4, 2025
It should be generated from whatever is most accurate.
geofft commentedon Oct 17, 2025
Oh, I just tracked down why this is happening specifically for
.suffixes.extensions. Tools/build/generate-build-details.py doeswhich is in turn implemented in
importlib._bootstrap_externalasEXTENSION_SUFFIXES = _imp.extension_suffixes(), which is in turn implemented in C by turningconst char *_PyImport_DynLoadFiletab[]into a Python list of strings.For many of the other things that generate-build-details.py looks at, they are set correctly set on cross builds because
$(PYTHON_FOR_BUILD)has a number of environment variables set that override the behavior of various functions, mostly in sysconfig. Specifically, configure.ac sets the variable to:But this particular call path doesn't go through anything with an override.
One option would be to try to read
_PyImport_DynLoadFiletabout of the just-built Python binary or libpython, e.g. by opening it with ctypes (but that would break in a number of circumstances) or using a binary parser (which would require a binary parser, but honestly the new remote debugging stuff might have enough of a parser to pull this off, at least for the platforms it supports). Probably a cleaner option is to get the info that's in that variable out of the build in some other way.Similarly,
['abi']['flags']is set fromsys.abiflags, which is set in C asSET_SYS_FROM_STRING("abiflags", ABIFLAGS);and also doesn't get impacted by these environment variables. (This one, at least, is available in the Makefile, so we could just pass it as an argument togenerate-build-details.py.)If it's helpful, python-build-standalone's interpreters exhibit both of these bugs: our debug and free-threaded variants use a
--with-build-pythonthat is neither of those, so we have the['abi']['flags']bug on those, and any of our cross-built interpreters (basically anything not x86_64 or arm64) has the['suffixes']['extensions']bug, as reported in astral-sh/python-build-standalone#816.(FWIW I just opened #140266 which is also about incorrect data in build-details.json and kind of involves all those environment variables set in
$(PYTHON_FOR_BUILD), but probably isn't super related, unless this is a motivation to more completely rework how the file is generated. As with that issue I'm happy to contribute a PR given guidance on how you want to approach this.)manueljacob commentedon Apr 7, 2026
This very likely has the same root cause as #90067.
mgorny commentedon Aug 13, 2026
Just hit this one as well. To be honest, I can only think of two ways of reliably solving this: we either mirror the whole logic from CPython in
generate-build-details.py(or somewhere else in the build system), or we play with some hard C parsing to get it out of the preprocessed C code used to build CPython.