Repository navigation
deriving configuration information for different builds with the same prefix #54016
Description
| BPO | 9807 |
|---|---|
| Nosy | @loewis, @freddrake, @warsaw, @doko42, @amauryfa, @pitrou, @ericvsmith, @merwok, @bitdancer, @davidmalcolm |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/warsaw'
closed_at = <Date 2010-11-24.19:44:04.672>
created_at = <Date 2010-09-09.01:24:24.167>
labels = ['type-bug', 'build']
title = 'deriving configuration information for different builds\twith the same prefix'
updated_at = <Date 2010-11-24.19:44:04.671>
user = 'https://github.com/doko42'bugs.python.org fields:
activity = <Date 2010-11-24.19:44:04.671>
actor = 'barry'
assignee = 'barry'
closed = True
closed_date = <Date 2010-11-24.19:44:04.672>
closer = 'barry'
components = ['Build']
creation = <Date 2010-09-09.01:24:24.167>
creator = 'doko'
dependencies = []
files = ['18807', '19272', '19776']
hgrepos = []
issue_num = 9807
keywords = ['patch']
message_count = 58.0
messages = ['115923', '115958', '115961', '115962', '115963', '115964', '115965', '115966', '115967', '115968', '116202', '116264', '116265', '116313', '116314', '116346', '116476', '116519', '116526', '116531', '116555', '116586', '116589', '116606', '117759', '117761', '117764', '117841', '117842', '117843', '117854', '118717', '118832', '118872', '118883', '118886', '118888', '118892', '118950', '119005', '119006', '119072', '119087', '119136', '119137', '119347', '120340', '120932', '121170', '121172', '121195', '121248', '121249', '121253', '121254', '122157', '122224', '122298']
nosy_count = 12.0
nosy_names = ['loewis', 'fdrake', 'barry', 'doko', 'amaury.forgeotdarc', 'pitrou', 'eric.smith', 'eric.araujo', 'rpetrov', 'Arfrever', 'r.david.murray', 'dmalcolm']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9807'
versions = ['Python 3.2']
Activity
doko42 commentedon Sep 9, 2010
currently, much of the configuration information is fetched by opening the config.h and the Makefile for the build. The locations of these files are derived from the prefix given at configure time.
If you want to have two distinct builds with the same prefix, but with different configuration options, this kind of configuration information is unsufficient. The current use case is a normal build and a debug build, as used for the Fedora and Debian/Ubuntu python packages.
Fedora and Debian/Ubuntu carry a patch, adding a sys.pydebug attribute and patching "some" things to get the correct information for these builds using the same prefix.
This kind of information should not be derived by following some paths names, but be contained in the python binary itself, without needing to read the Makefile and the config.h files.
Attached is a patch currently applied in Debian/Ubuntu (and some variant for Fedora) which introduces a sys.pydebug attribute and derives the correct locations for the config.h and Makefile files. It's not meant as a patch ready to apply, but to get an idea what is needed to have two different python installations with the same prefix.
amauryfa commentedon Sep 9, 2010
IMO there should not be any need to fetch information from config.h or the Makefile.
What about a sys.build_config dictionary containing all the necessary data?
doko42 commentedon Sep 9, 2010
+1
warsaw commentedon Sep 9, 2010
Amaury, you mention a sys.build_config dictionary, but I think it should actually be baked into the sysconfig module, possibly as a _sysconfig extension module. sysconfig is the new goodness for getting at this, and I don't think it ought to have to go through a sys dictionary to get the information.
I'd be willing to work on this, if we can get some consensus.
davidmalcolm commentedon Sep 9, 2010
For reference, the patch that I'm currently applying to Fedora's build of Python-3.2a1 can be seen at:
http://pkgs.fedoraproject.org/gitweb/?p=python3.git;a=blob_plain;f=python-3.2a1-debug-build.patch;hb=HEAD
It appears to be very similar to Matthias' patch (it was originally based on an earlier version of Debian's python 2 patch, which I fixed up against Fedora's python 2, changed some aspects I wasn't happy with, then ported to python 3.1, then fixed up to 3.2a1 IIRC)
For further reference, Fedora's python3.spec has these comments that I wrote on the patch:
# Patch to support building both optimized vs debug stacks DSO ABIs, sharing
# the same .py and .pyc files, using "_d.so" to signify a debug build of an
# extension module.
# Based on Debian's patch for the same,
# http://patch-tracker.debian.org/patch/series/view/python2.6/2.6.5-2/debug-build.dpatch
#
# (which was itself based on the upstream Windows build), but with some
# changes:
# * Debian's patch to dynload_shlib.c looks for module_d.so, then module.so,
# but this can potentially find a module built against the wrong DSO ABI. We
# instead search for just module_d.so in a debug build
# * We remove this change from configure.in's build of the Makefile:
# SO=$DEBUG_EXT.so
# so that sysconfig.py:customize_compiler stays with shared_lib_extension='.so'
# on debug builds, so that UnixCCompiler.find_library_file can find system
# libraries (otherwise "make sharedlibs" fails to find system libraries,
# erroneously looking e.g. for "libffi_d.so" rather than "libffi.so")
# * We change Lib/distutils/command/build_ext.py:build_ext.get_ext_filename
# to add the _d there, when building an extension. This way, "make sharedlibs"
# can build ctypes, by finding the sysmtem libffi.so (rather than failing to
# find "libffi_d.so"), and builds the module as _ctypes_d.so
#
# * Similarly, update build_ext:get_libraries handling of Py_ENABLE_SHARED by
# appending "_d" to the python library's name for the debug configuration
# * We modify Modules/makesetup to add the "_d" to the generated Makefile
# rules for the various Modules/*.so targets
# This may introduce issues when building an extension that links directly
# against another extension (e.g. users of NumPy?), but seems more robust when
# searching for external libraries
# * We don't change Lib/distutils/command/build.py: build.build_purelib to
# embed plat_specifier, leaving it as is, as pure python builds should be
# unaffected by these differences (we'll be sharing the .py and .pyc files)
# * We introduce DEBUG_SUFFIX as well as DEBUG_EXT:
# - DEBUG_EXT is used by ELF files (names and SONAMEs); it will be "_d" for
# a debug build
# - DEBUG_SUFFIX is used by filesystem paths; it will be "-debug" for a
# debug build
# Both will be empty in an optimized build. "_d" contains characters that
# are valid ELF metadata, but this leads to various ugly filesystem paths (such
# as the include path), and DEBUG_SUFFIX allows these paths to have more natural
# names. Changing this requires changes elsewhere in the distutils code.
# * We add DEBUG_SUFFIX to PYTHON in the Makefile, so that the two
# configurations build parallel-installable binaries with different names
# ("python-debug" vs "python").
# * Similarly, we add DEBUG_SUFFIX within python-config and
# python$(VERSION)-config, so that the two configuration get different paths
# for these.
# * Patch runtests.sh to support supplying a value for PYTHON, so that we can
# run the tests against each of the builds
amauryfa commentedon Sep 9, 2010
You are right; but the data could not be appended to sysconfig.py, because this file is shared by the different builds.
A new built-in C module is probably necessary.
warsaw commentedon Sep 9, 2010
@dmalcolm: I suspect you can reduce your diff for Python 3.2 now that PEP-3149 has landed.
doko42 commentedon Sep 9, 2010
On 09.09.2010 21:51, Dave Malcolm wrote:
right, by design/complaint: people did complain that locally built debug
extensions wouldn't be found in this case. This is now obsolete for 3.2,
because the debug extension has always added the "d" modifier.
[-]deriving configuration information for different builds with the same prefix[/-][+]deriving configuration information for different builds with the same prefix[/+]warsaw commentedon Sep 9, 2010
@Amaury: yes, that makes perfect sense. With PEP-3149, a debug (or any differently built) interpreter will pick up only the _sysconfig.blah.so that's appropriate for it, so baking it into there, with public API exposure through sysconfig seems like the right thing to do.
davidmalcolm commentedon Sep 9, 2010
In reply to Barry A. Warsaw:
loewis commentedon Sep 12, 2010
If the feature to be provided is "multiple Python installations with the same prefix", then I suggest to generalize this beyond the debug build. One approach would be:
merwok commentedon Sep 13, 2010
Agree with Barry: sysconfig is the new hotness for configuration info.
On bpo-4359, I said that future improvements in sysconfig (http://bitbucket.org/tarek/distutils2/src/tip/docs/design/wiki.rst) will address part of the issue: It will use a configparser file to store installation directories. There is nothing in the document about other build-time variables at the moment, but I’m +1 on any system that would remove the brittle Makefile parsing in sysconfig. The sysconfig.cfg does not look like the best place to put this info, since it’s not editable configuration but more state. +0 on generating a _sysconfig.c module.
I’ll have to read again the original patch and Martin’s proposal to understand the case of debug in the wider configure information picture.
36 remaining items
warsaw commentedon Nov 3, 2010
@RPetrov: reusing $LDVERSION does make sense, but then I think the variable is misnamed. I don't like $ABIVERSION either but maybe $BUILDVERSION?
Also I think we need to let the substitution for $LDVERSION happen in the Makefile rather than in the configure script. I tried it the way you suggested and IIRC, it had some problems, but I don't remember the details.
@Doko: Agreed. I'll have an updated patch that fixes python.pc. I'll look at INSTALL_SCHEME/headers too.
warsaw commentedon Nov 10, 2010
Attached is the remaining patch against py3k. This installs the symlinks, fixes the distutils 'install_headers' location, and uses abiflags in the python3.pc path.
I think this branch is done, pending approval and commit.
doko42 commentedon Nov 14, 2010
this adds the modifier to the <prefix>/python3.2<abi>/config directory.
Now you end up with both
<prefix>/python3.2<abi>/config
and
for anything else. This is not what Debian, Fedora and Ubuntu are currently doing. Is this really wanted? I'd rather like to see this as
so that everything is again in one prefix.
python-3.2mu.pc should be the file, python-3.2.pc the symlink.
doko42 commentedon Nov 14, 2010
the change to python.pc should make the abi change to includedir, not Cflags.
Arfrever commentedon Nov 14, 2010
Could you also fix issue bpo-10262, which is related to this issue?
warsaw commentedon Nov 15, 2010
Matthias,
you mean $prefix/lib/python3.2 and $prefix/lib/python3.2$abiflags, right? The latter has just a config directory, and you'd rather see that become $prefix/lib/python3.2/config-$abiflags, right?
warsaw commentedon Nov 15, 2010
Matthias,
When you say the python.pc change should change includedir instead of Cflags, that seems weird. includedir does not currently include the 'pythonX.Y' subdirectory so there's no place to put the abiflags. Or are you suggesting that we move the pythonX.Y<abiflags> subdir to includedir and remove it from Cflags?
doko42 commentedon Nov 15, 2010
On 15.11.2010 23:57, Barry A. Warsaw wrote:
yes.
[-]deriving configuration information for different builds with the same prefix[/-][+]deriving configuration information for different builds with the same prefix[/+]doko42 commentedon Nov 16, 2010
On 16.11.2010 00:01, Barry A. Warsaw wrote:
sorry, I'm wrong.
warsaw commentedon Nov 22, 2010
Here's an updated patch which address's Matthias's last concerns.
doko42 commentedon Nov 23, 2010
looks good. checked with a plain and a debug build and installation.
warsaw commentedon Nov 24, 2010
r86731