Repository navigation
python doesn't build if prefix contains non-ascii characters #50261
Description
| BPO | 6011 |
|---|---|
| Nosy | @vstinner, @tarekziade, @merwok, @nphilipp, @dstufft |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2014-03-24.21:44:06.329>
created_at = <Date 2009-05-13.11:01:48.448>
labels = ['type-bug', 'library']
title = "python doesn't build if prefix contains non-ascii characters"
updated_at = <Date 2014-03-24.21:44:06.327>
user = 'https://bugs.python.org/zegreek'bugs.python.org fields:
activity = <Date 2014-03-24.21:44:06.327>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2014-03-24.21:44:06.329>
closer = 'vstinner'
components = ['Distutils']
creation = <Date 2009-05-13.11:01:48.448>
creator = 'zegreek'
dependencies = ['9841']
files = ['13974', '13975', '13984', '18840', '19551', '20165', '21992']
hgrepos = []
issue_num = 6011
keywords = ['patch']
message_count = 47.0
messages = ['87674', '87675', '87766', '87772', '114568', '114571', '116060', '116087', '116263', '116297', '116390', '116391', '119074', '119102', '119353', '119368', '119383', '119389', '119409', '119420', '119449', '119847', '119848', '119891', '119894', '120178', '120818', '120821', '120827', '120829', '121551', '121564', '121566', '121810', '124571', '124597', '124646', '124647', '124648', '124650', '124651', '127016', '135750', '135755', '135903', '214727', '214738']
nosy_count = 6.0
nosy_names = ['vstinner', 'tarek', 'eric.araujo', 'zegreek', 'nils', 'dstufft']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6011'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']
Activity
zegreek commentedon May 13, 2009
I have tried to build python (version 3.1 beta 1) on linux and install
it to a non-standard prefix which contains non-ascii utf-8 characters
(my locale being utf-8). The build directory's path is ascii-only. The
exact configure line is given in the attached file 'tb.txt'.
Then the 'make' command fails at the stage where python extensions are
built, with the traceback displayed in file tb.txt (in short:
UnicodeDecodeError: 'ascii' codec can't decode byte ... ).
The problem is triggered when 'distutils.sysconfig.get_config_vars'
tries to parse the Makefile. The Makefile is opened with
'distutils.text_file.TextFile', which in turns calls 'io.open' with no
'encoding' parameter. At this stage of the build, the 'locale' module is
not available (due to '_collections' not being), so that
'locale.getprefferedencoding' cannot be called and the encoding falls
back to ascii (a quick look to 'Modules/_io/textio.c' suggests that this
fallback mechanism is already designed for being used at build time).
The solution I propose would be to use 'sys.getfilesystemencoding' as a
fallback first, as it is defined during build time on most systems:
windows, mac and on posix if 'CODESET' exists in 'langinfo.h'. Given
that in build routines, non-ascii characters are only likely to be
encountered in filesystem paths, this seems a reasonable behavior.
The attached patch 'text_file.diff' implements this strategy in
'distutils.text_file', and then calls 'io.open' with the appropriate
'encoding' parameter. It could be argued, however, that this new
fallback is of general interest and should be implemented directly in
'Modules/_io/textio.c'. If you deem so, I could try to come up with a
new patch.
The attached patch solves the problem on my system, and does not
introduce test failures (which is expected, as the new fallback should
only make a difference at build time).
Cheers,
Baptiste
zegreek commentedon May 13, 2009
And here comes the patch
zegreek commentedon May 14, 2009
OK, here is also the patch to 'Modules/_io/textio.c', as it is in fact
quite trivial. Choose which one you prefer :-)
Baptiste
tarekziade commentedon May 14, 2009
Thanks, I'll work on this during this week end hopefully
vstinner commentedon Aug 21, 2010
This issue may be related to bpo-9561.
vstinner commentedon Aug 21, 2010
Change _io.TextIOWrapper() heuristic to choose the encoding is a bad idea.
vstinner commentedon Sep 10, 2010
New patch:
Note: sys.getfilesystemencoding() is always set in Python 3.2 (but it may be None in Python 2.x and Python < 3.2).
zegreek commentedon Sep 11, 2010
Hello,
I just tried your patch on latest svn (r84707), but I found out that the problem I reported can no more be reproduced. First, '_locale' seems now to be built earlier. Also, a fallback has been introduced in 'locale.getpreferredencoding'. When '_locale' cannot be imported, the encoding is now parsed from the environment variables (cf 'Lib/locale.py', line 558 and below). It looks like 'locale.getpreferredencoding' is now no more likely to fail than 'sys.getfilesystemencoding'. So I'm not sure if a patch is still needed at all.
In case a patch still makes sense, pay attention that there now also is a 'Lib/sysconfig.py', which also has a '_parse_makefile' function. This function uses a logic similar to the one in 'Lib/distutils/sysconfig.py', even thought it uses the builtin 'open', so it would need the same fix, if one is needed.
Cheers, B.
merwok commentedon Sep 13, 2010
Victor: I would like bpo-9841 to be solved first so that we’re sure what file(s) need to be patched (and where to add a test for parsing).
Baptiste: Can you try to reproduce your bug with 2.7 and 3.1?
Tarek: I can take this over if you want.
zegreek commentedon Sep 13, 2010
Eric: the bug does not exist with 2.7, as the Makefile is read as bytes.
It exists with 3.1.2.
By the way, when I say the bug is solved on 3.2, I only mean the narrow
problem of using a
non-ascii prefix that *is* decodable with the current locale. I do not
mean the more general
problem that arises, for example, when building with the 'C' locale, as
is discussed in bpo-9561.
With a 'C' locale, the build fails also with 3.2.
vstinner commentedon Sep 14, 2010
For non-ascii directory name but ascii locale (eg. C locale), we have 3 choices:
a- read Makefile as a binary file
b- use the PEP-383
c- refuse to compile
(a) doesn't seem easy because it looks like distutils use the unicode type for all paths. (b) supposes to patch distutils to ensure that reading (and writing?) Makefile uses errors='surrogateescape'.
About (c), it can be a temporary solution. But I also think that non-ascii directory name and ascii locale encoding is a rare use case.
vstinner commentedon Sep 14, 2010
Warning: "use the PEP-383" may impact other distutils component because the path may be written into to other files, which mean that we have to use errors='surrogateescape' for these files too.
26 remaining items
merwok commentedon Dec 25, 2010
Baptiste: I meant that I couldn’t reproduce the bug, not that the patch had solved it.
Victor: Your patch uses os.fsencode, so porting to distutils2 won’t be easy. Tarek and I have instated this policy http://wiki.python.org/moin/Distutils/FixingBugs to make sure we don’t have to reopen 50 bugs when distutils2 gets into the stdlib.
d2 has to stay compatible with 2.4-2.7, and 3.1-3.2 in a near future.
vstinner commentedon Dec 25, 2010
Le samedi 25 décembre 2010 à 22:35 +0000, Éric Araujo a écrit :
In Python 3.1, you can replace name=os.fsencode(name) by
name=name.encode(sys.getfilesystemencoding(), 'surrogateescape'). The
mbcs codec doesn't support surrogateescape. In Python 3.2 it does now
raise an error (and so os.fsencode() uses strict error handler if the
encoding is mbcs), whereas Python 3.1 just ignores the error handler (it
uses 'ignore' to encode).
In Python 2, self.executable is already a byte string (you don't need
os.fsencode()).
merwok commentedon Dec 25, 2010
I suggest you wait for distutils2 to be 3.x-compatible and then adapt your patch to fix the bug when used with 3.2, keeping backward compat.
nphilipp commentedon Jan 25, 2011
NB: it's not the shell, but the kernel which interprets the shebang line (and subsequently calls the shell /bin/sh with it if it's missing, causing funny effects when it encounters the first import line and you happen to have ImageMagick installed).
vstinner commentedon May 10, 2011
New changeset 6ad356525381 by Victor Stinner in branch 'default':
Close bpo-10419, issue bpo-6011: build_scripts command of distutils handles correctly
http://hg.python.org/cpython/rev/6ad356525381
vstinner commentedon May 10, 2011
I fixed bpo-10419 in Python 3.2 and 3.3 (I applied my copy_script-2.patch fix). It is now possible to compile and install Python 3.2 and 3.3 with a non-ASCII prefix, so this issue can be done.
If you have issues when compiling Python with a non-ASCII prefix (and a locale encoding different than UTF-8), reopen the issue or open a more specific issue.
--
I don't want to backport fixes of this issue to Python 3.1, because Python 3.1 requires too much effort to handle correctly non-ASCII paths (not only for this specific issue). I consider that Python 3.1 is a stable release and should not be touched too much and non-ASCII path with a locale encoding different than UTF-8 is a corner case. If you have this issue with Python 3.1, please upgrade to Python 3.2 :-)
zegreek commentedon May 13, 2011
Indeed, I retried with 534a9e274d88 (that was the tip of 3.2 sometime
yesterday) and my original problem is solved. Thank you.
While I was at it, I ran "make test", and got 3 unusual skips and 1
failure.
The skips are test_sax, test_xml_etree and test_xml_etree_c and they are
skipped on purpose when the example XML filename is not encodable to
utf8. No problem here.
The failure is for test_distutils. 3 individual tests are failing:
test_simple_built, test_debug_mode and test_record. The cause of this
failure is that the "install" command installs a test distribution to a
path containing sys.prefix. This is not a problem per se, but later
test_simple_built tries to zip this distribution, and cannot construct a
valid archive name. A similar problem happens when test_record tries to
write the distribution's filenames to a record file (and test_debug_mode
fails because of test_record).
Imho those failures cannot be fixed, so the only possible improvement is
to skip those tests. The attached trivial patch does just that, but I'm
not sure if it's worth patching distutils for that.
Cheers,
Baptiste
merwok commentedon Mar 24, 2014
Victor, was this ticket kept open only for the backport to distutils2? If everything is fixed in Python stdlib and docs, then it could be closed, as distutils2 development has stopped.
vstinner commentedon Mar 24, 2014
I have no idea :) It's probably fine now. Open a new and fresh issue if it's not fixed yet, this issue has a too long history.