Repository navigation
distutils: set encoding to utf-8 for input and output files #53770
Description
| BPO | 9561 |
|---|---|
| Nosy | @malemburg, @vstinner, @tarekziade, @merwok, @abadger, @ikelos, @mgorny |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/merwok'
closed_at = <Date 2013-01-03.01:26:51.398>
created_at = <Date 2010-08-10.17:51:37.803>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'distutils: set encoding to utf-8 for input and output files'
updated_at = <Date 2013-01-03.01:26:51.398>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2013-01-03.01:26:51.398>
actor = 'vstinner'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2013-01-03.01:26:51.398>
closer = 'vstinner'
components = ['Distutils', 'Unicode', 'Distutils2']
creation = <Date 2010-08-10.17:51:37.803>
creator = 'vstinner'
dependencies = []
files = ['22523', '22524']
hgrepos = []
issue_num = 9561
keywords = ['patch']
message_count = 25.0
messages = ['113552', '113584', '113725', '116061', '116260', '116261', '116317', '120689', '121215', '136812', '136974', '136975', '138820', '138822', '139483', '139487', '139528', '139656', '141569', '141773', '143567', '143569', '143570', '143611', '144271']
nosy_count = 10.0
nosy_names = ['lemburg', 'vstinner', 'tarek', 'eric.araujo', 'hagen', 'a.badger', 'Arfrever', 'ikelos', 'mgorny', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9561'
versions = ['3rd party', 'Python 2.7', 'Python 3.2', 'Python 3.3']
Activity
vstinner commentedon Aug 10, 2010
While working on bpo-9425 (support non-ascii characters in python directory name with ascii locale), I wrote a patch for distutils.file_util(): set encoding to utf-8 and errors to surrogateescape. See the patch with comments at:
http://codereview.appspot.com/1874048/patch/1/9
(the patch is not enough, it should also patch *all* functions reading files)
I discussed with takek who told me that it is documented that distutils files have to be utf-8. I didn't found the documentation. I checked read_manifest() in sdist command: in Python2 and Python3, it uses open(name) syntax. It means that Python2 uses the binary API (bytes), whereas Python3 uses the text API (unicode characters) and Python3 relies on open() (TextIOWrapper) heuristic to *guess* the file encoding.
I think that it will be better to specify the encoding in Python3, and maybe use the text API in Python2.
Anyway, before going futher (work on patches), I would like the approval of distutils maintainer(s).
merwok commentedon Aug 11, 2010
There are different kind of files created by write_file:
PKG-INFO (METADATA in distutil2), that already uses a trick to support Unicode, but your change would replace it in a better way;
MANIFEST, which with your fix would gain the ability to handle non-ASCII paths, which is a feature or a bugfix depending on your point of view;
.def files, used by the compilers for the C linking step; I don’t know if it’s appropriate to allow UTF-8 there.
RPM spec files, which use ASCII or UTF-8 according to http://en.opensuse.org/openSUSE:Specfile_guidelines#Specfile_Encoding but it’s not confirmed in http://www.rpm.org/max-rpm/s1-rpm-build-creating-spec-file.html (linked from the LSB site), so there’s no guarantee this works for all RPM platforms. This sort of platform-specific thing is the reason why RPM support has been removed in distutils2.
record and .pth files created by the install command.
I agree that there is something to be fixed, but I don’t know if they can be fixed in distutils. Unicode in PKG-INFO is unrelated to files, whereas there are files or directories in MANIFEST, spec, record and .pth. If this is going to be fixed, write_file should not use UTF-8 unconditionally but grow a keyword argument IMO, so that use cases requiring ASCII continue to work.
When you say “patch *all* functions reading files”, I guess you mean all functions that read distutils files, i.e. MANIFEST and PKG-INFO.
Tarek, is this a bug fix or a feature? Could it break third-party tools?
vstinner commentedon Aug 13, 2010
Which "trick"?
Wait. Non encodable bytes is a separated issue. I would like to work on the
first problem: distutils in Python3 uses open() without encoding argument and
so the encoding depends on the user's locale. Said differently: if you produce
a file with distutils on a computer, you cannot be sure that the file can be
read with the same version of Python on other computer (if the locale encoding
is different). Eg. Windows uses mbcs encoding whereas utf-8 is the preferred
encoding on Linux.
What is the encoding of the MANIFEST file?
I don't know these files.
UTF-8 is a superset of ASCII. If you use utf-8 but only write ascii
characters, your output file will be written to utf-8... but it will be also
encoded to ascii. It's magical :-)
.pth contain directory names which can be non-ASCII.
You can use non-ASCII characters for other topics than filenames. Eg. in a
description of a package :-)
As written before, UTF-8 is a superset of ASCII. If you read a file using utf-8
encoding, you will be able to read ascii files. But if you use utf-8 and write
non-ascii characters, old version of distutils using ascii or other encoding
will not be able to read these files.
Anyway, I think that in most cases, all files only contain ASCII text. So it
doesn't really matter.
About the keyword solution: yes, it would be a smooth way to fix this issue.
I don't know distutils to answer to my own question.
vstinner commentedon Sep 10, 2010
I attached a patch to bpo-6011 to set the encoding to read the Makefile.
merwok commentedon Sep 13, 2010
[Toshio, I made you nosy for a question about RPM .spec files]
Some values are explicitly allowed to use Unicode and are encoded to UTF-8
when queried.
Python’s default encoding, unfortunately. Try listing “napoléon” in a MANIFEST
file and you’ll get a UnicodeEncodeError because the file wants ASCII.
So we’ll have to get advice from someone well-versed in C linking.
I know that, but it does not answer the question: Is it okay for these files
to use UTF-8?
Agreed.
See above: The description of a distribution is in UTF-8. Note that I don’t
really understand my comment anymore; I now think that this should be fixed
in distutils with the least intrusive change possible.
That’s what I meant: Don’t make write_file always use UTF-8 since some use cases are restricted to ASCII.
Let’s do it. (Make sys.getdefaultencoding() its default value for compat.)
You patch writing files, I’ll handle reading files :)
merwok commentedon Sep 13, 2010
Note that any change requires a test.
abadger commentedon Sep 13, 2010
rpm spec files are encoding agnostic similar to POSIX filesystems. This causes no end of troubles for people writing python code that deals with python of course, as they cannot rely on the bytes that they are dealing with from one package to another to have the same encoding (Remember that things like dependency solvers have to compare the information from multiple packages to make their decisions).
Individual distributions will have different policies about encoding and the use of unicode in spec files to try and mitigate the problems. For instance, Fedora specifies utf-8 in the spec files and additionally specifies that package names must be ascii. (So if there's a package name: python-café, we would likely transcribe it as python-cafe when we made a package for it).
utf-8 is a good default for locales on POSIX systems so it's a good default for encoding spec files but I know there's some people out there who make their own packages that aren't utf-8. I haven't checked but I also wouldn't be surprised if some Asian countries (where the bytes-per-character with utf-8 is high) have local distributions that use non-utf-8 encoding as well. Whether either of these use cases needs to be catered to in distutils (when the support is going away in distutils2) I'll leave to someone else to decide. My personal gut instinct is no but I'm not one of the people using a non-utf-8 locale.
vstinner commentedon Nov 7, 2010
This issue might be splitted in multiple issue: one issue per file type (eg. Makefile, RPM spec file, etc.).
hagen commentedon Nov 15, 2010
Created bpo-10419 for the encoding problem in "build_scripts".
3 remaining items
mgorny commentedon Jun 22, 2011
Now that installing scripts with unicode characters was fixed, shall I open a separate bug for writing egg files with utf8 chars in author name?
Arfrever commentedon Jun 22, 2011
Please file a separate issue.
vstinner commentedon Jun 30, 2011
pkginfo_utf8.patch: distutils uses UTF-8 to write PKG-INFO and .egg-info, instead of the locale encoding. It should be applied to 2.7, 3.2 and 3.3.
packaging_pkginfo_utf8.patch: packaging tests use UTF-8 to write PKG-INFO files, instead of the locale encoding (cosmetic change, the file content is an empty string :-)). It should only be applied to 3.3 (packaging has been introduced in Python 3.3).
merwok commentedon Jun 30, 2011
Okay. I guess you’ll use codecs.open in 2.7; please make sure there is no bootstrapping issue for the build of CPython itself.
It would be a good thing to have non-ASCII in the PGK-INFO/METADATA files in the tests; it’s how we caught bpo-12320.
vstinner commentedon Jun 30, 2011
Oh, Python 2.7... DistributionMetadata of distutils encodes most values to byte strings (get_xxx() methods calls self._encode_field). It would be possible to use codecs.open(), but an Unicode file expects Unicode strings. The problem is that the user may provide arbitrary byte strings, I mean strings not encoded to PKG_INFO_ENCODING. Even if such strings are *wrong* (not correctly encoded), is it a good idea to be more strict in a minor version (2.7.x)?
I don't want to be responsible of such tricky change, I prefer to leave distutils unchanged in Python 2.7 (at least for PKG-INFO).
I checked, there is not bootstrap issue.
merwok commentedon Jul 2, 2011
mgorny commentedon Aug 2, 2011
Ping. What's the progress on this? Will this ever be fixed?
vstinner commentedon Aug 8, 2011
Some functions has been fixed in the new packaging module, but not in
the distutils yet.
python-dev commentedon Sep 5, 2011
New changeset fb4d2e6d393e by Victor Stinner in branch '3.2':
Issue bpo-9561: distutils now reads and writes egg-info files using UTF-8
http://hg.python.org/cpython/rev/fb4d2e6d393e
New changeset 3c080bf75342 by Victor Stinner in branch 'default':
Merge 3.2: Issue bpo-9561: distutils now reads and writes egg-info files using UTF-8
http://hg.python.org/cpython/rev/3c080bf75342
vstinner commentedon Sep 5, 2011
I applied pkginfo_utf8.patch to Python 3.2 and 3.3. Python 2.7 is not affected, it does already encode Unicode to UTF-8.
python-dev commentedon Sep 5, 2011
New changeset 56ab3257ca13 by Victor Stinner in branch 'default':
Issue bpo-9561: packaging now writes egg-info files using UTF-8
http://hg.python.org/cpython/rev/56ab3257ca13
merwok commentedon Sep 6, 2011
If you apply patches to distutils, please add tests for the fixed behavior. (Sorry if I wasn’t reactive on this one.)
merwok commentedon Sep 19, 2011
I backported your last change to distutils2 as f5a74b1f9473.