Repository navigation
The new packaging module should not use the locale encoding #56321
Description
| BPO | 12112 |
|---|---|
| Nosy | @pitrou, @vstinner, @tarekziade, @merwok |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2013-01-03.02:57:56.754>
created_at = <Date 2011-05-19.11:54:27.874>
labels = ['library']
title = 'The new packaging module should not use the locale encoding'
updated_at = <Date 2013-01-03.02:57:56.753>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2013-01-03.02:57:56.753>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2013-01-03.02:57:56.754>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2011-05-19.11:54:27.874>
creator = 'vstinner'
dependencies = []
files = ['22028']
hgrepos = []
issue_num = 12112
keywords = ['patch']
message_count = 10.0
messages = ['136280', '136281', '136282', '136290', '136291', '136312', '136313', '136315', '136330', '141384']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'tarek', 'eric.araujo', 'Arfrever', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue12112'
versions = ['Python 3.3']
Activity
vstinner commentedon May 19, 2011
The locale encoding is not portable, packaging should use UTF-8 instead.
Attached patch is a draft to workaround "LANG=C ./python -m test test_packaging" failures. I'm not sure that my test in Metadata.write_file() of packaging.metadata is a good idea.
Moreover, packaging should also maybe use the surrogateescape error handler, but I propose to decide that later and in another issue. We may use this error handler only to read files (like the Python makefile).
For write a complete patch, *all* calls to open() in Lib/packaging/* (including Lib/packaging/tests/*) should be checked.
distutils uses the locale encoding, which is UTF-8 on Mac OS X and most Linux setup (but never on Windows). But distutils cannot be fixed, whereas packaging is a new module and can be fixed.
vstinner commentedon May 19, 2011
See also issue bpo-9561 (distutils: set encoding to utf-8 for input and output files) and bpo-6011 (python doesn't build if prefix contains non-ascii characters).
If you are curious, read also bpo-8611 (Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)) and bpo-9425 (Rewrite import machinery to work with unicode paths).
tarekziade commentedon May 19, 2011
Looks good, please commit this
vstinner commentedon May 19, 2011
I ported a distutils fix (for non-ASCII path) into packaging:
New changeset cc5cfeaa4a8d by Victor Stinner in branch 'default':
Issue bpo-10419, issue bpo-6011: port 6ad356525381 fix from distutils to packaging
http://hg.python.org/cpython/rev/cc5cfeaa4a8d
pitrou commentedon May 19, 2011
+ if not isinstance(fileobject, StringIO):
+ encoding = codecs.lookup(fileobject.encoding).name
IMO you should try to get the "encoding" attribute and silence the AttributeError instead.
(also, I'm not even sure why you're adding this check)
python-dev commentedon May 19, 2011
New changeset fe740c1cee02 by Victor Stinner in branch 'default':
Issue bpo-12112: packaging reads and writes setup.cfg using UTF-8
http://hg.python.org/cpython/rev/fe740c1cee02
New changeset 01d61096140a by Victor Stinner in branch 'default':
Issue bpo-12112: packaging reads/writes metadata using UTF-8
http://hg.python.org/cpython/rev/01d61096140a
vstinner commentedon May 19, 2011
What is the MainProgram.inspect_file() function in packaging.create? Which kind of file should it process? => What is the encoding of the input files?
tarekziade commentedon May 19, 2011
That's not used anymore, I am going to strip it
python-dev commentedon May 19, 2011
New changeset a636cb1b7f84 by Victor Stinner in branch 'default':
Issue bpo-12112: fix the encoding of setup.py in the packaging module
http://hg.python.org/cpython/rev/a636cb1b7f84
merwok commentedon Jul 29, 2011
I can run LANG=C python -m test test_packaging successfully. Can we close this?