Repository navigation
Add cp65001 codec #57425
Description
| BPO | 13216 |
|---|---|
| Nosy | @loewis, @amauryfa, @vstinner |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2011-10-26.23:43:05.986>
created_at = <Date 2011-10-18.22:44:30.671>
labels = ['expert-unicode']
title = 'Add cp65001 codec'
updated_at = <Date 2011-10-26.23:44:52.551>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2011-10-26.23:44:52.551>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2011-10-26.23:43:05.986>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2011-10-18.22:44:30.671>
creator = 'vstinner'
dependencies = []
files = ['23453']
hgrepos = []
issue_num = 13216
keywords = []
message_count = 10.0
messages = ['145871', '145872', '145891', '145894', '145901', '145922', '145932', '146463', '146464', '146466']
nosy_count = 4.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'vstinner', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue13216'
versions = ['Python 3.3']
Activity
vstinner commentedon Oct 18, 2011
Thanks to bpo-12281, it is now trivial to implement any Windows code page in Python. I don't know if existing code pages (e.g. cp932) should use codecs.code_page_encode/.code_page_decode on Windows, or continue to use the (portable) Python code.
Users want the code page 65001, even if I consider that it is useless to set the ANSI code page to 65001 in a console (see issue bpo-1602), but that's a different story. Attached patch implements this code page.
vstinner commentedon Oct 18, 2011
See issues bpo-6058, bpo-7441 and bpo-10920.
loewis commentedon Oct 19, 2011
We shouldn't use the MS codec if we have our own, as they may differ.
As for the 65001 bug: is that actually solved by this codec?
vstinner commentedon Oct 19, 2011
Ok, I agree. MS codec has a nice replacement behaviour (search for a similar
glyph): cp1252 encodes Ł to b'L' for example. Our codec raises a
UnicodeEncodeError on u'\u0141'.encode('cp1252').
Sorry, which bug?
See tests using CP_UTF8 in test_codecs. Depending on the Windows version, you
don't get the same behaviour on surrogates. Before Windows Vista, surrogates
were always encoded, whereas you can now choose the behaviour using the Python
error handler:
vstinner commentedon Oct 19, 2011
I did more tests on the Windows console, focused on output, see:
http://bugs.python.org/issue1602#msg145898
I was wrong, it *is* useful to change the code page to 65001. Even if we have fully Unicode compliant sys.stdout and sys.stderr, setting the code page to CP_UTF8 (65001) does still improve Unicode support in some cases:
loewis commentedon Oct 19, 2011
bpo-6501 and friends (isn't it interesting that the issue of code page
65001 is reported as bug 6501?)
vstinner commentedon Oct 19, 2011
Hum, this particular issue, bpo-6501, doesn't concern the code page 65001. The typical usecase (issues bpo-7441 and bpo-10920) is:
------------
C:\victor\cpython>chcp 65001
Page de codes active : 65001
C:\victor\cpython>pcbuild\python_d.exe
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: cp65001
------------
The console and console output code pages may be changed by something else.
The current workaround is to set PYTHONIOENCODING environment variable to utf-8, but as explained in msg132831, the workaround is not applicable if Python is embeded or if the program has been frozen by cx-freeze ("cx-freeze deliberately sets Py_IgnoreEnvironmentFlag").
--
The issue bpo-6501 was a bug in io.device_encoding(). I fixed it in Python 3.3 and I'm waiting... since 5 months... for Graham Dumpleton before backporting the fix. The issue suggests also to not fail if the encoding cannot be found (I dislike this idea).
python-dev commentedon Oct 26, 2011
New changeset 0eac706d82d1 by Victor Stinner in branch 'default':
Fix the issue number of my cp65001 commit: 13247 => issue bpo-13216
http://hg.python.org/cpython/rev/0eac706d82d1
vstinner commentedon Oct 26, 2011
New changeset 2cad20e2e588 by Victor Stinner in branch 'default':
Close bpo-13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8)
http://hg.python.org/cpython/rev/2cad20e2e588
vstinner commentedon Oct 26, 2011
Lib/encodings/cp65001.py uses a little trick to mark the codec as specific to Windows:
-----------------