Repository navigation
Unicode encoding failure #74093
Description
| BPO | 29907 |
|---|---|
| Nosy | @pfmoore, @vstinner, @tjguk, @ezio-melotti, @vadmium, @zware, @eryksun, @zooba |
| Superseder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2017-03-26.13:40:02.722>
created_at = <Date 2017-03-26.02:26:08.124>
labels = ['type-bug', 'expert-unicode', 'OS-windows']
title = 'Unicode encoding failure'
updated_at = <Date 2017-03-26.13:40:02.687>
user = 'https://bugs.python.org/RobertBaker'bugs.python.org fields:
activity = <Date 2017-03-26.13:40:02.687>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2017-03-26.13:40:02.722>
closer = 'eryksun'
components = ['Unicode', 'Windows']
creation = <Date 2017-03-26.02:26:08.124>
creator = 'Robert Baker'
dependencies = []
files = []
hgrepos = []
issue_num = 29907
keywords = []
message_count = 5.0
messages = ['290503', '290506', '290513', '290517', '290528']
nosy_count = 9.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ezio.melotti', 'martin.panter', 'zach.ware', 'eryksun', 'steve.dower', 'Robert Baker']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = '1602'
type = 'behavior'
url = 'https://bugs.python.org/issue29907'
versions = ['Python 2.7']
Activity
RobertBaker commentedon Mar 26, 2017
Using Python 2.7 (not IDLE) on Windows 10.
I have tried to use a Python 2.7 program to print the name of Czech composer Antonín Dvořák. I remembered to add the "u" before the string, but regardless of whether I encode the caron-r as a literal character (pasted from Windows Character Map) or as \u0159, it gives the error that character 0159 is undefined. This is incorrect; that character has been defined as "lower case r with caron above" for several years now. (The interpreter has no problem with the ANSI characters in the string.)
vadmium commentedon Mar 26, 2017
I presume you are trying to print to the normal Windows console. I understand the console was not well supported until Python 3.6 (see bpo-1602). Have you tried that version?
I’ll leave this open for someone more experienced to confirm, but I suspect what you want may not be possible with 2.7.
pfmoore commentedon Mar 26, 2017
Also, you need to:
Specifically, if you write your source in UTF-8, or use an encoding declaration or \u escapes, and you use Python 3.6, this problem will likely have gone away.
vstinner commentedon Mar 26, 2017
For Python 2, there is https://pypi.python.org/pypi/win_unicode_console
eryksun commentedon Mar 26, 2017
I'm closing this issue since Python's encodings in this case -- 852 (OEM) and 1250 (ANSI) -- both correctly map U+0159:
>>> u'\u0159'.encode('852') '\xfd' >>> u'\u0159'.encode('1250') '\xf8'You must be using an encoding that doesn't map U+0159. If you're using the console's default codepage (i.e. you haven't run chcp.com, mode.com, or called SetConsoleOutputCP), then Python started with stdout.encoding set to your locale's OEM codepage encoding. For example, if you're using a U.S. locale, it's cp437, and if you're using a Western Europe locale, it's cp850. Neither of these includes U+0159.
We're presented with this codepage hell because the WriteFile and WriteConsoleA functions write a stream of bytes to the console, and it needs to be told how to decode these bytes to get Unicode text. It would be nice if the console's UTF-8 implementation (codepage 65001) wasn't buggy, but Microsoft has never cared enough to fix it (at least not completely; it's still broken for input in Windows 10).
That leaves the wide-character UTF-16 function, WriteConsoleW, as the best alternative. Using this function requires bypassing Python's normal standard I/O implementation. This has been implemented as of 3.6. But for older versions you'll need to install and enable win_unicode_console.