windows console doesn't print or input Unicode

@malemburg

BPO	1602
Nosy	@malemburg, @mhammond, @terryjreedy, @pfmoore, @amauryfa, @ncoghlan, @pitrou, @giampaolo, @tjguk, @mark-summerfield, @ned-deily, @ezio-melotti, @florentx, @4kir4, @lilydjwg, @berkerpeksag, @vadmium, @eryksun, @zooba, @davispuh
Superseder	bpo-28217: Add interactive console tests
Files	sys_write_stdout.patch unicode2.py doc-patch.diff: Proposed changes to user-visible documentation unicode3.py win_console.patch test_win_console.py streams.py wincontest.py: Example io.TextIOWrapper sublcass using WideCharToMultiByte winconsoleio.diff 1602_2.patch 1602_3.patch 1602_4.patch 1602_5.patch 1602_6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2016-09-09.16:42:53.594>
created_at = <Date 2007-12-12.09:56:30.846>
labels = ['type-bug', 'expert-unicode', 'OS-windows']
title = "windows console doesn't print or input Unicode"
updated_at = <Date 2016-10-22.10:46:13.515>
user = 'https://github.com/mark-summerfield'

bugs.python.org fields:

activity = <Date 2016-10-22.10:46:13.515>
actor = 'THRlWiTi'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-09-09.16:42:53.594>
closer = 'steve.dower'
components = ['Unicode', 'Windows']
creation = <Date 2007-12-12.09:56:30.846>
creator = 'mark'
dependencies = []
files = ['19493', '20320', '20363', '23461', '23470', '23471', '36120', '40990', '44094', '44290', '44379', '44409', '44449', '44452']
hgrepos = []
issue_num = 1602
keywords = ['patch']
message_count = 148.0
messages = ['58487', '58621', '58651', '87086', '88059', '88077', '92854', '94445', '94480', '94483', '94496', '108173', '108228', '116801', '120414', '120415', '120416', '120700', '125823', '125824', '125826', '125833', '125852', '125877', '125889', '125890', '125898', '125899', '125938', '125942', '125947', '125956', '126286', '126288', '126303', '126304', '126308', '126319', '127782', '131657', '131854', '132060', '132061', '132062', '132064', '132065', '132067', '132184', '132191', '132208', '132266', '132268', '145898', '145899', '145963', '145964', '146471', '148990', '157569', '160812', '160813', '160897', '161151', '161153', '161308', '161651', '164572', '164578', '164580', '164618', '164619', '170899', '170915', '170999', '185135', '197700', '197751', '197752', '197773', '221175', '221178', '223403', '223404', '223507', '223509', '223945', '223946', '223947', '223948', '223949', '223951', '223952', '224019', '224086', '224095', '224596', '224605', '224690', '227329', '227330', '227332', '227333', '227337', '227338', '227347', '227354', '227373', '227374', '227441', '227450', '228191', '228208', '228210', '233347', '233350', '233916', '233937', '234019', '234020', '234096', '234371', '242884', '254405', '254407', '272596', '272605', '272645', '272662', '272675', '272716', '272718', '272720', '273999', '274449', '274673', '274884', '274906', '274912', '274939', '275003', '275004', '275005', '275157', '275362', '275510', '277047', '277048', '277050']
nosy_count = 38.0
nosy_names = ['lemburg', 'mhammond', 'terry.reedy', 'paul.moore', 'tzot', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'giampaolo.rodola', 'tim.golden', 'mark', 'ned.deily', 'christoph', 'ezio.melotti', 'v+python', 'hippietrail', 'flox', 'THRlWiTi', 'davidsarah', 'santoso.wijaya', 'akira', 'David.Sankel', 'python-dev', 'smerlin', 'lilydjwg', 'berker.peksag', 'martin.panter', 'piotr.dobrogost', 'eryksun', 'Drekin', 'steve.dower', 'wiz21', 'stijn', 'Jonitis', 'gurnec', 'escapewindow', 'dead1ne', 'davispuh']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '28217'
type = 'behavior'
url = 'https://bugs.python.org/issue1602'
versions = ['Python 3.6']

I am not sure if this is a Python bug or simply a limitation of cmd.exe.

I am using Windows XP Home.
I run cmd.exe with the /u option and I have set my console font to
"Lucida Console" (the only TrueType font offered), and I run chcp 65001
to set the utf8 code page.
When I run the following program:

for x in range(32, 2000):
    print("{0:5X} {0:c}".format(x))

one blank line is output.

But if I do chcp 1252 the program prints up to 7F before hitting a
unicode encoding error.

This is different behaviour from Python 2.5.1 which (with a suitably
modified print line) after chcp 65001 prints up to 7F and then fails
with "IOError: [Errno 0] Error".

I've looked into this a bit more, and from what I can see, code page
65001 just doesn't work---so it is a Windows problem not a Python problem.
A possible solution might be to read/write UTF16 which "managed" Windows
applications can do.

We are aware of multiple Windows related problems. We are planing to
rewrite parts of the Windows specific API to use the widechar variants.
Maybe that will help.

Yes, it is a Windows problem. There simply doesn't seem to be a true
Unicode codepage for command-line apps. Recommend closing.

Just in case it helps, this behaviour is on Win XP Pro, Python 2.5.1:

First, I added an alias for 'cp65001' to 'utf_8' in
Lib/encodings/aliases.py .

Then, I opened a command prompt with a bitmap font.

c:\windows\system32>python
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print u"\N{EM DASH}"
â€”

I switched the font to Lucida Console, and retried (without exiting the
python interpreter, although the behaviour is the same when exiting and
entering again: )

>>> print u"\N{EM DASH}"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 13] Permission denied

Then I tried (by pressing Alt+0233 for é, which is invalid in my normal
cp1253 codepage):

>> print u"née"

and the interpreter exits without any information. So it does for:

>> a=u"née"

Then I created a UTF-8 text file named 'test65001.py':

# -*- coding: utf_8 -*-
a=u"néeα"
print a

and tried to run it directly from the command line:

c:\windows\system32>python d:\src\PYTHON\test65001.py
néeαTraceback (most recent call last):
  File "d:\src\PYTHON\test65001.py", line 4, in <module>
    print a
IOError: [Errno 2] No such file or directory

You see? It printed all the characters before failing.

Also the following works:

c:\windows\system32>echo heéε
heéε

and

c:\windows\system32>echo heéε >D:\src\PYTHON\dummy.txt

creates successfully a UTF-8 file (without any UTF-8 BOM marks at the
beginning).

So it's possible that it is a python bug, or at least something can be
done about it.

an immediate thing to do is to declare cp65001 as an encoding:

Index: Lib/encodings/aliases.py
===================================================================

--- Lib/encodings/aliases.py    (revision 72757)
+++ Lib/encodings/aliases.py    (working copy)
@@ -511,6 +511,7 @@
     'utf8'               : 'utf_8',
     'utf8_ucs2'          : 'utf_8',
     'utf8_ucs4'          : 'utf_8',
+    'cp65001'            : 'utf_8',

 ## uu_codec codec
 #'uu'                 : 'uu_codec',

This is not enough unfortunately, because the win32 API function
WriteFile() returns the number of characters written, not the number of
(utf8) bytes:

>>> print("\u0124\u0102" + 'abc')
ĤĂabc
c
[44420 refs]
>>>

Additionally, there is a bug in the ReadFile, which returns an empty
string (and no error) when a non-ascii character is entered, which is
the behavior of an EOF condition...

Maybe the solution is to use the win32 console API directly...

Another note:
if one creates a dummy Stream object (having a softspace attribute and a
write method that writes using os.write, as in
http://stackoverflow.com/questions/878972/windows-cmd-encoding-change-causes-python-crash/1432462#1432462
) to replace sys.stdout and sys.stderr, then writes occur correctly,
without issues. Pre-requisites:
chcp 65001, Lucida Console font and cp65001 as an alias for UTF-8 in
encodings/aliases.py
This is Python 2.5.4 on Windows.

With Python 3.1.1, the following batch file seems to be necessary to use
UTF-8 successfully from an XP console:

set PYTHONIOENCODING=UTF-8
cmd /u /k chcp 65001
set PYTHONIOENCODING=
exit

the cmd line seems to be necessary because of Windows having
compatibility issues, but it seems that Python should notice the cp65001
and not need the PYTHONIOENCODING stuff.

Glenn Linderman's fix pretty well works for me on XP Home. I can print
every Unicode character up to and including U+D7FF (although most just
come out as rectangles, at least I don't get encoding errors).

It fails at U+D800 with message:

UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in
position 17: surrogates not allowed

I also tried U+D801 and got the same error.

Nonetheless, this is *much* better than before.

Mark Summerfield wrote:

Mark Summerfield <mark@qtrac.eu> added the comment:

Glenn Linderman's fix pretty well works for me on XP Home. I can print
every Unicode character up to and including U+D7FF (although most just
come out as rectangles, at least I don't get encoding errors).

It fails at U+D800 with message:

UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in
position 17: surrogates not allowed

I also tried U+D801 and got the same error.

That's normal and expected: D800 is the start of the surrogate
ranges which are only allows in pairs in UTF-8.

Updated patch based on some suggestions from Eryk.

The PEP has been accepted, so now I just need to land it in the next two days.

Currently "normal" usage here is fine, and some edge cases match the Python 3.5 behaviour. I'm going to go through now and bulk out the tests to try and catch more problems, but modulo that I hope the implementation is nearly ready.

I can't actually come up with many useful tests for this... so far I can validate that we can open the console IO object from 0, 1, 2, "CON", "CONIN$" and "CONOUT$", get fileno(), check readable()/writable() and close (multiple times without crashing).

Anything else requires a real console with a real person with a real keyboard.

But I fixed a couple of issues in fd handling as a result of the tests, so it's not a complete waste.

I left some minor comments for Doc/whatsnew/3.6.rst on Rietveld.

In Lib/test/test_winconsoleio.py:

self.assert_() (deprecated) can be replaced by self.assertTrue()
We can add

  if __name__ == '__main__':
      unittest.main()

Thanks! I've made the changes you suggested.

+++ b/Lib/test/test_winconsoleio.py
+to real people with real keyborads.
Should be keyboards
There are still assert_() calls in this file (1602_6.patch). Did you miss them?

+++ b/Lib/io.py
+from _io import WindowsConsoleIO
+all.append('WindowsConsoleIO')
I think you should either document this class, or remove it from __all__ to clarify it is just an implementation detail.

+++ b/Modules/_io/winconsoleio.c
+_io_WindowsConsoleIO___init___impl
+ PyObject *decodedname = Py_None;
+ Py_INCREF(decodedname);
+ int d = PyUnicode_FSDecoder(nameobj, (void*)&decodedname);
Won’t this leak a reference to Py_None?
(Also, I think needless casting like in the last line can mask mistakes that the compiler would otherwise pick up. Imagine if you got the parameters around the wrong way.)

+read_console_w(HANDLE handle, DWORD maxlen, DWORD *readlen) {
+ /* If we didn't read a full buffer that time, don't try
+ again or we will block a second time. */
I’m not familiar with the Windows APIs involved, but this doesn’t seem robust. What if there were exactly one full buffer waiting, would the next call block without returning anything?

Ah sorry I see Berker’s assert_() comment was _after_ you posted 1602_6.patch, so ignore that bit :)

Also as I understand it, the open() function can return this new class, so the documentation at <https://docs.python.org/3.6/library/functions.html#open\> needs updating.

New changeset 6142d2d3c471 by Steve Dower in branch 'default':
Issue bpo-1602: Windows console doesn't input or print Unicode (PEP-528)
https://hg.python.org/cpython/rev/6142d2d3c471

Martin, the console should be in line-input mode, in which case ReadConsole will block if there isn't at least one line in the input buffer. It reads up to the lesser of a complete line or the number of UTF-16 codes requested. If the previous call read the entire request size but didn't stop on '\n', then we know the next call shouldn't block because the input buffer has at least one '\n' in it.

I can validate that we can open the console IO object from
0, 1, 2, "CON", "CONIN$" and "CONOUT$", get fileno(), check
readable()/writable() and close (multiple times without
crashing).

I like the idea to have fileno() lazily get a file descriptor on demand, but _open_osfhandle is a low I/O function that uses _open flags -- not 'rb' (int 0x7262) or 'wb' (int 0x7762). ;-)

You can use _O_RDONLY | _O_BINARY or _O_WRONLY | _O_BINARY. But really those values would be ignored anyway. It's not actually opening the file, so it only cares about a few flags. Specifically, in lowio\osfinfo.cpp I see that it looks for _O_APPEND, _O_TEXT, and _O_NOINHERIT.

On line 329, the following assignment

    if (self->writable)
        access |= GENERIC_WRITE;

should be access = GENERIC_WRITE. Requesting both read and write access is an invalid parameter when opening "CON", as can be seen here:

    >>> f = open('CON', 'wb', buffering=0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: [WinError 87] The parameter is incorrect: 'CON'

CONOUT$ works, of course:

    >>> f = open('CONOUT$', 'wb', buffering=0)
    >>> f
    <_io._WindowsConsoleIO mode='wb' closefd=True>

Lastly, for a readall that starts with ^Z, you're still breaking out of the loop before incrementing len, which is thus 0 when subsequently checked. It ends up calling WideCharToMultiByte with len == 0, which fails.

    >>> sys.stdin.buffer.raw.read()
    ^Z
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: [WinError 87] The parameter is incorrect

I can't actually come up with many useful tests for this...

ctypes can be used to write to the input buffer and read from a screen buffer. For the latter it helps to first create and activate a scratch screen buffer, initialized to NULs to make it easy to read back everything that was written up to the current cursor position. I have existing ctypes code for this, written to solve the problem of a subprocess that stubbornly writes directly to the console instead of writing to stdout/stderr pipes.

Okay so regarding blocking reads with a full buffer, what you are saying is the second check to break the read loop should be sufficient:

+/* If the buffer ended with a newline, break out */
+if (buf[*readlen - 1] == '\n')
+ break;

Steve Dower (steve.dower)

[...]
Anything else requires a real console with a real person with a real keyboard.

FYI, not really, it is possible to fully automatically test console's output/input using WinAPI functions like WriteConsoleInput, GetConsoleScreenBufferInfo, ReadConsoleOutputCharacter

very recently I wrote such test, you can look at it as example http://review.source.kitware.com/gitweb?p=KWSys.git;a=blob;f=testConsoleBuf.cxx;hb=HEAD

it tests all 3 cases when output is actual console, redirected pipe and file.

Oh nice, I like that. We should definitely add some tests using that (though it seems like quite a big task... maybe I'll open a new issue for it).

Created bpo-28217 for adding these tests.

Navigation Menu

Repository navigation

windows console doesn't print or input Unicode #45943

131 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

windows console doesn't print or input Unicode #45943

Description

Activity

mark-summerfield commented on Dec 12, 2007

mark-summerfield commented on Dec 14, 2007

tiran commented on Dec 15, 2007

pitrou commented on May 3, 2009

tzot commented on May 19, 2009

amauryfa commented on May 19, 2009

tzot commented on Sep 19, 2009

vpython commented on Oct 25, 2009

mark-summerfield commented on Oct 26, 2009

malemburg commented on Oct 26, 2009

131 remaining items

zooba commented on Sep 7, 2016

zooba commented on Sep 7, 2016

berkerpeksag commented on Sep 7, 2016

zooba commented on Sep 8, 2016

vadmium commented on Sep 8, 2016

vadmium commented on Sep 8, 2016

vadmium commented on Sep 8, 2016

python-dev commented on Sep 8, 2016

eryksun commented on Sep 9, 2016

vadmium commented on Sep 10, 2016

davispuh commented on Sep 20, 2016

zooba commented on Sep 20, 2016

zooba commented on Sep 20, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions