Repository navigation
Py_Initialize breaks wprintf on Windows #60791
Description
| BPO | 16587 |
|---|---|
| Nosy | @vstinner, @tjguk, @ezio-melotti, @zware, @zooba |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2014-06-16.20:57:20.893>
created_at = <Date 2012-12-01.00:23:52.391>
labels = ['expert-unicode', 'type-bug', 'expert-IO', 'OS-windows']
title = 'Py_Initialize breaks wprintf on Windows'
updated_at = <Date 2014-06-23.09:59:21.196>
user = 'https://bugs.python.org/makegho'bugs.python.org fields:
activity = <Date 2014-06-23.09:59:21.196>
actor = 'makegho'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-16.20:57:20.893>
closer = 'vstinner'
components = ['Unicode', 'Windows', 'IO']
creation = <Date 2012-12-01.00:23:52.391>
creator = 'makegho'
dependencies = []
files = []
hgrepos = []
issue_num = 16587
keywords = []
message_count = 10.0
messages = ['176732', '176734', '176828', '176837', '186852', '220642', '220762', '221346', '221347', '221348']
nosy_count = 9.0
nosy_names = ['jpe', 'vstinner', 'tim.golden', 'ezio.melotti', 'Arfrever', 'BreamoreBoy', 'zach.ware', 'steve.dower', 'makegho']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16587'
versions = ['Python 3.3']
Activity
makegho commentedon Dec 1, 2012
In a C application on Windows, at least on MSVC 2010 and Windows 7, do this:
Output is:
Test
T e s t
I was able to track the issue to fileio.c to the following code block by searching where wprintf breaks:
This can be easily confirmed by adding wprintfs on both sides of _setmode.
This issue was also raised at http://mail.python.org/pipermail/python-list/2012-February/620528.html but no solution was provided back then.
makegho commentedon Dec 1, 2012
If the standard streams are not used through Python, this hack can be used to work around the bug on C side:
...
vstinner commentedon Dec 3, 2012
_setmode(self->fd, O_BINARY) change was done in Python 3.2: see the issue bpo-10841. This change introduced regressions:
In Python 3.1, _setmode(self->fd, O_BINARY) was already used when Python is called with the -u command line option.
_setmode() supports different options:
I didn't try wprintf(). This function is not used in the Python source code (except in the Windows launcher, which is not part of the main interpreter).
I don't know how to fix wprintf().
vstinner commentedon Dec 3, 2012
The main reason was to be able to read binary file from sys.stdin
using the CGI module: see the issue bpo-4953. In _O_TEXT mode, 0x0A byte
is replaced with 0x0A 0x0D (or the opposite, I never remember) which
corrupt binary files.
Articles about _setmode() and wprintf():
"A confluence of circumstances leaves a stone unturned..."
http://blogs.msdn.com/b/michkap/archive/2010/09/23/10066660.aspx
"Conventional wisdom is retarded, aka What the @#%&* is _O_U16TEXT?"
http://blogs.msdn.com/b/michkap/archive/2008/03/18/8306597.aspx
See also issue bpo-1602 (Windows console doesn't print or input Unicode).
jpe commentedon Apr 13, 2013
One way to fix this is to use the FileRead & FileWrite api functions directly as proposed in bpo-17723 I would regard this as a change in behavior and not a simple bug fix because there is probably code written for 3.3 that assumes the C level stdout is in binary after python is initialized so would target 3.4 for the change.
BreamoreBoy commentedon Jun 15, 2014
I'll let our Windows gurus fight over who gets this one :)
vstinner commentedon Jun 16, 2014
If I understood correctly, supporting the "wide mode" for wprintf() requires to modify all calls to functions like printf() in Python and so it requires to change a lot of code. Since this issue was the first time that I heard about wprintf(), I don't think that we should change Python. I'm not going to fix this issue except if much more users ask for it.
makegho commentedon Jun 23, 2014
It's quite common to use wide character strings to support Unicode in C and C++.
In C++ this often means using std::wstring and std::wcout. Maybe these are more common than wprintf? In any case the console output breaks as Py_Initialize hijacks the host application's standard output streams which sounds quite illegitimate to me.
I understand that Python isn't designed for embedding and it would be a lot of work to fix it, but I would still encourage everyone to take a look at this bug. For me, this was one of the reasons I ultimately had to decide against using Python as my application's scripting language, which is a shame.
vstinner commentedon Jun 23, 2014
"In C++ this often means using std::wstring and std::wcout. Maybe these are more common than wprintf? In any case the console output breaks as Py_Initialize hijacks the host application's standard output streams which sounds quite illegitimate to me."
On Linux, std::wcout doesn't use wprintf(). Do you mean that std::wcout also depends on the "mode" of stdout (_setmode)?
makegho commentedon Jun 23, 2014
Yes, exactly. I originally noticed this bug by using std::wcout on Windows.