Repository navigation
Fatal error on startup with invalid PYTHONIOENCODING #50750
Description
| BPO | 6501 |
|---|---|
| Nosy | @amauryfa, @abalkin, @pitrou, @vstinner, @merwok, @florentx |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2014-02-13.10:35:21.736>
created_at = <Date 2009-07-17.10:21:43.045>
labels = ['interpreter-core', 'OS-windows', 'type-crash']
title = 'Fatal error on startup with invalid PYTHONIOENCODING'
updated_at = <Date 2014-02-13.10:35:21.735>
user = 'https://bugs.python.org/grahamd'bugs.python.org fields:
activity = <Date 2014-02-13.10:35:21.735>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-02-13.10:35:21.736>
closer = 'vstinner'
components = ['Interpreter Core', 'Windows']
creation = <Date 2009-07-17.10:21:43.045>
creator = 'grahamd'
dependencies = []
files = ['15623', '21849']
hgrepos = []
issue_num = 6501
keywords = ['patch']
message_count = 19.0
messages = ['90616', '90618', '90620', '90640', '90813', '91311', '91317', '96685', '96703', '127762', '132743', '132831', '134947', '134948', '136667', '136670', '147666', '205396', '211140']
nosy_count = 10.0
nosy_names = ['amaury.forgeotdarc', 'atuining', 'belopolsky', 'pitrou', 'vstinner', 'eric.araujo', 'grahamd', 'flox', 'Daniel.Goertzen', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue6501'
versions = ['Python 3.3']
Activity
grahamd commentedon Jul 17, 2009
When using Python 3.1 for Apache/mod_wsgi (3.0c4) on Windows, Apache will
crash on startup because Python is forcing the process to exit with:
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: cp0
I first mentioned this on bpo-6393, but have now created it as a separate
issue as appears to be distinct from the issue on MacOS X, athough possibly
related.
In the Windows case there is actually an encoding, that of 'cp0' where as on
MacOS X, the encoding name was empty.
The same mod_wsgi code works fine under Python 3.1 on MacOS X.
amauryfa commentedon Jul 17, 2009
function returns zero (and os.device_encoding(1) returns 'cp0').
(fileno(stdin) < 0) is true, and the standard streams are all set to None.
test does not work there.
As a workaround, I suggest to set the environment variable
PYTHONIOENCODING before starting Apache, or before the call to
Py_Initialize.
grahamd commentedon Jul 17, 2009
Yes, Apache remaps stdout and stderr to the Apache error log to still
capture anything that errant modules don't log via the Apache error log
functions. In mod_wsgi it replaces sys.stdout and sys.stderr with special
file like objects that redirect via Apache error logging functions. This
though obviously happens after Python first initialises sys.stdout and
sys.stderr.
What would be an appropriate value to set PYTHONIOENCODING to on Windows
as a workaround?
amauryfa commentedon Jul 17, 2009
On a Western Windows, I suggest
PYTHONIOENCODING=cp1252:backslashreplace
But
PYTHONIOENCODING=mbcs
is also OK, except that characters outside the Windows code page will be
replaced with '?'
grahamd commentedon Jul 22, 2009
The workaround of using:
gets around the crash on startup.
I haven't done sufficient testing to know if this may introduce any other
problems given that one is overriding default I/O encoding for whole
process.
pitrou commentedon Aug 5, 2009
Graham, is the workaround ok for you or do you think this is something
Python itself should handle?
grahamd commentedon Aug 5, 2009
Python should be as robust as possible and thus should be fixed, but I am
happy with using the workaround so if this is more a question of what
priority to mark this, I wouldn't see it as being urgent.
florentx commentedon Dec 20, 2009
Patch to prevent crash when PYTHONIOENCODING is invalid:
~ $ PYTHONIOENCODING= ./python
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding:
Abandon
pitrou commentedon Dec 20, 2009
Well, this might prevent the crash but how does the system behave
afterwards? Do the standard streams use utf-8 by default?
At the minimum, we should still output a warning on stderr.
abalkin commentedon Feb 2, 2011
The issue is not Windows specific, so I am changing the title to reflect that. On OSX, for example, I get
I agree that abort() is too drastic for a typo in the environment variable setting, but ignoring it silently is not a good option either. Someone setting PYTHONIOENCODING most likely does it for a reason and giving him or her some sort of default behavior for mistyped encoding is not helpful. (Note that this is how many C libraries treat TZ environment variable setting and this is often very frustrating.)
I think errors in environment variables that can be detected on startup should be treated the same way as the command line typos: a descriptive message on C stderr and exit(1).
Currently different environment variables are treated differently. For example, mistakes in PYTHONHOME and PYTHONIOENCODING cause fatal error while an error in PYTHONSTARTUP is reported but does not terminate python:
[-]Fatal LookupError: unknown encoding: cp0 on Windows embedded startup.[/-][+]Fatal error on startup with invalid PYTHONIOENCODING[/+]DanielGoertzen commentedon Apr 1, 2011
I run into this problem when I start a Python app as a subprocess from Erlang (open_port() function). The PYTHONIOENCODING fix works when I launch my py app via pythonw.exe, but it does *not* work when I use the cx-freeze version of the app.
I am using the Win32GUI base for cx-freeze which appears to be a thin WinMain() wrapper around Py_Initialize(). I am going to continue investigating the cx-freeze related problems.
I am using Python 3.2 under Windows. The failure is basically silent under Windows (generic MSVC runtime error), so I wasted a lot time figuring out what the problem actually was. Python 2 doesn't seem to have this problem.
DanielGoertzen commentedon Apr 3, 2011
It turns out that cx-freeze deliberately sets Py_IgnoreEnvironmentFlag to ensure that the final executable is really an isolated, standalone executable (ie, it can't be subverted by setting PYTHONPATH.) Therefore the PYTHONIOENCODING work-around does not work in this situation.
I am currently using a cx-freeze work-around from the author to enable the PYTHONIOENCODING work-around. Altogether not that pleasant.
Could Python 3 could just default to some reasonable encoding and keep on chugging?
vstinner commentedon May 1, 2011
That's a bug in os.device_encoding(): os.device_encoding(sys.stdout.fileno()) should return None if the application has no console (if sys.stdout is not a Windows console stream).
Attached device_encoding.patch should fix this issue. (I didn't test the patch yet.)
vstinner commentedon May 1, 2011
Why using this very small charset whereas a web server can use UTF-8?
I don't think that using backslashreplace on stdout is a good idea.
Starting at Python 3.2, you should use mbcs:replace to replace unencodable characters by '?'. The strict error handler is now strict: it raises a UnicodeEncodeError if a character is not encodable to mbcs.
Note: mbcs is the ANSI code page.
--
Using device_encoding.patch, I suppose that sys.std* streams will use the ANSI code page (mbcs, which is the code page 1252 on a Western Windows setup) in grahamd's usecase (Python program running in Apache).
python-dev commentedon May 23, 2011
New changeset 5783a55a2418 by Victor Stinner in branch 'default':
Issue bpo-6501: os.device_encoding() returns None on Windows if the application
http://hg.python.org/cpython/rev/5783a55a2418
vstinner commentedon May 23, 2011
@Grahamd: Can you try the development version of Python 3.3, or try to patch your version using device_encoding.patch? You will not get cp0 encoding anymore.
If the patch fixes your issue, I will backport it. I don't see anything interesting to do for this issue.
merwok commentedon Nov 15, 2011
If PYTHONSTARTUP is the only envvar with non-fatal errors, I think it’s okay. PYTHONHOME contains vital information, PYTHONIOENCODING is set by the programmer/admin and their code probably depends on it, but PYTHONSTARTUP is just niceties for the interactive interpreter, so non-vital IMO.
BreamoreBoy commentedon Dec 6, 2013
Is there anything to backport as referred to in msg136670 or can this be closed?
vstinner commentedon Feb 13, 2014
The fix is present in Python 3.3 since Python 3.3.0 according to the changelog. Python 3.2 doesn't accept bugfixes anymore.
Python 2.7 doesn't have the function os.device_encoding(), I don't think that it is affected.
There is nothing more to do, I'm closing the bug. Thanks for the report.