Repository navigation
Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale #64176
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2014-04-27.23:57:39.901>
created_at = <Date 2013-12-13.16:40:20.152>
labels = ['type-bug', 'expert-unicode']
title = 'Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale'
updated_at = <Date 2017-12-18.14:31:51.323>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2017-12-18.14:31:51.323>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-04-27.23:57:39.901>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2013-12-13.16:40:20.152>
creator = 'vstinner'
dependencies = []
files = ['33122', '33124']
hgrepos = []
issue_num = 19977
keywords = ['patch']
message_count = 38.0
messages = ['206111', '206114', '206121', '206122', '206123', '206124', '206131', '206141', '206148', '206168', '207210', '207296', '207414', '210990', '213922', '213925', '213927', '213932', '213933', '215029', '215786', '215812', '215815', '217300', '217314', '217315', '217317', '217329', '217332', '217355', '217385', '217386', '217387', '284849', '284863', '284873', '284875', '308562']
nosy_count = 15.0
nosy_names = ['loewis', 'ishimoto', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'jwilk', 'ezio.melotti', 'a.badger', 'r.david.murray', 'Sworddragon', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'bkabrda']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19977'
versions = ['Python 3.5']
Activity
vstinner commentedon Dec 13, 2013
When LANG=C is used to get the english language (which is a mistake, LC_CTYPE=C should be used instead) or when Python is started with an empty environment (no environment variable), Python gets the POSIX locale (aka "C locale") for the LC_CTYPE (encoding) locale.
Standard streams use the locale encoding, which is usually ASCII with POSIX locale on most platforms (except on AIX: ISO 8859-1). In this case, data read from the OS (environment variables, command line arguments, filenames, etc.) may contain surrogate characters because of the internal usage of the surrogateescape error handler (see the PEP-383 for the rationale).
The problem is that standard output uses the strict error handler, and so print() fails to display OS data like filenames.
Example, "ls" command in Python:
---
Try it with "LANG=C python ls.py" in a directory containing non-ASCII characters and you will get unicode errors.
Issues bpo-19846 and bpo-19847 are examples of this annoyance.
I propose to use also the surrogateescape error handler for sys.stdout if the POSIX locale is used for LC_CTYPE at startup. Attached patch implements this idea.
With the patch, "LANG=C python ls.py" almost works as filenames and stdout are byte streams, even if the Unicode type is used.
vstinner commentedon Dec 13, 2013
Oh, in fact, sys.stdin is also modified by the patch (as I expected).
[-]Use "surrogateescape" error handler for sys.stdout on UNIX for the C locale[/-][+]Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale[/+]Sworddragon commentedon Dec 13, 2013
What would happen if we call this example script with LANG=C on the patch?:
---
Would it throw an exception on os.listdir('ä')?
vstinner commentedon Dec 13, 2013
test_ls.py: test script producing invalid filenames and then trying to display them into stdout.
Output with UTF-8 locale, UTF-8 terminal and Python 3.3 (or unpatched 3.4, it's the same):
ascii.txt
<UnicodeError 'invalid_utf8:\udcff.txt'>
<UnicodeError 'latin1:\udce9.txt'>
utf8:é€.txt
Output with C locale (ASCII), UTF-8 terminal and Python 3.3:
ascii.txt
<UnicodeError 'invalid_utf8:\udcff.txt'>
<UnicodeError 'latin1:\udce9.txt'>
<UnicodeError 'utf8:\udcc3\udca9\udce2\udc82\udcac.txt'>
Output with C locale (ASCII), UTF-8 terminal and patched Python 3.4:
ascii.txt
invalid_utf8:�.txt
latin1:�.txt
utf8:é€.txt
You get no Unicode error with LANG=C, but you get mojibake instead.
vstinner commentedon Dec 13, 2013
os.fsencode(text) always fail if text cannot be encoded to sys.getfilesystemencoding(). surrogateescape doesn't help here.
Your example is "artificial", you should not get 'ä'. All OS data is decoded from the filesystem encoding using the surrogateescape error handler (except on Windows, where strict is used, but it's a different story, Python uses Unicode functions when available so don't worry). So all these data can always be encoded back to bytes using os.fsencode().
More generally, os.fsencode(os.fsdecode(read_data)) == read_data is always true on Unix, with any filesystem (locale) encoding.
You may get Unicode data from other sources like files or a GUI, but I don't see what can be done here.
pitrou commentedon Dec 13, 2013
I think you mean LC_MESSAGES=C here.
(but it's not only about the English language; it's also about other locale parameters such as number formatting)
I think we should start thinking about making utf-8 the default filesystem encoding in 3.5 (under Unix).
bitdancer commentedon Dec 13, 2013
Reintroducing moji-bake intentionally doesn't sound like a particularly good idea, wasn't that what python3 was supposed to help prevent?
It does seem like a utf-8 default is the Way of the Future. Or even the present, most places.
abadger commentedon Dec 13, 2013
My impression was that python3 was supposed to help get rid of UnicodeError tracebacks, not mojibake. If mojibake was the problem then we should never have gone down the surrogateescape path for input.
serhiy-storchaka commentedon Dec 13, 2013
Mojibake in input can cause decoding error in other application which consumes output of Python script. In some cases this can be even worse thin UnicodeError in producer.
But for C locale this makes sense. I think we should try this experiment in 3.5. There will be much time for testing before 3.5 beta 1.
ncoghlan commentedon Dec 14, 2013
Getting rid of mojibake was the goal, surrogateescape was about dealing with cases where the "avoid mojibake" checks were spuriously breaking round-tripping between OS APIs due to other configuration errors (with LANG=C being set, or LANG not being set at all being the main problem). Other "high mojibake risk" power tools (like changing the encoding of an already open stream) are likely to return in the future, since there *are* cases where they're the right answer (e.g. you can't right an iconv equivalent in Python 3 at the moment, we need bpo-15216 implemented before that will be possible).
+1 for this solution - see bpo-19846 for the long discussion which got us to this point (there are a few unrelated tangents, a couple of them my fault, but this is definitely an improvement over the status quo.
ncoghlan commentedon Jan 3, 2014
Larry: I'm assuming it's way too late to make a change like this for the 3.4 release?
Slavek: assuming this change is made for 3.5 upstream, we may want to look at backporting it as a 3.4 patch in Fedora (as part of the Python-3-by-default project). Otherwise it's very easy to provoke Python 3 into throwing Unicode errors when attempting to print data provided by the OS.
larryhastings commentedon Jan 4, 2014
Yeah, unless there was a *huge* amount of support for changing this, it's way too late for 3.4.
13 remaining items
pitrou commentedon Apr 27, 2014
We should not overcomplicate this. I suggest that we simply use utf-8 under the C locale.
ncoghlan commentedon Apr 27, 2014
If you can convince Stephen Turnbull that's a good idea, sure. It's
probably more likely to be the right thing than "ASCII" or "ASCII +
surrogateescape", but in the absence of hard data, he's in a better
position than we are to judge the likely impact of that, at least in Japan.
I'm also going to hunt around on freedesktop.org to see if there's anything
more general there on the topic of encodings.
vstinner commentedon Apr 27, 2014
Do you mean utf8/strict or utf8/surrogateescape?
utf8/strict doesn't work (os.listdir raises an unicode error) if your
system is configured to use latin1 (ex: filenames are stored in this
encoding), but unfortunately your program is running in an empty
environment (so will use the POSIX locale).
vstinner commentedon Apr 27, 2014
Please open a new issue if you would prefer UTF-8. You will have to solve different technical issues. I tried to list some of them in issues bpo-19846 and bpo-19847.
In short, you should always decode and encode "OS data" with the same encoding. Python "file system encoding" is the locale encoding because in some places, PyUnicode_DecodeLocaleAndSize is used (ex: to decode PYTHONWARNINGS environment variable). A common location is PyUnicode_DecodeFSDefaultAndSize() before the Python codec is loaded. See also _Py_wchar2char() and _Py_char2wchar() functions which use the locale encoding and are used in many places.
I'm now closing the issue because the initial point (use surrogateescape error handler) is implemented in Python 3.5, and backporting such major change in Python 3.4 branch is risky right now.
pitrou commentedon Apr 28, 2014
The issue is about stdin and stdout, I'm not sure why os.listdir would
be affected.
ncoghlan commentedon Apr 28, 2014
Victor was referring to code like "print(os.listdir())". Those are the
motivating cases for ensuring round trips from system APIs to the standard
streams work correctly.
There's also the problem that sys.argv currently relies on the locale
encoding directly, because the filesystem encoding hasn't been worked out
at that point (see bpo-8776). So this current change will also make
"print(sys.argv)" work more reliably in the POSIX locale.
The conclusion I have come to is that any further decoupling of Python 3
from the locale encoding will actually depend on getting the PEP-432
bootstrapping changes implemented, reviewed and the PEP approved, so we
have more interpreter infrastructure in place by the time the interpreter
starts trying to figure out all these boundary encoding issues.
pitrou commentedon Apr 28, 2014
Yeah. My proposal had more to do with the fact that we should some day
switch to utf-8 by default on all POSIX systems, regardless of what the
system advertises as "best encoding".
ncoghlan commentedon Apr 28, 2014
Antoine Pitrou added the comment:
Yeah, that seems like a plausible future to me as well, and knowing it's a
step along that path actually gives me more motivation to get back to
working on the startup issues :)
Sworddragon commentedon Jan 6, 2017
Bug bpo-28180 has caused me to make a look at the "encoding" issue this and the tickets before have tried to solve more or less. Being a bit unsure what the root cause and intention for all this was I'm now at a point to actually check this ticket. Here is an example code (executed with Python 3.5.3 RC1 by having LANG set to C):
I thought with the surrogateescape error handler now being used for sys.stdout this would not throw an exception but I'm getting this:
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 0: ordinal not in range(128)
vstinner commentedon Jan 6, 2017
"I thought with the surrogateescape error handler now being used for sys.stdout this would not throw an exception but I'm getting this: (...)"
Please see the two recently proposed PEP: Nick's PEP-538 and my PEP-540, both propose (two different) solutions to your issue, especially for the POSIX locale (aka "C" locale).
Sworddragon commentedon Jan 6, 2017
The point is this ticket claims to be using the surrogateescape error handler for sys.stdout and sys.stdin for the C locale. I have never used surrogateescape explicitly before and thus have no experience for it and consulting the documentation mentions throwing an exception only for the strict error handler. I don't see anything that would make me think that surrogateescape would throw here an exception too. But maybe I'm just missing something.
vstinner commentedon Jan 6, 2017
This issue fixed exactly one use case: "List a directory into stdout" (similar to the UNIX "ls" or Windows "dir" commands):
https://www.python.org/dev/peps/pep-0540/#list-a-directory-into-stdout
Your use case is more "Display Unicode characters into stdout":
https://www.python.org/dev/peps/pep-0540/#display-unicode-characters-into-stdout
This use case is not supported by the issue. It should be fixed by PEP-538 or PEP-540.
Please join the happy discussion on the python-ideas mailing list to discuss how to "force UTF-8": this issue is closed, you shouldn't add new comments (other people will not see your comments).
vstinner commentedon Dec 18, 2017
Follow-up: the PEP-538 (bpo-28180) and PEP-540 (bpo-29240) have been accepted and implemented in Python 3.7!