Python 3 raises Unicode errors with the C locale

@malemburg

BPO	19846
Nosy	@malemburg, @loewis, @terryjreedy, @ncoghlan, @pitrou, @vstinner, @larryhastings, @jwilk, @abadger, @bitdancer, @serhiy-storchaka
Files	test.py: Example script asciilocale.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-12-09.10:42:16.755>
created_at = <Date 2013-11-30.21:40:45.188>
labels = ['type-bug', 'invalid', 'expert-IO']
title = 'Python 3 raises Unicode errors with the C locale'
updated_at = <Date 2017-12-18.14:38:09.078>
user = 'https://bugs.python.org/Sworddragon'

bugs.python.org fields:

activity = <Date 2017-12-18.14:38:09.078>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2013-12-09.10:42:16.755>
closer = 'vstinner'
components = ['IO']
creation = <Date 2013-11-30.21:40:45.188>
creator = 'Sworddragon'
dependencies = []
files = ['32914', '33026']
hgrepos = []
issue_num = 19846
keywords = ['patch']
message_count = 68.0
messages = ['204849', '204850', '204852', '205418', '205419', '205454', '205459', '205462', '205465', '205472', '205497', '205498', '205505', '205538', '205545', '205547', '205548', '205549', '205550', '205554', '205555', '205564', '205611', '205615', '205623', '205625', '205637', '205640', '205642', '205646', '205654', '205655', '205669', '205670', '205671', '205672', '205673', '205675', '205688', '205690', '205691', '205693', '205694', '205727', '205747', '205748', '205749', '205751', '205772', '205783', '205848', '205855', '205859', '205871', '206055', '206065', '206068', '206071', '206098', '206101', '206107', '206109', '206112', '206116', '206169', '232290', '283717', '308567']
nosy_count = 14.0
nosy_names = ['lemburg', 'loewis', 'terry.reedy', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'jwilk', 'a.badger', 'r.david.murray', 'Sworddragon', 'serhiy.storchaka', 'bkabrda', 'editor-buzzfeed']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19846'
versions = ['Python 3.3', 'Python 3.4']

Antoine Pitrou added the comment:

> Python uses the fact that the filesystem encoding is the locale
> encoding in various places.
The patch doesn't change that.

Nick Coghlan added the comment:

Note that the *only* change Antoine's patch makes is that:

*if* the locale encoding is ASCII (or an alias for ASCII)

*then* Python sets the filesystem encoding to UTF-8 instead

If the locale encoding is ASCII, filesystem encoding (UTF-8) is
different than the locale encoding.

Yes, that's the point. *Every* case I've seen where the locale encoding has been reported as ASCII on a modern Linux system has been because the environment has been configured to use the C locale, and that locale has a silly, antiquated, encoding setting.

This is particularly problematic when people remotely access a system with ssh and get given the C locale instead of something sensible, and then can't properly read the filesystem on that server.

The idea of using UTF-8 instead in that case is to *change* (and hopefully reduce) the number of cases where things go wrong.

if no non-ASCII data is encountered, the choice of ASCII vs UTF-8 doesn't matter
if it's a modern Linux distro, then the real filesystem encoding is UTF-8, and the setting it provides for LANG=C is just plain *wrong*
there may be other cases where ASCII actually *is* the filesystem encoding (in which case they're going to have trouble anyway), or the real filesystem encoding is something other than UTF-8

We're already approximating things on Linux by assuming every filesystem is using the *same* encoding, when that's not necessarily the case. Glib applications also assume UTF-8, regardless of the locale (http://unix.stackexchange.com/questions/2089/what-charset-encoding-is-used-for-filenames-and-paths-on-linux).

At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.

2013/12/8 Nick Coghlan <report@bugs.python.org>:

Yes, that's the point. *Every* case I've seen where the locale encoding has been reported as ASCII on a modern Linux system has been because the environment has been configured to use the C locale, and that locale has a silly, antiquated, encoding setting.

This is particularly problematic when people remotely access a system with ssh and get given the C locale instead of something sensible, and then can't properly read the filesystem on that server.

The solution is to fix the locale, not to fix Python. For example,
don't set LANG to C.

From the C locale, you cannot guess the "correct" encoding. In
Unicode, the general rule is to never try the encoding.

The idea of using UTF-8 instead in that case is to *change* (and hopefully reduce) the number of cases where things go wrong.

If the OS uses ISO-8859-1, forcing Python (filesystem) encoding to
UTF-8 would produce invalid filenames, display mojibake and more
generally produce data incompatible with other applicatons (who rely
on the C locale, and so the ASCII encoding).

there may be other cases where ASCII actually *is* the filesystem encoding (in which case they're going to have trouble anyway), or the real filesystem encoding is something other than UTF-8

As I wrote before, os.getfilesystemencoding() is *not* the filesystem
encoding. It's the "OS" encoding used to decode any kind of data
coming for the OS and used to encode back Python data to the OS. Just
some examples:

DNS hostnames
Environment variables
Command line arguments
Filenames
user/group entries in the grp/pwd modules
almost all functions of the os module, they return various type of
information (ttyname, ctermid, current working directory, login, ...)

We're already approximating things on Linux by assuming every filesystem is using the *same* encoding, when that's not necessarily the case. Glib applications also assume UTF-8, regardless of the locale (http://unix.stackexchange.com/questions/2089/what-charset-encoding-is-used-for-filenames-and-paths-on-linux).

If you use a different encoding but only just for filenames, you will
get mojibake when you pass a filename on the command line or in an
environment varialble.

At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.

Getting ASCII filesystem encoding is annoying, but I would not say
that it fundamentally breaks Python 3. If you want to do something,
you should write documentation explaining how to configure properly
Linux.

If you use a different encoding but only just for filenames, you will
get mojibake when you pass a filename on the command line or in an
environment varialble.

That's not what the patch does.

2013/12/8 Antoine Pitrou <report@bugs.python.org>:

> Python uses the fact that the filesystem encoding is the locale
> encoding in various places.

The patch doesn't change that.

You wrote: "-> With the patch: utf-8 utf-8 utf-8 ANSI_X3.4-1968", so
os.get sys.getfilesystemencoding() != locale.getpreferredencoding().
Or said differently, the filesystem encoding is different than the
locale encoding.

So please read again my following message which list real bugs:
https://mail.python.org/pipermail/python-dev/2010-October/104509.html

If you want to use a filesystem encoding different than the locale
encoding, you have to patch Python where Python assumes that the
filesystem encoding is the locale encoding, to fix all these bugs.
Starts with:

PyUnicode_DecodeFSDefaultAndSize()
PyUnicode_EncodeFSDefault()
_Py_wchar2char()
_Py_char2wchar()

It should be easier to change this function if the FS != locale only
occurs when FS is "UTF-8". On Mac OS X, Python always use UTF-8 for
the filesystem encoding, it doesn't care of the locale encoding. See
_Py_DecodeUTF8_surrogateescape() in unicodeobject.c, you may reuse it.

With a better patch, I can do more experiment to check if they are
other tricky bugs.

Does at least your patch pass the whole test suite with LANG=C?

Setting sys.stderr encoding to UTF-8 on ASCII locale is wrong. sys.stderr has the backslashreplace error handler by default, so it newer fails and should newer produce non-ASCII data on ASCII locale.

Antoine: are you characterizing this as a "bug" rather than a "new feature"?

I'd like to see more of a consensus before something like this gets checked in. Right now I see a variety of opinions.

When I think "conservative approach" and "knows about system encoding stuff", I think of Martin. Martin, can I ask you to form an opinion about this?

Or said differently, the filesystem encoding is different than the
locale encoding.

Indeed, but the FS encoding and the IO encoding are the same.
"locale encoding" doesn't really matter here, as we are assuming that it's wrong.

Victor, people set "LANG=C" for all sorts of reasons, and we have no
control over how operating systems define that locale. The user
perception is "Python 3 doesn't work properly when you ssh into
systems", not "Gee, I wish operating systems defined the C locale more
sensibly".

If you can come up with a more sensible guess than UTF-8, great, but
believing the nonsense claim of "ASCII" from the OS is a
not-insignificant usability issue on Linux, because it hoses *all* the
OS API interactions. Yes, theoretically, using UTF-8 can cause
problems, *if* the following all occur:

the OS *claims* the OS encoding is ASCII (so Python uses UTF-8 instead)
the OS encoding is *actually* something other than UTF-8
the program encounters non-ASCII data and writes it out to disk

For fear of doing the wrong thing in that incredibly rare scenario,
you're leaving Python broken under the C locale on *every* modern
Linux distro as soon as it encounters non-ASCII data in an OS
interface.

"haypo: title: Setting LANG=C breaks Python 3 -> print() and write() are relying on sys.getfilesystemencoding() instead of sys.getdefaultencoding()"

Oh, I didn't want to change the title of the issue, it's a bug in Roundup when I reply by email :-/

> Or said differently, the filesystem encoding is different than the
> locale encoding.

Indeed, but the FS encoding and the IO encoding are the same.
"locale encoding" doesn't really matter here, as we are assuming that
it's wrong.

Oh, I realized that "FS encoding" term in not clear. When I wrote "FS encoding", I mean sys.getfilesystemencoding() which is mbcs on Windows, UTF-8 on Mac OS X and (currently) the locale encoding on other platforms (UNIX, ex: Linux/FreeBSD/Solaris/AIX).

--

IMO there are two different points in this issue:

(a) which encoding should be used when the C locale is used: the encoding announced by the OS using nl_langinfo(CODESET) (current choice) or use an arbitrary optimistic "utf-8" encoding?

(b) for technical reasons, Python reuses the C codec during Python initialization to decode and encode OS data, and so currently Python *must* use the locale encoding for its "filesystem encoding"

Before being able to pronounce me on the point (a), I would like to see a patch fixing the point (b). I'm not against fixing point (b). I'm just saying that it's not trivial and obviously it must be fixed to change the status of point (a). I even gave clues to fix point (b).

--

asciilocale.patch has many issues. Try to run the Python test suite using this patch to see what I mean. Example of failures:

======================================================================
FAIL: test_non_ascii (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_cmd_line.py", line 140, in test_non_ascii
    assert_python_ok('-c', command)
  File "/home/haypo/prog/python/default/Lib/test/script_helper.py", line 69, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/haypo/prog/python/default/Lib/test/script_helper.py", line 55, in _assert_python
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is 1, stderr follows:
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 12: surrogates not allowed

======================================================================
FAIL: test_ioencoding_nonascii (test.test_sys.SysModuleTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_sys.py", line 603, in test_ioencoding_nonascii
    self.assertEqual(out, os.fsencode(test.support.FS_NONASCII))
AssertionError: b'' != b'\xc3\xa6'

======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_warnings.py", line 774, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

======================================================================
FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_warnings.py", line 774, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

test_warnings is probably bpo-9988, test_cmd_line failure is maybe bpo-9992.

There are maybe other issues, the Python test suite only have a few tests for non-ASCII characters.

--

If anything is changed, I would prefer to have more than a few months of test to make sure that it doesn't break anything. So I set the version field to Python 3.5.

On dim., 2013-12-08 at 22:22 +0000, STINNER Victor wrote:

(b) for technical reasons, Python reuses the C codec during Python
initialization to decode and encode OS data, and so currently Python
*must* use the locale encoding for its "filesystem encoding"

Ahhh! Well indeed that's a bummer :-)

asciilocale.patch has many issues. Try to run the Python test suite
using this patch to see what I mean.

I'm assuming much of this is due to (b) (all those tests seem to spawn
external processes).

It seems there is more work to do to get this right, but I'm not
terribly interested either. Feel free to take over.

It seems there is more work to do to get this right, but I'm not
terribly interested either. Feel free to take over.

If you are talking to me: I'm currently opposed to change anything, so I'm not interested to work on a patch. IMO Python works fine and you should try to workaround the current limitations :-)

If someone is interested to write an huge patch fixing all these issues, I would be able to reconsider my opinion on point (a).

End users tripping over this by setting LANG=C is one of the pain points of Python 3 relative to Python 2 for Fedora, so I've added a couple of Fedora folks to the nosy list.

My current understanding of the situation:

we should leave Windows and Mac OS X alone, since they ignore the locale when choosing the OS API encoding anyway
the main problem is on Linux (but potentially other *nix systems as well), where people set "LANG=C" for a variety of reasons, but this has the side effect of Python 3 choosing an inappropriate encoding (ASCII rather than UTF-8) when talking to the OS APIs.

Given the initialisation problems, this may be something that PEP-432 (the initialisation process rewrite) can help with (since it changes the initialisation order to create a more complete Python runtime before it starts to configure the OS interfaces).

Tangentially related, we may want to consider aliasing sys.getfilesystemencoding, os.fsencode and os.fsdecode as something like sys.getosapiencoding, os.apiencode and os.apidecode, since the current naming is misleading (the value is based on the platform and environment, not any particular filesystem, and is used for almost all bytes-based OS APIs, not just filesystem metadata)

End users tripping over this by setting LANG=C is one of the pain points of Python 3 relative to Python 2 for Fedora, so I've added a couple of Fedora folks to the nosy list.

Sorry, I'm not aware of such issue. Do you have examples?

the main problem is on Linux (but potentially other *nix systems as well), where people set "LANG=C" for a variety of reasons, but this has the side effect of Python 3 choosing an inappropriate encoding (ASCII rather than UTF-8) when talking to the OS APIs.

Why do you think that the issue is specific to Python 3? Try to open a
terminal with LC_ALL=C and try to type non-ASCII characters with your
keyboard. You can't because your terminal uses ASCII. Did you
applications written in another language handling Unicode, like Perl?
(Perl with Unicode support correctly enabled, it's "use utf8;" if I
remember correctly).

Can you explain the "various reasons" why users explictly force the
encoding to ASCII?

I use LANG=C to get manual pages and error messages in english. But
"LANG=en_US man ls" would be more correct, or "LC_MESSAGES=en_US man
ls" to be pedantic. (Env var priority: LC_ALL > LANG > LC_xxx).

IMO if you use LANG=C, you must not complain that Unicode stopped
working, but you should learn how to configure locales. Trivial
examples like the one which can be found in the initial message
(msg204849) are wrong: why would you force all locales to C and use
non-ASCII characters?

Given the initialisation problems, this may be something that PEP-432 (the initialisation process rewrite) can help with (since it changes the initialisation order to create a more complete Python runtime before it starts to configure the OS interfaces).

I don't see how it would help to solve my point (b).

Technically, this issue cannot be fixed. Or to be more specific, I
don't want to fix it, it's a waste of time. So I don't understand what
do you expect from this open issue?

I would prefer to close it as invalid or wontfix to be clear.

On 9 December 2013 12:08, STINNER Victor <report@bugs.python.org> wrote:

STINNER Victor added the comment:

> End users tripping over this by setting LANG=C is one of the pain points of Python 3 relative to Python 2 for Fedora, so I've added a couple of Fedora folks to the nosy list.

Sorry, I'm not aware of such issue. Do you have examples?

Armin's travails with remote shell access and Python 3 are just as
likely today as they were a couple of years ago:
http://lucumr.pocoo.org/2011/12/7/thoughts-on-python3/

(although technically that was a terminal ending up with the POSIX
locale, rather than specifically LANG=C)

> - the main problem is on Linux (but potentially other *nix systems as well), where people set "LANG=C" for a variety of reasons, but this has the side effect of Python 3 choosing an inappropriate encoding (ASCII rather than UTF-8) when talking to the OS APIs.

Why do you think that the issue is specific to Python 3? Try to open a
terminal with LC_ALL=C and try to type non-ASCII characters with your
keyboard. You can't because your terminal uses ASCII. Did you
applications written in another language handling Unicode, like Perl?
(Perl with Unicode support correctly enabled, it's "use utf8;" if I
remember correctly).

It's the fact this used to work transparently in Python 2 (since all
these interfaces were just bytes based on the Python side as well)
that's a problem. That makes the new sensitivity to the locale
encoding a usability regression, and that's a concern for distros that
are considering switching their default Python version.

Can you explain the "various reasons" why users explictly force the
encoding to ASCII?

testing applications for POSIX compliance
default settings on servers where you don't control the environment
because they never previously had to care, and it's only Python 3
deciding to pay attention to it which makes it relevent for them

I use LANG=C to get manual pages and error messages in english. But
"LANG=en_US man ls" would be more correct, or "LC_MESSAGES=en_US man
ls" to be pedantic. (Env var priority: LC_ALL > LANG > LC_xxx).

IMO if you use LANG=C, you must not complain that Unicode stopped
working, but you should learn how to configure locales. Trivial
examples like the one which can be found in the initial message
(msg204849) are wrong: why would you force all locales to C and use
non-ASCII characters?

And yet, in Python 2, people could do that, and Python didn't care.
*That's* the regression I'm worried about. If it hadn't round-tripped
cleanly in Python 2, I wouldn't care here either.

> Given the initialisation problems, this may be something that PEP-432 (the initialisation process rewrite) can help with (since it changes the initialisation order to create a more complete Python runtime before it starts to configure the OS interfaces).

I don't see how it would help to solve my point (b).

Having a Python runtime available makes things that are currently
tediously painful to deal with during startup easier to tweak. I'm not
sure it *will* help in this particular case, but it's now one I'm
going to keep an eye on.

Technically, this issue cannot be fixed. Or to be more specific, I
don't want to fix it, it's a waste of time. So I don't understand what
do you expect from this open issue?

A way to get Python 3 to cope as well with a misconfigured OS
environment as Python 2 did.

I would prefer to close it as invalid or wontfix to be clear.

It's a usability regression from Python 2, so I don't want to give up
on it. It may be that we just implement a "ignore what the OS claims,
it's misconfigured, just use UTF-8 for everything" flag. But OS
configuration errors shouldn't cripple the Python runtime.

You should keep things more simple:

Python and the operation system/filesystem are in a client-server relationship and Python should validate all.
It doesn't matter what you will finally decide to be the default encoding on various places - all will provide race-conditions with no exception.
The easiest way to fix this is to give the developer the ability to make a decision (like sys.use_strict_encoding(), sys.setfilesystemencoding(), sys.setdefaultencoding() etc.).

For example giving the developer control is especially needed if he wants to handle multiple different filesystems.

Why do you think that the issue is specific to Python 3? Try to open a
terminal with LC_ALL=C and try to type non-ASCII characters with your
keyboard. You can't because your terminal uses ASCII.

sworddragon@ubuntu:$ LANG=C
sworddragon@ubuntu:$ ä
bash: $'\303\244': command not found

The terminal doesn't pseudo-crash with an exception because it doesn't matter about encodings.
It allows to change the encoding at runtime.

Did you
applications written in another language handling Unicode, like Perl?

Compare C: It wouldn't matter like the terminal. For example fopen will simply return NULL if it can't open the file 'ä' because the filesystem is endoded with ISO-8859-1 and we wanted to open the utf-8 counterpart.

Can you explain the "various reasons" why users explictly force the
encoding to ASCII?

For example I'm using this for testcases to set the language uncomplicated to english.

And yet, in Python 2, people could do that, and Python didn't care.
*That's* the regression I'm worried about. If it hadn't round-tripped
cleanly in Python 2, I wouldn't care here either.

$ python2.7 -c "print u'\u20ac'"
€
$ LANG=C python2.7 -c "print u'\u20ac'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)

And even worse:

$ python2.7 -c "print u'\u20ac'" >/dev/null
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)

What the wart!

Other program can produces wrong (or even absolutely senseless) output with C locale.

$ LANG=C ls
???????????? ????????????????                                      ???????????????????? ??????????
?????????? ??????????????                                          ?????????????? ????????????????????????
?????????? ????????                                                ???????????? ??????????????
?????????????? ??????????????????                                  ???????? ????????????????????

What is better, silently produce corrupted output or raise an exception? If first, then let just set the "replace" or "backslashreplace" error handler for sys.stdout by default.

sworddragon@ubuntu:$ LANG=C
sworddragon@ubuntu:$ ä
bash: $'\303\244': command not found

The terminal doesn't pseudo-crash with an exception because it doesn't
matter about encodings. - It allows to change the encoding at runtime.

This is not a locale of your terminal. Try LANG=C xterm.

The "C" locale is part of the ANSI C standard. The "POSIX" locale is an alias for the "C" locale and a POSIX standard, so we cannot just replace the ASCII encoding with UTF-8 as we wish, so Antoine's patch won't work.

See e.g. http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html

The C and POSIX locale settings are the only locale settings that are guaranteed to always exist in C libraries. Python 3 should work with such locale settings. It doesn't have to be able to output non-ASCII code points, but it should run with ASCII data.

AFAIK, Python 3 does work with ASCII data in the C locale, so I'm not sure whether this is a bug at all.

I didn't understand Serhiy's "ls" example. I tried:

$ mkdir unicode
$ cd unicode
$ python3 -c 'open("ab\xe9.txt", "w").close()'
$ python3 -c 'open("euro\u20ac.txt", "w").close()'
$ ls
abé.txt  euro€.txt
$ LANG=C ls
ab??.txt  euro???.txt

Ah yes, I didn't remember that "ls" is aware of the locale encoding.

printf() and wprintf() behave differently on unencodable/undecoable characters:
http://unicodebook.readthedocs.org/en/latest/programming_languages.html#printf-functions-family

Again, the issue is not specific to Python. So it's time to learn how to configure correctly your locales.

About the "interoperability" point I mentionned in my first message ("This encoding is the best choice for interopability with other (python2 or non python) programs."): if you work around the annoying ASCII encoding by forcing UTF-8 encoding, Python may produce data which would be incompatible with other applications following POSIX and so using the ASCII encoding.

Nick> testing applications for POSIX compliance

Sorry but what do you mean by "POSIX compliance"? The POSIX standard only specify the ASCII encoding.

http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html
"The tables in Locale Definition describe the characteristics and behavior of the POSIX locale for data consisting entirely of characters from the portable character set and the control character set. For other characters, the behavior is unspecified. For C-language programs, the POSIX locale shall be the default locale when the setlocale() function is not called."

http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap06.html#tagtcjh_3
"Portable character set" = ASCII

Marc-Andre> AFAIK, Python 3 does work with ASCII data in the C locale, so I'm not sure whether this is a bug at all.

What do you mean? Python uses the surrogateescape encoding since Python 3.1, undecodable bytes are stored as surrogate characters.

Many bugs related to locales were fixed in Python 3.2, 3.3 and 3.4.

There are remaining bugs?

On 09.12.2013 11:19, STINNER Victor wrote:

STINNER Victor added the comment:

Marc-Andre> AFAIK, Python 3 does work with ASCII data in the C locale, so I'm not sure whether this is a bug at all.

What do you mean? Python uses the surrogateescape encoding since Python 3.1, undecodable bytes are stored as surrogate characters.

Many bugs related to locales were fixed in Python 3.2, 3.3 and 3.4.

There are remaining bugs?

I was referring to the original bug report on this ticket.

FWIW: I don't think you can expect Python to work without exceptions
if you use a C locale and write non-ASCII data to stdout. I also
don't think that the new ticket title is correct - or at least,
I fail to see which aspect of Python breaks with LANG=C :-)

I'm closing the issue as invalid, because Python 3 behaviour is correct and must not be changed.

Standard streams (sys.stdin, sys.stdout, sys.stderr) uses the locale encoding. sys.stdin and sys.stdout use the strict error handler, sys.stderr uses the backslashreplace error handler. These encodings and error handlers can be overriden by the PYTHONIOENCODING. Since Python 3.3, it's possible to only set the error handler using ":errors" syntax (ex: PYTHONIOENCODING=":replace").

Python uses sys.getfilesystemencoding() to decode data from / encode data to the operating system. Example of operating system data: command line arguments, environment variables, host names, filenames, user names, etc.

On Windows, Python tries to use the wide character (Unicode) API of Windows anywhere to avoid any conversion, to not loose data. The MBCS codec (ANSI code page) of Windows uses a replace error handler by default, it looses data. Try for example os.listdir() in a directory containing filenames not encodable to the ANSI code page in Python 2 (or os.listdir(b'.') in Python 3).

On Mac OS X, Python always use UTF-8 for sys.getfilesystemencoding() (with the surrogateescape error handler, see the PEP-383). The locale encoding is ignored for sys.getfilesystemencoding() (the locale encoding is still used in some functions).

On other operating systems... it's more complex. Python uses the locale encoding for sys.getfilesystemencoding() (with the surrogateescape error handler, see the PEP-383). For the POSIX locale (aka the "C" locale), you may get the ASCII encoding on Linux, ASCII on FreeBSD and Solaris (whereas these operating systems announce an alias of the ISO 8859-1 encoding, but use ASCII in practice), ISO 8859-1 on AIX etc. Using the locale encoding is the best choice for interoperability with other applications (which use also the locale encoding).

Even if an application uses "raw bytes" (like Python 2), these bytes are still "locale aware". For example, when "raw bytes" are written to the standard output, bytes are decoded to find the appropriate character in the font of the terminal. When "raw bytes" are written into a socket to generate a HTML document (ex: listing of a directory, so a list of filenames), the web brower will decode them from them encoding announced in the HTML page. Even if the encoding is not explicit, it does still exist. Read other comments of this issue for other examples.

Forcing the POSIX locale to get an user interface in english is wrong if you also expect from your application to still generate valid "raw bytes" in your "system" encoding (ISO 8859-1, ShiftJIS, UTF-8, whatever). To change the language, the correct environment variable is LC_CTYPE: use LC_CTYPE=C. Or better, use the real english locale which will probably handle better currency, numbers, etc. Example: LC_CTYPE=en_US.utf8 (on Fedora, "en_US" locale uses the ISO 8859-1 encoding).

I'm closing the issue as invalid, because Python 3 behaviour is correct > and must not be changed.

The fact that write() uses sys.getfilesystemencoding() is either a defect or a bad design (I leave the decision to you).

But I'm still missing a reply to my suggestion. As I'm seeing it has no disadvantages to give the developer optionally the control.

The fact that write() uses sys.getfilesystemencoding() is either a defect or a bad design (I leave the decision to you).

"Standard streams (sys.stdin, sys.stdout, sys.stderr) uses the locale encoding. sys.stdin and sys.stdout use the strict error handler, sys.stderr uses the backslashreplace error handler. These encodings and error handlers can be overriden by the PYTHONIOENCODING. Since Python 3.3, it's possible to only set the error handler using ":errors" syntax (ex: PYTHONIOENCODING=":replace")."

stdout uses the locale encoding (and if you read my whole message, you may understand why sys.getfilesystemencoding() is also the locale encoding on UNIX).

(FYI on Windows, the OEM code page is used for standard streams.)

sys.getdefaultencoding() is always utf-8, this is unrelated to standard streams and OS data: it's the default value of the encoding parameter of str.encode() and str.decode(). I'm surprised that it's not documented to be utf-8, it is hardcoded and so always utf-8 in Python 3.

But I'm still missing a reply to my suggestion. As I'm seeing it has no disadvantages to give the developer optionally the control.

"Standard streams (sys.stdin, sys.stdout, sys.stderr) uses the locale encoding. sys.stdin and sys.stdout use the strict error handler, sys.stderr uses the backslashreplace error handler. These encodings and error handlers can be overriden by the PYTHONIOENCODING. Since Python 3.3, it's possible to only set the error handler using ":errors" syntax (ex: PYTHONIOENCODING=":replace")."

If the environment variable is not enough, see also bpo-15216 which proposes to add a TextIOWrapper.set_encoding() method. (I'm not really a fan of this proposition, but it looks like some users ask for it.)

The fact that write() uses sys.getfilesystemencoding() is either
a defect or a bad design (I leave the decision to you).

I have good news for you. write() does not cal sys.getfilesystemencoding(), because the encoding is set at the time the file is opened.

But I'm still missing a reply to my suggestion. As I'm seeing it
has no disadvantages to give the developer optionally the control.

The programmer has all the control they need. They can open their own pipes using any encoding they like, and they can even reopen stdin/stdout with a different encoding if they wish.

If the environment variable is not enough

There is a big difference between environment variables and internal calls: Environment variables are user-space while builtin/library functions are developer-space.

I have good news for you. write() does not cal
sys.getfilesystemencoding(), because the encoding is set at the time > the file is opened.

Thanks for the clarification. I wished somebody had sayed me that after this sentence in my startpost: "It seems that print() and write() (and maybe other of such I/O functions) are relying on sys.getfilesystemencoding()."

In theory this makes already my ticket invalid. Well, but now I would wish print() would allow to choose the encoding like open() too^^

There is a big difference between environment variables and internal calls: Environment variables are user-space while builtin/library functions are developer-space.

You can reopen sys.stdout with a different encoding and replace sys.stdout. I don't remember the exact recipe, it's tricky if you want portable code (you have to take care of newline).

For example, I wrote:
http://hg.python.org/cpython/file/ebe28dba4a78/Lib/test/regrtest.py#l895

But you can avoid reopening the file using stdout.detach().

In theory this makes already my ticket invalid. Well, but now I would wish print() would allow to choose the encoding like open() too^^

Many options were already proposed. Another way, less convinient is to use sys.stdout.buffer.write("text".encode(encoding)) (you have to flush sys.stdout before, and flush the buffer after, to avoid inconsistencies between the TextIOWrapper and the BufferedWriter).

Ahh... added to the nosy list and bug closed all before I got up for the day ;-)

A few words:

I do think that python is broken here.

I do not think that translating everything to utf-8 if ascii is the locale's encoding is the solution.

As I would state it, the problem is that python's boundary with the OS is not yet uniform. If you set LC_ALL=C (note, LC_ALL=C is just one of multiple ways to beak things. For instance, LC_ALL=en_US.utf8 when dealing with latin-1 data will also break) then python will still *read* non-ascii data from the OS through some interfaces but it won't output it back to the OS. ie:

$ mkdir unicode && cd unicode
$ python3 -c 'open("ñ.txt".encode("latin-1"), "w").close()'
$ LC_ALL=en_US.utf8 python3
>>> import os
>>> dir_listing = os.listdir('.')
>>> for entry in dir_listing: print(entry)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf1' in position 0: surrogates not allowed

Note that currently, input() and sys.stdin.read() won't read undecodable data so this is somewhat symmetrical but it seems to me that saying "everything that interfaces with the OS except the standard streams will use surrogateescape on undecodable bytes" is drawing a line in an unintuitive location.

(A further note to serhiy.storchaka.... Your examples are not showing anything broken in other programs. xterm is refusing both input and output that is non-ascii. This is symmetric behaviour. ls is doing its best to display a *human-readable* representation of bytes that it cannot convert in the current encoding. It also provides the -b switch to see the octal values if you actually care. Think of this like opening a binary file in less or another pager.)

(Further note for haypo -- On Fedora, the default of en_US is utf8, not ISO8859-1.)

There's a wrong assumption here: glib applications on Linux use UTF-8
regardless of locale. That's the part I have a problem with: the assumption
that the locale will correctly specify the encoding to use for OS APIs on
modern Linux systems.

It's simply not always true: some Linux distros would be better handled
like OS X, where we always use UTF-8, regardless of what the locale says.

Nick: which glib functions are you specifically referring to? Many of them don't deal with strings at all, and of those that do, many are encoding-agnostic (i.e. it is correct to claim that they operate on UTF-8, but likewise also correct that they operate on Latin-1, simultaneously).

It's simply not always true: some Linux distros would be better handled
like OS X, where we always use UTF-8, regardless of what the locale says.

Perhaps by the 3.5 timeframe we can default to utf-8 on all Unix
systems?

I confess I didn't independently verify the glib claim in the Stack
Overflow post.

However, Toshio's post covers the specific error case we were discussing at
Flock (and I had misremembered), where the standard streams are classed as
"OS APIs" for the purpose of deciding which encoding to use, but as user
data APIs for the purpose of deciding which error handler to use. So the
standard streams are only "sort of" an OS API, since they don't participate
in the surrogateescape based round tripping guarantee by default.

From what I read, it appears that the SO posting is plain wrong. Consider, for example,

https://developer.gnome.org/glib/stable/glib-Character-Set-Conversion.html#g-filename-to-utf8

# Converts a string which is in the encoding used by GLib for filenames
# into a UTF-8 string. Note that on Windows GLib uses UTF-8 for filenames;
# on other platforms, this function indirectly depends on the current locale.

The SO author might have misread the part where it says that glib uses UTF-8 *on Windows* (instead of the braindead "ANSI" encoding indirection).

2013/12/10 Martin v. Löwis <report@bugs.python.org>:

>From what I read, it appears that the SO posting is plain wrong. Consider, for example,

https://developer.gnome.org/glib/stable/glib-Character-Set-Conversion.html#g-filename-to-utf8

Converts a string which is in the encoding used by GLib for filenames

into a UTF-8 string. Note that on Windows GLib uses UTF-8 for filenames;

on other platforms, this function indirectly depends on the current locale.

The SO author might have misread the part where it says that glib uses UTF-8 *on Windows* (instead of the braindead "ANSI" encoding indirection).

I wrote some notes about glib here:
http://unicodebook.readthedocs.org/en/latest/libraries.html#the-glib-library

g_filename_from_utf8() uses the g_get_filename_charsets() encoding.
g_get_filename_charsets() is the ANSI code page on Windows and the
locale encoding on Linux, except if G_FILENAME_ENCODING or
G_BROKEN_FILENAMES environment variables are set.

glib has a nice g_filename_display_name() function.

@Haypo

Looking at the glib code, this looks like the SO post is closer to the truth. The API documentation for g_filename_to_utf8() is over-simplified to the point of confusion. This section of the glib API document is closer to what the code is doing: https://developer.gnome.org/glib/stable/glib-Character-Set-Conversion.html#file-name-encodings

When encoding matters, glib and gtk functions will assume that char*'s that you pass to them point to strings which are encoded in utf-8.
When char* are not utf8 you are responsible for converting them to utf8 to be used by the glib functions (if encoding matters).
glib provides g_filename_to_utf8() for the special case of transforming filenames into the encoding that glib expects. (Presumably because glib and gtk deal with non-utf8 unicode filenames more often than the equivalent environment variables, command line switches, etc).
Contrary to the API docs for g_filename_to_utf8(), g_filename_to_utf8() will simply return a copy of the byte string it was passed unless G_FILENAME_ENCODING or G_BROKEN_FILENAMES is set. If those are set, then the value of G_FILENAME_ENCODING might be used to attempt to decode the filename or the encoding specified in the user's locale might be used.

@Haypo, I'm pretty sure from reading the code for g_get_filename_charsets() that you have the conditionals reversed. What I'm seeing is:

if G_FILENAME_ENCODING:
    charset = the first charset listed in G_FILENAME_ENCODING
    if charset == '@locale':
        charset = charset of user's locale
elif G_BROKEN_FILENAMES:
    charset = charset of user's locale
else:
    charset = 'UTF-8'

2013/12/10 Toshio Kuratomi <report@bugs.python.org>:

if G_FILENAME_ENCODING:
charset = the first charset listed in G_FILENAME_ENCODING
if charset == '@Locale':
charset = charset of user's locale
elif G_BROKEN_FILENAMES:
charset = charset of user's locale
else:
charset = 'UTF-8'

g_get_filename_charsets() returns a list of encodings. For the last
case (else:), it uses ['utf-8', local_encoding] on UNIX. It's reliable
because the utf-8 encoding has a nice feature, the utf-8 decoder fails
if the byte string is not a valid utf-8 string.

It would interesting to test this approach (try utf-8 or use the
locale encoding) in
PyUnicode_DecodeFSDefault/PyUnicode_EncodeFSDefault and
_Py_char2wchar/_Py_wchar2char.

It would interesting to test this approach (try utf-8 or use the locale encoding) ...

Oh, it may be easy to implement it for decoders, but what about
encoders? Should os.fsencode() always use UTF-8??

Yes, it returns a list but unless I'm missing something in the general case it's the caller's responsibility to loop through the charsets to test for failure and try again. This is not done automatically.

In the specific case we're talking about, first get_filename_charset() decides to only return the first entry in the list of charsets: list.https://git.gnome.org/browse/glib/tree/glib/gconvert.c#n1118

and then g_filename_to_utf8() disregards the charsets altogether because it sees that the filename is supposed to be utf-8 https://git.gnome.org/browse/glib/tree/glib/gconvert.c#n1160

> The fact that write() uses sys.getfilesystemencoding() is either
> a defect or a bad design (I leave the decision to you).

I have good news for you. write() does not cal sys.getfilesystemencoding(), because the encoding is set at the time the > file is opened.

Now after some researching I see I wasn't wrong at all. I should've been sayed:

"The fact that write() -> open() relies on sys.getfilesystemencoding() (respectively locale.getpreferredencoding()) at default as encoding is either a defect or a bad design (I leave the decision to you)."

Or am I overlooking something?

"The fact that write() -> open() relies on sys.getfilesystemencoding()
(respectively locale.getpreferredencoding()) at default as encoding is
either a defect or a bad design (I leave the decision to you)."

Or am I overlooking something?

First, you should probably just drop mentioning write() or print() or any of the functions that actually perform I/O. The crucial decisions about decoding are made inside open().

Second, open() is implemented in C. It cannot "rely on sys.getfilesystemencoding()" as it never calls it. Internally, sys.getfilesystemencoding() simply returns a C global called Py_FileSystemDefaultEncoding. But open() doesn't examine that, either.

Instead, open() determines the default encoding by calling the same function that's used to initialize Py_FileSystemDefaultEncoding: get_locale_encoding() in Python/pythonrun.c. Which on POSIX systems calls the POSIX function nl_langinfo().

If you want to see the actual mechanisms involved, you should read the C source code in Modules/_io in the Python trunk. open() is implemented as the C function io_open() in _iomodule.c. When it opens a file in text mode without an explicit encoding, it wraps it in a TextIOWrapper object; the __init__ function for this class is the C function textiowrapper_init() in textio.c.

As for your assertion that this is "either a defect or a bad design": I leave the critique of that to others.

Instead, open() determines the default encoding by calling the same function that's used to initialize Py_FileSystemDefaultEncoding: get_locale_encoding() in Python/pythonrun.c. Which on POSIX systems calls the POSIX function nl_langinfo().

open() will use at default the encoding of nl_langinfo() as sys.getfilesystemencoding() does on *nix. This is the part that looks dirty to me. As soon as LANG is set to C open() will rely on 'ascii' due to nl_langinfo() like sys.getfilesystemencoding() does too.

There's an alternative to trying to force a different encoding for the
standard streams when the OS claims ASCII as the OS encoding: we can
default to surrogateescape as the error handler, on the assumption that
whatever the *real* OS encoding is, it definitely isn't ASCII.

That means we'll still complain about displaying improperly encoded data
when the OS suggests a plausible encoding, but we won't fail entirely just
because someone enabled (deliberately or accidentally) the POSIX locale.

By the way I have found a valid use case for LANG=C. udev and Upstart are not setting LANG which will result in the ascii encoding for invoked Python scripts. This could be a problem since these applications are commonly dealing with non-ascii filesystems.

By the way, Java behaves as Python: with LANG=C, Java uses ASCII:

http://stackoverflow.com/questions/13415975/cant-read-utf-8-filenames-when-launched-as-an-upstart-service

udev and Upstart are not setting LANG

So it's an issue in udev and Upstart. See for example:
https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1235483
https://bugs.launchpad.net/ubuntu-translations/+bug/1208272

I found examples using "LANG=$LANG ..." when running a command in Upstart for example. I found another example using:

if [ -r /etc/default/locale ]; then
. /etc/default/locale
export LANG LANGUAGE
elif [ -r /etc/environment ]; then
. /etc/environment
export LANG LANGUAGE
fi

https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1235483

After opening many hundred tickets I would say: With luck this ticket will get a response within the next year. But in the worst case it will be simply refused.

I found examples using "LANG=$LANG

This Upstart script:

"exec echo LANG=$LANG > /tmp/test.txt"

Will result in the following:

root@ubuntu:# start test
test stop/waiting
root@ubuntu:# cat /tmp/test.txt
LANG=

At least in this example I'm getting on my system an empty LANG.

It's not a bug for upstart, systemd, sysvinit, cron, etc to use LANG=C. The POSIX locale is the only locale guaranteed to exist on a system. Therefore these low level services should be using LANG=C. Embedded systems, thin clients, and other low memory or low disk devices may benefit from shipping without any locales.

I created the issue bpo-19977 as a follow up of this one: "Use surrogateescape error handler for sys.stdout on UNIX for the C locale".

I propose to modify the error handler, the encoding cannot be modified. See my following message explaining why it's not possible to change the encoding:
http://bugs.python.org/issue19846#msg205675

Thanks Victor - I now agree that trying to guess another encoding is a bad idea, and that enabling surrogateescape for the standard streams under the C locale is a better way to go.

Since Viktor's alternative in bpo-19977 has been applied, should this issue be closed?

Also see http://bugs.python.org/issue28180 for a more recent proposal to tackle this by coercing the C locale to the C.UTF-8 locale

Follow-up: the PEP-538 (bpo-28180) and PEP-540 (bpo-29240) have been accepted and implemented in Python 3.7!

Navigation Menu

Repository navigation

Python 3 raises Unicode errors with the C locale #64045

Converts a string which is in the encoding used by GLib for filenames

into a UTF-8 string. Note that on Windows GLib uses UTF-8 for filenames;

on other platforms, this function indirectly depends on the current locale.

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Python 3 raises Unicode errors with the C locale #64045

Description

Activity

vstinner commented on Dec 8, 2013

ncoghlan commented on Dec 8, 2013

vstinner commented on Dec 8, 2013

pitrou commented on Dec 8, 2013

vstinner commented on Dec 8, 2013

serhiy-storchaka commented on Dec 8, 2013

larryhastings commented on Dec 8, 2013

pitrou commented on Dec 8, 2013

ncoghlan commented on Dec 8, 2013

vstinner commented on Dec 8, 2013

vstinner commented on Dec 8, 2013

pitrou commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

ncoghlan commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

ncoghlan commented on Dec 9, 2013

Sworddragon commented on Dec 9, 2013

serhiy-storchaka commented on Dec 9, 2013

serhiy-storchaka commented on Dec 9, 2013

malemburg commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

malemburg commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

Sworddragon commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

larryhastings commented on Dec 9, 2013

Sworddragon commented on Dec 9, 2013

vstinner commented on Dec 9, 2013

abadger commented on Dec 9, 2013

ncoghlan commented on Dec 9, 2013

loewis commented on Dec 9, 2013

pitrou commented on Dec 9, 2013

ncoghlan commented on Dec 9, 2013

loewis commented on Dec 10, 2013

vstinner commented on Dec 10, 2013

Converts a string which is in the encoding used by GLib for filenames

into a UTF-8 string. Note that on Windows GLib uses UTF-8 for filenames;

on other platforms, this function indirectly depends on the current locale.

abadger commented on Dec 10, 2013

vstinner commented on Dec 10, 2013

vstinner commented on Dec 10, 2013

abadger commented on Dec 10, 2013

Sworddragon commented on Dec 13, 2013

larryhastings commented on Dec 13, 2013

Sworddragon commented on Dec 13, 2013

ncoghlan commented on Dec 13, 2013

Sworddragon commented on Dec 13, 2013

vstinner commented on Dec 13, 2013

Sworddragon commented on Dec 13, 2013

abadger commented on Dec 13, 2013

vstinner commented on Dec 13, 2013

vstinner commented on Dec 13, 2013

ncoghlan commented on Dec 14, 2013

terryjreedy commented on Dec 7, 2014

ncoghlan commented on Dec 21, 2016

vstinner commented on Dec 18, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions