Repository navigation
locale.getdefaultlocale() fails on Mac OS X with default language set to English #62578
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-08-29.06:30:06.592>
created_at = <Date 2013-07-06.12:19:03.656>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'locale.getdefaultlocale() fails on Mac OS X with default language set to English'
updated_at = <Date 2019-08-29.06:30:06.586>
user = 'https://bugs.python.org/DmitryJemerov'bugs.python.org fields:
activity = <Date 2019-08-29.06:30:06.586>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-29.06:30:06.592>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2013-07-06.12:19:03.656>
creator = 'Dmitry.Jemerov'
dependencies = []
files = ['30807', '39384', '39385', '40014']
hgrepos = []
issue_num = 18378
keywords = ['patch', 'needs review']
message_count = 45.0
messages = ['192422', '192429', '192433', '192445', '192446', '192447', '192460', '192820', '192821', '192822', '192827', '209731', '214397', '214555', '214556', '214564', '215215', '239485', '239702', '243262', '247318', '247322', '247326', '247333', '247335', '247338', '247339', '247418', '263659', '268579', '278540', '285317', '285318', '285319', '285329', '285360', '285370', '296829', '333371', '347803', '347805', '350706', '350709', '350710', '350731']
nosy_count = 19.0
nosy_names = ['lemburg', 'loewis', 'barry-scott', 'ronaldoussoren', 'ncoghlan', 'mattheww', 'ned.deily', 'r.david.murray', 'Dmitry.Jemerov', 'larryv', 'serhiy.storchaka', 'wolma', 'Ilya.Kulakov', 'tsparber', 'karolyi', 'alexander.sturm', 'miss-islington', 'Tiger-222', 'rfrail3']
pr_nums = ['14736', '15569', '15570']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18378'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']
Activity
DmitryJemerov commentedon Jul 6, 2013
On Mac OS X 10.8 with the default language set to English (System Preferences | Language and Text), the default terminal application sets the LC_CTYPE environment variable to "UTF-8". If you run Python from the terminal and try to use locale.getdefaultlocate(), you get the following error:
> python Python 2.7.2 (default, Oct 11 2012, 20:14:37) [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import locale >>> locale.getdefaultlocale() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 496, in getdefaultlocale return _parse_localename(localename) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 428, in _parse_localename raise ValueError, 'unknown locale: %s' % localename ValueError: unknown locale: UTF-8(The stacktrace is from Python 2.7 but Python 3.3 suffers from the same problem.)
There are numerous workarounds for this problem (turning off the "Set locale environment variables on startup" option in the terminal settings, or adding "export LC_CTYPE=en_US.UTF8" to .bash_profile, selecting a language other than English in the Language & Text settings), but these require additional configuration from the user's side.
I think that the more useful behavior is for Python to handle this behavior of the system and not crash, even though it doesn't strictly comply to the POSIX standard.
The attached patch (against current Python 3.4 master branch) is one possible fix.
ronaldoussoren commentedon Jul 6, 2013
Strange, I have LANG=en_US.UTF-8 in my environment and no LC_CTYPE. A clean test account does have the same behavior as you are seeing.
ronaldoussoren commentedon Jul 6, 2013
The UTF-8 value seems suspect to me, but is actually supported by the system, changing it to a nonsense value results in failure in the C function setlocale.
As for the patch: I'd add this workaround only to the OSX platform (that is, test for sys.platform == 'darwin' before checking for UTF-8 as a value).
DmitryJemerov commentedon Jul 6, 2013
Judging from the results of Googling for the error message, I'm far from the only one seeing this problem.
What exactly would be the benefit of adding the code to check for the platform?
ronaldoussoren commentedon Jul 6, 2013
The test for darwin is needed because other platforms don't support "UTF-8" as a valid LC_CTYPE name, on a recent linux box:
(And just calling setlocale to check if the value is valid is not an option because that changes process-global state)
DmitryJemerov commentedon Jul 6, 2013
Why exactly does this matter? UTF-8 not being a valid LC_CTYPE value simply means that no one running Linux will ever have LC_CTYPE set to UTF-8, and the branch will never be hit.
OTOH, adding the check will make the code harder to test and simply larger (no code is always better than any non-zero amount of code).
DmitryJemerov commentedon Jul 6, 2013
A related issue (with a patch that touches the same locale parsing code) is http://bugs.python.org/issue5815
loewis commentedon Jul 10, 2013
Why do you need the "getdefaultlocale" function in the first place? I'd advise against using it, precisely because it can trigger problems like this one.
DmitryJemerov commentedon Jul 10, 2013
I personally don't, but the function is used by Sphinx, which is what I was trying to get to work when I ran into this problem.
bitdancer commentedon Jul 10, 2013
Regardless of the resolution here, the use of getdefaultlocale could be reported as a bug on the sphinx tacker.
loewis commentedon Jul 10, 2013
FWIW, I couldn't find any use of getdefaultlocale in any of the hg revisions (using hg grep) in
https://bitbucket.org/birkenfeld/sphinx/
Instead, it's (probably) docutils, which has this code:
I find that quite unfortunate, since locale.getpreferredencoding() would have don the right thing (IMO).
ronaldoussoren commentedon Jan 30, 2014
I just ran into this problem myself.
On fresh installs of OSX 10.9 LC_CTYPE is set to "UTF-8" (at least for english language users), and now sphinx won't work :-(
Is Dimitrys patch acceptable (either as is, or with my suggestion of checking for sys.platform == "darwin")?
ned-deily commentedon Mar 21, 2014
Ronald or Dmitry, can you elaborate under what conditions you start your login shell on 10.9? I cannot reproduce the behavior you observe. With 10.9 Terminal.app and the default language settings in System Preferences and with the default Terminal.app preferences, specifically Settings -> (Profile) -> Advanced -> Character encoding -> Unicode (UTF-8) and "Set LANG environment variable on startup" checked, login sessions have LANG=en_US.UTF-8 defined and LC_CTYPE is not defined at all. Are you sure that isn't begin created by a shell profile somewhere? (I can't check earlier OS X releases at the moment.) That said, I agree that, if OS X accepts "UTF-8" as a valid locale, the locale module should, too.
22 remaining items
malemburg commentedon Jan 12, 2017
I think Ronald's patch bpo-18378-2015-07-25-py36.txt with added darwin check would be the best way forward.
In the current form, it would allow using 'UTF-8' as locale string on all platforms - which is not such a good idea.
ncoghlan commentedon Jan 13, 2017
SSH environment forwarding will propagate this "LC_CTYPE=UTF-8" setting from Mac OS X clients to Linux servers.
At present, that breaks in multiple ways, as CPython will interpret it as being the "C" locale (since Linux servers don't offer a "UTF-8" locale, even when they do offer "C.UTF-8")
PEPs 538 and 540 aim to help CPython itself to deal with that case, but that won't be sufficient to help code that tries to pass the nominal LC_CTYPE setting to the locale module.
Accepting "UTF-8" and interpreting it as functionally equivalent to C.UTF-8 will mean that this setting will at least work as desired on servers that offer C.UTF-8.
malemburg commentedon Jan 13, 2017
On 13.01.2017 04:47, Nick Coghlan wrote:
I don't think that's within the scope of this patch. "UTF-8" is not
a valid locale setting on Linux and so Python should not allow
passing this through the locale normalization process on Linux.
Please also note that SSH does not forward arbitrary env vars.
Only a select few are forwarded and all others have to be
configured. The locale vars are not among the default ones
(see the ssh man page for details).
Aisde: While looking into this I found that the locale module
aliases C.UTF-8 to en_US.UTF-8. This was added as part of
issue bpo-20076 and originates from the X.org locale.alias file.
Time machine and all that :-)
mattheww commentedon Jun 25, 2017
That alias (C.UTF-8 to en_US.UTF-8) is surely a bug in itself nowadays. I've filed bpo-30755 .
rfrail3 commentedon Jan 10, 2019
I still have this issue on MacOS Mojave 10.14
Python 3.7.2 (default, Dec 27 2018, 07:35:06) [Clang 10.0.0 (clang-1000.11.45.5)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import locale >>> locale.getdefaultlocale() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/Cellar/python/3.7.2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/locale.py", line 568, in getdefaultlocale return _parse_localename(localename) File "/usr/local/Cellar/python/3.7.2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/locale.py", line 495, in _parse_localename raise ValueError('unknown locale: %s' % localename) ValueError: unknown locale: UTF-8 >>>ronaldoussoren commentedon Jul 13, 2019
Interestingly enough I get this error even when I unset the relevant environment variables. For some reason LC_CTYPE is reset when I start the interpreter, even if it is set to something else. This means the usual way of working around this problem no longer works.
I'll create a pull request with an up-to-date version of my latest patch for further discussion.
BTW. I'm testing with the current tip of the tree, but 3.7.3 fails in the same way.
ronaldoussoren commentedon Jul 13, 2019
As promised there is now a pull request.
I'd love a review (and a change to approve the pull request when reviewers are happy, I'm trying to get back into actively contributing).
---
I now understand why locale.getdefaultlocale() fails even when LC_CTYPE is not set: pylifecycle sets LC_CTYPE to UTF-8 in the UTF-8 coercion code.
ned-deily commentedon Aug 29, 2019
New changeset b0caf32 by Ned Deily (Ronald Oussoren) in branch 'master':
bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736)
b0caf32
miss-islington commentedon Aug 29, 2019
New changeset 554143e by Miss Islington (bot) in branch '3.7':
bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736)
554143e
miss-islington commentedon Aug 29, 2019
New changeset e471a54 by Miss Islington (bot) in branch '3.8':
bpo-18378: Recognize "UTF-8" as a valid name in locale._parse_localename (GH-14736)
e471a54
ned-deily commentedon Aug 29, 2019
Ronald's PR 14738 LGTM. I merged it to master and backported for 3.8.0b4 and 3.7.5. Thanks, everyone!