Repository navigation
CODESET error on AMD64 FreeBSD 10.x Shared 3.x caused by the PEP 538 #74832
Description
| BPO | 30647 |
|---|---|
| Nosy | @ncoghlan, @vstinner, @ned-deily, @ezio-melotti, @koobs |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2017-06-29.14:52:41.930>
created_at = <Date 2017-06-13.09:01:31.882>
labels = ['type-bug', '3.7', 'expert-unicode']
title = 'CODESET error on AMD64 FreeBSD 10.x Shared 3.x caused by the PEP 538'
updated_at = <Date 2017-06-29.14:52:41.929>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2017-06-29.14:52:41.929>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2017-06-29.14:52:41.930>
closer = 'ncoghlan'
components = ['Unicode']
creation = <Date 2017-06-13.09:01:31.882>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30647
keywords = []
message_count = 9.0
messages = ['295870', '295873', '295925', '295926', '295928', '296079', '296807', '297272', '297273']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'vstinner', 'ned.deily', 'ezio.melotti', 'koobs', 'bapt']
pr_nums = ['2374']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30647'
versions = ['Python 3.7']
Activity
vstinner commentedon Jun 13, 2017
Regression caused by the commit 6ea4186, bpo-28180: Implementation for PEP-538.
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/412/steps/compile/logs/stdio
Python detected LC_CTYPE=C: LC_CTYPE coerced to UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behavior).
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ValueError: CODESET is not set or empty
Current thread 0x0000000802006400 (most recent call first):
Abort trap (core dumped)
vstinner commentedon Jun 13, 2017
On my FreeBSD 11 VM, I only have the "C" locale, not "UTF-8 C" locale:
[haypo@freebsd ~/prog/python/master]$ locale -a|grep ^C
C
But CPython still asks me to use a non existent locale (newlines added for readability):
[haypo@freebsd ~/prog/python/master]$ ./python
Python runtime initialized with LC_CTYPE=C (a locale with default ASCII encoding), which may cause Unicode compatibility problems. Using C.UTF-8, C.utf8, or UTF-8 (if available) as alternative Unicode-compatible locales is recommended.
bapt commentedon Jun 13, 2017
Per POSIX, the C locale is only expected to be ASCII. C.UTF-8 is a linux only thing (actually I thought it was a debian only thing, but maybe not).
I was thinking about creating a C.utf8 locale on FreeBSD but it is not that simple to do (still doable and an interesting idea).
Note that if it fails here, it is probably due also failing on other OS. At minimum: Dragonfly and Illumos for sure, maybe NetBSD and OpenBSD as well.
haypo, do not hesitate to ping me on irc as usual if you want to discuss the issue.
ned-deily commentedon Jun 13, 2017
macOS is also BSD-like with regard to locales: it also does not have any C.* locales other than plain C. See, for example, the discussion at bpo-18378.
bapt commentedon Jun 13, 2017
More details here:
C.UTF-8 is a glibc only thing: https://sourceware.org/glibc/wiki/Proposals/C.UTF-8 not even mainstream.
The closest thing to a C locale with unicode would be to set everything to locale C but LC_CTYPE which would be set to en_US.UTF-8.
The problem is if your data for ctype comes from CLDR they are different per locales. On FreeBSD, Dragonfly and Illumos, we have extected it so LC_CTYPE is the same on all locales.
ncoghlan commentedon Jun 15, 2017
Note that the coercion logic includes a runtime check to see if 'setlocale(LC_CTYPE, "<locale_name>")' succeeds. That's how we skip over the non-existent C.UTF-8 and C.utf8 to get to "LC_CTYPE=UTF-8" on Mac OS X and FreeBSD.
That appears to work (and really does work on Mac OS X as far as CPython's test suite is concerned), but on FreeBSD we subsequently get the CODESET failure when we try to call
nl_langinfolater in the interpreter startup process.Victor's suggestion, which seems reasonable to me, is that we could also add the
nl_langinfocall in the coercion logic, so that we never implicitly configure a locale setting that breaks nl_langinfo.That way, instead of the interpreter failing to start, we'd just skip the locale coercion logic in that case (and update the test suite's expectations accordingly).
ncoghlan commentedon Jun 25, 2017
Current status of the PR:
Accordingly, I've revised the tests as follows:
At the locale coercion level, I've added an extra check where we save the initial locale (i.e. before we change anything), and if setlocale() succeeds, but nl_langinfo(CODESET) fails, we do setlocale(LC_CTYPE, initial_locale) to try to get things back to their original state.
This seems to *mostly* work on FreeBSD, but doesn't quite get readline back to where it is by default, so test_non_ascii in test_readline fails with the error:
My two current guesses as to what may be going wrong there are:
I'm leaning towards the former, as if it was the latter, I'd expect to have already seen the same error *without* locale coercion.
ncoghlan commentedon Jun 29, 2017
New changeset 18974c3 by Nick Coghlan in branch 'master':
bpo-30647: Check nl_langinfo(CODESET) in locale coercion (GH-2374)
18974c3
ncoghlan commentedon Jun 29, 2017
I was able to fix the test_readline failure by restoring the locale based on the environment settings with
setlocale(LC_CTYPE, "")rather than the return value from a preceding call tosetlocale(LC_CTYPE, NULL).That means we can leave the runtime coercion checks enabled on *BSD systems, and if/when any given BSD variant adds working Linux-style C.UTF-8 or OS-X-style UTF-8 locales, we'll automatically start using them.