Repository navigation
PEP 538: Unexpected locale behaviour on *BSD (including Mac OS X) #74857
Description
| BPO | 30672 |
|---|---|
| Nosy | @ronaldoussoren, @ncoghlan, @ned-deily |
| PRs | |
| Dependencies |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-06-15.09:23:53.311>
labels = ['3.7', 'type-bug', 'tests']
title = 'PEP 538: Unexpected locale behaviour on *BSD (including Mac OS X)'
updated_at = <Date 2020-06-29.10:36:19.078>
user = 'https://github.com/ncoghlan'bugs.python.org fields:
activity = <Date 2020-06-29.10:36:19.078>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2017-06-15.09:23:53.311>
creator = 'ncoghlan'
dependencies = ['30647', '34589']
files = []
hgrepos = []
issue_num = 30672
keywords = ['patch']
message_count = 13.0
messages = ['296076', '296239', '296248', '296250', '307784', '314646', '314648', '314652', '326710', '354503', '354994', '354995', '372505']
nosy_count = 3.0
nosy_names = ['ronaldoussoren', 'ncoghlan', 'ned.deily']
pr_nums = ['9257']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30672'
versions = ['Python 3.7']
Activity
ncoghlan commentedon Jun 15, 2017
To get the new PEP-538 tests passing on Mac OS X (see [1,2]), I ended up having to skip the following test scenarios:
However, I'm not sure whether that should be diagnosed as a pure testing problem, where we change the test's expectations to match the current behaviour, or a bug in the PEP-538 implementation, where we should be updating it to produce the behaviour that the tests were originally expecting.
[1] 4563099
[2] 7926516 )
ncoghlan commentedon Jun 17, 2017
For the case where POSIX is a distinct locale from the default C locale (rather than a simple alias), I'm leaning towards taking PEP-538 literally, and adjusting the affected test cases to assume that locale coercion *won't* happen for that case on Mac OS X.
I *think* we can check for the alias behaviourally by setting the "POSIX" locale and seeing if it reports itself back to us as "C", but if not, then I'll just add in a sys.platform check, and we can figure out a behavioural check later if the platform check turns out to cause problems.
ncoghlan commentedon Jun 18, 2017
It seems the "POSIX is not just an alias for the C locale" behaviour is inherited from *BSD, rather than being specific to Mac OS X: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%20custom/builds/12/steps/test/logs/stdio
[-]PEP 538: Unexpected locale behaviour on Mac OS X[/-][+]PEP 538: Unexpected locale behaviour on *BSD (including Mac OS X)[/+]ncoghlan commentedon Jun 18, 2017
Flagging bpo-30647 as a dependency, since the problem with breaking nl_langinfo will need to be fixed before these tests can be re-enabled.
ncoghlan commentedon Dec 7, 2017
As discussed in https://bugs.python.org/issue32238 and https://mail.python.org/pipermail/python-dev/2017-December/151105.html, I now think the right answer for the POSIX case is to ensure the legacy locale detection logic always treats that the same way as it does the C locale.
ncoghlan commentedon Mar 29, 2018
Ned, I'd forgotten about the part of this issue which amounts to "Check for 'POSIX' as a locale coercion trigger in addition to 'C', as not every platform aliases the former to the latter the way glibc does".
So while I don't think this is really a release blocker (as you have to explicitly request the POSIX locale to encounter the discrepancy, and I think fixing it in 3.7.1 would be OK), I *am* bumping it up to "critical", as I think it would be better to resolve this for 3.7.0b4 rather than letting it linger further.
ned-deily commentedon Mar 29, 2018
Thanks for the headsup, Nick. BTW, some of the discussion in languishing bpo-18378 might be relevant here.
ncoghlan commentedon Mar 29, 2018
The locale module has its own extra layer of oddities that I don't personally understand - bpo-29571 and bpo-20087 are another couple of issues along those lines.
ncoghlan commentedon Sep 30, 2018
Putting back to normal, as the difference between the C locale and the POSIX locale is that you never get the latter by default - you have to explicitly request it.
The underlying fix for this is in the PR for bpo-34589.
vstinner commentedon Oct 11, 2019
Since this issue has been created, I deeply reworked the Python Initialization with the PEP-587, and I made many changes related to locales and the UTF-8 Mode (PEP-540). What's the status of this issue?
6 remaining items
devdanzin commentedon Sep 21, 2025
This issue sounds valid and actionable, only requiring access to MacOS or *BSD.
pythongh-74857, PEP 538: Coerce POSIX locale to UTF-8 based locale
vstinner commentedon Sep 22, 2025
I wrote #139238 to coerce the POSIX locale to a UTF-8 based locale.
gh-74857, PEP 538: Coerce POSIX locale to UTF-8 based locale (#139238)
vstinner commentedon Sep 23, 2025
Python 3.15 now coerces also the POSIX locale.