Repository navigation
Implementation of the PEP 538: coerce C locale to C.utf-8 #72367
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2018-03-29.01:23:38.218>
created_at = <Date 2016-09-16.11:17:02.590>
labels = ['type-bug', '3.7', 'expert-unicode']
title = 'Implementation of the PEP 538: coerce C locale to C.utf-8'
updated_at = <Date 2020-03-22.16:25:00.648>
user = 'https://bugs.python.org/JanNiklasHasse'bugs.python.org fields:
activity = <Date 2020-03-22.16:25:00.648>
actor = 'mcepl'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2018-03-29.01:23:38.218>
closer = 'ncoghlan'
components = ['Unicode']
creation = <Date 2016-09-16.11:17:02.590>
creator = 'Jan Niklas Hasse'
dependencies = ['30565', '30635', '30647']
files = ['45907', '45951', '46059', '46121', '46190', '46205', '46329', '48991']
hgrepos = []
issue_num = 28180
keywords = ['patch']
message_count = 89.0
messages = ['276693', '276694', '276707', '276709', '276722', '276729', '277273', '277274', '282964', '282965', '282970', '282971', '282972', '282977', '282978', '282984', '283244', '283408', '283409', '283469', '283471', '283482', '283495', '283515', '283543', '283732', '284150', '284170', '284176', '284537', '284605', '284620', '284621', '284631', '284641', '284647', '284697', '284716', '284718', '284719', '284720', '284722', '284725', '284729', '284736', '284742', '284747', '284764', '284782', '284794', '284795', '284799', '284882', '284884', '284886', '284887', '284900', '284908', '284943', '284952', '285735', '286001', '289002', '289534', '295121', '295683', '295688', '295698', '295710', '295713', '295722', '295871', '295872', '295875', '295885', '295913', '295914', '296064', '296075', '296077', '305850', '306108', '314627', '364740', '364760', '364767', '364770', '364804', '364810']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'ned.deily', 'mcepl', 'ezio.melotti', 'r.david.murray', 'methane', 'akira', 'Sworddragon', 'xdegaye', 'yan12125', 'abarry', 'Jan Niklas Hasse']
pr_nums = ['659', '2130', '2155', '2208', '4334']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28180'
versions = ['Python 3.7']
Activity
JanNiklasHasse commentedon Sep 16, 2016
Working with Docker I often end up with an environment where the locale isn't correctly set. In these cases it would be great if sys.getfilesystemencoding() could default to 'utf-8' instead of 'ascii', as it's the encoding of the future and ascii is a subset of it anyway.
Related: http://bugs.python.org/issue19846
Vgr255 commentedon Sep 16, 2016
This is a duplicate of bpo-27781.
vstinner commentedon Sep 16, 2016
bpo-27781 is specific to Windows. I'm not sure that it's the base in this issue. So I reopen the issue.
@jan Niklas Hasse: What is your OS?
I proposed to add "-X utf8" command line option for UNIX to force utf8 encoding. Would it work for you?
JanNiklasHasse commentedon Sep 16, 2016
Unfortunately no, as this would mean I'll have to change all my python invocations in my scripts and it wouldn't work for executable files with
#!/usr/bin/env python3
would it?
bitdancer commentedon Sep 16, 2016
I thought we "fixed" this by using surrogate escape when the locale was ASCII? We certainly have discussed changing the default and posix and so far have decided not to (someday that will change...is this someday already?)
vstinner commentedon Sep 16, 2016
Not yet :-)
JanNiklasHasse commentedon Sep 23, 2016
Why not?
methane commentedon Sep 23, 2016
I want locale free Python which behaves like on C.UTF-8 locale.
(stdio encoding, preferred encoding, weekday in _strptime._strptime,
and more maybe)
But Python 3.6 is feature freeze already >_<;;
ncoghlan commentedon Dec 12, 2016
I think we're genuinely getting to the point now where the majority of "LANG=C" cases are misconfigurations rather than intended behaviour. We're also to the point where:
So I think for Python 3.7 it makes sense to do the following on other *nix systems:
I do think we actually want to *change* the C level locale in the process though, as otherwise we can expect to see weird interactions where CPython and extension modules disagree about the default text encoding.
ncoghlan commentedon Dec 12, 2016
Note also that if we say we're going to do this for 3.7, *and* go ahead and implement it, then distros may be more inclined to incorporate the same behavioural changes into distro-provided releases of 3.6, providing real world testing of the concept before we make it the default behaviour.
80 remaining items
mcepl commentedon Mar 21, 2020
Of course, I know that, but I just didn’t want to throw all my effort away, when I spent some hours on making it. And I guess, there may be somebody else who cares for 3.4 (ehm, RHEL-7 has 3.3, doesn’t it?).
vstinner commentedon Mar 21, 2020
RHEL 7.7 and RHEL 8 provides Python 3.6. PEP-538 was implemented in Python 3.7. PEP-538 feature was backported in RHEL 7.7 and RHEL 8 Python 3.6.
ncoghlan commentedon Mar 22, 2020
The test cases for locale coercion *not* triggering still assume that bpo-19977, using surrogateescape on the standard streams in the POSIX locale, has been implemented (since that was implemented in Python 3.5).
Hence the various test cases complaining that they found "ascii:strict" (Py 3.4 behaviour without bpo-19977) where they expected "ascii:surrogateescape" (the Py 3.5+ behaviour *with* bpo-19977).
To get a PEP-538 backport to work as intended on 3.4, you'd need to backport that earlier IO stream error handling change as well.
mcepl commentedon Mar 22, 2020
Thank you very much for the hint. Do I have to include the patch for bpo-19977 only (that would be easy), or also all twelve PRs for bpo-29240 (that would probably broke my will to do it)?
calestyo commentedon May 29, 2022
Not sure whether this is the appropriate place to mention/ask... but it seems that with this chance it's impossible to get the original environment Python was invoked with, or is there?
This in turn may however be important for e.g. wrapper programs, which would want to invoke other programs with the exact original locale they were invoked themselves.
methane commentedon May 29, 2022
You can set
PYTHONCOERCECLOCALE=0to disable coercion.calestyo commentedon May 29, 2022
Yes, but that just shifts the problem. Then one wouldn't know whether
PYTHONCOERCECLOCALEwas meant to be part of the real original environment, or whether it was just there to not mess around with that.methane commentedon May 29, 2022
What is the real use case for it?
locale affects many programs, but
PYTHONCOERCELOCALEaffects only Python. I don't think it is real problem for most users.If it is really your problem, you can disable the coercion by default by building your Python with
--with-c-locale-coercionoption.calestyo commentedon May 29, 2022
Take any program that shall be written in Python and which works just as a wrapper around some other program, with the specific intention that the environment is passed on exactly to that program. Just as an example, a wrapper-program that adds numbers to lines... or maybe counts characters in the stdout of the program.
The program the wrapper calls may even be another Python program... at that point it should become quite clear that it's apparently not possible to get that behaviour.
Either one does nothing respectively e.g. explicitly wants to run in a
Clocal environment and doesenv -i LC_CTYPE=C python3 wrapper.py commandor so... then Python would do the coercion andcommandwouldn't see the original environment.Same if one does
env -i python3in which case, as per POSIX,commandshould fall back to the “implementation-defined default locale”, but Python would override this toC.UTF-8.If one disables that behaviour with
PYTHONCOERCELOCALEone cannot know whether this was meant (just) for the wrapper script in order to not do the coercion, or whether it was meant forcommand.Or take something where
commandis something executed viassh,LC_CTYPEis often configured to be forwarded via that, so one may not wish to have that set by Python, OTOH, one may again execute a remote Python command and may not want to havePYTHONCOERCELOCALEin the environment either, unless it's really meant for the remote.calestyo commentedon May 29, 2022
Oh and, as unlikely as it is, there is no guarantee for that, at all. Changing the built-options doesn't really help either, as no real-world Python will have that.
The problem here really is that one seems to have no way to either get/restore the true original local (directly or indirectly).
methane commentedon May 29, 2022
I understand theorical problem. What I am asking for is how this issue affects to users in real world.
Please give us more concrete use cases.
And this issue is not good place for this discussion. Please move to discuss.python.org.
calestyo commentedon May 29, 2022
Well I wouldn't call it a theoretical(only) problem. There is a clear use-case described along with several examples (i.e. any program who wants to serve as a transparent wrapper)... which I think speaks for itself.
At least I wouldn't know how to "prove" validity of the use case any further - that feels a bit as if division by
834756387465723672348394923would fail any I'd need to prove why on earth anyone could ever want to divide by exactly that.However, I don't think any further discussion is really needed, at least not from my side,... I've simply used C now which is as good for me and does the job well enough.
As said, in the original post, I've just noted that this makes it impossible to get the original environment (except when using hacks)... and wanted to bring that to attention.
Whether Python wants to support that is up to that. A simple solution would perhaps be to provide something like a
os.original_environmentor so, which would solve the same problem should any other modifications done in the future.ncoghlan commentedon Jun 3, 2022
Wrapper program or no, setting 7-bit ASCII as the text encoding was deemed an unsupported system configuration error in 2018 when this PEP was accepted and implemented, and is still seen that way now. Given the availability of UTF-8 as an alternative, there's no good reason to run English-only systems any more. Hence the platform compatibility guidelines in PEP 11 being updated when these encoding related PEPs were accepted.
That said, setting
PYTHON_PRECOERCION_LOCALEin the environment when the coercion triggers would be straightforward, so I'd personally be +0 on providing that as a debugging aid, even though I'd advise wrapper script authors to resist the temptation to revert the coercion (all the wrapper scripts in the standard library intentionally leave the coerced locale in place)ncoghlan commentedon Jun 3, 2022
Note: any such a feature request should be filed as a new issue, and while I'd be happy to review a PR for such an addition, I wouldn't write it myself.