Repository navigation
Choose the filesystem encoding before Python initialization (add _PyCoreConfig.filesystem_encoding) #78704
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-08-29.22:51:48.219>
created_at = <Date 2018-08-27.23:44:03.090>
labels = ['interpreter-core', '3.8']
title = 'Choose the filesystem encoding before Python initialization (add _PyCoreConfig.filesystem_encoding)'
updated_at = <Date 2018-11-28.11:42:43.707>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2018-11-28.11:42:43.707>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-29.22:51:48.219>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-08-27.23:44:03.090>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 34523
keywords = ['patch']
message_count = 16.0
messages = ['324206', '324233', '324243', '324249', '324318', '324319', '324339', '324347', '324351', '324353', '324354', '328900', '330311', '330317', '330586', '330591']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'vstinner', 'methane', 'eric.snow']
pr_nums = ['8963', '8991', '8995', '8998', '9003', '9005', '10232', '10672', '10673', '10759', '10761']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34523'
versions = ['Python 3.8']
Activity
vstinner commentedon Aug 27, 2018
Currently, the Python filesystem encoding is get in the middle of the Python initialization. initfsencoding() is only called when the import machinery is ready. Technically, only getting the Python codec name requires a working import machinery.
We can get the locale encoding even before Py_Initialize(), but update the encoding to the Python codec name (ex: replace "ANSI_X3.4-1968" with "ascii") once the import machinery is ready.
With my change, for an application embedding Python, the filesystem encoding can now be forced using _PyCoreConfig.filesystem_encoding. If it's set, Python will use it (and ignore the locale encoding).
Attached PR implements this idea and adds _PyCoreConfig.filesystem_encoding.
The change move all code to select encoding and error handler inside config_init_fs_encoding(), rather than relying on Py_FileSystemDefaultEncoding constant (its default value is set using #ifdef) and initfsencoding() (to get the locale encoding at runtime).
With this change, it becomes more obvious than the interpreter core configuration is mutable. initfsencoding() modify the encoding/errors during Python initialization, and sys._enablelegacywindowsfsencoding() even modify it at runtime. Previously, I wanted to expose the full core_config at the Python level, as something like sys.flags. But since it's mutable, I'm not longer sure that it's a good idea. To *get* the filesystem encoding/errors, they are already sys.getfilesystemencoding() and sys.getfilesystemencodeerrors(). Modifying the filesystem encoding at runtime is a very bad idea tried in the past and it caused many issues.
For the long term, I would like to remove Py_HasFileSystemDefaultEncoding, since it's really an implementation detail and there is no need to expose it. More generally, I don't see the purpose of exposing directly the encoding and error handler at the C level: Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors. These symbols are constant constant, they cannot be modified when Python is embedded. But well, it's just a remark, technically, these 3 symbols don't cause any kind of trouble.
Commit message:
---
bpo-34485: Add _PyCoreConfig.filesystem_encoding
_PyCoreConfig_Read() is now responsible to choose the filesystem
encoding and error handler. Using Py_Main(), the encoding is now
selected even before calling Py_Initialize().
_PyCoreConfig.filesystem_encoding is now the reference instead of
Py_FileSystemDefaultEncoding for the Python filesystem encoding.
Changes:
system encoding. Coerce temporarly the locale or set temporarly the
LC_CTYPE locale to the user preferred locale.
now use the interpreter configuration rather than
Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors
global configuration variables.
private functions to only modify Py_FileSystemDefaultEncoding and
Py_FileSystemDefaultEncodeErrors in coreconfig.c.
---
[-]Choose the filesystem encoding earlier in Python initialization (add _PyCoreConfig.filesystem_encoding)[/-][+]Choose the filesystem encoding before Python initialization (add _PyCoreConfig.filesystem_encoding)[/+]vstinner commentedon Aug 28, 2018
The compilation failed on my PR when running PCbuild\amd64\_freeze_importlib.exe:
ValueError: only 'strict' and 'surrogateescape' error handlers are supported, not 'surrogatepass'
The error comes from locale_error_handler(). Before my change, PyUnicode_EncodeFSDefault() and PyUnicode_DecodeFSDefault() used Py_FileSystemDefaultEncodeErrors which is initialized to "surrogateescape", but only set to "surrogatepass" by initfsencoding().
With my change, the error handler is directly set to "surrogatepass", but currently, unicode_encode_locale() and unicode_decode_locale() don't accept this error handler.
vstinner commentedon Aug 28, 2018
This issue should now be fixed.
vstinner commentedon Aug 28, 2018
See also bpo-34485 which allows to select (using _PyCoreConfig) the encoding and error handlers of standard streams like sys.stdout.
vstinner commentedon Aug 29, 2018
New changeset b2457ef by Victor Stinner in branch 'master':
bpo-34523: Add _PyCoreConfig.filesystem_encoding (GH-8963)
b2457ef
vstinner commentedon Aug 29, 2018
New changeset 70fead2 by Victor Stinner in branch 'master':
bpo-34523: Fix config_init_fs_encoding() (GH-8991)
70fead2
vstinner commentedon Aug 29, 2018
New changeset c5989cd by Victor Stinner in branch 'master':
bpo-34523: Py_DecodeLocale() use UTF-8 on Windows (GH-8998)
c5989cd
vstinner commentedon Aug 29, 2018
New changeset 3d4226a by Victor Stinner in branch 'master':
bpo-34523: Support surrogatepass in locale codecs (GH-8995)
3d4226a
vstinner commentedon Aug 29, 2018
New changeset de42755 by Victor Stinner in branch 'master':
bpo-34523: Py_FileSystemDefaultEncoding NULL by default (GH-9003)
de42755
vstinner commentedon Aug 29, 2018
New changeset fbca908 by Victor Stinner in branch 'master':
bpo-34523: Use _PyCoreConfig instead of globals (GH-9005)
fbca908
vstinner commentedon Aug 29, 2018
It was much more complicated than expected to implement this idea, but it seems like it's now working! I close the issue.
vstinner commentedon Oct 30, 2018
New changeset 905f1ac by Victor Stinner in branch 'master':
bpo-34523: Fix config_init_fs_encoding() for ASCII (GH-10232)
905f1ac
vstinner commentedon Nov 23, 2018
New changeset 353933e by Victor Stinner in branch 'master':
bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672)
353933e
vstinner commentedon Nov 23, 2018
New changeset f6e323c by Victor Stinner in branch '3.7':
bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672) (GH-10673)
f6e323c
vstinner commentedon Nov 28, 2018
New changeset bde9d6b by Victor Stinner in branch 'master':
bpo-34523, bpo-35322: Fix unicode_encode_locale() (GH-10759)
bde9d6b
vstinner commentedon Nov 28, 2018
New changeset 85ab974 by Victor Stinner in branch '3.7':
bpo-34523, bpo-35322: Fix unicode_encode_locale() (GH-10759) (GH-10761)
85ab974