Skip to content

gh-91156: Fix encoding="locale" in UTF-8 mode #70056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
methane merged 5 commits into from
Apr 14, 2022
Merged

gh-91156: Fix encoding="locale" in UTF-8 mode #70056

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cba0836
bpo-47000: Fix `encoding="locale"` in UTF-8 mode
methane Apr 10, 2022
2e360d5
Add NEWS
methane Apr 10, 2022
da02718
Fix unused variable
methane Apr 11, 2022
713e9f4
Skip test_device_encoding
methane Apr 11, 2022
686490e
Update docs
methane Apr 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
bpo-47000: Fix encoding="locale" in UTF-8 mode
  • Loading branch information
@methane
methane committed Apr 10, 2022
commit cba083646baa7067640812ce8661370743275f89
6 changes: 4 additions & 2 deletions Lib/_pyio.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2021,7 +2021,9 @@ def __init__(self, buffer, encoding=None, errors=None, newline=None,
self._check_newline(newline)
encoding = text_encoding(encoding)

if encoding == "locale":
if encoding == "locale" and sys.platform == "win32":
# On Unix, os.device_encoding() returns "utf-8" instead of locale encoding
# in the UTF-8 mode. So we use os.device_encoding() only on Windows.
try:
encoding = os.device_encoding(buffer.fileno()) or "locale"
except (AttributeError, UnsupportedOperation):
Expand All @@ -2034,7 +2036,7 @@ def __init__(self, buffer, encoding=None, errors=None, newline=None,
# Importing locale may fail if Python is being built
encoding = "utf-8"
else:
encoding = locale.getpreferredencoding(False)
encoding = locale.getencoding()

if not isinstance(encoding, str):
raise ValueError("invalid encoding: %r" % encoding)
Expand Down
11 changes: 10 additions & 1 deletion Modules/_io/textio.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1061,6 +1061,7 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer,
_PyIO_State *state = NULL;
PyObject *res;
int r;
int use_locale_encoding = 0; // Use locale encoding even in UTF-8 mode.

self->ok = 0;
self->detached = 0;
Expand All @@ -1076,6 +1077,7 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer,
}
else if (strcmp(encoding, "locale") == 0) {
encoding = NULL;
use_locale_encoding = 1;
}

if (errors == Py_None) {
Expand Down Expand Up @@ -1113,6 +1115,11 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer,
self->encodefunc = NULL;
self->b2cratio = 0.0;

#ifdef MS_WINDOWS
// os.device_encoding() on Unix is the locale encoding or UTF-8
// according to UTF-8 Mode.
// Since UTF-8 mode shouldn't affect `encoding="locale"`, we call
// os.device_encoding() only on Windows.
if (encoding == NULL) {
/* Try os.device_encoding(fileno) */
PyObject *fileno;
Expand Down Expand Up @@ -1144,8 +1151,10 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer,
Py_CLEAR(self->encoding);
}
}
#endif

if (encoding == NULL && self->encoding == NULL) {
if (_PyRuntime.preconfig.utf8_mode) {
if (_PyRuntime.preconfig.utf8_mode && !use_locale_encoding) {
_Py_DECLARE_STR(utf_8, "utf-8");
self->encoding = Py_NewRef(&_Py_STR(utf_8));
}
Expand Down