ncoghlan · Sep 13, 2018 · Sep 20, 2018 · Sep 20, 2018 · Sep 20, 2018 · Sep 20, 2018
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
@@ -806,23 +806,23 @@ conflict.
 
 .. envvar:: PYTHONCOERCECLOCALE
 
-   If set to the value ``0``, causes the main Python command line application
-   to skip coercing the legacy ASCII-based C and POSIX locales to a more
-   capable UTF-8 based alternative.
-
-   If this variable is *not* set (or is set to a value other than ``0``), the
-   ``LC_ALL`` locale override environment variable is also not set, and the
-   current locale reported for the ``LC_CTYPE`` category is either the default
-   ``C`` locale, or else the explicitly ASCII-based ``POSIX`` locale, then the
-   Python CLI will attempt to configure the following locales for the
-   ``LC_CTYPE`` category in the order listed before loading the interpreter
-   runtime:
+   If set to the value ``0`` (encoded as ASCII bytes), causes the main Python
+   command line application to skip coercing the legacy ASCII-based C and
+   POSIX locales to a more capable UTF-8 based alternative. Locale coercion
+   is also skipped if the ``LC_ALL`` locale override environment variable is
+   set (as setting ``LC_CTYPE`` will have no effect in that case).
+
+   Otherwise, when the current locale reported for the ``LC_CTYPE`` category is
+   either the default ``C`` locale, or else the explicitly ASCII-based ``POSIX``
+   locale, then the Python CLI will attempt to configure the following locales
+   for the ``LC_CTYPE`` category in the order listed before beginning to
+   initalise the interpreter runtime:
 
    * ``C.UTF-8``
    * ``C.utf8``
    * ``UTF-8``
 
-   If setting one of these locale categories succeeds, then the ``LC_CTYPE``
+   If enabling one of these locale categories succeeds, then the ``LC_CTYPE``
    environment variable will also be set accordingly in the current process
    environment before the Python runtime is initialized. This ensures that in
    addition to being seen by both the interpreter itself and other locale-aware
@@ -842,18 +842,64 @@ conflict.
    For debugging purposes, setting ``PYTHONCOERCECLOCALE=warn`` will cause
    Python to emit warning messages on ``stderr`` if either the locale coercion
    activates, or else if a locale that *would* have triggered coercion is
-   still active when the Python runtime is initialized.
+   still active when the Python runtime is initialized. (Note: as with the
+   locale coercion disabling marker, both the environment variable name and
+   the ``warn`` value must be encoded as ASCII text in order to have the
+   described effect)
 
    Also note that even when locale coercion is disabled, or when it fails to
-   find a suitable target locale, :envvar:`PYTHONUTF8` will still activate by
-   default in legacy ASCII-based locales. Both features must be disabled in
-   order to force the interpreter to use ``ASCII`` instead of ``UTF-8`` for
-   system interfaces.
+   find a suitable target locale, UTF-8 mode (as described under
+   :envvar:`PYTHONUTF8`) will still activate by default in these legacy
+   ASCII-based locales. Both features must be explicitly disabled in order to
+   force the interpreter to use ``ASCII`` instead of ``UTF-8`` for system
+   interfaces.
 
    Availability: \*nix
 
+   .. note::
+
+      While UTF-8 mode is able to handle many of the issues that otherwise
+      arise, running in the legacy C locale is `not formally supported`_, as it
+      almost inevitably leads to text encoding and decoding problems at either
+      operating system interfaces or else between the interpreter runtime and
+      locale-aware extension modules. As such, the ability to disable locale
+      coercion or emit warnings when it occurs is provided solely as a debugging
+      aid when behavioural differences are encountered across different platforms
+      (where some platforms may not define a suitable coercion target), or
+      embedding applications (where some applications, including older versions
+      of CPython, will manage the locale differently from the way the CPython
+      3.7+ CLI does).
+
+   .. _not formally supported: https://www.python.org/dev/peps/pep-0011/#legacy-c-locale
+
+   .. note::
+
+      As a locale configuration variable, ``PYTHONCOERCECLOCALE`` is processed
+      in a way that is similar to the C level locale configuration variables
+      (``LANG``, ``LC_CTYPE``, and ``LC_ALL``): to have any effect, both the
+      key and the value must be encoded as ASCII bytes, and the setting is
+      processed even when :option:`-E` or :option:`-I` option is passed on the
+      command line.
+
    .. versionadded:: 3.7
-      See :pep:`538` for more details.
+      See :pep:`538` for more details. Note that the 3.7 implementation differs
+      from the PEP in a few key aspects due to some unintended interactions
+      with the :pep:`540` implementation: 1) ``LC_ALL=C`` must be used to
+      disable locale coercion when the :option:`-E` or :option:`-I` option is
+      specified (``PYTHONCOERCECLOCALE=0`` won't work); 2) calling
+      :c:func:`Py_Initialize` in an embedding application will trigger locale
+      coercion when running in the C locale;  2) calling :c:func:`Py_Main` in
+      an embedding application will trigger locale coercion when running in the
+      C locale.
+
+   .. versionchanged:: 3.8
+      Locale coercion is now triggered by the POSIX locale, even on platforms
+      where that isn't a simple alias for the C locale.
+      The locale coercion implementation has also been brought back into line
+      with the approach described in :pep:`538`: calling :c:func:`Py_Initialize`
+      and :c:func:`Py_Main` will never implicitly trigger locale coercion, and
+      ``PYTHONCOERCECLOCALE=0`` and ``PYTHONCOERCECLOCALE=warn`` take effect
+      even when the :option:`-E` or :option:`-I` option is specified.
 
 
 .. envvar:: PYTHONDEVMODE

diff --git a/Include/coreconfig.h b/Include/coreconfig.h
@@ -63,8 +63,7 @@ typedef struct {
     int show_alloc_count;   /* -X showalloccount */
     int dump_refs;          /* PYTHONDUMPREFS */
     int malloc_stats;       /* PYTHONMALLOCSTATS */
-    int coerce_c_locale;    /* PYTHONCOERCECLOCALE, -1 means unknown */
-    int coerce_c_locale_warn; /* PYTHONCOERCECLOCALE=warn */
+    int warn_on_c_locale;   /* PYTHONCOERCECLOCALE=warn */
 
     /* Python filesystem encoding and error handler:
        sys.getfilesystemencoding() and sys.getfilesystemencodeerrors().
@@ -314,7 +313,7 @@ typedef struct {
         .use_hash_seed = -1, \
         .faulthandler = -1, \
         .tracemalloc = -1, \
-        .coerce_c_locale = -1, \
+        .warn_on_c_locale = -1, \
         .utf8_mode = -1, \
         .argc = -1, \
         .nmodule_search_path = -1, \

diff --git a/Include/pylifecycle.h b/Include/pylifecycle.h
@@ -175,7 +175,8 @@ PyAPI_FUNC(int) _PyOS_URandomNonblock(void *buffer, Py_ssize_t size);
 
 /* Legacy locale support */
 #ifndef Py_LIMITED_API
-PyAPI_FUNC(void) _Py_CoerceLegacyLocale(int warn);
+PyAPI_FUNC(int) _Py_CoerceLegacyLocale(const char **coercion_target,
+                                       const char **coercion_warning);
 PyAPI_FUNC(int) _Py_LegacyLocaleDetected(void);
 PyAPI_FUNC(char *) _Py_SetLocaleFromEnv(int category);
 #endif

diff --git a/Lib/test/test_c_locale_coercion.py b/Lib/test/test_c_locale_coercion.py
@@ -15,7 +15,7 @@
 )
 
 # Set the list of ways we expect to be able to ask for the "C" locale
-EXPECTED_C_LOCALE_EQUIVALENTS = ["C", "invalid.ascii"]
+EXPECTED_C_LOCALE_EQUIVALENTS = ["C", "invalid.ascii", "POSIX"]
 
 # Set our expectation for the default encoding used in the C locale
 # for the filesystem encoding and the standard streams
@@ -31,12 +31,6 @@
         # Android defaults to using UTF-8 for all system interfaces
         EXPECTED_C_LOCALE_STREAM_ENCODING = "utf-8"
         EXPECTED_C_LOCALE_FS_ENCODING = "utf-8"
-    else:
-        # Linux distros typically alias the POSIX locale directly to the C
-        # locale.
-        # TODO: Once https://bugs.python.org/issue30672 is addressed, we'll be
-        #       able to check this case unconditionally
-        EXPECTED_C_LOCALE_EQUIVALENTS.append("POSIX")
 elif sys.platform.startswith("aix"):
     # AIX uses iso8859-1 in the C locale, other *nix platforms use ASCII
     EXPECTED_C_LOCALE_STREAM_ENCODING = "iso8859-1"
@@ -53,7 +47,6 @@
 # Note that the above expectations are still wrong in some cases, such as:
 # * Windows when PYTHONLEGACYWINDOWSFSENCODING is set
 # * Any platform other than AIX that uses latin-1 in the C locale
-# * Any Linux distro where POSIX isn't a simple alias for the C locale
 # * Any Linux distro where the default locale is something other than "C"
 #
 # Options for dealing with this:
@@ -86,7 +79,7 @@ def _set_locale_in_subprocess(locale_name):
         # If there's no valid CODESET, we expect coercion to be skipped
         cmd_fmt += "; import sys; sys.exit(not locale.nl_langinfo(locale.CODESET))"
     cmd = cmd_fmt.format(locale_name)
-    result, py_cmd = run_python_until_end("-c", cmd, PYTHONCOERCECLOCALE='')
+    result, py_cmd = run_python_until_end("-c", cmd, __isolated=True)
     return result.rc == 0
 
 
@@ -152,7 +145,7 @@ def get_child_details(cls, env_vars):
         """
         result, py_cmd = run_python_until_end(
             "-X", "utf8=0", "-c", cls.CHILD_PROCESS_SCRIPT,
-            **env_vars
+            __isolated=True, **env_vars
         )
         if not result.rc == 0:
             result.fail(py_cmd)
@@ -166,15 +159,15 @@ def get_child_details(cls, env_vars):
 
 # Details of the shared library warning emitted at runtime
 LEGACY_LOCALE_WARNING = (
-    "Python runtime initialized with LC_CTYPE=C (a locale with default ASCII "
-    "encoding), which may cause Unicode compatibility problems. Using C.UTF-8, "
+    "Python runtime initialized with a legacy locale that uses a default ASCII "
+    "encoding, which may cause Unicode compatibility problems. Using C.UTF-8, "
     "C.utf8, or UTF-8 (if available) as alternative Unicode-compatible "
     "locales is recommended."
 )
 
 # Details of the CLI locale coercion warning emitted at runtime
 CLI_COERCION_WARNING_FMT = (
-    "Python detected LC_CTYPE=C: LC_CTYPE coerced to {} (set another locale "
+    "Python detected a legacy locale: LC_CTYPE coerced to {} (set LC_ALL "
     "or PYTHONCOERCECLOCALE=0 to disable this locale coercion behavior)."
 )
 
@@ -257,7 +250,6 @@ def test_external_target_locale_configuration(self):
             "LANG": "",
             "LC_CTYPE": "",
             "LC_ALL": "",
-            "PYTHONCOERCECLOCALE": "",
         }
         for env_var in ("LANG", "LC_CTYPE"):
             for locale_to_set in AVAILABLE_TARGETS:

diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py
@@ -277,8 +277,7 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase):
         'filesystem_errors': None,
 
         'utf8_mode': 0,
-        'coerce_c_locale': 0,
-        'coerce_c_locale_warn': 0,
+        'warn_on_c_locale': 0,
 
         'pycache_prefix': NULL_STR,
         'program_name': './_testembed',

diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-09-20-21-36-57.bpo-34589.9zERCD.rst b/Misc/NEWS.d/next/Core and Builtins/2018-09-20-21-36-57.bpo-34589.9zERCD.rst
@@ -0,0 +1,7 @@
+The locale coercion implementation has been adjusted back to being closer to
+the design documented in PEP 538. This means locale coercion is once again
+solely the responsibility of the application embedding the CPython runtime
+(in CPython's case, the CLI wrapper), with the runtime only providing
+support to detect legacy locales, actually apply the changes needed to
+coerce the locale to a UTF-8 based one, and to emit warnings when requested
+via PYTHONCOERCECLOCALE=warn.
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
@@ -4701,10 +4701,8 @@ get_coreconfig(PyObject *self, PyObject *Py_UNUSED(args))
              PyLong_FromLong(config->dump_refs));
     SET_ITEM("malloc_stats",
              PyLong_FromLong(config->malloc_stats));
-    SET_ITEM("coerce_c_locale",
-             PyLong_FromLong(config->coerce_c_locale));
-    SET_ITEM("coerce_c_locale_warn",
-             PyLong_FromLong(config->coerce_c_locale_warn));
+    SET_ITEM("warn_on_c_locale",
+             PyLong_FromLong(config->warn_on_c_locale));
     SET_ITEM("filesystem_encoding",
              FROM_STRING(config->filesystem_encoding));
     SET_ITEM("filesystem_errors",

diff --git a/Modules/main.c b/Modules/main.c
@@ -318,6 +318,10 @@ typedef struct {
     char **bytes_argv;
     wchar_t **wchar_argv;
 
+    /* Allow the embedding app to emit a locale coercion message */
+    const char *locale_coercion_warning;
+    const char *locale_coercion_target;
+
     /* Exit status or "exit code": result of pymain_main() */
     int status;
     /* Error message if a function failed */
@@ -1292,7 +1296,6 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config,
 #endif
     _PyCoreConfig save_config = _PyCoreConfig_INIT;
     int res = -1;
-    int locale_coerced = 0;
     int loops = 0;
 
     if (_PyCoreConfig_Copy(&save_config, config) < 0) {
@@ -1303,6 +1306,9 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config,
     /* Set LC_CTYPE to the user preferred locale */
     _Py_SetLocaleFromEnv(LC_CTYPE);
 
+    /* TODO: With locale coercion moved back to _PyUnix_Main, this can
+     * potentially be simplified now that it only needs to handle UTF-8 mode
+     */
     while (1) {
         int utf8_mode = config->utf8_mode;
         int encoding_changed = 0;
@@ -1332,22 +1338,6 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config,
             goto done;
         }
 
-        /* The legacy C locale assumes ASCII as the default text encoding, which
-         * causes problems not only for the CPython runtime, but also other
-         * components like GNU readline.
-         *
-         * Accordingly, when the CLI detects it, it attempts to coerce it to a
-         * more capable UTF-8 based alternative.
-         *
-         * See the documentation of the PYTHONCOERCECLOCALE setting for more
-         * details.
-         */
-        if (config->coerce_c_locale && !locale_coerced) {
-            locale_coerced = 1;
-            _Py_CoerceLegacyLocale(config->coerce_c_locale_warn);
-            encoding_changed = 1;
-        }
-
         if (utf8_mode == -1) {
             if (config->utf8_mode == 1) {
                 /* UTF-8 Mode enabled */
@@ -1367,15 +1357,13 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config,
         /* Reset the configuration before reading again the configuration,
            just keep UTF-8 Mode value. */
         int new_utf8_mode = config->utf8_mode;
-        int new_coerce_c_locale = config->coerce_c_locale;
         if (_PyCoreConfig_Copy(config, &save_config) < 0) {
             pymain->err = _Py_INIT_NO_MEMORY();
             goto done;
         }
         pymain_clear_cmdline(pymain, cmdline);
         memset(cmdline, 0, sizeof(*cmdline));
         config->utf8_mode = new_utf8_mode;
-        config->coerce_c_locale = new_coerce_c_locale;
 
         /* The encoding changed: read again the configuration
            with the new encoding */
@@ -1729,6 +1717,12 @@ pymain_init(_PyMain *pymain, PyInterpreterState **interp_p)
 
     pymain_init_stdio(pymain, config);
 
+    if (config->warn_on_c_locale && pymain->locale_coercion_warning != NULL) {
+        fprintf(stderr, pymain->locale_coercion_warning, pymain->locale_coercion_target);
+        pymain->locale_coercion_warning = NULL;
+        pymain->locale_coercion_target = NULL;
+    }
+
     PyInterpreterState *interp;
     pymain->err = _Py_InitializeCore(&interp, config);
     if (_Py_INIT_FAILED(pymain->err)) {
@@ -1739,6 +1733,7 @@ pymain_init(_PyMain *pymain, PyInterpreterState **interp_p)
     pymain_clear_config(config);
     config = &interp->core_config;
 
+
     if (pymain_init_python_main(pymain, config, interp) < 0) {
         _Py_FatalInitError(pymain->err);
     }
@@ -1789,6 +1784,42 @@ _Py_UnixMain(int argc, char **argv)
     pymain.argc = argc;
     pymain.bytes_argv = argv;
 
+    /* The legacy C locale assumes ASCII as the default text encoding, which
+        * causes problems not only for the CPython runtime, but also other
+        * components like GNU readline.
+        *
+        * Accordingly, when the CLI detects it, it attempts to coerce it to a
+        * more capable UTF-8 based alternative.
+        *
+        * See the documentation of the PYTHONCOERCECLOCALE setting for more
+        * details.
+        */
+    _Py_SetLocaleFromEnv(LC_CTYPE);
+    if (_Py_LegacyLocaleDetected()) {
+            /* We ignore the Python -E and -I flags here, as the CLI needs to sort out
+             * the locale settings *before* we try to do anything with the command
+             * line arguments. For cross-platform debugging purposes, we also need
+             * to give end users a way to force even scripts that are otherwise
+             * isolated from their environment to use the legacy ASCII-centric C
+             * locale (otherwise a user on platform with a valid coercion target,
+             * like Fedora, can't readily debug a script that runs with -I or -E
+             * on a platform without a valid coercion target, like CentOS 7).
+             *
+             * Ignoring -E and -I is safe from a security perspective, as we only use
+             * the setting to turn *off* the implicit locale coercion, and anyone with
+             * access to the process environment already has the ability to set
+             * `LC_ALL=C` to override the C level locale settings anyway.
+             */
+            const char *coerce_c_locale = getenv("PYTHONCOERCECLOCALE");
+            if (coerce_c_locale == NULL || strncmp(coerce_c_locale, "0", 2) != 0) {
+                _Py_CoerceLegacyLocale(
+                    &pymain.locale_coercion_target,
+                    &pymain.locale_coercion_warning
+                );
+            }
+    }
+
+
     return pymain_main(&pymain);
 }
 

diff --git a/Programs/_testembed.c b/Programs/_testembed.c
@@ -330,8 +330,7 @@ dump_config(void)
 
     printf("filesystem_encoding = %s\n", config->filesystem_encoding);
     printf("filesystem_errors = %s\n", config->filesystem_errors);
-    printf("coerce_c_locale = %i\n", config->coerce_c_locale);
-    printf("coerce_c_locale_warn = %i\n", config->coerce_c_locale_warn);
+    printf("warn_on_c_locale = %i\n", config->warn_on_c_locale);
     printf("utf8_mode = %i\n", config->utf8_mode);
 
     printf("pycache_prefix = %ls\n", config->pycache_prefix);
@@ -607,7 +606,7 @@ static int test_init_isolated(void)
     _PyCoreConfig config = _PyCoreConfig_INIT;
 
     /* Set coerce_c_locale and utf8_mode to not depend on the locale */
-    config.coerce_c_locale = 0;
+    config.warn_on_c_locale = 0;
     config.utf8_mode = 0;
     /* Use path starting with "./" avoids a search along the PATH */
     config.program_name = L"./_testembed";