Repository navigation
Add new EncodingWarning warning category: emitted when the locale encoding is used implicitly #81395
Description
| BPO | 37214 |
|---|---|
| Nosy | @vstinner |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-06-18.10:13:54.250>
created_at = <Date 2019-06-10.10:39:37.707>
labels = ['type-feature', 'library', '3.9']
title = 'Add new EncodingWarning warning category: emitted when the locale encoding is used implicitly'
updated_at = <Date 2019-06-18.10:13:54.248>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2019-06-18.10:13:54.248>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-18.10:13:54.250>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-06-10.10:39:37.707>
creator = 'vstinner'
dependencies = []
files = ['48408']
hgrepos = []
issue_num = 37214
keywords = ['patch']
message_count = 6.0
messages = ['345121', '345123', '345124', '345125', '345567', '345976']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['13938']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37214'
versions = ['Python 3.9']
Activity
vstinner commentedon Jun 10, 2019
Spin-off of INADA-san's PEP-597: I propose to emit a new EncodingWarning warning when the locale encoding is used implicitly.
I propose to add a new warning to be able to control how they are handled: either make them hard error (python3.9 -W error::EncodingWarning) or ignore them (python3.9 -W ignore::EncodingWarning).
Displaying these warnings by default or not is an open question discussed in PEP-597 discussion:
https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819
I propose to only emit a warning which can be turned off easily to not bother people who don't care of encodings. There are many legit cases where a Python application works well in a control environment whereas it would break on any other environment, but the application is only used in one specific environment. For example, an application may only be run in a container which has a well defined locale and locale encoding, but will never be run on Windows, macOS, FreeBSD or HP-UX.
Writing portable code is *great*. But I'm not sure that *all* code must be portable. For example, it's also fine to use Python to write applications specific to Linux, again, for code only run in a controlled environment. Many companies have private projects which will never run outside their "walls" (outside their datacenters ;-)).
vstinner commentedon Jun 10, 2019
Building Python (make) with PR 13938 emits 7 EncodingWarning warnings:
/home/vstinner/prog/python/master/Lib/sysconfig.py:228: EncodingWarning: encoding=None
with open(filename, errors="surrogateescape") as f:
/home/vstinner/prog/python/master/Lib/sysconfig.py:371: EncodingWarning: encoding=None
with open(config_h) as f:
./setup.py:559: EncodingWarning: encoding=None
with open(tmpfile) as fp:
./setup.py:691: EncodingWarning: encoding=None
with open(config_h) as file:
./setup.py:893: EncodingWarning: encoding=None
with open(tmpfile) as fp:
./setup.py:1369: EncodingWarning: encoding=None
with open(f) as file:
./setup.py:1496: EncodingWarning: encoding=None
with open(zlib_h) as fp:
The test suite cannot be run if EncodingWarning are treated as error, an exception fails before running the first test:
To be able to run the test suite, I turned off the warning in subprocess.Popen using this change:
Using PR 13938 and subprocess.Popen change, at least 136 tests are failing. I have to interrupt the test suite since 7 tests hang:
$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -j0 -r ... 7 tests omitted: test_cmd_line_script test_concurrent_futures test_httpservers test_multiprocessing_fork test_multiprocessing_spawn test_ssl test_subprocess258 tests OK.
136 tests failed:
test___all__ test__xxsubinterpreters test_argparse test_asyncio
test_atexit test_base64 test_baseexception test_bdb test_bool
test_builtin test_bz2 test_calendar test_capi test_cgi test_cgitb
test_cmath test_cmd_line test_compile test_compileall test_complex
test_configparser test_contextlib test_coroutines test_cprofile
test_csv test_ctypes test_dbm test_dbm_dumb test_decimal
test_defaultdict test_deque test_difflib test_distutils
test_doctest test_eintr test_email test_embed test_eof
test_exceptions test_faulthandler test_file test_file_eintr
test_filecmp test_fileinput test_float test_gc test_gdb test_gzip
test_hash test_http_cookiejar test_httplib test_idle test_imp
test_import test_importlib test_inspect test_io test_iter
test_json test_lib2to3 test_linecache test_list test_lltrace
test_logging test_lzma test_mailbox test_mailcap test_math
test_module test_modulefinder test_multiprocessing_forkserver
test_multiprocessing_main_handling test_netrc test_ntpath
test_opcodes test_os test_parser test_pathlib test_pdb test_pickle
test_pipes test_pkg test_pkgimport test_pkgutil test_platform
test_poll test_popen test_posix test_posixpath test_profile
test_py_compile test_pydoc test_quopri test_random test_readline
test_regrtest test_repl test_runpy test_sax test_script_helper
test_select test_set test_shutil test_signal test_site test_socket
test_source_encoding test_stat test_support test_symbol test_sys
test_sysconfig test_tabnanny test_tarfile test_tcl test_tempfile
test_threading test_tools test_trace test_traceback
test_tracemalloc test_turtle test_unicodedata test_unittest
test_univnewlines test_userlist test_utf8_mode test_venv
test_warnings test_weakref test_webbrowser test_xml_etree
test_xml_etree_c test_zipfile test_zipimport
test_zipimport_support
For example, if EncodingWarning is treated as an error, test_cmd_line_script.test_repl_stderr_flush() hangs:
$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -v test_cmd_line_script -m test_repl_stderr_flush ... 0:00:00 load avg: 0.83 [1/1] test_cmd_line_script test_repl_stderr_flush (test.test_cmd_line_script.CmdLineTest) ...I would like to believe that Python code base is sane in general and that we handle encodings properly (I spent a significant amount of time to ensure that it's the case ;-)). So even if Python itself emits so many warnings, I don't think that it would be a good idea to introduce this warning and make it an hard error (exception) in Python 3.9.
I even don't think that such warning must be displayed by default. As DeprecationWarning, it's more a notice that should only be displayed to developers. Users don't control the code and so cannot fix the warning just by tuning their locale or locale encoding.
I don't see the point of flooding users with such warnings just to be pedantic.
As I wrote, they are many legit cases for encoding=None.
vstinner commentedon Jun 10, 2019
Another test: try to build a venv to get pip:
Building a venv is ok. Good, our stdlib is not so bad :-)
But ensurepip fails soon:
Note: an "up to date" pip still emits many SyntaxWarning and DeprecationWarning warnings :-(
New attempt using "always" action:
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:157: EncodingWarning: encoding=None
changing mode of /home/vstinner/prog/python/master/env/bin/easy_install-3.9 to 775
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:140: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:157: EncodingWarning: encoding=None
changing mode of /home/vstinner/prog/python/master/env/bin/pip3 to 775
changing mode of /home/vstinner/prog/python/master/env/bin/pip3.9 to 775
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
Successfully installed pip-19.0.3 setuptools-40.8.0
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-kb535zvl'
Hum, there are *a lot* of EncodingWarning warnings :-/
Trying to use pip fails with the same distro error:
Installing requests using pip emits even more EncodingWarning warnings in many functions of pip, but also in distro and msgpack dependencies:
vstinner commentedon Jun 10, 2019
My subprocess patch was wrong: you must use attached subprocess.patch.
vstinner commentedon Jun 14, 2019
I wrote this issue to discuss https://www.python.org/dev/peps/pep-0597/ but it's unrelated to the implementation of this PEP :-)
vstinner commentedon Jun 18, 2019
I reject this issue: see PEP-597 for the follow-up.