Repository navigation
Many regtest failures on Windows with non-ASCII account name #89339
Description
| BPO | 45176 |
|---|---|
| Nosy | @pfmoore, @vstinner, @tjguk, @ezio-melotti, @zware, @serhiy-storchaka, @eryksun, @zooba |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2021-09-12.09:50:10.188>
labels = ['type-bug', 'OS-windows', '3.10', '3.11', 'tests', 'expert-unicode', '3.9']
title = 'Many regtest failures on Windows with non-ASCII account name'
updated_at = <Date 2021-09-21.16:32:42.185>
user = 'https://bugs.python.org/minghua'bugs.python.org fields:
activity = <Date 2021-09-21.16:32:42.185>
actor = 'steve.dower'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Tests', 'Unicode', 'Windows']
creation = <Date 2021-09-12.09:50:10.188>
creator = 'minghua'
dependencies = []
files = []
hgrepos = []
issue_num = 45176
keywords = []
message_count = 9.0
messages = ['401659', '401663', '401665', '401667', '402250', '402288', '402305', '402314', '402324']
nosy_count = 9.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'minghua']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45176'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
minghua commentedon Sep 12, 2021
Background:
Since at least Windows 8, it is possible to invoke the input method engine (IME) when installing Windows and creating accounts. So at least among simplified Chinese users, it's not uncommon to have a Chinese account name.
Issue:
After successful installation using the 64-bit .exe installer for Windows, just to be paranoid (and to get familiar with Python's test framework), I decided to run the bundled regression tests. To my surprise I got many failures. The following is the summary of "python.exe -m test" with 3.8 some months ago (likely 3.8.6):
371 tests OK.
11 tests failed:
test_cmd_line_script test_compileall test_distutils test_doctest
test_locale test_mimetypes test_py_compile test_tabnanny
test_urllib test_venv test_zipimport_support
43 tests skipped:
test_asdl_parser test_check_c_globals test_clinic test_curses
test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_smtpnet
test_socketserver test_spwd test_syslog test_threadsignals
test_timeout test_tix test_tk test_ttk_guionly test_urllib2net
test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
test_xxtestfuzz test_zipfile64
Total duration: 59 min 49 sec
Tests result: FAILURE
The failures all look similar though, it seems Python on Windows assumes the home directory of the user, "C:\Users\<username>\", is either in ASCII or UTF-8 encoding, while it is actually in Windows native codepage, in my case cp936 for simplified Chinese (zh-CN).
To take a couple of examples (these are from recent testing with 3.10.0 rc2):
Ran 44 tests in 29.769s
FAILED (failures=2, errors=5)
test test_cmd_line_script failed
test_cmd_line_script failed (5 errors, 2 failures) in 30.4 sec
== Tests result: FAILURE ==
In the above test_directory_error AssertionError message I redacted part of the path as my account name is my real name. Hope the issue is clear enough despite the redaction, since the "\xe5<5 bytes redacted>" part is 6 bytes and apparently in UTF-8 (for two Chinese characters) and the "\xbb<3 bytes redacted>" part is 4 bytes and apparently in cp936.
Postscript:
As I've said above, I discovered this issue some time ago, but only have time now to report it. I believe I've see these failures in 3.8.2/6, 3.9.7, and 3.10.0 rc2. It shouldn't be hard to reproduce for people with ways to create account with non-ASCII name on Windows. If reproducing turns out to be difficult though, I'm happy to provide more information and/or run more tests.
2 remaining items
eryksun commentedon Sep 12, 2021
In Windows, the standard I/O encoding of the spawn_python() child defaults to the process active code page, i.e. GetACP(). In Windows 10, the active code page can be set to UTF-8 at the system or application level, but most systems and applications still use a legacy code page. Python's default can be overridden to UTF-8 for standard I/O via PYTHONIOENCODING, or for all I/O via PYTHONUTF8 or "-X utf8=1". I would recommend using one of these UTF-8 options instead of trying to make a test work with the legacy code page. There is no guarantee, and should be no guarantee, that a filesystem path, which is Unicode, can be encoded using a legacy code page.
minghua commentedon Sep 12, 2021
Eryk Sun (eryksun) posted:
FWIW, I did test with "-X utf8" option and it wasn't any better. Just tested "python.exe -X utf8=1 -m test -W test_cmd_line_script" with 3.10.0 rc2 again, and got 6 errors and 2 failures this way (1 more error than without "-X utf8=1"). There is also this new error message:
eryksun commentedon Sep 12, 2021
I was suggesting to modify the tests to use the UTF-8 mode option in the spawn_python() command line. It doesn't help to run the parent process in UTF-8 mode since it isn't inherited. It could be inherited via PYTHONUTF8, but it turns out that environment variables won't help in this case due to the use of the -E and -I command-line options.
zooba commentedon Sep 20, 2021
I'd guess that these tests are assuming that sys.executable contains only ASCII characters. All the tests run in a non-ASCII working directory, so it's only the runtime that is not tested propersy here.
The easiest way for Ming Hua to test this is to install for all users (into Program Files), and run tests with the same user account.
If that's the case, we probably have to just go through the tests and make them Unicode-aware.
minghua commentedon Sep 21, 2021
Steve Dower (steve.dower) posted:
I've already installed for all users, just not into the default "C:\Program Files\", but instead "C:\Programs\Python\". I don't think it's the executable's path that is problematic, but the temporary directory where the tests are run (%LOCALAPPDATA%\Temp\tmpcwkfn9ct, where %LOCALAPPDATA% is C:\Users\<account name>\AppData\Local and therefore contains non-ASCII characters).
Both of these paths are shown in the error/failure logs posted in the first message.
I doubt installing into "C:\Program Files\" would make a difference.
serhiy-storchaka commentedon Sep 21, 2021
Not only sys.executable. Sources of non-ASCII paths:
The last one is the most common in these failures.
Tests fail when a non-ASCII path is written to the stdout or a file with the default encoding (which differs from the filesystem encoding) and then read with implying:
Fixing tests is not enough, because it is often an issue of scripts which write paths to the stdout. This problem does not have simple and general solution.
eryksun commentedon Sep 21, 2021
I see no problem with changing a test -- such as test_consistent_sys_path_for_direct_execution() -- to spawn the child interpreter with
-X utf8when the I/O encoding itself is irrelevant to the test -- except for forcing a common Unicode encoding to ensure the integrity of test data (i.e. no mojibake) and prevent encoding/decoding failures.zooba commentedon Sep 21, 2021
Ah yes, that indeed rules out my first suspicion.
Sure, but the ones currently failing here are ours, so we are the ones who need to fix them :) And they all seem to be in our test suite.
Fixing the tests doesn't make all the problems go away, just the specific ones we are responsible for on this issue.
vstinner commentedon Sep 22, 2023
I fixed a few issues recently. What's the status of this issue on Python 3.13 (main branch)?
vstinner commentedon Sep 22, 2023
See also issue GH-69368.