Repository navigation
test_builtin.PtyTests fail on non-ASCII characters if the readline module is loaded #58094
Description
| BPO | 13886 |
|---|---|
| Nosy | @doko42, @vstinner, @ezio-melotti, @akheron, @vadmium, @serhiy-storchaka, @miss-islington, @iritkatriel |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2022-01-17.13:48:59.793>
created_at = <Date 2012-01-27.10:23:39.123>
labels = ['type-bug', 'tests', '3.9', '3.10', '3.11']
title = 'test_builtin.PtyTests fail on non-ASCII characters if the readline module is loaded'
updated_at = <Date 2022-01-17.13:48:59.792>
user = 'https://bugs.python.org/nadeemvawda'bugs.python.org fields:
activity = <Date 2022-01-17.13:48:59.792>
actor = 'vstinner'
assignee = 'nadeem.vawda'
closed = True
closed_date = <Date 2022-01-17.13:48:59.793>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2012-01-27.10:23:39.123>
creator = 'nadeem.vawda'
dependencies = []
files = ['24337', '24338', '41653', '45771', '46296', '46340']
hgrepos = []
issue_num = 13886
keywords = ['patch']
message_count = 41.0
messages = ['152080', '152082', '164253', '180732', '180766', '180767', '180769', '180770', '180837', '181200', '181201', '181205', '181207', '181208', '191272', '211077', '258517', '258564', '282399', '282496', '283932', '283961', '285539', '285810', '303770', '303772', '317353', '317757', '318018', '410714', '410715', '410759', '410761', '410766', '410767', '410772', '410774', '410777', '410782', '410784', '410786']
nosy_count = 11.0
nosy_names = ['doko', 'vstinner', 'nadeem.vawda', 'ezio.melotti', 'Arfrever', 'python-dev', 'petri.lehtinen', 'martin.panter', 'serhiy.storchaka', 'miss-islington', 'iritkatriel']
pr_nums = ['7133', '30631', '30634', '30635']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13886'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
nadeemvawda commentedon Jan 27, 2012
I've recently come across a strange failure in the tests for the input()
built-in function:
$ ./python -E -m test -v test_readline test_builtinTraceback (most recent call last): File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1079, in test_input_tty_non_ascii self.check_input_tty("prompté", b"quux\xe9", "utf-8") File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1070, in check_input_tty self.assertEqual(input_result, expected) AssertionError: 'quux' != 'quux\udce9' - quux + quux\udce9 ? +Traceback (most recent call last): File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1083, in test_input_tty_non_ascii_unicode_errors self.check_input_tty("prompté", b"quux\xe9", "ascii") File "/home/nadeem/src/cpython/def/Lib/test/test_builtin.py", line 1070, in check_input_tty self.assertEqual(input_result, expected) AssertionError: 'quux' != 'quux\udce9' - quux + quux\udce9 ? +The failure only manifests itself if the readline module is loaded before
test_builtin runs (hence the presence of test_readline above). It will
not occur if regrtest is run with either of the -j or -W flags (which is
why it hasn't been seen on the buildbots).
The problem seems to be that readline assumes that its input should use
the locale encoding, and silently strips out any undecodable chars. This
breaks the tests mentioned above, since they set up sys.stdin to use the
surrogateescape error handler, expecting invalid characters to be escaped
rather than discarded.
This problem doesn't crop up if readline is *not* loaded, because in that
case PyOS_Readline() falls back to a stdio-based implementation
(PyOS_StdioReadline()) that preserves invalid characters, allowing them
to be handled properly by sys.stdin's encoding and error handler.
I have been able to fix the test failures with the attached patch, which
stops readline from eating invalid characters, making it consistent with
the stdio-based fallback. Can someone with more knowledge of readline
and/or locale issues advise whether the change is a good idea?
nadeemvawda commentedon Jan 27, 2012
Here's another patch that ensures the test always exercises the GNU
readline code path (rather than the stdio fallback). This will cause the
failure to occur when running just test_builtin (no need to also run
test_readline before it).
Ideally we'd want to test both code paths, but I'm not sure how to
accomplish that reliably, short of running the test in a subprocess.
akheron commentedon Jun 28, 2012
I had exactly the same error on 3.3b1 when running the test suite with randomized order.
ezio-melotti commentedon Jan 27, 2013
This problem still exists on 3.4.
The attached patch seems to solve it.
python-dev commentedon Jan 27, 2013
New changeset cf0f450b3299 by Nadeem Vawda in branch '3.2':
Issue bpo-13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/cf0f450b3299
python-dev commentedon Jan 27, 2013
New changeset 5c8732049dd5 by Nadeem Vawda in branch '3.3':
Issue bpo-13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/5c8732049dd5
python-dev commentedon Jan 27, 2013
New changeset 9774721bfc32 by Nadeem Vawda in branch 'default':
Issue bpo-13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/9774721bfc32
python-dev commentedon Jan 27, 2013
New changeset 12223782031f by Nadeem Vawda in branch '2.7':
Issue bpo-13886: Fix input() to not strip out supposedly-invalid input bytes.
http://hg.python.org/cpython/rev/12223782031f
akheron commentedon Jan 28, 2013
+ setlocale(LC_CTYPE, "C");
This looks dangerous to me. Are you sure readline's behavior doesn't change because of this?
nadeemvawda commentedon Feb 2, 2013
You're right; it breaks backspacing over multibyte characters. I should
have tested it more carefully before committing. I'll revert the changes.
26 remaining items
iritkatriel commentedon Jan 16, 2022
See also bpo-44887.
vstinner commentedon Jan 17, 2022
On Fedora 35, I still reproduce the initial issue on the main branch of Python:
$ ./python -E -m test -v test_readline test_builtin (...)======================================================================
FAIL: test_input_tty_non_ascii (test.test_builtin.PtyTests)
----------------------------------------------------------------------
Traceback (most recent call last): File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2095, in test_input_tty_non_ascii self.check_input_tty("prompté", b"quux\xe9", "utf-8") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2086, in check_input_tty self.assertEqual(input_result, expected) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 'quux' != 'quux\udce9' - quux + quux\udce9 ? +======================================================================
FAIL: test_input_tty_non_ascii_unicode_errors (test.test_builtin.PtyTests)
----------------------------------------------------------------------
Traceback (most recent call last): File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2099, in test_input_tty_non_ascii_unicode_errors self.check_input_tty("prompté", b"quux\xe9", "ascii") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2086, in check_input_tty self.assertEqual(input_result, expected) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 'quux' != 'quux\udce9' - quux + quux\udce9 ? + (...)Fedora 35 uses readline 8.1:
vstinner commentedon Jan 17, 2022
Oh, the test_builtin.test_input_tty_non_ascii() fails just if test_readline is loaded previously:
$ ./python -E -m test -m test.test_builtin.PtyTests.test_input_tty_non_ascii -v test_readline test_builtin == CPython 3.11.0a4+ (heads/main:7f4b69b9076, Jan 17 2022, 12:28:15) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] == Linux-5.15.12-200.fc35.x86_64-x86_64-with-glibc2.34 little-endian == cwd: /home/vstinner/python/main/build/test_python_49429æ == CPU count: 8 == encodings: locale=UTF-8, FS=utf-8 0:00:00 load avg: 0.48 Run tests sequentially 0:00:00 load avg: 0.48 [1/2] test_readlineRan 0 tests in 0.000s
OK
0:00:00 load avg: 0.48 [2/2] test_builtin -- test_readline ran no tests
test_input_tty_non_ascii (test.test_builtin.PtyTests) ... FAIL
======================================================================
FAIL: test_input_tty_non_ascii (test.test_builtin.PtyTests)
----------------------------------------------------------------------
Traceback (most recent call last): File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2095, in test_input_tty_non_ascii self.check_input_tty("prompté", b"quux\xe9", "utf-8") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2086, in check_input_tty self.assertEqual(input_result, expected) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 'quux' != 'quux\udce9' - quux + quux\udce9 ? +Ran 1 test in 0.013s
FAILED (failures=1)
test test_builtin failed
test_builtin failed (1 failure)
== Tests result: FAILURE ==
1 test failed:
test_builtin
1 test run no tests:
test_readline
Total duration: 559 ms
Tests result: FAILURE
In just, just importing readline is enough to make the test fails:
$ ./python -E -m test -m test.test_builtin.PtyTests.test_input_tty_non_ascii -v test_builtin (...)======================================================================
FAIL: test_input_tty_non_ascii (test.test_builtin.PtyTests)
----------------------------------------------------------------------
Traceback (most recent call last): File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2097, in test_input_tty_non_ascii self.check_input_tty("prompté", b"quux\xe9", "utf-8") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/test/test_builtin.py", line 2088, in check_input_tty self.assertEqual(input_result, expected) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 'quux' != 'quux\udce9' - quux + quux\udce9 ? +(...)
vstinner commentedon Jan 17, 2022
Since nobody managed to fix this issue in 10 years and the test still fails if the readline module is loaded, I wrote python/issues-test-cpython#30631 to skip the test if the readline module is loaded.
[-]readline-related test_builtin failure[/-][+]test_builtin.PtyTests fail on non-ASCII characters if the readline module is loaded[/+]vstinner commentedon Jan 17, 2022
rl-locale.diff changes the readline implementation of the PyOS_Readline() to set LC_CTYPE locale to "C": setlocale(LC_CTYPE, "C"), rather to the user preferred locale: setlocale(LC_CTYPE, ""). IMO it's a bad idea. Python made great progress in Unicode support, readline has a good Unicode support, and in most cases, it just works like a charm. This change looks a hack just to get these 2 specific tests to pass, but it breaks any other usage of readline.
rl-test.diff skips the test if the readline module can be imported, and it always import the readline module. It's different than that python/issues-test-cpython#30631 which only checks if the readline module is currently imported: my change doesn't import readline in test_builtin.
"input-readline*.patch" patches and #51382 spawn a fresh Python process to make sure that the readline is not imported or to import readline explicitly. They are better than my fix, but they are more complicated. It seems likle these changes also fix test_input_tty_non_ascii() but I don't understand how.
vstinner commentedon Jan 17, 2022
I marked bpo-41034 "test_builtin: PtyTests fail when run twice" as a duplicate of this issue. Moreover, I tested manually: my change python/issues-test-cpython#30631 fix the "./python -m test -R 3:3 test_builtin" command.
vstinner commentedon Jan 17, 2022
New changeset ad6e640 by Victor Stinner in branch 'main':
bpo-13886: Skip PTY non-ASCII tests if readline is loaded (GH-30631)
ad6e640
vstinner commentedon Jan 17, 2022
With my change, the two following commands now pass successfully:
miss-islington commentedon Jan 17, 2022
New changeset 1345b46 by Miss Islington (bot) in branch '3.10':
bpo-13886: Skip PTY non-ASCII tests if readline is loaded (GH-30631)
1345b46
vstinner commentedon Jan 17, 2022
New changeset 0fbb9af by Miss Islington (bot) in branch '3.9':
bpo-13886: Skip PTY non-ASCII tests if readline is loaded (GH-30631) (GH-30635)
0fbb9af
vstinner commentedon Jan 17, 2022
Ok, the initial issue is now fixed: the test pass.
If someone wants to write test input() with non-ASCII input and readline, I suggest to open a new issue and add the test in the test_readline module instead.