Repository navigation
test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions #78584
Copy link
Copy link
Closed
Labels
3.7 (EOL)end of lifeend of life3.8 (EOL)end of lifeend of lifestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytestsTests in the Lib/test dirTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
| BPO | 34403 |
|---|---|
| Nosy | @terryjreedy, @vstinner, @aixtools, @michael-o |
| PRs | |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-08-29.13:49:04.403>
created_at = <Date 2018-08-14.12:57:12.659>
labels = ['3.7', 'tests', '3.8', 'type-bug', 'library']
title = 'test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions'
updated_at = <Date 2018-10-30.15:43:10.318>
user = 'https://github.com/michael-o'bugs.python.org fields:
activity = <Date 2018-10-30.15:43:10.318>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-29.13:49:04.403>
closer = 'vstinner'
components = ['Library (Lib)', 'Tests']
creation = <Date 2018-08-14.12:57:12.659>
creator = 'michael-o'
dependencies = ['34207']
files = ['47767', '47770', '47771']
hgrepos = []
issue_num = 34403
keywords = ['patch', '3.7regression']
message_count = 39.0
messages = ['323516', '323682', '323729', '323797', '324070', '324085', '324100', '324172', '324175', '324176', '324196', '324211', '324213', '324214', '324215', '324216', '324217', '324219', '324221', '324225', '324236', '324238', '324242', '324253', '324260', '324261', '324273', '324274', '324289', '324322', '324323', '324463', '324502', '328901', '328902', '328903', '328916', '328924', '328932']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'vstinner', 'Michael.Felt', 'michael-o']
pr_nums = ['8966', '8967', '8969', '10232', '10233', '10235']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34403'
versions = ['Python 3.7', 'Python 3.8']Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
3.7 (EOL)end of lifeend of life3.8 (EOL)end of lifeend of lifestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytestsTests in the Lib/test dirTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
michael-o commentedon Aug 14, 2018
Running from 3.7 branch on HP-UX 11.31 ia64, 32 bit, big endian.
The test output is:
> Traceback (most recent call last): > File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 230, in test_cmd_line > check('utf8=0', [c_arg], LC_ALL='C') > File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check > self.assertEqual(args, ascii(expected), out) > AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']" > - ['h\xc3\xa9\xe2\x82\xac'] > + ['h\udcc3\udca9\udce2\udc82\udcac'] > : roman8:['h\xc3\xa9\xe2\x82\xac'] > >I tried to understand the issue, but my Python knowledge is too low, especially I do not understand by a byte array "arg = 'h\xe9\u20ac'.encode('utf-8')" is passed as one arg to the forked process.
I highly assume that this is related to the non-standard, default character encoding on HP-UX: https://en.wikipedia.org/wiki/HP_Roman#HP_Roman-8 (roman8).
A stupid 8 bit encoding. The very same snippet on FreeBSD says:
Willing to test and modify if someone tells what to do.
terryjreedy commentedon Aug 17, 2018
You might get more information asking questions on python-list.
michael-o commentedon Aug 18, 2018
Thanks, I'll do that. Hopefully I can provide a patch for. Though, I am convinced that I have to write a custom codec for roman8 to make all at stuff work flawlessly.
aixtools commentedon Aug 20, 2018
Although the default is different (i.e., roman8 versus latin1 (iso8859-1)) both HP-UX and AIX (like Windows, cp1252) this issue and bpo-33347 are related.
As I mentioned in https://bugs.python.org/issue34347#msg323319 the string seen by self.get_output() is not the same string as "expected".
If I recall, there may be a way to almost get the two be the same - excect "expected" is a bytes object and the value returned as CLI output is a regular string.
I am thinking, maybe the "easy" way will be to add AIX, HP-UX, and others to skip this test. Rather than hard-code, do a query to see what the default is, and it it is not UTF-8 - skip the test.
In any case, it seems to be broken for any system that does not have UTF-8 as default.
aixtools commentedon Aug 25, 2018
It might be as simple as what I saw for AIX:
so, adding below might be all that is needed:
+ elif sys.platform == "hpux":
+ c_arg = arg.decode('roman8')
aixtools commentedon Aug 25, 2018
As the AIX complaint is (was once the PR merges):
AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
+ ['h\udcc3\udca9\udce2\udc82\udcac']
And the HP-UX complaint is: File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check self.assertEqual(args, ascii(expected), out) AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']" - ['h\xc3\xa9\xe2\x82\xac'] + ['h\udcc3\udca9\udce2\udc82\udcac']Maybe a change such as:
I mention this because it seems neither roman8 nor roman9 have 'official' iso names or alias (correct me if I am wrong).
michael-o commentedon Aug 25, 2018
I think you are absoltely right.
You likely mean ASCII. Python assumes that LANG=C is ASCII which is not the case for AIX and HP-UX.
Your patch looks reasonable, I will try this on Monday. The problem is that there is no roman8 codec in Python. Maybe ISO-8859-1 will do it for the test, but I am still eager to add one.
There are no ISO names because this is not an ISO encoding. This is an HP invention aka hp-roman8 (roman8, ibm-1051, r8, Cp1051).
Edit: there is roman8 support: https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Lib/encodings/hp_roman8.py as well as aliases.
There are a few aliases missing: cp1051, ibm1051 and hp-roman8. This needs an additonal PR.
michael-o commentedon Aug 27, 2018
So I changed the test code to:
and the output is:
======================================================================
FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
----------------------------------------------------------------------
Traceback (most recent call last): File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 224, in test_cmd_line check('utf8=0', [c_arg], LC_ALL='C') File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 217, in check self.assertEqual(args, ascii(expected), out) AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\xfb\\u02cb\\xe3\\x82\\u02dc']" - ['h\xc3\xa9\xe2\x82\xac'] + ['h\xfb\u02cb\xe3\x82\u02dc'] : roman8:['h\xc3\xa9\xe2\x82\xac']I still don't understand that.
I believe that surrogate escape only works for ASCII and nothing else. If so, this test must be skipped on HP-UX and AIX.
michael-o commentedon Aug 27, 2018
Maybe skipping the test is the best thing:
diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 26e2e13ec5..d6c4b321be 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -12,7 +12,7 @@ from test.support.script_helper import assert_python_ok, assert_python_failure
+ @unittest.skipIf(HPUX, 'test specific to Unix with ASCII default locale')
def test_cmd_line(self):
arg = 'h\xe9\u20ac'.encode('utf-8')
arg_utf8 = arg.decode('utf-8')
michael-o commentedon Aug 27, 2018
Maybe Victor Stinner has some insights here.
16 remaining items
aixtools commentedon Aug 28, 2018
On 28/08/2018 13:20, STINNER Victor wrote:
Seems to work well as far as AIX and test_utf8_mode (as you had already
merged I pulled master and built that anew).
Attached is the output with LC_ALL=C in the prefix. If you were hoping
for "dangling processes - your hopes are affirmed.
Perhaps also noteworthy:
root@x066:[/data/prj/python/git/python3-3.8]set | grep LC
LC__FASTMSG=true
MAILCHECK=600
root@x066:[/data/prj/python/git/python3-3.8]set | grep LANG
LANG=en_US
aixtools commentedon Aug 28, 2018
On 28/08/2018 20:43, Michael Felt wrote:
Previous mail ended with:
== Tests result: FAILURE ==
375 tests OK.
13 tests failed:
test__xxsubinterpreters test_asyncio test_concurrent_futures
test_ctypes test_distutils test_embed test_httpservers
test_multiprocessing_fork test_os test_pkg test_socket
test_subprocess test_time
30 tests skipped:
test_curses test_dbm_gnu test_devpoll test_epoll test_gdb
test_idle test_kqueue test_lzma test_msilib test_ossaudiodev
test_readline test_smtpnet test_socketserver test_spwd test_sqlite
test_startfile test_tcl test_timeout test_tix test_tk
test_ttk_guionly test_ttk_textonly test_turtle test_urllib2net
test_urllibnet test_winconsoleio test_winreg test_winsound
test_xmlrpc_net test_zipfile64
Total duration: 14 min 53 sec
Tests result: FAILURE
root@x066:[/data/prj/python/git/python3-3.8]
Without LC_ALL=C summary is (different):
== Tests result: FAILURE ==
376 tests OK.
10 tests failed:
test__xxsubinterpreters test_asyncio test_ctypes test_distutils
test_embed test_httpservers test_multiprocessing_forkserver
test_os test_socket test_time
32 tests skipped:
test_curses test_dbm_gnu test_devpoll test_epoll test_gdb
test_idle test_kqueue test_lzma test_msilib test_ossaudiodev
test_readline test_smtpnet test_socketserver test_spwd test_sqlite
test_startfile test_tcl test_timeout test_tix test_tk
test_ttk_guionly test_ttk_textonly test_turtle test_unicode_file
test_unicode_file_functions test_urllib2net test_urllibnet
test_winconsoleio test_winreg test_winsound test_xmlrpc_net
test_zipfile64
Total duration: 11 min 1 sec
And, rather than dangling processes, I see BrokenBarrierErrors
FYI
vstinner commentedon Aug 28, 2018
My policy is to focus on the master branch to support a new platform. Then add a buildbot and find a core developer to maintain this platform. See the PEP-11 for details.
I would prefer to see a full test suite passing before discussing which changes should or should not be backported.
I would also prefer to first see a more general discussion about who is going to support HP-UX.
IMHO HP-UX is not officially supported today. My list of supported platforms:
https://pythondev.readthedocs.io/cpython.html#supported-platforms
See the test_utf8_mode now pass on HP-UX, I close the issue. Please open more specific issues for other failures. You might open a meta issue to track all HP-UX issues.
michael-o commentedon Aug 29, 2018
Please close, issue fixed. Thank you very much.
vstinner commentedon Aug 29, 2018
You're welcome ;-)
aixtools commentedon Sep 1, 2018
On 28/08/2018 23:14, STINNER Victor wrote:
So I was not the one asking. IMHO - as the PEP was new, if I understood
correctly, in 3.7 - would be "nice" to see it back ported.
However, like you - my goal is to get the tests passing on master, and
worry about backport later.
vstinner commentedon Sep 3, 2018
Oh, I didn't notice that you two have the same first name :-)
*My position didn't change since my last comment, same position for AIX and HP-UX: msg324289. I also updated my website to write down this policy:
https://pythondev.readthedocs.io/cpython.html#i-want-cpython-to-support-my-platform
By the way, please don't comment issues that are closed.
vstinner commentedon Oct 30, 2018
New changeset 905f1ac by Victor Stinner in branch 'master':
bpo-34523: Fix config_init_fs_encoding() for ASCII (GH-10232)
905f1ac
vstinner commentedon Oct 30, 2018
Michael Osipov: Oops, my commit b2457ef broke the filesystem encoding on HP-UX. It should be fixed by my commit 905f1ac.
vstinner commentedon Oct 30, 2018
New changeset 21220bb by Victor Stinner in branch '3.7':
bpo-34403: Fix initfsencoding() for ASCII (GH-10233)
21220bb
vstinner commentedon Oct 30, 2018
New changeset 7d35f79 by Victor Stinner in branch '3.7':
bpo-34403: Always implement _Py_GetForceASCII() (GH-10235)
7d35f79
michael-o commentedon Oct 30, 2018
Victor, looks good to me: 0:00:26 [ 23/419/3] test_utf8_mode passed.
I don't know wether it is related, but test_unicode crash dumps here:
0:00:22 [ 16/419/2] test_unicode crashed (Exit code -11)
Fatal Python error: Segmentation fault
Current thread 0x00000001 (most recent call first):
File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2465 in PyUnicode_FromFormat
File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2468 in check_format
File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2472 in test_from_format
File "/var/osipovmi/cpython/Lib/unittest/case.py", line 610 in run
File "/var/osipovmi/cpython/Lib/unittest/case.py", line 658 in __call__
File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
File "/var/osipovmi/cpython/Lib/test/support/testresult.py", line 162 in run
File "/var/osipovmi/cpython/Lib/test/support/init.py", line 1928 in _run_suite
File "/var/osipovmi/cpython/Lib/test/support/init.py", line 2022 in run_unittest
File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 175 in test_runner
File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 134 in runtest
File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 587 in _main
File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 571 in main
File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 627 in main
File "/var/osipovmi/cpython/Lib/test/regrtest.py", line 46 in _main
File "/var/osipovmi/cpython/Lib/test/regrtest.py", line 50 in <module>
File "/var/osipovmi/cpython/Lib/runpy.py", line 85 in _run_code
File "/var/osipovmi/cpython/Lib/runpy.py", line 192 in _run_module_as_main
Is that related to your PEP?
vstinner commentedon Oct 30, 2018
Please open a new issue to track this bug.