Skip to content

test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions #78584

Closed

Description

@michael-o
mannequin
BPO 34403
Nosy @terryjreedy, @vstinner, @aixtools, @michael-o
PRs
  • bpo-34403: Skip test_utf8_mode.test_cmd_line() on HP-UX #8966
  • bpo-34403: Fix test_utf8_mode.test_cmd_line() on HP-UX #8967
  • bpo-34403: On HP-UX, force ASCII for C locale #8969
  • bpo-34523: Fix config_init_fs_encoding() for ASCII #10232
  • [3.7] bpo-34403: Fix initfsencoding() for ASCII #10233
  • [3.7] bpo-34403: Always implement _Py_GetForceASCII() #10235
  • Dependencies
  • bpo-34207: test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD 3.x (3.8) buildbots
  • Files
  • c_locale.c
  • py3.8-LC-C.text
  • py3.8-default.text
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-08-29.13:49:04.403>
    created_at = <Date 2018-08-14.12:57:12.659>
    labels = ['3.7', 'tests', '3.8', 'type-bug', 'library']
    title = 'test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions'
    updated_at = <Date 2018-10-30.15:43:10.318>
    user = 'https://github.com/michael-o'

    bugs.python.org fields:

    activity = <Date 2018-10-30.15:43:10.318>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-08-29.13:49:04.403>
    closer = 'vstinner'
    components = ['Library (Lib)', 'Tests']
    creation = <Date 2018-08-14.12:57:12.659>
    creator = 'michael-o'
    dependencies = ['34207']
    files = ['47767', '47770', '47771']
    hgrepos = []
    issue_num = 34403
    keywords = ['patch', '3.7regression']
    message_count = 39.0
    messages = ['323516', '323682', '323729', '323797', '324070', '324085', '324100', '324172', '324175', '324176', '324196', '324211', '324213', '324214', '324215', '324216', '324217', '324219', '324221', '324225', '324236', '324238', '324242', '324253', '324260', '324261', '324273', '324274', '324289', '324322', '324323', '324463', '324502', '328901', '328902', '328903', '328916', '328924', '328932']
    nosy_count = 4.0
    nosy_names = ['terry.reedy', 'vstinner', 'Michael.Felt', 'michael-o']
    pr_nums = ['8966', '8967', '8969', '10232', '10233', '10235']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue34403'
    versions = ['Python 3.7', 'Python 3.8']

    Activity

    1. michael-o commented on Aug 14, 2018

      michael-omannequin
      MannequinAuthor

      Running from 3.7 branch on HP-UX 11.31 ia64, 32 bit, big endian.
      The test output is:

      Re-running failed tests in verbose mode
      Re-running test 'test_utf8_mode' in verbose mode
      test_cmd_line (test.test_utf8_mode.UTF8ModeTests) ... FAIL
      test_env_var (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_filesystemencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_io (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_io_encoding (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_locale_getpreferredencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_optim_level (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_posix_locale (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_stdio (test.test_utf8_mode.UTF8ModeTests) ... ok
      test_xoption (test.test_utf8_mode.UTF8ModeTests) ... ok

      ======================================================================
      FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
      ----------------------------------------------------------------------

      > Traceback (most recent call last):
      >   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 230, in test_cmd_line
      >     check('utf8=0', [c_arg], LC_ALL='C')
      >   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check
      >     self.assertEqual(args, ascii(expected), out)
      > AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
      > - ['h\xc3\xa9\xe2\x82\xac']
      > + ['h\udcc3\udca9\udce2\udc82\udcac']
      >  : roman8:['h\xc3\xa9\xe2\x82\xac']
      > 
      > 

      Ran 10 tests in 2.595s

      FAILED (failures=1)
      test test_utf8_mode failed
      1 test failed again:
      test_utf8_mode

      == Tests result: FAILURE then FAILURE ==

      1 test failed:
      test_utf8_mode

      1 re-run test:
      test_utf8_mode

      Total duration: 7 sec 265 ms
      Tests result: FAILURE then FAILURE
      Makefile:1066: recipe for target 'test' failed
      gmake: *** [test] Error 2

      I tried to understand the issue, but my Python knowledge is too low, especially I do not understand by a byte array "arg = 'h\xe9\u20ac'.encode('utf-8')" is passed as one arg to the forked process.

      I highly assume that this is related to the non-standard, default character encoding on HP-UX: https://en.wikipedia.org/wiki/HP_Roman#HP_Roman-8 (roman8).

      A stupid 8 bit encoding. The very same snippet on FreeBSD says:

      $ LC_ALL=C python3.6 test_utf8.py
      US-ASCII:[]

      Willing to test and modify if someone tells what to do.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      testsTests in the Lib/test dir
      type-bugAn unexpected behavior, bug, or error
      on Aug 14, 2018
    3. terryjreedy commented on Aug 17, 2018

      @terryjreedy
      Member

      You might get more information asking questions on python-list.

    4. michael-o commented on Aug 18, 2018

      michael-omannequin
      MannequinAuthor

      Thanks, I'll do that. Hopefully I can provide a patch for. Though, I am convinced that I have to write a custom codec for roman8 to make all at stuff work flawlessly.

    5. aixtools commented on Aug 20, 2018

      @aixtools
      Contributor

      Although the default is different (i.e., roman8 versus latin1 (iso8859-1)) both HP-UX and AIX (like Windows, cp1252) this issue and bpo-33347 are related.

      As I mentioned in https://bugs.python.org/issue34347#msg323319 the string seen by self.get_output() is not the same string as "expected".

      If I recall, there may be a way to almost get the two be the same - excect "expected" is a bytes object and the value returned as CLI output is a regular string.

      I am thinking, maybe the "easy" way will be to add AIX, HP-UX, and others to skip this test. Rather than hard-code, do a query to see what the default is, and it it is not UTF-8 - skip the test.

      In any case, it seems to be broken for any system that does not have UTF-8 as default.

    6. aixtools commented on Aug 25, 2018

      @aixtools
      Contributor

      It might be as simple as what I saw for AIX:

      diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
      index 26e2e13ec5..3e918fd54c 100644
      --- a/Lib/test/test_utf8_mode.py
      +++ b/Lib/test/test_utf8_mode.py
      @@ -219,6 +219,8 @@ class UTF8ModeTests(unittest.TestCase):
               check('utf8', [arg_utf8])
               if sys.platform == 'darwin' or support.is_android:
                   c_arg = arg_utf8
      +        elif sys.platform.startswith("aix"):
      +            c_arg = arg.decode('iso-8859-1')
               else:
                   c_arg = arg_ascii
               check('utf8=0', [c_arg], LC_ALL='C')

      so, adding below might be all that is needed:
      + elif sys.platform == "hpux":
      + c_arg = arg.decode('roman8')

    7. aixtools commented on Aug 25, 2018

      @aixtools
      Contributor

      As the AIX complaint is (was once the PR merges):
      AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"

      • ['h\xc3\xa9\xe2\x82\xac']
        + ['h\udcc3\udca9\udce2\udc82\udcac']
      And the HP-UX complaint is:
        File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check
          self.assertEqual(args, ascii(expected), out)
      AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
      - ['h\xc3\xa9\xe2\x82\xac']
      + ['h\udcc3\udca9\udce2\udc82\udcac']

      Maybe a change such as:

      --- a/Lib/test/test_utf8_mode.py
      +++ b/Lib/test/test_utf8_mode.py
      @@ -219,6 +219,8 @@ class UTF8ModeTests(unittest.TestCase):
               check('utf8', [arg_utf8])
               if sys.platform == 'darwin' or support.is_android:
                   c_arg = arg_utf8
      +        elif (platform.system == "AIX") or
      +              sys.platform.startswith("hp-ux"):
      +            c_arg = arg.decode('iso-8859-1')
               else:
                   c_arg = arg_ascii
               check('utf8=0', [c_arg], LC_ALL='C')

      I mention this because it seems neither roman8 nor roman9 have 'official' iso names or alias (correct me if I am wrong).

    8. michael-o commented on Aug 25, 2018

      michael-omannequin
      MannequinAuthor

      I think you are absoltely right.

      In any case, it seems to be broken for any system that does not have UTF-8 as default.

      You likely mean ASCII. Python assumes that LANG=C is ASCII which is not the case for AIX and HP-UX.

      Your patch looks reasonable, I will try this on Monday. The problem is that there is no roman8 codec in Python. Maybe ISO-8859-1 will do it for the test, but I am still eager to add one.

      I mention this because it seems neither roman8 nor roman9 have 'official' iso names or alias (correct me if I am wrong).

      There are no ISO names because this is not an ISO encoding. This is an HP invention aka hp-roman8 (roman8, ibm-1051, r8, Cp1051).

      Edit: there is roman8 support: https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Lib/encodings/hp_roman8.py as well as aliases.

      There are a few aliases missing: cp1051, ibm1051 and hp-roman8. This needs an additonal PR.

    9. michael-o commented on Aug 27, 2018

      michael-omannequin
      MannequinAuthor

      So I changed the test code to:

      diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
      index 26e2e13ec5..d9f8a3ed8b 100644
      --- a/Lib/test/test_utf8_mode.py
      +++ b/Lib/test/test_utf8_mode.py
      @@ -208,7 +208,7 @@ class UTF8ModeTests(unittest.TestCase):
           def test_cmd_line(self):
               arg = 'h\xe9\u20ac'.encode('utf-8')
               arg_utf8 = arg.decode('utf-8')
      -        arg_ascii = arg.decode('ascii', 'surrogateescape')
      +        arg_ascii = arg.decode('roman8', 'surrogateescape')
               code = 'import locale, sys; print("%s:%s" % (locale.getpreferredencoding(), ascii(sys.argv[1:])))'
               def check(utf8_opt, expected, **kw):

      and the output is:
      ======================================================================
      FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 224, in test_cmd_line
          check('utf8=0', [c_arg], LC_ALL='C')
        File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 217, in check
          self.assertEqual(args, ascii(expected), out)
      AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\xfb\\u02cb\\xe3\\x82\\u02dc']"
      - ['h\xc3\xa9\xe2\x82\xac']
      + ['h\xfb\u02cb\xe3\x82\u02dc']
       : roman8:['h\xc3\xa9\xe2\x82\xac']

      I still don't understand that.

      I believe that surrogate escape only works for ASCII and nothing else. If so, this test must be skipped on HP-UX and AIX.

    10. michael-o commented on Aug 27, 2018

      michael-omannequin
      MannequinAuthor

      Maybe skipping the test is the best thing:
      diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
      index 26e2e13ec5..d6c4b321be 100644
      --- a/Lib/test/test_utf8_mode.py
      +++ b/Lib/test/test_utf8_mode.py
      @@ -12,7 +12,7 @@ from test.support.script_helper import assert_python_ok, assert_python_failure

       MS_WINDOWS = (sys.platform == 'win32')
      -
      +HPUX = (sys.platform.startswith('hp-ux'))
      
       class UTF8ModeTests(unittest.TestCase):
           DEFAULT_ENV = {
      @@ -205,6 +205,7 @@ class UTF8ModeTests(unittest.TestCase):
               self.assertEqual(out, 'UTF-8 UTF-8')
       @unittest.skipIf(MS_WINDOWS, 'test specific to Unix')
      

      + @unittest.skipIf(HPUX, 'test specific to Unix with ASCII default locale')
      def test_cmd_line(self):
      arg = 'h\xe9\u20ac'.encode('utf-8')
      arg_utf8 = arg.decode('utf-8')

    11. michael-o commented on Aug 27, 2018

      michael-omannequin
      MannequinAuthor

      Maybe Victor Stinner has some insights here.

    12. 16 remaining items

    13. aixtools commented on Aug 28, 2018

      @aixtools
      Contributor

      On 28/08/2018 13:20, STINNER Victor wrote:

      I updated my PR 8969 to implement properly my idea. With this PR, on HP-UX with C or POSIX locale, Python now uses ASCII for its "filesystem encoding": sys.getfilesystemencoding() returns "ascii".

      Michael: can you please try my updated PR 8969?

      • apply the updated change
      • recompile Python
      • run the test suite using: LC_ALL=C ./python -m test -j0 -r

      You may also test with the current locale: ./python -m test -j0 -r

      Seems to work well as far as AIX and test_utf8_mode (as you had already
      merged I pulled master and built that anew).

      Attached is the output with LC_ALL=C in the prefix. If you were hoping
      for "dangling processes - your hopes are affirmed.

      Perhaps also noteworthy:

      root@x066:[/data/prj/python/git/python3-3.8]set | grep LC
      LC__FASTMSG=true
      MAILCHECK=600
      root@x066:[/data/prj/python/git/python3-3.8]set | grep LANG
      LANG=en_US

    14. aixtools commented on Aug 28, 2018

      @aixtools
      Contributor

      On 28/08/2018 20:43, Michael Felt wrote:

      Attached is the output with LC_ALL=C in the prefix. If you were hoping
      for "dangling processes - your hopes are affirmed.

      Previous mail ended with:

      == Tests result: FAILURE ==

      375 tests OK.

      13 tests failed:
          test__xxsubinterpreters test_asyncio test_concurrent_futures
          test_ctypes test_distutils test_embed test_httpservers
          test_multiprocessing_fork test_os test_pkg test_socket
          test_subprocess test_time

      30 tests skipped:
          test_curses test_dbm_gnu test_devpoll test_epoll test_gdb
          test_idle test_kqueue test_lzma test_msilib test_ossaudiodev
          test_readline test_smtpnet test_socketserver test_spwd test_sqlite
          test_startfile test_tcl test_timeout test_tix test_tk
          test_ttk_guionly test_ttk_textonly test_turtle test_urllib2net
          test_urllibnet test_winconsoleio test_winreg test_winsound
          test_xmlrpc_net test_zipfile64

      Total duration: 14 min 53 sec
      Tests result: FAILURE
      root@x066:[/data/prj/python/git/python3-3.8]

      Without LC_ALL=C summary is (different):

      == Tests result: FAILURE ==

      376 tests OK.

      10 tests failed:
          test__xxsubinterpreters test_asyncio test_ctypes test_distutils
          test_embed test_httpservers test_multiprocessing_forkserver
          test_os test_socket test_time

      32 tests skipped:
          test_curses test_dbm_gnu test_devpoll test_epoll test_gdb
          test_idle test_kqueue test_lzma test_msilib test_ossaudiodev
          test_readline test_smtpnet test_socketserver test_spwd test_sqlite
          test_startfile test_tcl test_timeout test_tix test_tk
          test_ttk_guionly test_ttk_textonly test_turtle test_unicode_file
          test_unicode_file_functions test_urllib2net test_urllibnet
          test_winconsoleio test_winreg test_winsound test_xmlrpc_net
          test_zipfile64

      Total duration: 11 min 1 sec

      And, rather than dangling processes, I see BrokenBarrierErrors

      FYI

    15. vstinner commented on Aug 28, 2018

      @vstinner
      Member

      Can we backport this to 3.7 at least?

      My policy is to focus on the master branch to support a new platform. Then add a buildbot and find a core developer to maintain this platform. See the PEP-11 for details.

      I would prefer to see a full test suite passing before discussing which changes should or should not be backported.

      I would also prefer to first see a more general discussion about who is going to support HP-UX.

      IMHO HP-UX is not officially supported today. My list of supported platforms:
      https://pythondev.readthedocs.io/cpython.html#supported-platforms

      See the test_utf8_mode now pass on HP-UX, I close the issue. Please open more specific issues for other failures. You might open a meta issue to track all HP-UX issues.

    16. michael-o commented on Aug 29, 2018

      michael-omannequin
      MannequinAuthor

      Please close, issue fixed. Thank you very much.

    17. vstinner commented on Aug 29, 2018

      @vstinner
      Member

      Please close, issue fixed. Thank you very much.

      You're welcome ;-)

    18. aixtools commented on Sep 1, 2018

      @aixtools
      Contributor

      On 28/08/2018 23:14, STINNER Victor wrote:

      STINNER Victor <vstinner@redhat.com> added the comment:

      > Can we backport this to 3.7 at least?
      I am the AIX(tools) Michael, Michael O is the HP-UX Michael :p

      So I was not the one asking. IMHO - as the PEP was new, if I understood
      correctly, in 3.7 - would be "nice" to see it back ported.

      However, like you - my goal is to get the tests passing on master, and
      worry about backport later.

      My policy is to focus on the master branch to support a new platform. Then add a buildbot and find a core developer to maintain this platform. See the PEP-11 for details.

      I would prefer to see a full test suite passing before discussing which changes should or should not be backported.

      I would also prefer to first see a more general discussion about who is going to support HP-UX.

      IMHO HP-UX is not officially supported today. My list of supported platforms:
      https://pythondev.readthedocs.io/cpython.html#supported-platforms

      See the test_utf8_mode now pass on HP-UX, I close the issue. Please open more specific issues for other failures. You might open a meta issue to track all HP-UX issues.

      ----------


      Python tracker <report@bugs.python.org>
      <https://bugs.python.org/issue34403\>


    19. vstinner commented on Sep 3, 2018

      @vstinner
      Member

      I am the AIX(tools) Michael, Michael O is the HP-UX Michael :p

      Oh, I didn't notice that you two have the same first name :-)

      So I was not the one asking. IMHO - as the PEP was new, if I understood
      correctly, in 3.7 - would be "nice" to see it back ported. However, like you - my goal is to get the tests passing on master, and
      worry about backport later.

      *My position didn't change since my last comment, same position for AIX and HP-UX: msg324289. I also updated my website to write down this policy:
      https://pythondev.readthedocs.io/cpython.html#i-want-cpython-to-support-my-platform

      By the way, please don't comment issues that are closed.

    20. vstinner commented on Oct 30, 2018

      @vstinner
      Member

      New changeset 905f1ac by Victor Stinner in branch 'master':
      bpo-34523: Fix config_init_fs_encoding() for ASCII (GH-10232)
      905f1ac

    21. vstinner commented on Oct 30, 2018

      @vstinner
      Member

      Michael Osipov: Oops, my commit b2457ef broke the filesystem encoding on HP-UX. It should be fixed by my commit 905f1ac.

    22. vstinner commented on Oct 30, 2018

      @vstinner
      Member

      New changeset 21220bb by Victor Stinner in branch '3.7':
      bpo-34403: Fix initfsencoding() for ASCII (GH-10233)
      21220bb

    23. vstinner commented on Oct 30, 2018

      @vstinner
      Member

      New changeset 7d35f79 by Victor Stinner in branch '3.7':
      bpo-34403: Always implement _Py_GetForceASCII() (GH-10235)
      7d35f79

    24. michael-o commented on Oct 30, 2018

      michael-omannequin
      MannequinAuthor

      Victor, looks good to me: 0:00:26 [ 23/419/3] test_utf8_mode passed.

      I don't know wether it is related, but test_unicode crash dumps here:
      0:00:22 [ 16/419/2] test_unicode crashed (Exit code -11)
      Fatal Python error: Segmentation fault

      Current thread 0x00000001 (most recent call first):
      File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2465 in PyUnicode_FromFormat
      File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2468 in check_format
      File "/var/osipovmi/cpython/Lib/test/test_unicode.py", line 2472 in test_from_format
      File "/var/osipovmi/cpython/Lib/unittest/case.py", line 610 in run
      File "/var/osipovmi/cpython/Lib/unittest/case.py", line 658 in __call__
      File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
      File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
      File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 122 in run
      File "/var/osipovmi/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/var/osipovmi/cpython/Lib/test/support/testresult.py", line 162 in run
      File "/var/osipovmi/cpython/Lib/test/support/init.py", line 1928 in _run_suite
      File "/var/osipovmi/cpython/Lib/test/support/init.py", line 2022 in run_unittest
      File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 175 in test_runner
      File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
      File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest.py", line 134 in runtest
      File "/var/osipovmi/cpython/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
      File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 587 in _main
      File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 571 in main
      File "/var/osipovmi/cpython/Lib/test/libregrtest/main.py", line 627 in main
      File "/var/osipovmi/cpython/Lib/test/regrtest.py", line 46 in _main
      File "/var/osipovmi/cpython/Lib/test/regrtest.py", line 50 in <module>
      File "/var/osipovmi/cpython/Lib/runpy.py", line 85 in _run_code
      File "/var/osipovmi/cpython/Lib/runpy.py", line 192 in _run_module_as_main

      Is that related to your PEP?

    25. vstinner commented on Oct 30, 2018

      @vstinner
      Member

      0:00:22 [ 16/419/2] test_unicode crashed (Exit code -11)

      Please open a new issue to track this bug.

    26. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions