Skip to content

test_re fails on Alpine Linux (with musl) #131342

Closed
Closed
test_re fails on Alpine Linux (with musl)#131342
Labels
OS-unsupportedtestsTests in the Lib/test dirtopic-regextype-bugAn unexpected behavior, bug, or error

Description

@prashanthallu
prashanthallu
opened on Mar 17, 2025

Bug report

Bug description:

718.6 0:01:10 load avg: 0.97 [35/44] test_re
719.7 test test_re failed
719.7 0:01:11 load avg: 0.98 [36/44] test_set -- test_re failed (2 failures)
724.7 0:01:16 load avg: 0.98 [37/44] test_sqlite3
725.6 0:01:17 load avg: 0.98 [38/44] test_statistics
733.0 0:01:24 load avg: 0.98 [39/44] test_str
735.1 0:01:26 load avg: 0.98 [40/44] test_struct
735.8 0:01:27 load avg: 0.98 [41/44] test_tabnanny
736.3 0:01:28 load avg: 0.98 [42/44] test_time
739.5 0:01:31 load avg: 0.98 [43/44] test_xml_etree
740.2 0:01:32 load avg: 0.98 [44/44] test_xml_etree_c
741.3
741.3 Total duration: 1 min 33 sec
741.3 Total tests: run=9,178 failures=3 skipped=203
741.3 Total test files: run=44/44 failed=2 skipped=2
741.3 Result: FAILURE
741.3 make: *** [Makefile:886: profile-run-stamp] Error 2

And while running in verbose mode the following is the output of it.



 ./python -m unittest -v test.test_re
(...)
======================================================================
FAIL: test_locale_caching (test.test_re.ReTests.test_locale_caching)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Python-3.13.2/Lib/test/test_re.py", line 2195, in test_locale_caching
    self.check_en_US_iso88591()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Python-3.13.2/Lib/test/test_re.py", line 2204, in check_en_US_iso88591
    self.assertTrue(re.match(b'\xc5', b'\xe5', re.L|re.I))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: None is not true

======================================================================
FAIL: test_locale_compiled (test.test_re.ReTests.test_locale_compiled)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Python-3.13.2/Lib/test/test_re.py", line 2240, in test_locale_compiled
    self.assertTrue(p.match(b'\xe5\xe5'))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: None is not true

----------------------------------------------------------------------
Ran 165 tests in 0.838s

FAILED (failures=2, skipped=1)

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Activity

  1. prashanthallu
    added
    type-bugAn unexpected behavior, bug, or error
    on Mar 17, 2025
  2. vstinner
    changed the title [-]test_locale_caching and test_locale_compiled failing while installing python3.13[/-] [+]test_re fails on Alpine Linux (with musl)[/+] on Mar 17, 2025

  3. StanFromIreland commented on Mar 17, 2025

    @StanFromIreland
    StanFromIreland
    on Mar 17, 2025
    Member

    Do you have en_US_iso88591 installed? Check with:

    locale -a | grep en_US

    Both of the tests depend on it.

    I presume it is setting a locale that is not available for some reason.

            for loc in 'en_US.iso88591', 'en_US.utf8':
            try:
                locale.setlocale(locale.LC_CTYPE, loc)
            except locale.Error:
                # Unsupported locale on this system
                self.skipTest('test needs %s locale' % loc)

    I am unable to reproduce after deleting locale:

    test_locale_caching (test.test_re.ReTests.test_locale_caching) ... skipped 'test needs en_US.iso88591 locale'
test_locale_compiled (test.test_re.ReTests.test_locale_compiled) ... skipped 'test needs en_US.iso88591 locale'
  4. picnixz
    added
    testsTests in the Lib/test dir
    topic-regex
    OS-unsupported
    on Mar 17, 2025
  5. github-project-automation
    added this to Unsupported platformson Mar 17, 2025

  6. vstinner commented on Mar 17, 2025

    @vstinner
    vstinner
    on Mar 17, 2025
    Member

    @StanFromIreland: Do you link Python to the musl C library?

  7. StanFromIreland commented on Mar 17, 2025

    @StanFromIreland
    StanFromIreland
    on Mar 17, 2025
    Member

    @vstinner: I should have specified, I did not. I am not sure whether it is even packaged for Fedora.

  8. vstinner commented on Mar 17, 2025

    @vstinner
    vstinner
    on Mar 17, 2025
    Member

    musl is package in Fedora, but I failed to build Python using musl-gcc wrapper. I built Python in Alpine to get musl.

  9. serhiy-storchaka commented on Mar 17, 2025

    @serhiy-storchaka
    serhiy-storchaka
    on Mar 17, 2025
    Member

    Please add the following lines at the beginning of test_locale_compiled() and show the output.

            print()
        print(f'LC_CTYPE: {locale.setlocale(locale.LC_CTYPE)}')
        print(f'LC_COLLATE: {locale.setlocale(locale.LC_COLLATE)}')
        print(f'LC_ALL: {locale.setlocale(locale.LC_ALL)}')

  10. vstinner commented on Mar 17, 2025

    @vstinner
    vstinner
    on Mar 17, 2025
    Member

    Please add the following lines at the beginning of test_locale_compiled() and show the output.

    Here you have:

    test_locale_compiled (test.test_re.ReTests.test_locale_compiled) ... 
LC_CTYPE: C.UTF-8
LC_COLLATE: C
LC_ALL: C.UTF-8;C;C;C;C;C
FAIL

  11. vstinner commented on Mar 17, 2025

    @vstinner
    vstinner
    on Mar 17, 2025
    Member

    Using musl, setlocale() doesn't fail even if the locale isn't supported:

    $ ./python 
Python 3.14.0a5+ (heads/main-dirty:fe186d76cfd, Mar 11 2025, 08:54:51) [GCC 14.2.0] on linux
>>> import locale
>>> locale.setlocale(locale.LC_CTYPE, 'en_US.iso88591')
'en_US.iso88591'
>>> locale.getencoding()
'UTF-8'

    Another example:

    >>> import locale
>>> locale.setlocale(locale.LC_CTYPE, 'donotexist')
'donotexist'
>>> locale.getencoding()
'UTF-8'

  12. vstinner commented on Mar 17, 2025

    @vstinner
    vstinner
    on Mar 17, 2025
    Member

    See also #131313

  13. StanFromIreland commented on Mar 17, 2025

    @StanFromIreland
    StanFromIreland
    on Mar 17, 2025
    Member

    The PR covers both the tests here already. I guess the PR can be assigned this issue?

  14. deepwzh commented on Mar 17, 2025

    @deepwzh
    deepwzh
    on Mar 17, 2025
    Contributor

    I have roughly identified the reasons:

    1. Alpine does not include the en_US_iso88591 encoding by default.
    2. The implementations of the setlocale function in musl and glibc are inconsistent. For musl, passing any value will not cause an error. Therefore, even if locale.setlocale(locale.LC_CTYPE, 'en_US.iso88591') does not throw an exception, it will not work properly.

    When I run the code in two environments, the behavior is different

    locale.setlocale(locale.LC_CTYPE, "xxx.xxx")

    glibc:

    locale.setlocale(locale.LC_CTYPE, "xxx.xxx")  
Traceback (most recent call last):  
  File "/home/deepwzh/opensource/python_demo/test.py", line 9, in <module>  
    locale.setlocale(locale.LC_CTYPE, "xxx.xxx")  
  File "/home/deepwzh/miniconda3/lib/python3.12/locale.py", line 615, in setlocale  
    return _setlocale(category, locale)  
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    musl:
    No error is thrown.

    Further, you can refer to the musl source code here:
    https://github.com/bminor/musl/blob/c47ad25ea3b484e10326f933e927c0bc8cded3da/src/locale/locale_map.c#L99
    From this, it can be seen that if a non-existent locale is passed, it will actually default to UTF-8 encoding instead of throwing an error.

  15. vstinner commented on Mar 17, 2025

    @vstinner
    vstinner
    on Mar 17, 2025
    Member

    The PR covers both the tests here already. I guess the PR can be assigned this issue?

    PR #131313 was assigned to the older issue #90548.

  16. vstinner commented on Mar 19, 2025

    @vstinner
    vstinner
    on Mar 19, 2025
    Member

    Issue fixed by change 6146295.

  17. vstinner
    closed this as completedon Mar 19, 2025
  18. github-project-automation
    moved this to Done in Unsupported platformson Mar 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS-unsupportedtestsTests in the Lib/test dirtopic-regextype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions