Repository navigation
Python 3.13: test_xml_etree fails on expat 2.2.5 (RHEL8 / Rocky linux 8 / Ubuntu 22.04) #125067
Description
Activity
Are you able to test with pre releases, or attempt git bisect to help us narrow down the issue?
A
Are you able to test with pre releases, or attempt git bisect to help us narrow down the issue?
Yes - for the pre-releases, are there specific versions I should attempt?
I would attempt a manual bisection -- start with eg eg beta 1, then iterate through depending on if the failures are present.
Reacted by Jacob HennerI would attempt a manual bisection -- start with eg eg beta 1, then iterate through depending on if the failures are present.
I've reproduced the
test_xml_etreeerror on all rc releases, and also a1. I'm testing some beta releases now.Oddly, the
test_hashliberror only occurs within our CI build environment, but not when building locally. The reason this is odd is because the same Dockerfile is used in both cases, with no known difference in build context that'd explain why it'd fail in one case and succeed in the other. I'll follow up on this one next.Reacted by Vladimir KurilinThe failing tests (
test_xml_etree*) were all touched in https://github.com/python/cpython/pull/115623/files#diff-cd2315ff25ab49e653d363de2d213e390ecb1b01bbc86e2be3b700db4b0bf454. Perhaps the changes in 9f74e86, intended to address test failures for expat < 2.6, were incomplete?Have you tried with a more recent version of expat (2.6+)? Do the tests succeed?
Have you tried with a more recent version of expat (2.6+)? Do the tests succeed?
Just did -
test_xml_etree*succeeds with expat 2.6.3 built from source on Rocky 8, with all else remaining equal.Oddly, the test_hashlib error only occurs within our CI build environment, but not when building locally.
+1 to @JacobHenner – I have the exact same issue with test_hashlib. For 3.13.0 in CI, tests are failing both on rhel and debian. However, for all versions from 3.8 to 3.12, the tests pass without errors.
Locally, the Dockerfile builds without errors.Oddly, the test_hashlib error only occurs within our CI build environment, but not when building locally.
+1 to @JacobHenner – I have the exact same issue with test_hashlib. For 3.13.0 in CI, tests are failing both on rhel and debian. However, for all versions from 3.8 to 3.12, the tests pass without errors.
Locally, the Dockerfile builds without errors.What are you using to build your containers? Docker, podman, Kaniko?
33 remaining items
Thank you for the replies.
However, there's been a bit of a misunderstanding. By "distro maintainer" I was referring to my not being a "maintainer of a distribution of the Linux OS", and not anything about being a "distributor of Python binaries".
Ideally I wouldn't have to be building Python binaries at all, however:
- Our use-case (like eg pyenv's) requires the ability to select arbitrary Python versions (so using eg Ubuntu-provided Python isn't an option)
- Python.org doesn't distribute official binaries for Linux (xref https://discuss.python.org/t/publish-linux-installer-on-python-org/53760/1)
Perhaps things will improve in the future, given recent news about
python-build-standalone, but for now we're stuck with building Python binaries.Given our use-case, it seems likely we'll pick:
a) use the CPython bundled Expat (and be vulnerable until that bundle is updated and your CPython is updated)
...especially given that this is the configuration that CPython defaults to, and tests in its own CI.
Reacted by Tianon GraviReacted by Tianon GraviDo you also distribute Python tests? If yes, then you should distribute modified tests which ignore such failure (on selected platforms or always). If no, then you should not worry at all.
Do you also distribute Python tests? If yes, then you should distribute modified tests which ignore such failure (on selected platforms or always).
No we don't distribute tests to reduce resultant container image size (we build with
--disable-test-modules).If no, then you should not worry at all.
The failing tests with Ubuntu
expatare still something that affects us, since:- We need PGO, and PGO now requires the tests to pass (quite reasonably so!)
- If we patch the tests so that the failing tests are skipped, then it could just be masking a current or future expat incompatibility, per:
(More significantly) Is there an actual functionality regression when using xml.parsers.expat with Ubuntu 22.04's expat (that has partial backports), or is this a false-positive-test-only issue? Plus even if it's only a false positive now, how will we know there won't be a real regression in the future?
(from #125067 (comment))
Do you also distribute Python tests? [..]. If no, then you should not worry at all.
I believe this picture is missing that CPython compiled against backported system Expat will use less of the recent Expat API internally (because the CPython build system assumes that API to be missing based on its version, which is legit) and so then even outside of tests pull requests like #115623 may not work as expected any more because system Expat is behaving differently than what CPython is rightfully expecting from the given Expat version number. So if I'm not missing something it would need patching where binaries are built against backported Expat, and not just the tests but also C code (or even just C code). A cheap trick that may work is to fake
XML_COMBINED_VERSIONto a different value but it only works when the given backport did not omit any behavior-changing patches in between. It will need close evaluation (and then patching) in any case. Compared to the pile of patches to CPython that other distributors need to apply, one patch is not too bad for a distribution of CPython with regard to cost. On a personal note, I think it's interesting that 99% of the time backporting appears to work just fine and then that other 1% shows how flawed the backporting approach is by design and that most of the time it just doesn't show.Reacted by Ed MorleyFWIW, this issue exists on RHEL9/Rocky9 as well which ships expat 2.5.0. I've tested builds on Python 3.13.0-3.13.5 thus far.
Also exists on Azure Linux 3 with Python 3.13.5.
I do not think that this can be fixed on our side.
Reacted by Sebastian Pippingany solutions?
@whybeyoung We (like
docker-library/python) have had to give up on using--with-system-expatand instead use thelibexpatversion bundled with CPython as the least-worst alternative. (And given it matches the CPython configure defaults and what's tested in CPython's own CI, it feels fine as a long term solution to me.)Reacted by Tianon Gravi and Anthon van der NeutReacted by Tianon GraviClosing, as Serhiy noted above this isn't something we can fix, and there are workarounds.
Bug report
Bug description:
I'm attempting to build Python 3.13.0 for Rocky Linux 8, and several tests are failing:
These tests are required to succeed in order for
--enable-optimizationsto work as expected.I am using the following build options:
3.9-3.12.6 do not exhibit this behavior when built with the same commands (aside from the Python version).
I'm curious whether there might be a regression for certain versions of expat, similar to #117187.
expat-devel version is
expat-2.2.5-15.el8.The specific test failures are:
CPython versions tested on:
3.13
Operating systems tested on:
Linux