Repository navigation
Cython 0.29.x Python 3.12.0a5+ tracker #5285
Description
Hello,
I am trying o make Cython 0.29.x work with Python 3.12.0a5+, so @hrnciar can continue testing Fedora packages with pre-releases of Python 3.12.
The idea is that once I make it work I'll contribute backports here and/or other interested folks contribute backports here and I'll test them in Fedora.
Note: We have considered upgrading to Cython 3.0.0b1 but far too many of our packages are not buildable with it (I've asked @hrnciar to provide a summary of the failures to you, in case you'd like that).
Following up on #5238 (comment) and 58d5fed61c I came up with this WIP patch for 0.29.33:
diff --git a/Cython/Utility/ModuleSetupCode.c b/Cython/Utility/ModuleSetupCode.c
index 33aac2c4a..b3de5dd75 100644
--- a/Cython/Utility/ModuleSetupCode.c
+++ b/Cython/Utility/ModuleSetupCode.c
@@ -210,6 +210,10 @@
#elif !defined(CYTHON_USE_PYLONG_INTERNALS)
#define CYTHON_USE_PYLONG_INTERNALS 1
#endif
+ #if PY_VERSION_HEX >= 0x030C00A5
+ #undef CYTHON_USE_PYLONG_INTERNALS
+ #define CYTHON_USE_PYLONG_INTERNALS 0
+ #endif
#ifndef CYTHON_USE_PYLIST_INTERNALS
#define CYTHON_USE_PYLIST_INTERNALS 1
#endif
@@ -252,7 +256,7 @@
#define CYTHON_USE_TP_FINALIZE (PY_VERSION_HEX >= 0x030400a1)
#endif
#ifndef CYTHON_USE_DICT_VERSIONS
- #define CYTHON_USE_DICT_VERSIONS (PY_VERSION_HEX >= 0x030600B1)
+ #define CYTHON_USE_DICT_VERSIONS ((PY_VERSION_HEX >= 0x030600B1) && (PY_VERSION_HEX < 0x030C00A5))
#endif
#if PY_VERSION_HEX >= 0x030B00A4
#undef CYTHON_USE_EXC_INFO_STACKThis makes Cython at least build itself. However, the tests still fail.
Full logs: builder-live.log.gz
The test failures I could recognize are:
Exception hints
Expected:
Traceback (most recent call last):
...
AttributeError: 'special_methods_T561_py3.VerySpecial' object has no attribute '__xxx__'
Got:
Traceback (most recent call last):
...
AttributeError: 'special_methods_T561_py3.VerySpecial' object has no attribute '__xxx__'. Did you mean: '__yyy__'?
This looks like a test-expectation problem only and when I look at the master branch, I see that possibly the same failures would happen there as well.
Confirmed: This also happens with 3.0.0b1. Fixed in d883ccd
getattribute examples
Failed example:
g11 = object.__getattribute__(GetAttribute(), '__getattribute__')
Expected nothing
Got:
GetAttribute getattribute '__dict__'
GetAttribute getattribute '__class__'
GetAttribute getattribute '__dict__'
GetAttribute getattribute '__class__'
This also happens with 3.0.0b1.
Failed example:
(a.called_getattr, a.called_getattribute)
Expected:
(2, 11)
Got:
(6, 15)
This also happens with 3.0.0b1.
builtins.bool size changed (fixed in 3.0.0b1)
ERROR: runTest (__main__.CythonRunTestCase.runTest)
[0] compiling (cpp) and running type_inference
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/cython-0.29.33/runtests.py", line 1270, in run
self.run_tests(result, ext_so_path)
File "/builddir/build/BUILD/cython-0.29.33/runtests.py", line 1288, in run_tests
self.run_doctests(self.module, result, ext_so_path)
File "/builddir/build/BUILD/cython-0.29.33/runtests.py", line 1302, in run_doctests
run_forked_test(result, run_test, self.shortDescription(), self.fork)
File "/builddir/build/BUILD/cython-0.29.33/runtests.py", line 1307, in run_forked_test
run_func(result)
File "/builddir/build/BUILD/cython-0.29.33/runtests.py", line 1294, in run_test
module = import_ext(module_or_name, ext_so_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/builddir/build/BUILD/cython-0.29.33/runtests.py", line 502, in import_ext
return imp.load_dynamic(module_name, file_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/imp.py", line 343, in load_dynamic
return _load(spec)
^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 872, in _load
File "<frozen importlib._bootstrap>", line 841, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 1314, in exec_module
File "<frozen importlib._bootstrap>", line 400, in _call_with_frames_removed
File "tests/run/type_inference.pyx", line 1, in init type_inference (type_inference.cpp:27226)
# cython: infer_types = True
ValueError: builtins.bool size changed, may indicate binary incompatibility. Expected 32 from C header, got 24 from PyObject
This is only happening with 0.29.x.
Backported in 43ce558
undefined symbols
ImportError: /builddir/build/BUILD/cython-0.29.33/TEST_TMP/1/run/c/py_unicode_strings/py_unicode_strings.cpython-312-x86_64-linux-gnu.so: undefined symbol: PyUnicode_FromUnicode
It's only used if you use the old Py_UNICODE type explicitly in your code. We disable the respective tests on CPython versions that no longer support it (without deprecation warnings).
Activity
hroncok commentedon Mar 1, 2023
I'm now building 3.0.0b1 again to see what test failures from above (if any) are missing.
matusvalo commentedon Mar 1, 2023
Cython 0.29.x will need some work in order to work with 3.12 release because there will be several removals breaking cython 0.29.X:
impmodule is going to be removed in 3.12 [1]. Cython 3 is not usingimpmodule in latest versions of python but 0.29.X does.distutilspackage is going to be removed in 3.12 [2]. Cython 3 still depends on distutils but removal is planned in Cython 3.1 - see [ENH] Switch to using onlysetuptools(PEP 632) #4405.setup.py install,setDaemon(),unittest.makeSuite()andPy_SetProgramName#5134 but not sure, I have not investigated in deep.[1] https://docs.python.org/3.11/library/imp.html
[2] https://docs.python.org/3/library/distutils.html
hroncok commentedon Mar 2, 2023
builtins.bool size changed
Possibly python/cpython#101394
I might need to backport #4894 which builds on top of #4869
Draft PR: #5289
hroncok commentedon Mar 2, 2023
That removal already happened and using distutils shipped via setuptools seems to work, at least for Fedora.
hroncok commentedon Mar 3, 2023
PyUnicode_FromUnicode still appears to be used in master, not sure why I didn't get the errors when I built b1:
cython/Cython/Utility/TypeConversion.c
Lines 99 to 101 in 94bea66
matusvalo commentedon Mar 4, 2023
In order to support python 3.12 in Cython 0.29.X we need to backport following PRs (imp module is going to be removed - python/cpython#98573):
#4941
#4625
I can create backport PRs to 0.29.X branch.
scoder commentedon Mar 4, 2023
scoder commentedon Mar 5, 2023
This looks small enough for a backport.
This seems stretching it. I'd rather not risk issues with Py<3.12 in 0.29.x any more. I can live with pyximport being broken in Py3.12 and later in 0.29.x, it's really only a dev tool, not meant for production systems.
da-woods commentedon Mar 5, 2023
If we get too many issues, we could possibly split off a
pyximport_futurepackage so that people have something to use with Py3.12 and Cy 0.29.x. I'd be tempted not to do that until actually needed, but it's an option.hroncok commentedon Mar 5, 2023
9d38bf8 / #5154
scoder commentedon Mar 7, 2023
I disabled the usage of PyLong internals and dict versioning for Py3.12 in 3397705, only for the 0.29.x branch, similar to your WIP patch above. Both are optimisations, it's ok to disable them for 0.29.x. Users who want fast support for Py3.12 should migrate to Cython 3.0.
scoder commentedon Mar 7, 2023
I picked the change from 9d38bf8 to disable the
py_unicode_stringstest and also worked around the exception output changes.21e64ec
d883ccd
hroncok commentedon Mar 7, 2023
Awesome, thanks. I planned to open more PRs today, so let me test your commits instead.
hroncok commentedon Mar 7, 2023
Testing this combination on top of 0.29.33:
hroncok commentedon Mar 7, 2023
With that I get:
mattip commentedon May 15, 2023
From numpy/numpy#23766
In
__Pyx_PyThreadState_Currentthetstatefields used no longer exist in the struct on CPython 3.12.0a7.mattip commentedon May 15, 2023
The exception fields were replaced due to python/cpython#101578 with a single
PyObject *current_exceptionscoder commentedon May 16, 2023
These fields shouldn't be used in 0.29.x any more. We disabled the
CYTHON_FAST_THREAD_STATEfeature there. (BTW, the change came in 3.12a6 already.)dev-python/cython: enable py3.12, use PYTHON_TESTED logic from 3.0.0_…
da-woods commentedon Feb 2, 2024
I'm going to close this because I think Python 3.12 works as well as it's going to with Cython. The only remaining issue is tracing, and we've concluded we can't fix it.