Repository navigation
C API: _PyObject_VisitManagedDict() function should be public #107073
Description
tl; dr: _PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() function should be public.
If a C extension implements a type as a heap type, the type must support the GC protocol: it must implement visit and clear functions, and the dealloc function must DECREF the type.
It seems like if a heap type uses the new Py_TPFLAGS_MANAGED_DICT flag, the visit function must call _PyObject_VisitManagedDict() and the clear function must call _PyObject_ClearManagedDict(). Correct me if I'm wrong.
Problem: _PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() functions are private. IMO they must be public.
Either Py_TPFLAGS_MANAGED_DICT flag must be private, or it should be public and all related helper functions should be public as well.
Py_TPFLAGS_MANAGED_DICT was added to Python 3.11.
_PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() functions were added to Python 3.12.
In Python 3.12, typing.TypeVar, typing.ParamSpec, typing.TypeVarTuple are implemented with Py_TPFLAGS_MANAGED_DICT (and use the 2 helper functions).
In Python 3.13, _asyncio.Future and _asyncio.Task are also implemented with Py_TPFLAGS_MANAGED_DICT (and use the 2 helper functions).
Note: In Objects/typeobject.c, subtype_traverse() calls _PyObject_VisitManagedDict() in some cases, and subtype_clear() calls _PyObject_ClearManagedDict() in some cases.
Linked PRs
- gh-107073: Make PyObject_VisitManagedDict() public #108763
- gh-107073: Mention pythoncapi-compat for PyObject_VisitManagedDict() #110291
- gh-107073: fix relevant typo in
PyObject_ClearManagedDict#140032 - [3.14] gh-107073: fix relevant typo in
PyObject_ClearManagedDict(GH-140032) #140033 - [3.13] gh-107073: fix relevant typo in
PyObject_ClearManagedDict(GH-140032) #140034
Activity
vstinner commentedon Jul 22, 2023
cc @brandtbucher @iritkatriel: Tell me if I'm correct. If yes, I can create a PR to fix this issue.
vstinner commentedon Jul 22, 2023
The doc doesn't say much about how a heap type should be implemented:
vstinner commentedon Jul 22, 2023
pybind11 (C++) uses a strange hack using
*_PyObject_GetDictPtr()expression (note the*at the start) which uses the private _PyObject_GetDictPtr() function:https://github.com/pybind/pybind11/blob/8d08dc64ca300342852ceaa7b1d65fe9f69dab06/include/pybind11/detail/class.h#L523-L539
traverse:
clear:
By the way, should we also make _PyObject_GetDictPtr() function public?
vstinner commentedon Jul 23, 2023
cc @colesbury @ambv
pythongh-107073: Make PyObject_VisitManagedDict() public
pythongh-107073: Make PyObject_VisitManagedDict() public
vstinner commentedon Sep 1, 2023
I'm not sure if
_PyObject_GetDictPtr()should be made public or not.pythongh-107073: Make PyObject_VisitManagedDict() public
pythongh-107073: Make PyObject_VisitManagedDict() public
pythongh-107073: Make PyObject_VisitManagedDict() public
markshannon commentedon Oct 2, 2023
Just to be clear, the root problem here is that if a class is defined using
PyType_FromSpec()andPy_TPFLAGS_MANAGED_DICTis set, there is no way to implementtp_visitandtp_clear.That can be fixed by adding
_PyObject_VisitManagedDict()and_PyObject_ClearManagedDict()(or something with equivalent functionality) to the API so thattp_visitandtp_clearcan be implemented.However,
PyType_FromSpec()inherits many of the flaws of statically declaringPyTypeObjects.We should offer a better way to build classes, so maybe add a
PyUnstable_prefix, as we want to come up with a better approach?markshannon commentedon Oct 2, 2023
_PyObject_GetDictPtr()should not be made public.We should remove it, but not while anyone still uses
tp_dictoffset.markshannon commentedon Oct 2, 2023
If the
PyUnstable_prefix had existed when these functions were added, I would have used it.Having managed dicts is stable, so the functions will continue to have meaning if even the implementation changes.
In that sense they are stable.
However, we want to move the responsibility for visiting and clearing from C extensions to the VM. Once we have done that we will no longer need
_PyObject_VisitManagedDict()and_PyObject_ClearManagedDict(), so there is an argument for making themPyUnstable.On balance, I think they should be public. Making object layout the full responsibility of the VM won't happen for a few releases and even when it does it will need a longish deprecation to remove all the old API. So, we might as well add
_PyObject_VisitManagedDict()and_PyObject_ClearManagedDict()to the stable API.gh-107073: Make PyObject_VisitManagedDict() public (#108763)
vstinner commentedon Oct 3, 2023
Py_VISIT() should only be called once by the GC. So either calling PyObject_VisitManagedDict() should be the responsibility of the C extension, or the VM. But it cannot be both. And it cannot change without adding a new type flag, otherwise it would break the backward compatibility, since PyObject_VisitManagedDict() is public in Python 3.13. No?
That sounds like a good practical trade-off.
vstinner commentedon Oct 3, 2023
I added PyObject_VisitManagedDict() and PyObject_ClearManagedDict() functions to pythoncapi-compat for Python 3.11 and Python 3.12 using _PyObject_GetDictPtr(): python/pythoncapi-compat@a594354
pythongh-107073: Mention pythoncapi-compat for PyObject_VisitManagedD…
gh-107073: Mention pythoncapi-compat for PyObject_VisitManagedDict() (#…
pythongh-107073: Make PyObject_VisitManagedDict() public (python#108763)
pythongh-107073: Mention pythoncapi-compat for PyObject_VisitManagedD…
gh-107073: fix relevant typo in `PyObject_ClearManagedDict` (#140032)
pythongh-107073: fix relevant typo in `PyObject_ClearManagedDict` (py…
pythongh-107073: fix relevant typo in `PyObject_ClearManagedDict` (py…
[3.13] gh-107073: fix relevant typo in `PyObject_ClearManagedDict` (G…
[3.14] gh-107073: fix relevant typo in `PyObject_ClearManagedDict` (G…