Repository navigation
C API: _PyObject_VisitManagedDict() function should be public #107073
Description
Activity
cc @brandtbucher @iritkatriel: Tell me if I'm correct. If yes, I can create a PR to fix this issue.
The doc doesn't say much about how a heap type should be implemented:
pybind11 (C++) uses a strange hack using
*_PyObject_GetDictPtr()expression (note the*at the start) which uses the private _PyObject_GetDictPtr() function:
https://github.com/pybind/pybind11/blob/8d08dc64ca300342852ceaa7b1d65fe9f69dab06/include/pybind11/detail/class.h#L523-L539traverse:
PyObject *&dict = *_PyObject_GetDictPtr(self); Py_VISIT(dict);
clear:
PyObject *&dict = *_PyObject_GetDictPtr(self); Py_CLEAR(dict);
By the way, should we also make _PyObject_GetDictPtr() function public?
cc @colesbury @ambv
Reacted by Donghee NaI'm not sure if
_PyObject_GetDictPtr()should be made public or not.- added a commit that references this issue
on Sep 4, 2023 Just to be clear, the root problem here is that if a class is defined using
PyType_FromSpec()andPy_TPFLAGS_MANAGED_DICTis set, there is no way to implementtp_visitandtp_clear.That can be fixed by adding
_PyObject_VisitManagedDict()and_PyObject_ClearManagedDict()(or something with equivalent functionality) to the API so thattp_visitandtp_clearcan be implemented.However,
PyType_FromSpec()inherits many of the flaws of statically declaringPyTypeObjects.
We should offer a better way to build classes, so maybe add aPyUnstable_prefix, as we want to come up with a better approach?_PyObject_GetDictPtr()should not be made public.
We should remove it, but not while anyone still usestp_dictoffset.If the
PyUnstable_prefix had existed when these functions were added, I would have used it.Having managed dicts is stable, so the functions will continue to have meaning if even the implementation changes.
In that sense they are stable.However, we want to move the responsibility for visiting and clearing from C extensions to the VM. Once we have done that we will no longer need
_PyObject_VisitManagedDict()and_PyObject_ClearManagedDict(), so there is an argument for making themPyUnstable.On balance, I think they should be public. Making object layout the full responsibility of the VM won't happen for a few releases and even when it does it will need a longish deprecation to remove all the old API. So, we might as well add
_PyObject_VisitManagedDict()and_PyObject_ClearManagedDict()to the stable API.Reacted by Petr ViktorinHowever, we want to move the responsibility for visiting and clearing from C extensions to the VM.
Py_VISIT() should only be called once by the GC. So either calling PyObject_VisitManagedDict() should be the responsibility of the C extension, or the VM. But it cannot be both. And it cannot change without adding a new type flag, otherwise it would break the backward compatibility, since PyObject_VisitManagedDict() is public in Python 3.13. No?
On balance, I think they should be public. Making object layout the full responsibility of the VM won't happen for a few releases
That sounds like a good practical trade-off.
I added PyObject_VisitManagedDict() and PyObject_ClearManagedDict() functions to pythoncapi-compat for Python 3.11 and Python 3.12 using _PyObject_GetDictPtr(): python/pythoncapi-compat@a594354
- added a commit that references this issue
on Oct 3, 2023
tl; dr: _PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() function should be public.
If a C extension implements a type as a heap type, the type must support the GC protocol: it must implement visit and clear functions, and the dealloc function must DECREF the type.
It seems like if a heap type uses the new Py_TPFLAGS_MANAGED_DICT flag, the visit function must call _PyObject_VisitManagedDict() and the clear function must call _PyObject_ClearManagedDict(). Correct me if I'm wrong.
Problem: _PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() functions are private. IMO they must be public.
Either Py_TPFLAGS_MANAGED_DICT flag must be private, or it should be public and all related helper functions should be public as well.
Py_TPFLAGS_MANAGED_DICT was added to Python 3.11.
_PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() functions were added to Python 3.12.
In Python 3.12, typing.TypeVar, typing.ParamSpec, typing.TypeVarTuple are implemented with Py_TPFLAGS_MANAGED_DICT (and use the 2 helper functions).
In Python 3.13, _asyncio.Future and _asyncio.Task are also implemented with Py_TPFLAGS_MANAGED_DICT (and use the 2 helper functions).
Note: In Objects/typeobject.c, subtype_traverse() calls _PyObject_VisitManagedDict() in some cases, and subtype_clear() calls _PyObject_ClearManagedDict() in some cases.
Linked PRs
PyObject_ClearManagedDict#140032PyObject_ClearManagedDict(GH-140032) #140033PyObject_ClearManagedDict(GH-140032) #140034