Repository navigation
Make PyObject and other runtime structures optionally fully opaque #6
Copy link
Copy link
Closed
Description
Full opaqueness in stable ABI would be one of the most useful features for alternative Pythons and I believe also for any larger changes inside CPython VM. The most problematic part is the exposure of the PyObject head:
struct {
Py_ssize_t ob_refcnt;
PyTypeObject *ob_type;
};
There several problems with making PyObject fully opaque:
PyObject_HEADis embedded in user defined types, but only the right size of it is necessary for that- macros such as
Py_INCREF,PY_TYPEdirectly access the fields - user code may directly access the fields
- without direct access to those performance critical fields, i.e., when
Py_INCREFis a function call, the extensions may be slower
Plan for gradual migration to optional opaqueness in stable ABI (stable NI):
- Optional function call for tagged pointers, but keep the fast direct access version for regular pointers:
- Keep the definition of
PyObject, but allow implementation to tag some pointers (lest significant bit of the pointer is set) - Change
Py_INCREF,Py_TYPEmacros such that they work as before unless the pointer is tagged in which case they call to runtime - No pointers will be really tagged at this point (direct access to
ob_typecontinues to work)
- Keep the definition of
- Migrate extensions to avoid direct access, but use macros/inline functions, nothing changes from performance perspective
- check of the tag should be extremely fast, it would happen just before the pointer is about to be dereferenced anyway
- note: here we are talking only about extensions that target or want to target stable ABI
- Make
PyObjectopaque- the macros would cast it to non-public struct that gives them access to the fields (so the original
PyObjectstruct is still part of the ABI, but not part of API) PyObject_HEADwould be changed to be a struct of the right size, but would not expose any useful fields (e.g.,bytearray of the right size)
- the macros would cast it to non-public struct that gives them access to the fields (so the original
- Users cannot access the fields anyone, Python runtimes can opt-in to function calls by tagging the pointer
- (memory pointed by the) tagged pointers do not need to follow any particular memory layout
- the tagged pointers do not actually have to be pointers at all
In summary:
- optional pointer tagging will become part of the ABI (almost fully backward compatible change modulo code that tags pointers internally)
- new functions, such as
Py_IncRef_Taggedwould become part of the ABI PyObjectlayout will not be part of API, but will stay as part of ABI,Py_INCREFstays as fast as before- none of this would apply when user does not choose to build for stable ABI (i.e., tagged pointers would not be expected in non-stable builds)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
encukou commentedon Oct 10, 2023
For the migration to opaque PyObject, we can also use @colesbury's
abi4proposal: https://discuss.python.org/t/34018(When Sam turns it into a PEP it'll be easier to follow.)
gvanrossum commentedon Oct 10, 2023
I migrated this issue to the api-revolution repo -- it looks like this would require a PEP (and so does NI).
@steve-s Can you find an issue in the problems repo with a discussion of the problems caused by the non-opacity of structs? Since it's a well-known problem I assume it's been discussed there.
steve-s commentedon Oct 10, 2023
Related problems:
encukou commentedon Jul 11, 2025
A non-revolutionary approach would be first adding the API to support this (and making sure that API is sufficient). The API can be useful for alternate implementations until CPython needs it (which it probably will, for free-threading).
See:
PyObjectopaque)PyModuleDef-- this struct also needs to be opaque)In #136505, the macro is private (
_Py_OPAQUE_PYOBJECT) and designed to be easy to remove if we don't want to make it public.[-]Make PyObject* and other runtime structures optionally fully opaque[/-][+]Make PyObject and other runtime structures optionally fully opaque[/+]encukou commentedon Apr 14, 2026
Solved in
abi3t(PEP 803). DefinePy_TARGET_ABI3Tto turn this option on.Pointer tagging is now possible for implementations that don't need to support
abi3nor CPython version-specific ABI. Making this possible in CPython would need a new API layer, see e.g. #7.