Repository navigation
Add a new Include/cpython/ subdirectory for the "CPython API" with implementation details #79315
Description
Activity
The PEP-384 "Defining a Stable ABI" introduced Py_LIMITED_API define to exclude functions from the Python C API. The problem is when a new API is introduced, it has to explicitly be excluded using "#ifndef Py_LIMITED_API". If the author forgets it, the function is added to be stable API by mistake.
I propose to move the API which should be excluded from the stable ABI to a new subdirectory: Include/pycapi/.
To not break the backward compatibility, I propose to automatically include new header files from existing header files. For example, Include/pycapi/pyapi_objimpl.h would be automatically included by Include/pycapi/pycapi_objimpl.h.
New header files would have a "pycapi_" prefix to avoid conflict Include/ header files, if Include/pycapi/ directory is in the header search paths.
This change is a follow-up of bpo-35081 which moved Py_BUILD_CORE code to Include/internal/.
It is also part of a larger project to cleanup the C API, see:
The change is backward compatible: #include <Python.h> will still provide exactly the same API.
- changed the title
[-]Move Py_LIMITED_API to Include/pycapi/[/-][+]Move !Py_LIMITED_API to Include/pycapi/[/+]on Nov 1, 2018 There are not just two sides. It is common to wrap new stable C API with something like:
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000
What will you do with this?
There are not just two sides. It is common to wrap new stable C API with something like:
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000
What will you do with this?objimpl.h always includes pycapi/pycapi_objimpl.h, so I don't think that we need a strong rules. I propose to always add move code using "#if ... Py_LIMITED_API" to the pycapi/ subdirectory, even if it uses "#if !defined(Py_LIMITED_API)".
Do you want to keep only stable ABI v.3.2 and move both newer stable API and non-stable API to the pycapi/ subdirectory? Sorry, I don't found a sense in this.
Do you want to keep only stable ABI v.3.2 and move both newer stable API and non-stable API to the pycapi/ subdirectory? Sorry, I don't found a sense in this.
The raw definition could be that Include/.h is part of the stable ABI, and Include/pycapi/.h are the definitions using Py_LIMITED_API and so can be stable or not stable depending on Py_LIMITED_API value :-)
To be honest, I'm not sure that I understand how "Py_LIMITED_API+0 >= 0x03050000" works and should be used.
I understand that you would prefer to leave PyObject_Calloc() in Include/objimpl.h. Honestly, I have no strong opinion on that. We can leave it there if you prefer.
--
Maybe the rule "move everything using Py_LIMITED_API to pycapi" is misleading.
My intent is that API in Include/*.h should not leak implementation details. It should be the starting point to design a new C API which does not leak any implementation detail:
http://pythoncapi.readthedocs.io/It's easier with an example:
#define _PyObject_GC_TRACK(o) do { \ PyGC_Head *g = _Py_AS_GC(o); \ if (g->_gc_next != 0) { \ Py_FatalError("GC object already tracked"); \ } \ assert((g->_gc_prev & _PyGC_PREV_MASK_COLLECTING) == 0); \ ...
This macro is private: it starts with "_Py", so it doesn't belong to Include/*.h. Moreover, it access private fields like PyGC_Head._gc_prev.
From my point of view, the ideal API would not access *any* structure field and PyGC_Header structure must not be used nor part of the C API.
--
After saying that, I looked again at my PR, and I still see private functions in objimpl.h. Example:
PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *); PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t); #define PyObject_New(type, typeobj) \ ( (type *) _PyObject_New(typeobj) ) #define PyObject_NewVar(type, typeobj, n) \ ( (type *) _PyObject_NewVar((typeobj), (n)) )
These functions are not excluded from Py_LIMITED_API. Since they are private, we are free to remove them whenever we want, so IMHO it's fine to exclude from Py_LIMITED_API right now if we want.
Another example:
static inline PyObject* PyObject_INIT(PyObject *op, PyTypeObject *typeobj) { assert(op != NULL); Py_TYPE(op) = typeobj; _Py_NewReference(op); return op; }
It's a public function but it calls the private function _Py_NewReference(). So _Py_NewReference() must be part of Py_LIMITED_API somehow...
In release mode (if Py_TRACE_REFS is not defined), _Py_NewReference() is defined like that:
/* Without Py_TRACE_REFS, there's little enough to do that we expand code inline. */ static inline void _Py_NewReference(PyObject *op) { if (_Py_tracemalloc_config.tracing) { _PyTraceMalloc_NewReference(op); } _Py_INC_TPALLOCS(op); _Py_INC_REFTOTAL; Py_REFCNT(op) = 1; }
It does access to the private _Py_tracemalloc_config variable and private macros/functions _Py_INC_TPALLOCS(op) and _Py_INC_REFTOTAL.
We *can* always define _Py_NewReference() as a function call if Py_LIMITED_API is defined, but it would have an impact on performance.
Right now, I don't want to risk to introduce a performance slowdown.
I have a "Proof-of-concept" implementation of my proposed "new C API":
https://github.com/pythoncapi/cpython/My implementation currently uses 3 defines:
- Py_NEWCAPI_NO_MACRO: replace macros with function calls PyTuple_GET_SIZE() becomes PyTuple_Size()
- Py_NEWCAPI_NO_STRUCT: must not use PyObject.ob_refcnt or any other field of Python object structures; structures should hide their fields: compilation error.
- Py_NEWCAPI: new C API without borrowed references, without macro, without struct
But this project is highly experimental and I don't want to make it upstream before we measured properly the impact on the performance, the API has been properly reviewed and discussed, and the overall project has been approved by core developers. For example, by writing a PEP :-)
--
In short, I'm not sure of what can or should be done right now for Include/pycapi/ :-)
I wrote the PR to open the discussion :-)
To be honest, I'm not sure that I understand how "Py_LIMITED_API+0 >= 0x03050000" works and should be used.
It's described here: https://docs.python.org/3/c-api/stable.html
If a stable ABI consumer just declares "#define PY_LIMITED_API 1", then they'll get the original stable ABI as defined in Python 3.2.
If they don't care about versions prior to 3.6, they can instead declare "#define PY_LIMITED_API 0x03060000", and get access to the functions added to the stable ABI in 3.3, 3.4, 3.5, and 3.6.
For this PR though, I think it's OK to ignore that detail, as once all the internal APIs are in "Include/internal", and all the APIs that don't offer ABI stability guarantees are in "Include/TBD" (see note below), then the general rule to follow is that everything added to the headers directly in "Include/" needs a Py_LIMITED_API guard that matches the upcoming release.
Note: I wrote "TBD" rather than "pycapi" above, as "pycapi" sounds like the name of a preferred public API to me, rather than "code compiled against this API is not portable to later versions, and may not be portable to other implementations". Given the name of the macro, "Include/unlimited/.h" may make sense, especially if those header files are all written to error out at compile time if PY_LIMITED_API is defined. "Include/unstable_abi/.h" would be another self-describing name.
On actually looking at the initial changes in the PR:
-
declarations that aren't part of the stable ABI in any version (i.e. "#ifndef PY_LIMITED_API", "#if !defined(PY_LIMITED_API)") should move to the new directory
-
declarations that are part of the stable ABI in *some* version should remain where they are (i.e. in "Include/*.h")
In your initial PR, the only API that subtle distinction affects is PyObject_Calloc (since that's a new addition to the stable ABI in 3.5+), and moving that back to the public header means you can add the desired "Py_LIMITED_API is not defined" check to the header in the new directory.
-
I created a new PR 10624:
- move "#ifndef Py_LIMITED_API" code to a new unstable/objimpl.h header file
- Include/unstable/ files (Include/unstable/objimpl.h) are no longer prefixed with "unstable_".
Include/unstable/ directory must not be added to the search paths for headers (gcc -I Include/unstable/). unstable/objimpl.h must not be included directly: it fails with a compiler error.
I propose the following organization:
- Include/*.h should be the "stable API"
- Include/unstable/.h is the "unstable API" (if Py_LIMITED_API is *not defined at all)
- Include/internal/pycore_*.h is the "internal" API
It should become easier to see what is exposed or not to the stable ABI just by looking at Include.*.h.
It should also become easier to spot in a review when a pull request something to the stable ABI, whereas it should be added to the unstable or internal API.
- changed the title
[-]Move !Py_LIMITED_API to Include/pycapi/[/-][+]Add a new Include/unstable/ subdirectory for the "unstable" API[/+]on Nov 20, 2018 Just to avoid the risk of name conflict, would it make sense to rename "unstable" to "pyunstable" or something else with "py" inside? I'm not sure if #include "unstable/objimpl.h" first looks the same directory than the header file that does the include?
Note: I tested "make install" and I get a /opt/py38/include/python3.8dm/unstable/ directory which contains a single file (yet): objimpl.h.
I think the rules for C includes are that
"path/header.h"looks next to the current file first, whereas<path/header.h>looks only in include directories.However, given your technique of mostly hiding the new directory name from API consumers, what do you think of calling the new directory "cpython" rather than "unstable"?
The idea there would be that the "unstable ABI" eventually become known as "the CPython C API" (since it exposes a lot of CPython implementation details", while the limited API could become known as "the portable cross-implementation Python C API".
(I know, I know, you were aiming to avoid further bikeshedding on the name, but "cpython" would namespace things nicely even if a compiler does something weird with header file lookups, and helps make it clearer to CPython contributors that we still need to care about public API stability in that directory, we just don't need to care about cross-implementation portability)
55 remaining items
- added a commit that references this issue
on Nov 23, 2022 - added 9 commits that reference this issue
on Oct 7, 2025 I merged 4 more changes. It seems like the separation between the limited C API and the "CPython" C API is now complete, project started in 2018 :-)
Reacted by Mickaël Schoentgen, Mikhail Efimov, Eric Snow and Sergey MiryanovAn important job well done!
Reacted by Victor Stinner
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs