Soft-deprecate parts of the undocumented C API

@encukou

Hi,

I've begun working on getting the C API documentation up to date: python/cpython#141004. Unfortunately, several parts of the undocumented C API aren't very helpful. For example, there's a bunch of undocumented Py*Iter_Type for the container classes. We don't really want users to start using these, but we can't just remove them either. @encukou and I believe that the best option is to soft-deprecate them.

The list of APIs is available below.

Please consider also

PyLong_BASE, PyLong_MASK and PyLong_SHIFT
Py_INFINITY/NAN
Py_MATH_PIl/Py_MATH_El could be just deprecated. Just one example I found on the Github was misuse of these macros.

Edit:
4. Py_RETURN_NAN/INF

I ran a code searchs on PyPI top 15,000 projects (at 2025-09-22).

Py*Iter types

It found a single project using 2 types: Nuitka (2.7.16) uses PySeqIter_Type and PyCallIter_Type.

Type objects for some builtin classes (PyEnum_Type, PyMap_Type, PyReversed_Type, PyZip_Type).

Regex: PyEnum_Type|PyMap_Type|PyReversed_Type|PyZip_Type

4 projects are using PyReversed_Type and PyEnum_Type:

Nuitka (2.7.16)
lazy_object_proxy (1.12.0)
newrelic (10.17.0)
wrapt (1.17.3)

Details

PYPI-2025-09-22/lazy_object_proxy-1.12.0.tar.gz: lazy_object_proxy-1.12.0/src/lazy_object_proxy/cext.c: return PyObject_CallFunctionObjArgs((PyObject *)&PyReversed_Type,
PYPI-2025-09-22/newrelic-10.17.0.tar.gz: newrelic-10.17.0/newrelic/packages/wrapt/_wrappers.c: return PyObject_CallFunctionObjArgs((PyObject *)&PyReversed_Type,
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/Serialization.py: return "(PyObject *)&PyEnum_Type"
PYPI-2025-09-22/wrapt-1.17.3.tar.gz: wrapt-1.17.3/src/wrapt/_wrappers.c: return PyObject_CallFunctionObjArgs((PyObject *)&PyReversed_Type,

PyRange* APIs and PyLongRangeIter_Type.

Regex: PyRange_Type|PyRangeIter_Type|PyRange_Check|PyLongRangeIter_Type.

4 projects are using PyRange_Type and PyRange_Check()

Nuitka (2.7.16)
datatable (1.1.0)
mypy (1.18.2)
mypy_dev-1.19.0a2

Details

PYPI-2025-09-22/datatable-1.1.0.tar.gz: datatable-1.1.0/src/core/python/obj.cc: bool _obj::is_range()         const noexcept { return v && PyRange_Check(v); }
PYPI-2025-09-22/datatable-1.1.0.tar.gz: datatable-1.1.0/src/core/python/range.cc: v = PyObject_CallObject(reinterpret_cast<PyObject*>(&PyRange_Type),
PYPI-2025-09-22/mypy-1.18.2.tar.gz: mypy-1.18.2/mypyc/primitives/misc_ops.py: load_address_op(name="builtins.range", type=object_rprimitive, src="PyRange_Type")
PYPI-2025-09-22/mypy-1.18.2.tar.gz: mypy-1.18.2/mypyc/test-data/irbuild-basic.test: r0 = load_address PyRange_Type
PYPI-2025-09-22/mypy_dev-1.19.0a2.tar.gz: mypy_dev-1.19.0a2/mypyc/primitives/misc_ops.py: load_address_op(name="builtins.range", type=object_rprimitive, src="PyRange_Type")
PYPI-2025-09-22/mypy_dev-1.19.0a2.tar.gz: mypy_dev-1.19.0a2/mypyc/test-data/irbuild-basic.test: r0 = load_address PyRange_Type
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/Serialization.py: return "(PyObject *)&PyRange_Type"
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/build/static_src/CompiledCodeHelpers.c: struct _rangeobject2 *result = (struct _rangeobject2 *)PyObject_New(struct _rangeobject2, &PyRange_Type);
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/build/static_src/CompiledCodeHelpers.c: struct _rangeobject3 *result = (struct _rangeobject3 *)PyObject_New(struct _rangeobject3, &PyRange_Type);
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/build/static_src/CompiledCodeHelpers.c: struct _rangeobject3 *result = (struct _rangeobject3 *)PyObject_New(struct _rangeobject3, &PyRange_Type);
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/build/static_src/HelpersDeepcopy.c: PyDict_SetItem(_deep_copy_dispatch, (PyObject *)&PyRange_Type, _deep_noop);
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/build/static_src/HelpersDeepcopy.c: PyBytes_CheckExact(value) || PyRange_Check(value) || PyType_Check(value) || PySlice_Check(value) ||
PYPI-2025-09-22/Nuitka-2.7.16.tar.gz: Nuitka-2.7.16/nuitka/build/static_src/HelpersDeepcopy.c: PyBool_Check(value) || PyRange_Check(value) || PySlice_Check(value) || PyCFunction_Check(value)) {

Soft-deprecate parts of the undocumented C API

If an API is unused and we don't want to document it, I would prefer to hard deprecate it, rather than soft deprecating it.

Py*Iter types.

We can deprecate PyODictIter_Type, PyByteArrayIter_Type, PyRangeIter_Type, PyLongRangeIter_Type, PySetIter_Type, PyTupleIter_Type, PyBytesIter_Type, PyListIter_Type, PyListRevIter_Type, PyUnicodeIter_Type.

(I would prefer to not deprecate PySeqIter_Type and PyCallIter_Type which are used by Nuitka.)

Type objects for some builtin classes (PyEnum_Type, PyMap_Type, PyReversed_Type, PyZip_Type).

We can deprecate PyMap_Type and PyZip_Type.

PyRange* APIs and PyLongRangeIter_Type.

We can deprecate PyRangeIter_Type and PyLongRangeIter_Type.

Anything under pyport.h.

Would you mind to elaborate? Many pyport.h macros are needed to implement the C API (in header files). For example, _Py_TYPEOF() is used by Py_CLEAR(), and _Py_NO_RETURN is used to defined Py_FatalError().

Internal type flags (Py_TPFLAGS_HAVE_VERSION_TAG, Py_TPFLAGS_INLINE_VALUES, Py_TPFLAGS_IS_ABSTRACT).

I don't think that we should deprecate them.

Macros exposing implementation details (there are a lot of these).

Please elaborate.

PyLong_FromPid & PyLong_AsPid, in favour of PyLong_AsNativeBytes with sizeof(pid_t)

Is it really an issue to keep these 2 functions?

PyLong_BASE, PyLong_MASK and PyLong_SHIFT

Python 3.14 just added PyLong_GetNativeLayout(). It might be early to deprecate these 3 macros.

Py_INFINITY/NAN

Py_NAN looks special, it has special code for Solaris.

For Py_INFINITY, it looks reasonable to use INFINITY instead.

Py_MATH_PIl/Py_MATH_El could be just deprecated. Just one example I found on the Github was misuse of these macros.

Ok to deprecate these two macros.

Nuitka copies our internal APIs for optimization's sake, so they're probably just copying the code for things like range.__new__ and similar. And, as long as it's a soft deprecation, Nuitka won't care anyway.

Would you mind to elaborate? Many pyport.h macros are needed to implement the C API (in header files). For example, _Py_TYPEOF() is used by Py_CLEAR(), and _Py_NO_RETURN is used to defined Py_FatalError().

Sorry, I meant deprecating anything under pyport.h with a Py prefix (implying they're public). For example, Py_ALIGNED, Py_MEMCPY, Py_CAN_START_THREADS.

I don't think that we should deprecate them.

Why not? Users shouldn't touch these.

Please elaborate.

Things such as PySet_MINSIZE expose implementation details.

@encukou

Sorry, I meant deprecating anything under pyport.h with a Py prefix (implying they're public). For example, Py_ALIGNED, Py_MEMCPY, Py_CAN_START_THREADS.

Py_MEMCPY() is already soft deprecated.

Py_ALIGNED() doesn't seem to be used by Python anymore: _Py_ALIGNED_DEF() is used instead. @encukou may have an opinion on this macro.

Py_CAN_START_THREADS looks useful.

I don't think that we should deprecate them.

Why not? Users shouldn't touch these.

Debuggers and profilers might want to inspect these flags (Py_TPFLAGS_HAVE_VERSION_TAG, Py_TPFLAGS_INLINE_VALUES, Py_TPFLAGS_IS_ABSTRACT). IMO it's too late to move them to the internal C API.

Things such as PySet_MINSIZE expose implementation details.

It's needed to define PySetObject structure. Since PySetObject is part of the public C API, I don't think that exposing PySet_MINSIZE is an issue.

Python 3.14 just added PyLong_GetNativeLayout(). It might be early to deprecate these 3 macros.

It depends. Now PEP 387 recommends 5 years before removal, i.e. something like 3.20. Though, I suggest do soft-deprecation.

Py_NAN looks special, it has special code for Solaris.

It seems this is for system, which doesn't fit in build requirements. See python/cpython#141004 for details. Do we want to relax build requirements?

I've updated the list based on Victor's feedback.

@encukou

Soft-deprecate parts of the undocumented C API

If an API is unused and we don't want to document it, I would prefer to hard deprecate it, rather than soft deprecating it.

Soft-deprecation requires writing docs, so, yes, we'd need to do that.
Though I'd still generally prefer documenting APIs as part of normal deprecation, especially if we know use cases of the deprecated API. When people get deprecation warnings they should go search the docs for advice.

PyLong_FromPid & PyLong_AsPid, in favour of PyLong_AsNativeBytes with sizeof(pid_t)

Is it really an issue to keep these 2 functions?

Definitely keep them, but I think soft-deprecation is a good way to document that we don't want to add similar new API, and that there's a replacement. Basically, avoid people pointing to it as a precedent.

Py_ALIGNED() doesn't seem to be used by Python anymore: _Py_ALIGNED_DEF() is used instead. @encukou may have an opinion on this macro.

Soft-deprecation is the way to go here. Alignment control is in C now; you should use that directly if you (unlike Python) can.

Things such as PySet_MINSIZE expose implementation details.

It's needed to define PySetObject structure. Since PySetObject is part of the public C API, I don't think that exposing PySet_MINSIZE is an issue.

We can use the _PySet_MINSIZE name for that.

please consider soft deprecating the following macros defined in cpython/pyctype.h:

Py_ISALNUM ...

Why do you think these should be soft deprecated ?

Those macros provide a good way to get locale independent Latin-1 character type information. The C lib variants usually are locale aware, which is not always what you want when parsing text based on a fixed set of rules.

It does not seem like something for CPython to provide, it should be a private internal (we actually call it „internal” in the docs already) utility IMO.

It is useful for Python C extensions and why should they have to reimplement something we already have (and need) in CPython ?

It's more of a future-proofing case. If we exposed everything that CPython needed, we'd be stuck maintaining all sorts of APIs that aren't directly related to Python. Soft deprecation would allow us to reject requests for features or fixes, easing the maintenance burden.

That said, I'm not opposed to keeping them, especially if there are extensions out there that actually use them. I had Stan bring it up here because they jumped out at me as something CPython doesn't need to provide.

we'd be stuck maintaining all sorts of APIs that aren't directly related to Python.

Is cost is high in the given case?

Py_RETURN_NAN/INF macros aren't used in the CPython (and, probably, never be).

Are we getting close to a list to vote on?

Yes, I'll compile a full list sometime in the next few days.

Here's the full list of things that I'd like to soft deprecate:

PyFile_NewStdPrinter: Only used by Python initialization and doesn't have much use to anyone other than us.
PyStdPrinter_Type: Same as above.
PyAsyncGenASend_CheckExact: Comes with no other APIs and exposes a CPython implementation detail.
PyDescr_COMMON: Exposes descriptor object internals.
PyWrapperFlag_KEYWORDS: No other APIs and exposes descriptor object internals.
PySet_MINSIZE: Exposes set object internals.
PyLong_SHIFT: Obsolete since PyLong_GetNativeLayout.
PyLong_BASE: Same as above, also probably not useful.
PyLong_MASK: Same as above.
PyException_HEAD: Exposes exception object internals.
~~Py_INVALID_SIZE: Same as -1, not very useful and not used anywhere in CPython.~~ See below.
Py_TPFLAGS_HAVE_VERSION_TAG: Exposes version tags, which are an implementation detail.
~~All PyDTrace* APIs: Not useful outside of CPython.~~

I'm fine with soft deprecating these APIs, except of PyDTrace.

All PyDTrace* APIs: Not useful outside of CPython.

I would prefer to keep these functions (not deprecate them). They are used by SystemTap.

These functions are not part of the Python C API: Python.h doesn't include pydtrace.h. You have to include explicitly pydtrace.h.

Here's the full list of things that I'd like to soft deprecate

So you don't want to soft deprecate the other APIs, such as PySetIter_Type or PyErr_RangedSyntaxLocationObject()?

I would prefer to keep these functions (not deprecate them). They are used by SystemTap.

For users who are profiling with SystemTap, they still won't ever need these functions. They're merely wrappers that CPython uses to compile DTrace calls away when configured without --with-dtrace.

So you don't want to soft deprecate the other APIs, such as PySetIter_Type or PyErr_RangedSyntaxLocationObject()?

The *Iter_Type APIs don't have a great replacement, so I don't see much benefit in deprecating them. I can add it to the list if others want them deprecated.

PyErr_RangedSyntaxLocationObject is useful; that was my mistake. Despite the name, PyErr_SyntaxLocationObjectEx is not actually exposed in our C API, so PyErr_RangedSyntaxLocationObject provides actual functionality.

Could you humour me and keep Py_INVALID_SIZE fully supported, preferably documented together with PY_SSIZE_T_MIN and PY_SSIZE_T_MAX?

The Py_ssize_t type is weird -- it's a widely used type that doesn't come from the C standard, and CPython currently requires it to do more than whan POSIX guarantees for ssize_t.
We don't support systems with weird signed size types, and porting CPython to such a system would be hard. But we've seen hard platforms added (hello WASM!). If such a platform does come up, then I'd accept PRs that change -1's to the constant.

I'm not familiar with DTrace; others look good to soft-deprecate (and for Py_OMIT_LEGACY_API if PEP 743 is accepted). *Iter_Type can be another vote.

What is Py_INVALID_SIZE use case?

It's just a replacement for -1 with Py_ssize_t. I think the future-proofing case is enough to warrant keeping it in.

@encukou

Soft-deprecate the bulk of the proposed APIs: PyFile_NewStdPrinter, PyStdPrinter_Type, PyAsyncGenASend_CheckExact, PyDescr_COMMON, PyWrapperFlag_KEYWORDS, PySet_MINSIZE, PyLong_SHIFT, PyLong_BASE, PyLong_MASK, PyException_HEAD, Py_TPFLAGS_HAVE_VERSION_TAG:

(if anything needs more discussion, it'll be split out and get a separate vote later)

Soft-deprecate Py_INVALID_SIZE:

Soft-deprecate PyDTrace* APIs:

@erlend-aasland

Ping @erlend-aasland @serhiy-storchaka

@serhiy-storchaka

ping @serhiy-storchaka

@serhiy-storchaka

ping @serhiy-storchaka

@serhiy-storchaka

@serhiy-storchaka counts as abstaining due to inactivity.
The APIs (except Py_INVALID_SIZE & PyDTrace*) should be soft-deprecated.

Navigation Menu

Repository navigation

Soft-deprecate parts of the undocumented C API #86

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Soft-deprecate parts of the undocumented C API #86

Description

Activity

skirpichev commented on Nov 5, 2025

vstinner commented on Nov 5, 2025

vstinner commented on Nov 5, 2025

vstinner commented on Nov 5, 2025

ZeroIntensity commented on Nov 5, 2025

vstinner commented on Nov 5, 2025

skirpichev commented on Nov 5, 2025

ZeroIntensity commented on Nov 5, 2025

encukou commented on Nov 6, 2025

StanFromIreland commented on Nov 8, 2025

malemburg commented on Nov 9, 2025

StanFromIreland commented on Nov 9, 2025

malemburg commented on Nov 9, 2025

ZeroIntensity commented on Nov 9, 2025

skirpichev commented on Nov 10, 2025

10 remaining items

encukou commented on Nov 24, 2025

ZeroIntensity commented on Nov 24, 2025

ZeroIntensity commented on Nov 26, 2025

vstinner commented on Nov 26, 2025

ZeroIntensity commented on Nov 26, 2025

encukou commented on Nov 27, 2025

vstinner commented on Nov 27, 2025

ZeroIntensity commented on Nov 27, 2025

encukou commented on Dec 1, 2025

vstinner commented on Dec 10, 2025

encukou commented on Dec 17, 2025

vstinner commented on Jan 6, 2026

encukou commented on Jan 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions