Skip to content
Prev Previous commit
Next Next commit
PEP 630: Remove spurious heading level and conform heading breaks/chars
  • Loading branch information
CAM-Gerlach committed Apr 15, 2022
commit 778e9b10e2c978293441fe29347fa8660d38deee
75 changes: 44 additions & 31 deletions pep-0630.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,8 @@ Created: 25-Aug-2020
Post-History: 16-Jul-2020


Isolating Extension Modules
===========================

Abstract
--------
========

Traditionally, state of Python extension modules was kept in C
``static`` variables, which have process-wide scope. This document
Expand All @@ -25,8 +22,9 @@ possible. The switch involves allocating space for that state, potentially
switching from static types to heap types, and—perhaps most
importantly—accessing per-module state from code.


About this document
-------------------
===================

As an :pep:`informational PEP <1#pep-types>`,
this document does not introduce any changes: those should be done in
Expand Down Expand Up @@ -64,8 +62,9 @@ specific to CPython.
As with any Informational PEP, this text does not necessarily represent
a Python community consensus or recommendation.


Motivation
----------
==========

An *interpreter* is the context in which Python code runs. It contains
configuration (e.g. the import path) and runtime state (e.g. the set of
Expand Down Expand Up @@ -96,8 +95,9 @@ Unfortunately, *per-interpreter* state is not easy to achieve: extension
authors tend to not keep multiple interpreters in mind when developing,
and it is currently cumbersome to test the behavior.


Rationale for Per-module State
------------------------------
==============================

Instead of focusing on per-interpreter state, Python's C API is evolving
to better support the more granular *per-module* state. By default,
Expand All @@ -112,8 +112,9 @@ module object is created, and clean up when it's freed. In this regard,
a module is just like any other ``PyObject *``; there are no “on
interpreter shutdown” hooks to think about—or forget about.


Goal: Easy-to-use Module State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
------------------------------

It is currently cumbersome or impossible to do everything the C API
offers while keeping modules isolated. Enabled by :pep:`384`, changes in
Expand All @@ -128,16 +129,18 @@ per-thread or per-task state. The goal is to treat these as exceptional
cases: they should be possible, but extension authors will need to
think more carefully about them.


Non-goals: Speedups and the GIL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------------------------

There is some effort to speed up CPython on multi-core CPUs by making the GIL
per-interpreter. While isolating interpreters helps that effort,
defaulting to per-module state will be beneficial even if no speedup is
achieved, as it makes supporting multiple interpreters safer by default.


How to make modules safe with multiple interpreters
---------------------------------------------------
===================================================

There are many ways to correctly support multiple interpreters in
extension modules. The rest of this text describes the preferred way to
Expand All @@ -153,7 +156,7 @@ This section assumes that “*you*” are an extension module author.


Isolated Module Objects
~~~~~~~~~~~~~~~~~~~~~~~
-----------------------

The key point to keep in mind when developing an extension module is
that several module objects can be created from a single shared library.
Expand All @@ -179,8 +182,9 @@ While some modules could do with less stringent restrictions, isolated
modules make it easier to set clear expectations (and guidelines) that
work across a variety of use cases.


Surprising Edge Cases
~~~~~~~~~~~~~~~~~~~~~
---------------------

Note that isolated modules do create some surprising edge cases. Most
notably, each module object will typically not share its classes and
Expand All @@ -194,7 +198,7 @@ separate objects. In the following code, the exception is *not* caught::
... old_binascii.unhexlify(b'qwertyuiop')
... except binascii.Error:
... print('boo')
...
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
binascii.Error: Non-hexadecimal digit found
Expand All @@ -206,8 +210,9 @@ The goal is to make extension modules safe at the C level, not to make
hacks behave intuitively. Mutating ``sys.modules`` “manually” counts
as a hack.


Managing Global State
~~~~~~~~~~~~~~~~~~~~~
---------------------

Sometimes, state of a Python module is not specific to that module, but
to the entire process (or something else “more global” than a module).
Expand All @@ -229,8 +234,9 @@ avoid issues with multiple interpreters is to explicitly prevent a
module from being loaded more than once per process—see “Opt-Out:
Limiting to One Module Object per Process” below.


Managing Per-Module State
~~~~~~~~~~~~~~~~~~~~~~~~~
-------------------------

To use per-module state, use `multi-phase extension module
initialization <https://docs.python.org/3/c-api/module.html#multi-phase-initialization>`__
Expand Down Expand Up @@ -265,7 +271,7 @@ example module initialization shown at the bottom of the file.


Opt-Out: Limiting to One Module Object per Process
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------------------------------------------------

A non-negative ``PyModuleDef.m_size`` signals that a module supports
multiple interpreters correctly. If this is not yet the case for your
Expand All @@ -286,8 +292,9 @@ process. For example::
// ... rest of initialization
}


Module State Access from Functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
----------------------------------

Accessing the state from module-level functions is straightforward.
Functions get the module object as their first argument; for extracting
Expand All @@ -308,8 +315,9 @@ exception if there is no module state, i.e. ``PyModuleDef.m_size`` was
zero. In your own module, you're in control of ``m_size``, so this is
easy to prevent.)


Heap types
----------
==========

Traditionally, types defined in C code are *static*, that is,
``static PyTypeObject`` structures defined directly in code and
Expand Down Expand Up @@ -347,7 +355,7 @@ or inherited slots). Always test the details that are important to you.


Defining Heap Types
~~~~~~~~~~~~~~~~~~~
-------------------

Heap types can be created by filling a ``PyType_Spec`` structure, a
description or “blueprint” of a class, and calling
Expand All @@ -364,7 +372,7 @@ Python code).


Garbage Collection Protocol
~~~~~~~~~~~~~~~~~~~~~~~~~~~
---------------------------

Instances of heap types hold a reference to their type.
This ensures that the type isn't destroyed before its instance,
Expand Down Expand Up @@ -403,7 +411,7 @@ and ``tp_clear``.


Module State Access from Classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--------------------------------

If you have a type object defined with ``PyType_FromModuleAndSpec()``,
you can call ``PyType_GetModule`` to get the associated module, then
Expand All @@ -417,8 +425,9 @@ these two steps with ``PyType_GetModuleState``, resulting in::
return NULL;
}


Module State Access from Regular Methods
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
----------------------------------------

Accessing the module-level state from methods of a class is somewhat
more complicated, but possible thanks to changes introduced in :pep:`573`.
Expand All @@ -445,7 +454,6 @@ that subclass, which may be defined in different module than yours.
class Sub(Base):
pass


For a method to get its “defining class”, it must use the
``METH_METHOD | METH_FASTCALL | METH_KEYWORDS`` `calling
convention <https://docs.python.org/3.9/c-api/structures.html?highlight=meth_o#c.PyMethodDef>`__
Expand Down Expand Up @@ -488,8 +496,9 @@ For example::
{NULL},
}


Module State Access from Slot Methods, Getters and Setters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
----------------------------------------------------------

.. note::

Expand Down Expand Up @@ -535,7 +544,7 @@ module.


Lifetime of the Module State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
----------------------------

When a module object is garbage-collected, its module state is freed.
For each pointer to (a part of) the module state, you must hold a reference
Expand All @@ -550,39 +559,43 @@ libraries.


Open Issues
-----------
===========

Several issues around per-module state and heap types are still open.

Discussions about improving the situation are best held on the `capi-sig
mailing list <https://mail.python.org/mailman3/lists/capi-sig.python.org/>`__.


Type Checking
~~~~~~~~~~~~~
-------------

Currently (as of Python 3.10), heap types have no good API to write
``Py*_Check`` functions (like ``PyUnicode_Check`` exists for ``str``, a
static type), and so it is not easy to ensure whether instances have a
particular C layout.


Metaclasses
~~~~~~~~~~~
-----------

Currently (as of Python 3.10), there is no good API to specify the
*metaclass* of a heap type, that is, the ``ob_type`` field of the type
object.


Per-Class scope
~~~~~~~~~~~~~~~
---------------

It is also not possible to attach state to *types*. While
``PyHeapTypeObject`` is a variable-size object (``PyVarObject``),
its variable-size storage is currently consumed by slots. Fixing this
is complicated by the fact that several classes in an inheritance
hierarchy may need to reserve some state.


Lossless conversion to heap types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---------------------------------

The heap type API was not designed for “lossless” conversion from static types,
that is, creating a type that works exactly like a given static type.
Expand All @@ -591,7 +604,7 @@ known “gotchas”.


Copyright
---------
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.