Repository navigation
ENH: Please support subinterpreters #24755
Description
Activity
- changed the title
[-]Support for subinterpreters[/-][+]ENH: Please support subinterpreters[/+]on Sep 21, 2023 PEP 554 states:
To mitigate that impact and accelerate compatibility, we will do the following:
- be clear that extension modules are not required to support use in multiple interpreters
- raise ImportError when an incompatible module is imported in a subinterpreter
- provide resources (e.g. docs) to help maintainers reach compatibility
- reach out to the maintainers of Cython and of the most used extension modules (on PyPI) to get feedback and possibly provide assistance
The PEP also links to Isolating Extensions which has a lot of theory, but does not clearly state how to migrate a large existing c-extension library like NumPy to support subinterpreters. I think we would need to:
- move to HeapTypes
- move all static state into module state
- carefully analyze code for possible shared state.
I am a bit unclear whether subinterpreters share a single GIL, if not we would also have to carefully examine the code for possible race conditions.
This is a lot of work, and may have performance implications. What is your use case for subinterpreters? Do you think you could help with the effort or find funding for this effort?
Relevant mailing list threads and issues:
This is a lot of work, and may have performance implications. What is your use case for subinterpreters? Do you think you could help with the effort or find funding for this effort?
I suspect the vast majority of changes to be relatively easy, but there is still the same problem that we need someone to explicitly dedicate time on this, and I doubt it will be one of the current core devs.
We even added a warning a long time back saying exactly that, but it seems CPython changes to makesubinterpretersupport better in the long-run now enforces an error rather than a warning.PEP 554 states: …
FWIW the recent CPython changes should be from PEP 684 “Per-Interpreter GIL”; PEP 554, for the Python API and subinterpreter management features, is still in draft status.
There's a very strong use case for subinterpreters since PEP 684 for parallel processing that I expect would be useful to a lot of numpy client code: using subinterpreters in separate threads will enable shared memory (at least in a read-only case) with significantly less hassle than multiprocessing.
Reacted by AlexisM, Danny Kim, Stella Laurenzo and Mike R.Reacted by a-reichI’m also very excited about the potential opportunities of using subinterpreters with numpy, and agree with what @mdekstrand said. In particular, the latest draft of PEP 734 discusses sharing data via the buffer protocol (and already implemented in the private interpreters module since 3.13a1). Since ndarrays can export their buffer or be created from one without copies, this could be a very nice pattern:
- pickle your array with protocol 5 to get some serialized metadata plus the memoryview,
- pass that view to a bunch of interpreters (which is basically instant) as well as the small metadata,
- and unpickle: now all of them are sharing the data in each of their arrays
- And if you don’t want to worry about data races, seems like np can handle that by setting the readonly flag.
You get concurrency with performant, opt-in data sharing, without the hassles of managing subprocesses and using multiprocessing.shared_memory where you have to create a shared buffer of fixed size ahead of time and only create arrays using that. With interpreters you can take any random array you got and easily share it.
Reacted by Donghyun-EditInterpreterPoolExecutor's are to be introduced in 3.14 124548.
At present, numpy imports fail due to "ImportError: module numpy._core._multiarray_umath does not support loading in subinterpreters":
See following example: TPE and PPE work, IPE does not.
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ProcessPoolExecutor from interpreters_backport.concurrent.futures.interpreter import InterpreterPoolExecutor def try_tpe(): with ThreadPoolExecutor() as executor: executor.submit(exec, "import numpy as np;print('TPE:', np.random.rand(2))") def try_ppe(): with ProcessPoolExecutor() as executor: executor.submit(exec, "import numpy as np;print('PPE:', np.random.rand(2))") def try_ipe(): with InterpreterPoolExecutor() as executor: f = executor.submit(exec, "import numpy as np;print('IPE:', np.random.rand(2))") f.result() if __name__ == "__main__": try_tpe() try_ppe() try_ipe()
Footnote: I noted this recent comment, which perhaps closed the door on hope here; #27192 (comment)
It's a sad thing that even the possibilities are closed. Now Python is heading towards true multithreaded parallelism with
InterpreterPoolExecutorand Numpy will not be able to deal with new demands..Reacted by Lilo HuangIt's a sad thing that even the possibilities are closed. Now Python is heading towards true multithreaded parallelism with
InterpreterPoolExecutorand Numpy will not be able to deal with new demands..They aren't closed. A lot of work has happened (and is still happening) to supported free-threaded CPython over the past 6 months. Pretty much all that work is also directly relevant to support for subinterpreters. The extra work needed for subinterpreters - primarily moving to heap types I believe - will need someone to dig in and do that work though. None of the active maintainers are working on this, however contributions are very much welcome.
Reacted by AlexisMReacted by Danny Kim, Amit Singh and Julianus PfeufferThe list of tasks is still this, although the documentation has gotten better:
- move all static types to HeapTypes (i.e. search for
static PyTypeObjectand replace that withPyType_FromModuleAndSpec()or some other heap allocation of the type. This can be done one type at a time, and must be benchmarked for performance implications. - move all static state into module state. Much of the groundwork for this has been done in the free-threading code fixes. Again, better to go in small increments and to benchmark for performance implications.
- carefully analyze code for other possible shared state.
- move all static types to HeapTypes (i.e. search for
@mattip I imagine cython support for subinterpreters would also be required? cython/cython#6445
Yes, that is part of the "carefully analyze code for other possible shared state". Cython does support the constructs needed for generating code compatible with subinterpreters, the question is more "are we using unsafe coding practices in our cython code?"
Another area that will need adaptation is the f2py generated wrappers.
The list of tasks is still this, although the documentation has gotten better:
* move all [static types to HeapTypes](https://docs.python.org/3/howto/isolating-extensions.html#changing-static-types-to-heap-types) (i.e. search for `static PyTypeObject` and replace that with [`PyType_FromModuleAndSpec()`](https://docs.python.org/3/c-api/type.html#c.PyType_FromModuleAndSpec) or some other heap allocation of the type. This can be done one type at a time, and must be benchmarked for performance implications. * move all [static state into module state](https://docs.python.org/3/howto/isolating-extensions.html#module-state-access-from-functions). Much of the groundwork for this has been done in the free-threading code fixes. Again, better to go in small increments and to benchmark for performance implications. * carefully analyze code for other possible shared state.With that it is then theoretically possible to compile numpy with
Py_LIMITED_API=[hex value for 3.14 here]But then one cannot use the free-threaded define yet with the limited API (😭).In fact, I think the day when both defines are allowed it would be the best day for Numpy as then single builds from that version on could be built with that version minimally with both of those defines and work out-of-box in all the future versions of CPython provided they do not extend the size of the
PyObjectstructure and cause a warning to display when loading the C extensions.Reacted by Danny KimWith that it is then theoretically possible to compile numpy with Py_LIMITED_API=[hex value for 3.14 here]
I don't think that is correct or necessary. Neither is a possible change of the struct any issue at all for NumPy as we don't use the limited API.
Just wanted to drop a reference to python/cpython#138045 here. This is a more subtle issue: Due to a Python 3.13 regression, numpy currently crashes on load inside a subinterpreter, even if that subinterpreter is the only one loading numpy and it is not using full isolation (that's the
.check_multi_interp_extensions = 1from OP, which would be 0 in this case).I believe this previously worked, as long as only a single subinterpreter ever imports numpy.
I also believe the crash/badness happens even if you configure things like OP, requesting full isolation. That would previously fail cleanly with an exception, but now causes interpreter state corruption.
- added a commit that references this issue
on May 9, 2026 - added a commit that references this issue
on May 20, 2026
Proposed new feature or change:
Version 1.25.1, Python 3.12.re02
After enabling interpreters in Python C Api:
PyInterpreterConfig config = {
.check_multi_interp_extensions = 1,
.gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
if (PyStatus_Exception(status)) {
return -1;
}
Import numpy throws an exception:
module numpy.core._multiarray._umath does not support loading in subinterpreters