Repository navigation
Disable importing legacy single-phase init extensions within subinterpreters in --disable-gil build #117649
Description
Feature or enhancement
When importing a single-phase init extension in a subinterpreter, Python will make a shallow copy of the module's dictionary, which can share (non-immortal) objects between interpreters.
This does not work properly in the --disable-gil build, and we should disable it by raising an ImportError, at least for now. We can investigate how to support this in the future.
There are currently some unit tests for this case. Those tests pass, but that's mostly because they are simple and small changes to things like the GC will cause them to crash.
The underlying problems are not directly related to the GIL, but rather because the GC and our mimalloc integration in the --disable-gil build assume that non-immortal objects are isolated by interpreter:
- The GC assumes that all tracked objects reachable via
tp_traverseare also reachable from the per-interpreter mimalloc heaps. Violating this assumption can cause flags to be set to an inconsistent state. - The mimalloc pool of abandoned segments is per-interpreter. If a non-immortal object outlives its creating interpreter, this can cause use-after-free problems.
Activity
ericsnowcurrently commentedon Apr 8, 2024
Note that this only relates to legacy subinterpreters (not isolated, created by
Py_NewInterpreter()). With isolated interpreters single-phase init modules already triggerImportError.Raising
ImportErrorunder--disable-gilseems like a reasonable short-term solution.[-]Disable importing single-phase init extensions in subinterpreters in `--disable-gil` build[/-][+]Disable importing legacy single-phase init extensions within subinterpreters in `--disable-gil` build[/+]pythongh-117649: Raise ImportError for unsupported modules in free-th…
pythongh-117649: Raise ImportError for unsupported modules in free-th…
encukou commentedon Apr 9, 2024
You might want use the current mechanism: require
PyInterpreterConfig.check_multi_interp_extensionsto be 1 forPy_NewInterpreterFromConfig, and set it to 1 forPy_NewInterpreter(i.e._PyInterpreterConfig_LEGACY_INIT).And perhaps disable
_imp._override_multi_interp_extensions_check.Note that there are two ways to implement the limitation:
The first was easier to implement; we might want to switch to the second later. If/when we do,
--disable-gilbuilds might want to follow along.Merge branch 'main' into pythongh-117649-singlephase-subinterp-nogil
gh-117649: Raise ImportError for unsupported modules in free-threaded…
pythongh-117649: Fix file descriptor leak in (expected) failing test …
gh-117649: Fix file descriptor leak in (expected) failing test case (#…
pythongh-117649: Raise ImportError for unsupported modules in free-th…
pythongh-117649: Fix file descriptor leak in (expected) failing test …