Repository navigation
Python 3.14 free-threaded support #2721
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 36 commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
1d93b30
Thread-safety prep for free-threading builds
greateggsgreg e029137
Initialise pythonnet on free-threaded Python (#2720)
greateggsgreg fd56aa4
Make extension/CLR-object registries thread-safe
greateggsgreg b727a15
Atomic type creation in ReflectedClrType.GetOrCreate / TypeManager.Ge…
greateggsgreg debba0d
Add free-threaded thread-stress tests and 3.14t to CI matrix
greateggsgreg 585916e
Atomic GCHandle ownership and finalizer-thread shutdown guards
greateggsgreg c0ddb29
Make additional internal registries thread-safe
greateggsgreg 096e466
test_thread: join worker threads before returning
greateggsgreg 5005f36
test_thread: cover ModuleObject thread-safe registries
greateggsgreg 3ad586a
Wider thread-safety audit fixes for free-threaded Python
greateggsgreg 6b1f3f4
Document lock acquisition sites and strong->weak GCHandle swap
greateggsgreg 732d97e
Preserve InternString single-write invariant under DEBUG
greateggsgreg 3d56611
test_thread: cover real-world consumer patterns
greateggsgreg 7094b12
Auto-detect free-threaded libpython in venv home
greateggsgreg f49ed1b
Snapshot pypath, use ConcurrentDictionary for thunks and slot holders
greateggsgreg fa89ff0
Fix handling of python runtime suffixes m/t
greateggsgreg a373e79
Fix threadtest race
greateggsgreg 43a4237
Fix double-free in chained ClassDerived Finalize
greateggsgreg b42377e
Enable Mono CI jobs on free-threaded Python 3.14
greateggsgreg 526a297
Inline freethreaded_only as pytest.mark.skipif at call sites
greateggsgreg 8485837
Fix InterruptTest assertion on free-threaded Python 3.14
greateggsgreg a459772
Add concurrent stress tests for PyBuffer.Dispose and CLR-cycle gc.col…
greateggsgreg 3bf6d68
Trim concurrent overhead on hot paths from free-threading prep
greateggsgreg 3433446
Pre-warm ctor binder in concurrent-gc test to avoid first-call race
greateggsgreg 4a034f7
Make MethodBinder.GetMethods lazy init thread-safe under free-threading
greateggsgreg afd5799
Precompute method precedence to avoid quadratic GetParameters allocat…
greateggsgreg 95e4811
Zero the slot in ClassDerived.tp_dealloc when tp_clear already ran to…
greateggsgreg a799c44
Keep ClassDerived wrapper alive across the NewObjectToPython slot dem…
greateggsgreg 220b4b4
Document private helpers added during free-threading prep
greateggsgreg dfda807
Add debug echoes and a 6-minute step timeout to the Mono test job
greateggsgreg 1130f6a
Drop the per-loop CLR GC.Collect from concurrent-gc test to avoid Mon…
greateggsgreg f75cbab
Add temporary Mono-step diagnostics on Linux/macOS to locate the x64-…
greateggsgreg 64ea342
Revert temporary Mono-step diagnostics now that the underlying race i…
greateggsgreg fda8211
Add user-facing threading guide covering GIL, free-threading, and com…
greateggsgreg 356d76f
Harden CollectBasicObject against .NET-GC timing differences
greateggsgreg f0ee4b2
Adjust the header offset on 32bit systems
filmor 3445b08
Drop broken and unnecessary exclude
filmor 8e6861e
Be strict about not loading on Python 3.13
filmor File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,214 @@ | ||
| Threading | ||
| ========= | ||
|
|
||
| This page explains how Python.NET interacts with the Python Global Interpreter | ||
| Lock (GIL) and with managed threads, and what guarantees the runtime makes | ||
| when your code is multi-threaded. It covers both classic CPython builds and | ||
| the free-threaded build introduced in CPython 3.13 (``Py_GIL_DISABLED``). | ||
|
|
||
| The model in one paragraph | ||
| -------------------------- | ||
|
|
||
| Python.NET embeds CPython, so every interaction with a Python object — | ||
| including reading a ``PyObject``'s attributes, calling a Python callable, | ||
| constructing a Python value, or letting a ``PyObject`` go out of scope — must | ||
| happen while the calling thread is *attached* to the interpreter. On a | ||
| classic (GIL-enabled) CPython build "attached" means "holds the GIL"; on a | ||
| free-threaded build it means "has an active thread state". In both cases the | ||
| attachment API is the same: ``Py.GIL()`` on the C# side and | ||
| ``threading.Thread`` / ``_thread`` on the Python side. Forgetting to attach | ||
| will crash the process or corrupt memory. | ||
|
|
||
| Acquiring the GIL from C# | ||
| ------------------------- | ||
|
|
||
| When .NET code calls into Python it must hold the GIL. Use the ``Py.GIL()`` | ||
| disposable to acquire and release it:: | ||
|
|
||
| using (Py.GIL()) | ||
| { | ||
| dynamic np = Py.Import("numpy"); | ||
| var arr = np.array(new[] { 1, 2, 3 }); | ||
| // ... interact with arr ... | ||
| } | ||
|
|
||
| ``Py.GIL()`` is re-entrant: nesting calls on the same thread is harmless and | ||
| cheap. Always pair acquisition with disposal — the ``using`` form does this | ||
| automatically, and you must release the GIL on the same thread that acquired | ||
| it. | ||
|
|
||
| If you need a Python object to outlive the ``using`` block, copy what you | ||
| need (e.g. ``.As<int[]>()`` or ``new PyObject(value)``) before releasing the | ||
| GIL. | ||
|
|
||
| Releasing the GIL for long-running .NET work | ||
| -------------------------------------------- | ||
|
|
||
| If a managed call holds the GIL but then does long-running work that does not | ||
| touch Python (heavy CPU, blocking I/O, native interop), release the GIL so | ||
| other Python threads can run:: | ||
|
|
||
| IntPtr threadState = PythonEngine.BeginAllowThreads(); | ||
| try | ||
| { | ||
| DoCpuHeavyWork(); // safe: no Python C API calls | ||
| } | ||
| finally | ||
| { | ||
| PythonEngine.EndAllowThreads(threadState); | ||
| } | ||
|
|
||
| Inside the ``BeginAllowThreads``/``EndAllowThreads`` block you must not touch | ||
| any Python object. If you need to talk to Python from worker threads spawned | ||
| in this region, those threads must acquire the GIL themselves with | ||
| ``Py.GIL()``. | ||
|
|
||
| Calling .NET from Python threads | ||
| -------------------------------- | ||
|
|
||
| Calling a managed method from a Python ``threading.Thread`` works | ||
| transparently — Python.NET handles GIL acquisition/release around the | ||
| managed call. The managed code sees the GIL held on entry and is free to | ||
| release it via ``BeginAllowThreads`` if it does its own blocking work. | ||
|
|
||
| Calling Python from CLR threads | ||
| ------------------------------- | ||
|
|
||
| A CLR thread that was *not* spawned by Python (a thread-pool task, a | ||
| ``Thread`` started in C#, an ``async`` continuation that resumed on a | ||
| different thread, etc.) must acquire the GIL before touching any | ||
| ``PyObject``:: | ||
|
|
||
| Task.Run(() => | ||
| { | ||
| using (Py.GIL()) | ||
| { | ||
| // safe to use PyObjects here | ||
| } | ||
| }); | ||
|
|
||
| Forgetting this is the most common pythonnet threading bug. Symptoms range | ||
| from immediate segfaults to subtle refcount corruption that crashes much | ||
| later. | ||
|
|
||
| Reference counting and finalizers | ||
| --------------------------------- | ||
|
|
||
| ``PyObject`` follows the .NET ``IDisposable`` pattern. ``Dispose()`` (or the | ||
| end of a ``using`` block) drops the underlying Python reference; the GC | ||
| finalizer queues the same release for the next time Python.NET is on the GIL. | ||
|
|
||
| Two practical consequences: | ||
|
|
||
| * **Don't share a single ``PyObject`` instance across threads without | ||
| serialising access.** ``PyObject`` is not internally locked. If multiple | ||
| threads concurrently dispose the same instance, the underlying refcount can | ||
| go negative. | ||
|
|
||
| * **Don't rely on the GC finalizer running promptly.** The PyObject is only | ||
| freed when a Python.NET API later reacquires the GIL. If your application | ||
| shuts down without that happening, finalizable PyObjects can be reported as | ||
| leaked. | ||
|
|
||
| Free-threaded Python (PEP 703) | ||
| ------------------------------ | ||
|
|
||
| Starting with the free-threaded CPython 3.13+ build (``Py_GIL_DISABLED``), | ||
| the GIL is no longer the serialisation point for Python C API calls. | ||
| Python.NET is tested against the ``3.14t`` (free-threaded) interpreter and | ||
| behaves as follows under that build: | ||
|
|
||
| * ``Py.GIL()`` still acquires a thread state. It is functionally a no-op | ||
| for mutual exclusion but is still required for thread-state attachment. | ||
| Existing code that uses ``using (Py.GIL())`` continues to work without | ||
| changes. | ||
| * ``PythonEngine.BeginAllowThreads`` / ``EndAllowThreads`` similarly | ||
| manage the thread state and are still needed if you want the GC and | ||
| other Python threads to run while you're in long-running unmanaged code. | ||
| * Internal Python.NET caches (the reflection cache, generic-type binding | ||
| cache, dynamic-dispatch cache, module attribute cache, the interned- | ||
| string table, etc.) are thread-safe. You may read and call CLR types | ||
| concurrently from any number of threads without external locking. | ||
| * The reference-counting protocol uses CPython's ``Py_REFCNT`` symbol on | ||
| 3.14+, which returns the merged biased + shared refcount; values you read | ||
| from ``PyObject.Refcount`` are correct under free-threading. | ||
|
|
||
| Behaviour that is *unchanged* between GIL and free-threaded builds: | ||
|
|
||
| * A managed object exposed to Python (e.g. via ``System.Object`` or a | ||
| Python subclass of a CLR type) is still owned by a single CLR side: you | ||
| must not mutate its plain CLR fields from multiple threads without your | ||
| own locking. Python.NET only protects its own bookkeeping, not your | ||
| domain data. | ||
| * Operations on a single ``PyObject`` instance still require external | ||
| serialisation — see "Reference counting" above. | ||
|
|
||
| Patterns | ||
| -------- | ||
|
|
||
| Concurrent CLR access from Python | ||
| """"""""""""""""""""""""""""""""" | ||
|
|
||
| Hammering CLR attributes / generic types from many threads is supported:: | ||
|
|
||
| from threading import Thread | ||
| import System | ||
| from System.Collections.Generic import List | ||
|
|
||
| def worker(): | ||
| for _ in range(1000): | ||
| _ = System.String.Empty | ||
| _ = List[int]() | ||
|
|
||
| threads = [Thread(target=worker) for _ in range(8)] | ||
| for t in threads: t.start() | ||
| for t in threads: t.join() | ||
|
|
||
| This works on both GIL and free-threaded builds. | ||
|
|
||
| Python callback invoked from a managed thread | ||
| """"""""""""""""""""""""""""""""""""""""""""" | ||
|
|
||
| If a managed component calls back into a Python delegate from a thread it | ||
| spawned, that callback path acquires the GIL internally — you do not need to | ||
| add ``Py.GIL()`` around the Python code in the delegate. | ||
|
|
||
| Spawning a managed thread from inside ``Py.GIL()`` | ||
| """""""""""""""""""""""""""""""""""""""""""""""""" | ||
|
|
||
| If you start a managed thread while holding the GIL and the thread needs to | ||
| call back into Python, release the GIL first so the new thread can acquire | ||
| it:: | ||
|
|
||
| using (Py.GIL()) | ||
| { | ||
| var pyCallback = scope.Get("on_done"); | ||
| PythonEngine.BeginAllowThreads(); // let workers acquire the GIL | ||
| try | ||
| { | ||
| // spawn workers, wait for them... | ||
| } | ||
| finally | ||
| { | ||
| PythonEngine.EndAllowThreads(...); | ||
| } | ||
| } | ||
|
|
||
| Without the ``BeginAllowThreads`` the spawned thread blocks forever waiting | ||
| for the GIL the parent thread is still holding. | ||
|
|
||
| Common pitfalls | ||
| --------------- | ||
|
|
||
| * Holding ``Py.GIL()`` across ``Task.Run`` / ``await`` boundaries. Async | ||
| continuations can resume on a different thread; the GIL handle is | ||
| thread-bound and must be released on the same thread that acquired it. | ||
| * Passing a ``PyObject`` to a managed worker without taking ownership. If | ||
| the producer disposes its handle while the consumer is still using it, | ||
| the worker will operate on a freed object. Wrap the producer's | ||
| ``PyObject`` with ``new PyObject(value)`` before handing it off, or use | ||
| ``NewReference()``. | ||
| * Calling a Python callable that does CPU-bound work without releasing the | ||
| GIL. Other Python threads cannot make progress in that case, even on a | ||
| free-threaded build where the GIL is otherwise a no-op (the callable | ||
| itself may still touch contended Python state). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.