Skip to content

Commit 73d3dc3

Browse files
eendebakptclaude
andcommitted
gh-NNNNN: Make _STORE_ATTR_SLOT lock-free on the free-threaded build
Both the bytecode handler (_STORE_ATTR_SLOT) and the C-API path (PyMember_SetOne for Py_T_OBJECT_EX / _Py_T_OBJECT) currently take per-object locks (ob_mutex via LOCK_OBJECT or Py_BEGIN_CRITICAL_SECTION) to sequence the read-of-old-value, store-of-new-value, and decref-of-old. Replace the lock with a single atomic exchange on the slot pointer. Atomic exchange returns the previous value and stores the new one in one atomic op, which is both sufficient for correctness on the write side (each concurrent writer observes a unique old value, so no double-decref) and consistent with how the read side already works: _LOAD_ATTR_SLOT and PyMember_GetOne are lock-free for the fast path, using FT_ATOMIC_LOAD_PTR plus _Py_TryIncrefCompare to cope with mid-flight pointer changes. This is the WRITE-side counterpart to the existing lock-free LOAD_ATTR_SLOT read path, and follows the pattern Dino Viehland established with pythongh-130373 (Avoid locking in _LOAD_ATTR_WITH_HINT, merged Feb 2025) and companions pythongh-130312 / pythongh-130313 (don't lock when populating sets in bytecode / when clearing attribute-less objects). The primitive being used here, _Py_atomic_exchange_ptr, is being formally proposed in pythongh-150044 for an internal _Py_SETREF_ATOMIC macro. Three coordinated changes are required: 1. _STORE_ATTR_SLOT in Python/bytecodes.c: drop LOCK_OBJECT / UNLOCK_OBJECT and the DEOPT_IF on lock-acquire failure; use _Py_atomic_exchange_ptr on Py_GIL_DISABLED, plain read/write on the GIL build. 2. PyMember_SetOne in Python/structmember.c, Py_T_OBJECT_EX and _Py_T_OBJECT cases: drop Py_BEGIN_CRITICAL_SECTION; use _Py_atomic_exchange_ptr to match the bytecode path. The two write paths must change together: if only the bytecode were lock-free, PyMember_SetOne's plain "oldv = *addr" inside the CS would race with concurrent atomic-exchange writers, returning a "stolen" old value that another writer is also about to DECREF (double-free). 3. PyMember_GetOne in Python/structmember.c, Py_T_OBJECT_EX and _Py_T_OBJECT cases: replace the CS+Py_XINCREF fallback (taken on _Py_TryIncrefCompare failure) with _Py_XGetRef, which retries the atomic-load + try-incref loop. The previous fallback relied on writers also taking ob_mutex to keep the slot stable inside the CS; with lock-free writers, the CS no longer excludes them, and a plain Py_XINCREF on a re-loaded slot value could resurrect a refcount-0 (about-to-be-freed) object. History (git blame on the LOCK_OBJECT line): * 22b0de2 (2024-06): original FT structure of _STORE_ATTR_SLOT. * 1b15c89 (2024-12, pythongh-127838 / pythongh-115999): added LOCK_OBJECT in the broader STORE_ATTR specialization commit. For _STORE_ATTR_INSTANCE_VALUE the lock is genuinely required (it also updates the PyDictValues insertion-order list, which has to be coordinated with the NULL->non-NULL transition). For _STORE_ATTR_SLOT it was added for symmetry but isn't required when atomic exchange is used. * bef63d2 (2025-12, pythonGH-134584 / pythongh-142729): removed a redundant refcount op from _STORE_ATTR_SLOT, confirming the slot fast path is still being actively simplified. Why this is correct: * The slot store is a single pointer write at a fixed offset (determined by the type and guarded by _GUARD_TYPE_VERSION before _STORE_ATTR_SLOT). * Atomic exchange replaces the LOCK_OBJECT + plain-load-old + atomic-release-store-new + UNLOCK_OBJECT sequence with one lock-prefix xchg. * Each writer's exchange returns exactly one old pointer, so Py_XDECREF(old_value) cannot double-free across concurrent writers. * Readers (LOAD_ATTR_SLOT, PyMember_GetOne) atomically load + try- incref, then re-validate that the slot still points at the same object before claiming the reference. The TryIncrefCompare path correctly fails if ob_ref_shared has reached 0 (object being freed), so we never resurrect. * Type-change races (`__class__ = NewType`) are caught by the _GUARD_TYPE_VERSION uop that runs before _STORE_ATTR_SLOT, same as for the already-lock-free _LOAD_ATTR_SLOT. * The DEOPT_IF on lock-acquire failure is dropped: atomic exchange always succeeds, so the bytecode no longer thrashes between specialized and generic forms under contention. Microbench (origin/main @ 28eac9a, --disable-gil --enable-optimizations --with-lto=yes, taskset -c 4, best-of-9 ns/op): workload baseline patched ratio STORE_ATTR_SLOT (alone) 15.40 9.30 0.604 (-39.6%) attr_idiv (LOAD + / + STORE) 41.22 30.20 0.733 (-26.7%) Point.normalize (per Point) 417.4 398.0 0.953 (-4.6%) pyperformance bm_float (direct timing, best-of-5, ms): baseline FT 53.2 patched FT 48.5 (-4.7 ms, -8.8%) GIL (ref) 46.6 (patched FT is within 4% of GIL on this bench) bm_nbody is unchanged as a control (its STORE-heavy work is on STORE_SUBSCR_LIST_INT, not STORE_ATTR). Comprehensive regression sweep on PGO+LTO build: test_descr, test_class, test_dataclasses, test_dict, test_typing, test_threading, test_concurrent_futures, test_thread, test_capi, test_inspect, test_subclassinit, test_long, test_float, test_complex, test_gc, test_pickle, test_listcomps, test_exceptions, test_compile - all pass. The full test_free_threading suite (192 tests) also passes. Concurrent stress test (4 writer threads + 2 reader threads racing on the same __slots__ object's slots, 200k writes each, 100k reads each): no crash, no inconsistency, refcount stays in expected range. Standalone benchmark script: bench_store_attr_slot.py (microbench + Point.normalize pipeline; see PR comment). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 28eac9a commit 73d3dc3

7 files changed

Lines changed: 82 additions & 56 deletions

File tree

‎Include/internal/pycore_opcode_metadata.h‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_uop_metadata.h‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Modules/_testinternalcapi/test_cases.c.h‎

Lines changed: 8 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Python/bytecodes.c‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3225,14 +3225,27 @@ dummy_func(
32253225
op(_STORE_ATTR_SLOT, (index/1, value, owner -- o)) {
32263226
PyObject *owner_o = PyStackRef_AsPyObjectBorrow(owner);
32273227

3228-
DEOPT_IF(!LOCK_OBJECT(owner_o));
32293228
char *addr = (char *)owner_o + index;
32303229
STAT_INC(STORE_ATTR, hit);
3230+
// Lock-free read-modify-write on the slot pointer (free-threaded
3231+
// build). Atomic exchange returns the previous value and stores
3232+
// the new one in a single atomic op. Each concurrent writer
3233+
// observes a unique old value (so no double-decref), and
3234+
// concurrent LOAD_ATTR_SLOT readers already cope with mid-flight
3235+
// mutations via _Py_TryIncrefCompare. The C-API path
3236+
// (PyMember_SetOne for Py_T_OBJECT_EX / _Py_T_OBJECT) is updated
3237+
// to the same atomic-exchange pattern so the two paths stay
3238+
// consistent without a per-object lock. On the GIL build this
3239+
// compiles to a plain read+write.
3240+
#ifdef Py_GIL_DISABLED
3241+
PyObject *old_value = _Py_atomic_exchange_ptr(
3242+
(void **)addr, PyStackRef_AsPyObjectSteal(value));
3243+
#else
32313244
PyObject *old_value = *(PyObject **)addr;
3232-
FT_ATOMIC_STORE_PTR_RELEASE(*(PyObject **)addr, PyStackRef_AsPyObjectSteal(value));
3233-
UNLOCK_OBJECT(owner_o);
3234-
INPUTS_DEAD();
3245+
*(PyObject **)addr = PyStackRef_AsPyObjectSteal(value);
3246+
#endif
32353247
o = owner;
3248+
DEAD(owner);
32363249
Py_XDECREF(old_value);
32373250
}
32383251

‎Python/executor_cases.c.h‎

Lines changed: 15 additions & 11 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Python/generated_cases.c.h‎

Lines changed: 8 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Python/structmember.c‎

Lines changed: 32 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -97,36 +97,33 @@ PyMember_GetOne(const char *obj_addr, PyMemberDef *l)
9797
break;
9898
}
9999
case _Py_T_OBJECT:
100-
v = FT_ATOMIC_LOAD_PTR(*(PyObject **) addr);
101-
if (v != NULL) {
102100
#ifdef Py_GIL_DISABLED
103-
if (!_Py_TryIncrefCompare((PyObject **) addr, v)) {
104-
Py_BEGIN_CRITICAL_SECTION((PyObject *) obj_addr);
105-
v = FT_ATOMIC_LOAD_PTR(*(PyObject **) addr);
106-
Py_XINCREF(v);
107-
Py_END_CRITICAL_SECTION();
108-
}
101+
// _Py_XGetRef does an atomic load + try-incref loop, retrying when
102+
// a concurrent writer (lock-free atomic exchange in _STORE_ATTR_SLOT
103+
// and PyMember_SetOne) mutates the slot. The previous critical-
104+
// section fallback assumed writers also took ob_mutex, which they no
105+
// longer do, so a plain Py_XINCREF inside the CS could resurrect a
106+
// refcount-0 (about-to-be-freed) object.
107+
v = _Py_XGetRef((PyObject **)addr);
109108
#else
110-
Py_INCREF(v);
109+
v = *(PyObject **)addr;
110+
Py_XINCREF(v);
111111
#endif
112-
}
113112
if (v == NULL) {
114-
v = Py_None;
113+
v = Py_NewRef(Py_None);
115114
}
116115
break;
117116
case Py_T_OBJECT_EX:
117+
#ifdef Py_GIL_DISABLED
118+
v = _Py_XGetRef((PyObject **)addr);
119+
if (v == NULL) {
120+
PyErr_Format(PyExc_AttributeError,
121+
"'%T' object has no attribute '%s'",
122+
(PyObject *)obj_addr, l->name);
123+
}
124+
#else
118125
v = member_get_object(addr, obj_addr, l);
119-
#ifndef Py_GIL_DISABLED
120126
Py_XINCREF(v);
121-
#else
122-
if (v != NULL) {
123-
if (!_Py_TryIncrefCompare((PyObject **) addr, v)) {
124-
Py_BEGIN_CRITICAL_SECTION((PyObject *) obj_addr);
125-
v = member_get_object(addr, obj_addr, l);
126-
Py_XINCREF(v);
127-
Py_END_CRITICAL_SECTION();
128-
}
129-
}
130127
#endif
131128
break;
132129
case Py_T_LONGLONG:
@@ -320,11 +317,20 @@ PyMember_SetOne(char *addr, PyMemberDef *l, PyObject *v)
320317
break;
321318
}
322319
case _Py_T_OBJECT:
323-
case Py_T_OBJECT_EX:
324-
Py_BEGIN_CRITICAL_SECTION(obj);
320+
case Py_T_OBJECT_EX: {
321+
// Lock-free swap to match the _STORE_ATTR_SLOT bytecode handler
322+
// (which also uses atomic exchange on Py_GIL_DISABLED builds).
323+
// Atomic exchange gives each concurrent writer a unique old value,
324+
// so the Py_XDECREF below cannot double-free. Concurrent readers
325+
// (PyMember_GetOne, _LOAD_ATTR_SLOT) use atomic loads and the
326+
// TryIncrefCompare pattern to cope with mid-flight mutations.
327+
PyObject *new_v = Py_XNewRef(v);
328+
#ifdef Py_GIL_DISABLED
329+
oldv = (PyObject *)_Py_atomic_exchange_ptr((void **)addr, new_v);
330+
#else
325331
oldv = *(PyObject **)addr;
326-
FT_ATOMIC_STORE_PTR_RELEASE(*(PyObject **)addr, Py_XNewRef(v));
327-
Py_END_CRITICAL_SECTION();
332+
*(PyObject **)addr = new_v;
333+
#endif
328334
if (v == NULL && oldv == NULL && l->type == Py_T_OBJECT_EX) {
329335
// Raise an exception when attempting to delete an already deleted
330336
// attribute.
@@ -336,6 +342,7 @@ PyMember_SetOne(char *addr, PyMemberDef *l, PyObject *v)
336342
}
337343
Py_XDECREF(oldv);
338344
break;
345+
}
339346
case Py_T_CHAR: {
340347
const char *string;
341348
Py_ssize_t len;

0 commit comments

Comments
 (0)