Repository navigation
BUG: SeedSequence.spawn and np.random.set_bit_generator are not thread-safe #32063
Description
Describe the issue:
Two independent thread-safety problems in numpy.random, found while reviewing the fix for gh-32059.
1. SeedSequence.spawn hands out duplicate spawn keys
SeedSequence.spawn reads self.n_children_spawned, constructs the children (Python-level work, so the GIL can be released between bytecodes), and only then increments the counter. Two threads spawning from the same SeedSequence can both observe the same starting index and return children with identical spawn keys, i.e. bit generators that produce identical streams. That silently breaks the central guarantee of the spawning API for parallel work.
The BitGenerator lock can't help here: a SeedSequence is an independent object that can be shared by multiple bit generators (or none), so it needs its own synchronization. Note that a per-instance lock interacts with pickling (the auto-generated cdef pickling would try to serialize it).
BitGenerator.spawn and Generator.spawn delegate to SeedSequence.spawn, so they inherit the bug.
Reproduce:
import sys
import threading
import numpy as np
# not required, just makes it reproduce quickly on the GIL-enabled build
sys.setswitchinterval(1e-5)
ss = np.random.SeedSequence(0)
barrier = threading.Barrier(4)
def spawn(out):
barrier.wait()
out.extend(ss.spawn(500))
results = [[] for _ in range(4)]
threads = [threading.Thread(target=spawn, args=(r,)) for r in results]
for t in threads:
t.start()
for t in threads:
t.join()
keys = [c.spawn_key for r in results for c in r]
print(f"spawned {len(keys)} children, {len(set(keys))} unique spawn keys, "
f"n_children_spawned={ss.n_children_spawned}")Observed (3 out of 3 runs on a GIL build):
spawned 2000 children, 500 unique spawn keys, n_children_spawned=2000
i.e. all four threads returned the same 500 children.
2. np.random.set_bit_generator races with concurrent draws from the singleton
set_bit_generator calls RandomState._initialize_bit_generator with no synchronization. That method rewrites, non-atomically with respect to running draws:
self._bit_generator(dropping the reference to the old one),self._bitgen, the copiedbitgen_tstruct of function pointers plus state pointer,self._aug_state.bit_generator, andself.lockitself.
Distribution fills run with self.lock, nogil: — without the GIL — reading those function/state pointers, so a concurrent swap lets a fill observe a mixed struct (e.g. MT19937's next_double called on a pcg64_state*, which reads/writes far past the end of the smaller struct) or a state pointer into an already-deallocated bit generator.
Taking the lock inside set_bit_generator would not be sufficient: the lock attribute is replaced by the swap, so a thread still blocked on the old lock and a thread acquiring the new lock can be inside the critical section simultaneously. A real fix needs a stable guard owned by RandomState that is never swapped or the API has to explicitly require that replacement happen only while no other thread uses the singleton.
IMO the simplest fix is to detect and raise an error when we detect that set_bit_generator is used on a shared RandomState.
Reproduce:
import threading
import numpy as np
barrier = threading.Barrier(2)
stop = threading.Event()
def draw():
barrier.wait()
while not stop.is_set():
np.random.random(100_000)
def swap():
barrier.wait()
for i in range(500):
bg = np.random.PCG64(i) if i % 2 else np.random.MT19937(i)
np.random.set_bit_generator(bg)
stop.set()
threads = [threading.Thread(target=draw), threading.Thread(target=swap)]
for t in threads:
t.start()
for t in threads:
t.join()
print("survived")This script segfaults most times I run it on the GIL-enabled build in my testing.
Python and NumPy Versions:
NumPy main, reproduced using Python 3.14.5.
cc @rkern since fixing these issues requires answering some design questions for the RNG module
Activity
rkern commentedon Jul 22, 2026
What do you mean GIL-enabled (I have not really kept up with the free-threading developments)? When I run that last script on 3.11 (which is definitely GIL-enabled to my understanding), I get no problem.
Did Cython change how it offers up GIL switches? I thought it was still held at the method call boundary without
with nogil:even if the statement was nominally equivalent to pure Python.ngoldbaum commentedon Jul 22, 2026
I mean Python classic. So, no "t" in the version.
I think it's that Python changed the eval-breaker semantics even on the GIL-enabled build for 3.13 and that makes this race more likely to trigger crashes on 3.13 and newer, at least on a Mac. I can trigger a crash on 3.11 on Linux FWIW.
You're right about Cython but there is Python code firing as well. For example,
SeedSequence.__init__callsget_assembled_entropywhich callsnp.concatenate. That makes ordinary Python-dispatched calls via e.g.__array_function__so the GIL can be released there mid-spawn, between reading and incrementing n_children_spawned.kumaraditya303 commentedon Jul 23, 2026
I think for 2) the update can happen under the old lock so that no other thread could be using the old generator while it is being replaced.
rkern commentedon Jul 23, 2026
Yeah, I think
SeedSequenceneeds its own lock. I think implementing__reduce_ex__appropriately will solve the pickling issue.I think it's correct that holding the old
BitGenerator's lock before doing the swap would work, but it's worth thinking through both the cases where theRandomStatemethod body has thewith lockand when it uses one of the_common.pyxhelpers likedisc()orcont()that are passed theself._bitgenandself.lockseparately and it's those helper functions that acquire the lock.ngoldbaum commentedon Jul 24, 2026
OK, that sounds like a plan. I'm not planning to take on fixes for these issues because there's other higher-priority stuff on my plate. I'd happily review someone else's PR though.
shbhmbnsl1 commentedon Oct 1, 2026
Hi @ngoldbaum @seberg
I would be happy to work on this as part of the sustain-2026 series. I am planning to first make a code change for the first issue here and then build from there. For that I think adding a lock of its own on
SeedSequencemakes sense to me and to handle pickling I am planning to add an implementation of__reduce_ex__insideSeedSequenceas mentioned above.