Skip to content

Segfault in _pickle.c:batch_dict_exact when pickling dict with concurrent mutation (free-threading) #146452

Closed
Closed
Segfault in _pickle.c:batch_dict_exact when pickling dict with concurrent mutation (free-threading)#146452
Labels
extension-modulesC modules in the Modules dirtopic-free-threadingtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@overlorde
overlorde
opened on Mar 26, 2026

Bug report

Bug description

I've been testing the free-threaded build for thread safety issues in unaudited modules (ref #116738 — _pickle.c is unchecked). Found a segfault when pickle.dumps() runs concurrently with dict mutations on the same dict.

The crash happens in batch_dict_exact at Modules/_pickle.c:3496 — it iterates dict items via _PyDict_Next which gives borrowed references, but there's no critical section on the dict. If another thread pops or replaces a key mid-iteration, the borrowed reference goes stale and Py_INCREF hits a freed/NULL pointer.

Sometimes it raises RuntimeError: dictionary changed size during iteration cleanly, but other times the reference gets invalidated before the size check and it just segfaults.

Reproducer

import pickle
import threading

shared = {str(i): list(range(50)) for i in range(100)}

for trial in range(2000):
    barrier = threading.Barrier(3)

    def dumper():
        try:
            barrier.wait()
            pickle.dumps(shared)
        except Exception:
            pass

    def mutator():
        try:
            barrier.wait()
            for j in range(200):
                key = str(j % 100)
                shared[key] = list(range(j % 50))
                if j % 10 == 0:
                    shared.pop(str(j % 100), None)
                    shared[str(j % 100)] = [j]
        except Exception:
            pass

    threads = [
        threading.Thread(target=dumper),
        threading.Thread(target=dumper),
        threading.Thread(target=mutator),
    ]
    for t in threads: t.start()
    for t in threads: t.join()

Crashes within a few hundred iterations usually. I ran it 10 times, got 9 segfaults.

Tested on

3.14.0a4 free-threading (stock install, no sanitizers):

$ python3.14t reproducer.py
Segmentation fault (core dumped)

3.15.0a7+ free-threading (built with --with-address-sanitizer):

AddressSanitizer:DEADLYSIGNAL
==1334802==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000000c
    #0 _Py_atomic_load_uint32_relaxed Include/cpython/pyatomic_gcc.h:367
    #1 Py_INCREF Include/refcount.h:267
    #2 batch_dict_exact Modules/_pickle.c:3496
    #3 save_dict Modules/_pickle.c:3563
    #4 save Modules/_pickle.c:4568
    #5 dump Modules/_pickle.c:4768
    #6 _pickle_dumps_impl Modules/_pickle.c:7992

Root cause

  • batch_dict_exact calls PyDict_Next() which returns borrowed references to key and value
  • Between PyDict_Next() returning and Py_INCREF(key) / Py_INCREF(value), another thread can mutate the dict
  • The borrowed reference points to a freed object, Py_INCREF dereferences it and segfaults
  • The size check at line 3512 (PyDict_GET_SIZE(obj) != dict_size) only catches mutations that change dict size, and only after the batch — too late to prevent the stale reference
  • Same issue exists in both the single-item path (line 3474) and the batch loop (line 3494)

Possible fix

  • Wrap PyDict_Next() + Py_INCREF(key) + Py_INCREF(value) in Py_BEGIN_CRITICAL_SECTION(obj) / Py_END_CRITICAL_SECTION() to prevent dict mutation while converting borrowed refs to owned refs
  • Keep save() calls outside the critical section since they can reenter Python
  • Same approach as the set iteration path at line 3644 which already uses Py_BEGIN_CRITICAL_SECTION(obj) with _PySet_NextEntryRef
  • Both PyDict_Next call sites in batch_dict_exact need this (line 3474 and line 3494)

CPython versions tested on

3.14, CPython main branch

Operating systems tested on

Linux

Linked PRs

Activity

  1. picnixz
    added
    extension-modulesC modules in the Modules dir
    type-crashA hard crash of the interpreter, possibly with a core dump
    topic-free-threading
    on Mar 26, 2026
  2. picnixz
    added this to Pickle and copy issues 🥒on Mar 26, 2026

  3. overlorde commented on Mar 26, 2026

    @overlorde
    overlorde
    on Mar 26, 2026
    ContributorAuthor

    Would like to get assigned to that issue.

  4. overlorde
    added a commit that references this issue on Mar 26, 2026

    pythongh-146452: Fix pickle segfault when pickling dict with concurre…

    ab4d60e

  5. overlorde commented on Mar 27, 2026

    @overlorde
    overlorde
    on Mar 27, 2026
    ContributorAuthor

    The PR patches the two PyDict_Next call sites in batch_dict_exact, but the same borrowed reference problem exists wherever PyDict_Next is used without a critical section — there are 87 call sites across the codebase.

    For sets, this was solved by adding _PySet_NextEntryRef which returns owned references and asserts the caller holds the critical section (used in _pickle.c at line 3660). There's no dict equivalent yet.

    Would it make sense to add a _PyDict_NextRef following the same pattern as _PySet_NextEntryRef? That way each call site can be migrated with a simple function swap instead of wrapping every PyDict_Next + Py_INCREF in a critical section independently. Happy to work on that if it sounds reasonable.

  6. overlorde
    added a commit that references this issue on Mar 27, 2026

    pythongh-146452: Fix pickle segfault when pickling dict with concurre…

    8f55561
  7. picnixz
    marked Segfault in _pickle.c:batch_list_exact when pickling list with concurrent mutation (free-threading) #146533 as a duplicate of this issue on Mar 27, 2026

  8. mjbommar commented on Apr 16, 2026

    @mjbommar
    mjbommar
    on Apr 16, 2026
    Contributor

    Possibly useful finding from adjacent issue:

    A similar crash class fires on free-threaded builds when two threads share a single Pickler instance, with no external mutator.

    Lock-test on 3.14.3t over 10 subprocesses each:

    • shared Pickler with no Python lock = 8-10/10 SIGSEGV;
    • shared Pickler with threading.Lock around dump() = 0/10;
    • fresh Pickler per dump() = 0/10.

    Crashes land in save_dict per faulthandler, same neighborhood as your fix. Repro is two threads calling p.dump(obj_a) / p.dump(obj_b) on a shared pickle.Pickler(io.BytesIO()).

    Root cause looks like BEGIN_USING_PICKLER at lines 710-720 being a non-atomic check-then-set on running; under PEP 703 two threads can both pass the guard (race window is in C, below Python instrumentation resolution, but the lock-test contrast nails it). Once both threads slip past, they hit the same PyDict_Next + Py_INCREF borrowed-ref window on self->memo that #146470 wraps for the object dict. Atomic compare-exchange on the running flag (and the analogous BEGIN_USING_UNPICKLER) would restore the documented RuntimeError: "Pickler object is already used" and prevent the second-thread entry.

    Happy to help but seems like you have the ball in this PR.

  9. kumaraditya303
    added 2 commits that reference this issue on May 17, 2026

    gh-146452: Fix pickle segfault on concurrent mutation of dict in pick…

    e62a611

    [3.14] gh-146452: Fix pickle segfault on concurrent mutation of dict …

    f380f85
  10. JelleZijlstra
    added 3 commits that reference this issue on May 18, 2026

    gh-146452: Improve locking granularity in pickle's batch_dict_exact a…

    57a0e57

    [3.15] gh-146452: Improve locking granularity in pickle's batch_dict_…

    66ade28

    [3.14] gh-146452: Improve locking granularity in pickle's batch_dict_… (

    1243cd6
  11. vstinner
    added a commit that references this issue on May 22, 2026

    Revert "gh-146452: Improve locking granularity in pickle's batch_dict…

    b770b23
  12. kumaraditya303
    added 3 commits that reference this issue on May 23, 2026

    [3.15] Revert "[3.15] gh-146452: Improve locking granularity in pickl…

    795dd3b

    [3.14] Revert "[3.14] gh-146452: Improve locking granularity in pickl…

    0bd80a1

    [3.15] gh-146452: Fix pickle segfault on concurrent mutation of dict …

    77cc442
  13. kumaraditya303
    closed this as completedon May 23, 2026
  14. github-project-automation
    moved this to Done in Pickle and copy issues 🥒on May 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtopic-free-threadingtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions