An object referenced only through it's own __dict__ can get collected too early.

Bug report

Bug description:

After reusing one object's __dict__ for another object, and adding a reference to the original object in a dictionary on the target object, the original object might get collected too early (and cause a whole other dictionary to get cleared). I hope the code below describes it better than my words.

I've bisected this problem to c32dc47.

This is the smallest reproduction I've found so far:

import gc
import sys

print(sys.version_info)

class A:
    def __del__(self):
        print("del", hex(id(self)))

a = A()
a.attr_a = "test value"
b = A()
b.__dict__ = a.__dict__
b.other_attr = {"self reference": a, "other data": 42}
del a
print("before gc:", b.__dict__)
gc.collect()

print("after gc:", b.__dict__)

Results on 3.13:

sys.version_info(major=3, minor=13, micro=0, releaselevel='alpha', serial=5)
before gc: {'attr_a': 'test value', 'other_attr': {'self reference': <__main__.A object at 0x76cc6317a900>, 'other data': 42}}
del 0x76cc6317a900
after gc: {'attr_a': 'test value', 'other_attr': {}}
del 0x76cc6256ccd0

Results on 3.12, and what I expected to see:

sys.version_info(major=3, minor=12, micro=7, releaselevel='final', serial=0)
before gc: {'attr_a': 'test value', 'other_attr': {'self reference': <__main__.A object at 0x77496b531130>, 'other data': 42}}
after gc: {'attr_a': 'test value', 'other_attr': {'self reference': <__main__.A object at 0x77496b531130>, 'other data': 42}}
del 0x77496b531160
del 0x77496b531130

I've noticed this change in behaviour while working with Locust.io, the original issue for reference: locustio/locust#3050

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Linux

Linked PRs

I suspect it's probably something about __dict__ not getting properly overwritten for objects with inline values. I'm looking into it.

I haven't fully figured it out, but it seems the problem is (somewhat) here:

cpython/Objects/dictobject.c

Line 7174 in 5a13faa

FT_ATOMIC_STORE_PTR(_PyObject_ManagedDictPointer(obj)->dict, new_dict);

The dictionary is being stored in the object's managed pointer, but the original object still assumes it has ownership, so it clears it upon being deallocated. We need to detach ownership upon doing that, or copy the inline values of a to b. I'll put up a PR (hopefully) later today, assuming that my analysis is correct.

I think the issue is in PyObject_VisitManagedDict. The problem is not any tp_clear or tp_dealloc handlers, because nothing should be getting cleared or deallocated during GC in the example. There are cycles, but no cyclic trash. All the relevant objects, a, b, and the shared dictionary should stay alive.

cpython/Objects/dictobject.c

Lines 7146 to 7164 in 6c450f4

    
           int 
        
           PyObject_VisitManagedDict(PyObject *obj, visitproc visit, void *arg) 
        
           { 
        
               PyTypeObject *tp = Py_TYPE(obj); 
        
               if((tp->tp_flags & Py_TPFLAGS_MANAGED_DICT) == 0) { 
        
                   return 0; 
        
               } 
        
               if (tp->tp_flags & Py_TPFLAGS_INLINE_VALUES) { 
        
                   PyDictValues *values = _PyObject_InlineValues(obj); 
        
                   if (values->valid) { 
        
                       for (Py_ssize_t i = 0; i < values->capacity; i++) { 
        
                           Py_VISIT(values->values[i]); 
        
                       } 
        
                       return 0; 
        
                   } 
        
               } 
        
               Py_VISIT(_PyObject_ManagedDictPointer(obj)->dict); 
        
               return 0; 
        
           }

If there's a managed dictionary, we should visit the dictionary, not the inline values, even if the values are "valid".

Thanks for the insight, Sam. I haven't dived into these parts of the codebase before, so bear with me here. I'm not sure it's possible for an object to have valid inline values while also having a non-null managed dictionary.

The relevant code that sets the managed dictionary pointer also immediately invalidates the inline values:

cpython/Objects/dictobject.c

Lines 7174 to 7181 in 5a13faa

    
           FT_ATOMIC_STORE_PTR(_PyObject_ManagedDictPointer(obj)->dict, new_dict); 
        
           if (values->valid) { 
        
               FT_ATOMIC_STORE_UINT8(values->valid, 0); 
        
               for (Py_ssize_t i = 0; i < values->capacity; i++) { 
        
                   Py_CLEAR(values->values[i]); 
        
               } 
        
           }

The problem is not any tp_clear or tp_dealloc handlers, because nothing should be getting cleared or deallocated during GC in the example

Hmm. If not tp_clear (or subtype_clear, I guess), what might clear a managed dictionary?

I'm not sure it's possible for an object to have valid inline values while also having a non-null managed dictionary

Yes, this is allowed and fairly common. It allows fast access to the attributes via LOAD_ATTR_INSTANCE_VALUE even if there are also accesses via obj.__dict__. For example:

class MyObj: pass

a = MyObj()
a.attr = "value"
d = a.__dict__  # a now has a managed dictionary and still has valid inline values
tmp = a.attr # this will be specialized to `LOAD_ATTR_INSTANCE_VALUE` if executed enough times

The relevant code that sets the managed dictionary pointer also immediately invalidates the inline values

That's the code path for changing the managed dictionary. For example, obj.__dict__ = new_dict will invalidate any previous inline values of obj. The most common way way to create a manage dictionary is just the (read) access obj.__dict__, which will call ensure_managed_dict and doesn't invalidate inline values.

So when we have:

a = A()
b = A()
b.__dict__ = A.__dict__

Both a and b have the same (non-NULL) managed dictionary
That dictionary's values point to the inline values in a
a's inline values are valid
b's inline values are not valid

Hmm. If not tp_clear (or subtype_clear, I guess), what might clear a managed dictionary

Yes, subtype_clear is clearing the managed dictionary, but the bug is not in subtype_clear. The gc.collect() call shouldn't be calling subtype_clear() or any other tp_clear hooks. Although there are cycles in the repro, there isn't any cyclic trash. Even though there's a del a statement, that object is still reachable via b. The bug is that the GC incorrectly thinks it's cyclic trash and calls subtype_clear on it.

The bugs are in PyObject_VisitManagedDict and also dict_traverse.

Concretely, what's happening is:

The GC determines the b is reachable (good)
It then tries to transitively mark objects as reachable via b (this is in move_unreachable in gc.c or mark_reachable in gc_free_threading.c), which eventually calls PyObject_VisitManagedDict on b, which visits b's dictionary (good)
The dictionary is traversed, but this doesn't traverse the values because they're embedded (in a). (bad!)

So now a (and any other values in the dictionary) incorrectly look like cyclic trash to the GC because they're not marked as reachable.

What should happen?

There's ambiguity here because for objects like a, the dictionary and the inline values share a reference count for each object in the inline values. So it's not clear who "owns" the objects in the inline values and who should traverse them.

I think the logic should be:

If there's a non-NULL managed dictionary, PyObject_VisitManagedDict should traverse the dictionary and NOT traverse the inline values.
dict_traverse should traverse its values even if they are embedded

Basically, we should treat it as if the dictionary owns the values, not the object.

Ok, got it. Thanks for all your help with this!

So it's not clear who "owns" the objects in the inline values and who should traverse them.

Out of curiosity--what happens if two objects share a reference to A, and both objects traverse A? Does the garbage collector require that the number of GC referrers is <= the object's reference count?

pythongh-130327: Only set the managed dictionary if there aren't inli…

Revert "pythongh-130327: Only set the managed dictionary if there are…

Out of curiosity--what happens if two objects share a reference to A, and both objects traverse A? Does the garbage collector require that the number of GC referrers is <= the object's reference count?

Yes, the GC requires that the number of referrers is <= the object's reference count. Otherwise, you may trigger an assert like ""refcount is too small".

The GC computes object's reference count minus the number of referrers as "gc_refs". If this values is >0, then the object has some external reference and is definitely reachable and shouldn't be collected. If "gc_refs" is zero, then it doesn't have external references, but it may still be reachable from some other live object considered by the GC. And if "gc_refs" is <0, then we trigger an assertion failure like the one above.

So if we have too many Py_VISIT() calls we may:

incorrectly make "gc_refs" zero and call a tp_clear() hook on an object that shouldn't actually be cleared.
incorrectly make "gc_refs" negative and trigger an assertion failure
or nothing may happen because "gc_refs" is still positive or the relevant object was otherwise still reachable.

Merge branch 'main' into pythongh-130327-managed-dict-inline

I believe this is the root cause (per ZeroIntensity in 135552, which we thought was responsible originally) of eventlet/eventlet#1032 and https://bugs.launchpad.net/nova/+bug/2103413 if you need another example of this bug having real-world impacts.

Fix neutron client dict grabbing

I've stumbled upon this bug as well, causing much head-scratching - so bumping this bug report.

In fact, you don't even need to share dicts in order to reproduce this issue, the following reproduces it just as well - all you have to do is keep a ref to an object's dict beyond the object's lifetime. (And the object should have circular references to itself - think trees)

import gc

class Shenanigans:
    pass

to_be_deleted = Shenanigans()
to_be_deleted.attr = "whatever"
dictus = to_be_deleted.__dict__
to_be_deleted.ref = {"circular": to_be_deleted, "data": 42}

del to_be_deleted
gc.collect()
assert(dictus['ref']['data'] == 42)
assert(dictus['attr'] == "whatever")

Fix neutron client dict grabbing

Merge branch 'main' into pythongh-130327-managed-dict-inline

gh-130327: Always traverse managed dictionaries, even when inline val…

pythongh-130327: Always traverse managed dictionaries, even when inli…

[3.13] pythongh-130327: Always traverse managed dictionaries, even wh…

[3.14] gh-130327: Always traverse managed dictionaries, even when inl…

[3.13] gh-130327: Always traverse managed dictionaries, even when inl…

@ktosiek

This is fixed now and should be in upcoming 3.13 and 3.14 patch releases. Thanks for the bug report @ktosiek and fix by @ZeroIntensity.

pythongh-130327: Always traverse managed dictionaries, even when inli…

Navigation Menu

Repository navigation

An object referenced only through it's own `dict` can get collected too early. #130327

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

What should happen?

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	int
	PyObject_VisitManagedDict(PyObject obj, visitproc visit, void arg)
	{
	PyTypeObject *tp = Py_TYPE(obj);
	if((tp->tp_flags & Py_TPFLAGS_MANAGED_DICT) == 0) {
	return 0;
	}
	if (tp->tp_flags & Py_TPFLAGS_INLINE_VALUES) {
	PyDictValues *values = _PyObject_InlineValues(obj);
	if (values->valid) {
	for (Py_ssize_t i = 0; i < values->capacity; i++) {
	Py_VISIT(values->values[i]);
	}
	return 0;
	}
	}
	Py_VISIT(_PyObject_ManagedDictPointer(obj)->dict);
	return 0;
	}

	FT_ATOMIC_STORE_PTR(_PyObject_ManagedDictPointer(obj)->dict, new_dict);

	if (values->valid) {
	FT_ATOMIC_STORE_UINT8(values->valid, 0);
	for (Py_ssize_t i = 0; i < values->capacity; i++) {
	Py_CLEAR(values->values[i]);
	}
	}

An object referenced only through it's own __dict__ can get collected too early. #130327

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

ZeroIntensity commented on Feb 20, 2025

ZeroIntensity commented on Feb 20, 2025

colesbury commented on Feb 20, 2025

ZeroIntensity commented on Feb 21, 2025

colesbury commented on Feb 21, 2025

What should happen?

ZeroIntensity commented on Feb 22, 2025

colesbury commented on Feb 25, 2025

jayofdoom commented on Jun 19, 2025

3 remaining items

thisismypassport commented on Feb 10, 2026

thisismypassport commented on Feb 10, 2026

colesbury commented on Mar 2, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

An object referenced only through it's own `dict` can get collected too early. #130327