Add PyObject_GetDict() function

@erlend-aasland

Python has a private _PyObject_GetDictPtr() function which is used by 30 projects of the PyPI top 15,000 projects. It has multiple issues:

It returns a PyObject** type (pointer to an object) which is an unusual type.
It ignores silently (memory allocation) errors: very bad :-(
Using the PyObject** pointer to set an object dictionary is dangerous, because assigning directly to the pointer returned by this function could result in the object and its __dict__ getting out of sync.
Using the PyObject** pointer to set or clear the object dictionary is unsafe with free threading. Free-threading stores and reads dictionary pointer atomically.
The function name is misleading as it suggests that it is getting a pointer, without side effects, and that pointer can be written to. It also perpetuates the idea that all objects have a dictionary (at the C level), and that accessing that dictionary is cheap.
It's private :-)

I propose adding a public PyObject_GetDict() function to get an object __dict__ dictionary.

API: int PyObject_GetDict(PyObject *obj, PyObject **dict)

Get a pointer to __dict__ of the object obj.
- If there is a __dict__, set *dict to a strong reference to the dictionary and return 1.
- If there is no __dict__, set *dict to NULL without setting an exception and return 0.
- On error, set an exception and return -1.

The function is similar to the existing PyObject* PyObject_GenericGetDict(PyObject *o, void *context) function, but it does not raise an exception if the object has no __dict__.

The idea is to replace _PyObject_GetDictPtr() usage with PyObject_GetDict(). Other possible replacements are:

PyObject_GetAttr()
PyObject_SetAttr()
PyObject_ClearManagedDict()
PyObject_VisitManagedDict()

See also:

issue discussions
pull request

Vote:

At the moment I frequently see Rust / PyO3 code do the equivalent of getattr(obj, "__dict__") (listed as one of your replacements). Is the motivation for this specialized API performance?

Would PyObject_GetDict() materialize the dict if an object uses Py_TPFLAGS_INLINE_VALUES?

I don't have a strong opinion about this function, but I looked at a few random projects listed on the issue and this would not replace the usages I saw:

frozendict: Usage of _PyObject_GetDictPtr is verbatim dictobject.c code copied from CPython 3.7-3.10
pybind11: Calls Py_CLEAR() on the field - can't use PyObject_GetDict
cvxpy: no direct uses (uses pybind11)
catboost: Calls Py_CLEAR on the pointer

@davidhewitt

@davidhewitt:

Is the motivation for this specialized API performance?

Yes. But also to not raise an exception if the object has no __dict__.

Object with no dictionary:

class NoDict:
    __slots__ = ()
obj = NoDict()

@colesbury:

Would PyObject_GetDict() materialize the dict if an object uses Py_TPFLAGS_INLINE_VALUES?

Yes, it does:

    PyDictObject *dict = _PyObject_GetManagedDict(obj);
    if (dict == NULL && Py_TYPE(obj)->tp_flags & Py_TPFLAGS_INLINE_VALUES) {
        dict = _PyObject_MaterializeManagedDict(obj);
        ...

Cython may benefit of PyObject_GetDict() in these two specialized functions.

Cython __Pyx_PyObject_GetMethod():

    dictptr = _PyObject_GetDictPtr(obj);
    if (dictptr != NULL && (dict = *dictptr) != NULL) {
        Py_INCREF(dict);
        attr = __Pyx_PyDict_GetItemStr(dict, name);
        ...

Cython __Pyx_get_object_dict_version():

static CYTHON_INLINE PY_UINT64_T __Pyx_get_object_dict_version(PyObject *obj) {
    PyObject **dictptr = NULL;
    Py_ssize_t offset = Py_TYPE(obj)->tp_dictoffset;
    if (offset) {
#if CYTHON_COMPILING_IN_CPYTHON
        dictptr = (likely(offset > 0)) ? (PyObject **) ((char *)obj + offset) : _PyObject_GetDictPtr(obj);
#else
        dictptr = _PyObject_GetDictPtr(obj);
#endif
    }
    return (dictptr && *dictptr) ? __PYX_GET_DICT_VERSION(*dictptr) : 0;
}

__Pyx_PyObject_GetMethod looks like it's only used when compiling for older versions of CPython and ma_version_tag (__PYX_GET_DICT_VERSION) is pending removal.

I really need to be more persuaded that PyObject_GetAttributeString(obj, "__dict__") isn't a good enough alternative.

"It might not have a dict" - then it's not your own type and you shouldn't be messing with its internals. Just use its attributes normally.

What are the use cases for clearing the dict? Are they uses that we want to encourage, or should be we trying to lead people into better ways of achieving the same thing? (All I can think of is doing a type transform on an already-constructed object, which I don't think we want to encourage...)

How is this different from PyObject_GenericGetDict?

Also, _PyObject_GetDictPtr is a documented API. Wouldn't it be better to just make it public with minor changes (make it raise an exception on memory errors, fix the return type to ensure that the returned pointer is not writable)

How is this different from PyObject_GenericGetDict?

"The function is similar to the existing PyObject* PyObject_GenericGetDict(PyObject *o, void *context) function, but it does not raise an exception if the object has no __dict__."

Also, _PyObject_GetDictPtr is a documented API. Wouldn't it be better to just make it public with minor changes (make it raise an exception on memory errors, fix the return type to ensure that the returned pointer is not writable)

The API has multiple issues that I listed in my first message. C extensions use it to set or clear the dict which is bad.

Do you know any project that uses _PyObject_GetDictPtr, doesn't write to the pointer, and couldn't switch to PyType_Lookup?

PyObject_GetDict() doesn't seem to replace properly the private _PyObject_GetDictPtr() function. There are other existing alternatives: PyObject_GenericGetDict() and PyObject_GetAttributeString(obj, "__dict__"). So I don't think that it's worth it to add PyObject_GetDict() function, and I close this issue.

Thanks for your feedback and the constructive discussion.

Navigation Menu

Repository navigation

Add `PyObject_GetDict()` function #82

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add PyObject_GetDict() function #82

Description

Activity

davidhewitt commented on Oct 16, 2025

colesbury commented on Oct 16, 2025

vstinner commented on Oct 16, 2025

colesbury commented on Oct 16, 2025

zooba commented on Oct 16, 2025

ronaldoussoren commented on Oct 17, 2025

vstinner commented on Oct 17, 2025

encukou commented on Oct 20, 2025

vstinner commented on Nov 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Add `PyObject_GetDict()` function #82