Infinite recursion in Pickler.persistent_id

@serhiy-storchaka

BPO	45687
Nosy	@serhiy-storchaka, @embe
Files	pickle-bug.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-02.08:21:56.849>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11', 'library']
title = 'Infinite recursion in Pickler.persistent_id'
updated_at = <Date 2021-11-02.09:16:16.272>
user = 'https://github.com/embe'

bugs.python.org fields:

activity = <Date 2021-11-02.09:16:16.272>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2021-11-02.08:21:56.849>
creator = 'embe-navalgo'
dependencies = []
files = ['50418']
hgrepos = []
issue_num = 45687
keywords = []
message_count = 1.0
messages = ['405494']
nosy_count = 2.0
nosy_names = ['serhiy.storchaka', 'embe-navalgo']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45687'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

The following code, which seems reasonable:

import io
import pickle

class Pickler(pickle.Pickler):
  def persistent_id(self, obj):
    return super().persistent_id(obj)

Pickler(io.BytesIO()).dump(42)

crashes with:
RecursionError: maximum recursion depth exceeded while calling a Python object

It works perfectly when inheriting from pickle._Pickler (the Python implementation).

I think it's because in the C version persistent_id can be NULL (but it's still in dir()):

>>> pickle.Pickler(io.BytesIO()).persistent_id
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: persistent_id
>>> dir(pickle.Pickler(io.BytesIO()))
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'bin', 'clear_memo', 'dispatch_table', 'dump', 'fast', 'memo', 'persistent_id']
>>>

on the other hand:

>>> pickle._Pickler(io.BytesIO()).persistent_id
<bound method _Pickler.persistent_id of <pickle._Pickler object at 0x10ce42220>>

@serhiy-storchaka

(gdb) p _PyObject_Dump(self->pers_func)
object address  : 0x7ffff74b90d0
object refcount : 3
object type     : 0x555555a6f020
object type name: function
object repr     : <function Pickler.persistent_id at 0x7ffff74b90d0>
$1 = void
(gdb) c
Continuing.

Breakpoint 1, Pickler_get_persid (self=0x7ffff75c13e0, _unused_ignored=0x0) at ./Modules/_pickle.c:5004
5004        if (self->pers_func == NULL) {
(gdb) p _PyObject_Dump(self->pers_func)
object address  : 0x7ffff74b90d0
object refcount : 4
object type     : 0x555555a6f020
object type name: function
object repr     : <function Pickler.persistent_id at 0x7ffff74b90d0>
$2 = void
(gdb) c
Continuing.

Breakpoint 1, Pickler_get_persid (self=0x7ffff75c13e0, _unused_ignored=0x0) at ./Modules/_pickle.c:5004
5004        if (self->pers_func == NULL) {
(gdb) p _PyObject_Dump(self->pers_func)
object address  : 0x7ffff74b90d0
object refcount : 5
object type     : 0x555555a6f020
object type name: function
object repr     : <function Pickler.persistent_id at 0x7ffff74b90d0>
$3 = void
(gdb)

The refcount of self->pers_func appears to be increasing which causes the recursion error. I tried to decrease the refcount (I reverted the changes for this function on this PR, by the way, I didn't see any reference cycles @serhiy-storchaka ) Recursion error is fixed but I am getting TypeError

Traceback (most recent call last):
  File "/home/furkan/cpython/bug.py", line 8, in <module>
    Pickler(io.BytesIO()).dump(42)
  File "/home/furkan/cpython/bug.py", line 6, in persistent_id
    return super().persistent_id(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Pickler.persistent_id() missing 1 required positional argument: 'obj'

pythongh-89850: Add default C implementations of persistent_id() and …

The C implementation saves references to persistent_id and persistent_load methods for performance reason. It saves a method lookup and creation. Also, since the C implementation of Pickler does not have the persistent_id method by default, it allows to avoid calling the trivial method at all.

Since saving a reference to a method creates a reference loop, it uses a trick: deconstruct it if it is a bound method and save only reference to an unbound method. When read the persistent_id attribute, the bound method is reconstructed back.

But it does not work with super(). super() simply skips the current class when looks up the attribute in the MRO. But in this case it cames to the persistent_id descriptor in the base class which returns a bound method created from an unbound method of the subclass. So calling it leads to an infinite recursion.

There is the same issue with persistent_load method.

PR #113579 uses a different approach, similar to the one used for reducer_override. It removes persistent_id and persistent_load descriptors and caches the bound methods, but only for the time of execution dump() and load() methods. It also adds the default implementations of persistent_id and persistent_load methods in the C classes as well as in the Python classes.

This approach may have disadvantage: calling dump() and load() for small objects may now be slower due to added constant overhead. But I hope that the overhead is insignificant in comparison with other operations (IO, memory management).

gh-89850: Add default C implementations of persistent_id() and persis…

pythongh-89850: Add default C implementations of persistent_id() and …

Navigation Menu

Repository navigation

Infinite recursion in Pickler.persistent_id #89850

Linked PRs

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Infinite recursion in Pickler.persistent_id #89850

Description

Linked PRs

Activity

embe commented on Nov 2, 2021

4 remaining items

iritkatriel commented on Jun 1, 2022

furkanonder commented on Dec 20, 2022

serhiy-storchaka commented on Dec 30, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions