Fix pickling exceptions with multiple arguments #76877

Activity

1. slallum commented on Jan 28, 2018

   

   slallummannequin

   on Jan 28, 2018

   MannequinAuthor

   More actions

   Pickling exceptions fails when a custom exception class requires multiple arguments.

   import pickle
   class MultipleArgumentsError(Exception):
       def __init__(self, a, b):
           self.a = a
           self.b = b
   
   pickle.loads(pickle.dumps(MultipleArgumentsError('a', 'b')))

   this code produces the following error:

   Traceback (most recent call last):
     File "/tmp/multiple_arguments_exception.py", line 8, in <module>
       pickle.loads(pickle.dumps(MultipleArgumentsError('a', 'b')))
     File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1388, in loads
       return Unpickler(file).load()
     File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 864, in load
       dispatch[key](self)
     File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1139, in load_reduce
       value = func(*args)
   TypeError: __init__() takes exactly 3 arguments (2 given)

   The same error occurs when using a built-in module like subprocess:
   >>> import subprocess, pickle
   >>> try:
   ...   subprocess.check_call(['python', '-c', 'raise SystemExit(1)'])
   ... except Exception as e:
   ...   pickle.loads(pickle.dumps(e))
   ... 

   Related issue: https://bugs.python.org/issue1692335
2. slallum

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jan 28, 2018
3. KirillMatsaberydze commented on Apr 12, 2018

   

   KirillMatsaberydzemannequin

   on Apr 12, 2018

   Mannequin

   More actions

   Hi, I encounter similar behavior in python 3.6.5 with following code:

   import pickle
   class CustomException(Exception):
       def __init__(self, arg1, arg2):
           msg = "Custom message {} {}".format(arg1, arg2)
           super().__init__(msg)
   
   
   obj_dump = pickle.dumps(CustomException("arg1", "arg2"))
   obj = pickle.loads(obj_dump)

   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   TypeError: __init__() missing 1 required positional argument: 'arg2'

   So it looks like it not only python 2.7 problem
4. jaraco commented on Apr 12, 2018

   

   jaraco

   on Apr 12, 2018

   Member

   More actions

   Kirill, see https://bugs.python.org/issue1692335#msg310951 in the related issue for one possible way to work around the issue on Python 3.
5. iritkatriel commented on Jun 18, 2021

   

   iritkatriel

   on Jun 18, 2021

   Member

   More actions

   This is still the same in 3.11:

   >>> import pickle
   >>> class CustomException(Exception):
   ...     def __init__(self, arg1, arg2):
   ...         msg = "Custom message {} {}".format(arg1, arg2)
   ...         super().__init__(msg)
   ...
   >>> obj_dump = pickle.dumps(CustomException("arg1", "arg2"))
   >>> obj = pickle.loads(obj_dump)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   TypeError: CustomException.__init__() missing 1 required positional argument: 'arg2'
   >>>
6. iritkatriel

   added

   3.11only security fixesonly security fixes

   on Jun 18, 2021
7. Cryvate commented on Jun 18, 2021

   

   Cryvatemannequin

   on Jun 18, 2021

   Mannequin

   More actions

   It seems like the example in the OP now works (persist both arguments), but Irit's/Kirill's (create a msg) fails (I am running 3.9).
8. iritkatriel commented on Jun 27, 2021

   

   iritkatriel

   on Jun 27, 2021

   Member

   More actions

   See also bpo-43460, bpo-30005, bpo-29466
9. iritkatriel commented on Jan 6, 2022

   

   iritkatriel

   on Jan 6, 2022

   Member

   More actions

   This isn't really an issue with exceptions, it's just that __reduce__ goes out of sync with the object's constructor signature. Here's a simple example of the same:

   class Base:
      def __init__(self, msg):
          self.msg = msg
   
      def __reduce__(self):
          return type(self), (self.msg,)
   
   class Derived(Base):
      def __init__(self, a, b):
          super().__init__(f'{a}|{b}')
   
   x = Derived('a', 'b')
   assert x.msg == 'a|b'
   y = pickle.dumps(x, -1)
   z = pickle.loads(y)

   ---

   Output:

   Traceback (most recent call last):
     File "/Users/iritkatriel/src/cpython/zz.py", line 22, in <module>
       z = pickle.loads(y)
           ^^^^^^^^^^^^^^^
   TypeError: Derived.__init__() missing 1 required positional argument: 'b'

   If I define __reduce__ on Derived to return an arg tuple of the right length for its __init__, it works:

   class Derived(Base):
      def __init__(self, a, b):
          super().__init__(f'{a}|{b}')
   
      def __reduce__(self):
          return type(self), tuple(self.msg.split('|'))

   But note that this is not something we could make the base class do, it has no way of finding out what the semantics of the derived class constructor args are.

   👍React with 👍1Reacted by Luke Yeager
10. iritkatriel

    closed this as completedon Jan 6, 2022
11. iritkatriel

    added

    invalid

    on Jan 6, 2022
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022
13. youtux commented on Oct 30, 2022

    

    youtux

    on Oct 30, 2022

    Last edited by youtux

    More actions

    Should it be documented in the docs how to implement __reduce__ on custom exceptions?