Argumentless super() fails in classes constructed with type() #74130

Activity

1. mr-nfamous commented on Mar 30, 2017

   

   mr-nfamousmannequin

   on Mar 30, 2017

   MannequinAuthor

   More actions

   The simplest example:

   def mydec(cls):
       return type(cls.__name__, cls.__bases__, dict(cls.__dict__))
   
   @mydec
   class MyList(list):
       
       def extend(self, item):
           super(MyList, self).extend(item)
           
       def insert(self, index, object):
           super().insert(index, object)

   >>> lst = MyList()
   >>> lst.extend([2,3])
   >>> lst.insert(0, 1)
   TypeError: super(type, obj): obj must be an instance or subtype of type
   >>> lst
   [2, 3]

   If this is intended behavior, at least the error message could be fixed.

   👍React with 👍1Reacted by Ryan Siemens
2. mr-nfamous

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Mar 30, 2017
3. ncoghlan commented on Mar 30, 2017

   

   ncoghlan

   on Mar 30, 2017

   Contributor

   More actions

   Interestingly, even types.new_class misbehaves in this case:

   >>> def mydec(cls):
   ...     return types.new_class(cls.__name__, cls.__bases__, exec_body=lambda ns: ns.update(cls.__dict__))
   ... 
   >>> @mydec
   ... class MyList(list):
   ...     def insert(self, idx, obj):
   ...         super().insert(idx, obj)
   ... 
   >>> MyList().insert(0, 1)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "<stdin>", line 4, in insert
   TypeError: super(type, obj): obj must be an instance or subtype of type
   

   The error message is confusing here because it's using 'type' as a metavariable to refer to the first argument of super(), *not* to the builtin called "type".

   If we poke around in the MyList.insert closure, we can see the origin of the problem:

   >>> MyList.insert.__closure__[0].cell_contents
   <class '__main__.MyList'>
   >>> MyList.insert.__closure__[0].cell_contents is MyList
   False
   

   The class cell is still bound to the originally created class object, but the decorator threw that away and returned a completely different object. As a result, the "self" passed to the bound method is *not* an instance of the type stored in the "__class__" cell, and super() complains.

   Given that super() doesn't take keyword arguments, perhaps we should propose on python-dev that "type" be changed to "starting_type" in the super() error messages and documentation?

   👍React with 👍1Reacted by Ryan Siemens
4. MojoVampire commented on Mar 30, 2017

   

   MojoVampiremannequin

   on Mar 30, 2017

   Mannequin

   More actions

   It looks like this is a general problem caused by the fact that a function that is:

   1. Defined in a class
   2. References the name "super" (even if it's a local variable name, even if it's never called)

   isn't "just" a plain function. Even though Python 3 officially removed the concept of unbound methods, there is still some unbound method-like behavior going on, which makes that sort of function intrinsically tied to where it executes.

   Specifically, a function of that sort is effectively a closure, where the closure scope defines __class__ (insert.__code__.co_freevars will be non-empty with the string '__class__' in it); the value of that closure variable cell is set when the class finishes being defined (that is, the closure cell is empty when the def block finishes, but when the class block itself completes, the cell is populated with a reference to the class).

   No argument super (specifically, super_init) relies on this assistance, as it looks up the __class__ cell in the caller's frame when not provided a class explicitly. In your repro, the problem is that __class__ is set to the original MyList (it was set at the moment the class MyList(list): block finished, before the decorator was invoked), but at call time, the self instance is of the new type you created (which has the same name, but the name isn't looked up, it has a cached reference to the class).

   The same problem applies if you do:

   class MyList(list):
       def insert(self, index, object):
           super().insert(index, object)
   
   class MyOtherList(list):
       insert = MyList.insert

   because MyList.insert is permanently tied to MyList, it doesn't get "reclosured" as a result of MyOtherList using it. Similarly:

   class MyList(list):
       pass
   
   def insert(self, index, object):
       super().insert(index, object)
   
   MyList.insert = insert

   fails when insert is called with error: "RuntimeError: super(): __class__ cell not found", because functions def-ed outside a class block entirely don't have __class__ as "virtual closure" scope.

   It's arguable whether this should be changed: If you're using super() without arguments, the implicit behavior is that it works with "whatever class I'm currently defining", so trying to reuse it with some other class is trying to extend that implicit behavior to "whatever class I was eventually assigned to". Making it work would mean that already "class bound" methods must be rebound when assigned to a new class, and non-bound methods must be bound (in both cases, you'd need to handle this when implicitly assigned in the class body, or explicitly assigned later as an attribute on the constructed class).

   Doing so would break at least one pattern someone might be using already, where they have multiple inheritance, and while most methods should prioritize the first class in the MRO, they want a specific method to bypass one or more classes in the MRO without calling it. For example:

   class Stuff:  # Top of diamond, just so super().stuff() valid in all children
       def stuff(self):
           print("Stuff")
   
   class Spam:
       def stuff(self):
           print("Spam")
           super().stuff()
   
   class Eggs:
       def stuff(self):
           print("Eggs")
           super().stuff()
   
   class SpamAndEggsWithoutSpam(Spam, Eggs):
       stuff = Eggs.stuff  # I'd like my stuff without Spam

   Under the current design, Eggs.stuff is already bound to Eggs, so this works; you see:

   Eggs
   Stuff

   printed, but not Spam. If rebinding were made a thing, you'd end up with:

   Eggs
   Spam
   Eggs
   Stuff

   with the first Eggs being from the rebound method.
5. MojoVampire commented on Mar 30, 2017

   

   MojoVampiremannequin

   on Mar 30, 2017

   Mannequin

   More actions

   This is what I get for leaving a response up and working on it intermittently for three hours, and not rechecking: I found the same basic problem.

   For the record, I also found a terribly hacky way of doing this sort of rebinding manually, should you actually need to do so under the current design:

   def make_class_closure(__class__):
       '''Makes a one-tuple closure with a cell for the provided class'''
       return (lambda: super).__closure__
   
   class MyList(list):
       def insert(self, index, object):
           super().insert(index, object)
   
   class MyList2(list):
       pass
   
   MyList2.insert = types.FunctionType(MyList.insert.__code__, globals(), closure=make_class_closure(MyList2))

   It would need further improvements to recognize when it needs to run, deal with existing closures, handle top-level functions without __class__ in their closure co_freevars, etc., and I won't even pretend that's a sane workaround for the problem, but it does demonstrate what is happening/what would need to be changed to make it work, if indeed that's desirable.
6. ncoghlan commented on Mar 31, 2017

   

   ncoghlan

   on Mar 31, 2017

   Contributor

   More actions

   Right, there's a very similar instance-method-reuse related problem described in http://bugs.python.org/issue29270, where ctypes re-uses a class namespace is define a swapped-endianness version of the originally defined class.

   The easiest place to deflect conflicts is at the point where bound instance methods are created:

   >>> bound_method = MyList().insert
   >>> bound_method.__self__.__class__
   <class '__main__.MyList'>
   >>> bound_method.__func__.__closure__[0].cell_contents
   <class '__main__.MyList'>
   >>> bound_method.__self__.__class__ is bound_method.__func__.__closure__[0].cell_contents
   False

   However, that method of detection only works for plain instance methods that only close over the __class__ variable: as soon as you wrap them in a decorator, you may not have easy access to the __closure__ attribute any more, and if the method has closure references to more than just __class__, then it's a bit more work to find the right closure index from outside the function.
7. terryjreedy

   changed the title [-]Argumentless super() calls do not work in classes constructed with type()[/-] [+]Argumentless super() fails in classes constructed with type()[/+] on Apr 1, 2017
8. vadmium commented on Apr 9, 2017

   

   vadmium

   on Apr 9, 2017

   Member

   More actions

   In bpo-23674, I posted a patch that changes to consistent parameter names (subclass, self). The exception message would avoid “type”, becoming

   super(subclass, self): self must be an instance or subtype of subclass
9. mr-nfamous commented on Apr 9, 2017

   

   mr-nfamousmannequin

   on Apr 9, 2017

   MannequinAuthor

   More actions

   Just wanted to add that I found this when I was trying to find a way to decorate all methods in a class without using a metaclass or modifying __getattr__.

   Using Josh's workaround, here's a simple demonstration that (at least at first glance) prints a message when a method is called and when it is finished:

   def mydec(*, enter=True, exit=True):
       def make_closure(__class__):
           return (lambda: super).__closure__
       def outer(cls):
           def wrapper(method):
               @functools.wraps(method)
               def inner(*args, **kwargs):
                   if enter:
                       print('entering', method.__qualname__)
                   r = method(*args, **kwargs)
                   if exit:
                       print('exiting', method.__qualname__)
                   return method(*args, **kwargs)
               return inner    
           for name, value in cls.__dict__.items():
               if isinstance(value, types.FunctionType):
                   method = types.FunctionType(getattr(cls, name).__code__,
                                               globals(),
                                               closure=make_closure(cls))
                   setattr(cls, name, wrapper(method))
           return cls
       return outer
10. vstinner commented on Mar 28, 2022

    

    vstinner

    on Mar 28, 2022

    Member

    More actions

    See also bpo-47143 "Add functools.copy_class() which updates closures".
11. ezio-melotti

    transferred this issue fromon Apr 10, 2022
12. iritkatriel

    added

    interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

    on Nov 23, 2023