Repository navigation
Argumentless super() fails in classes constructed with type() #74130
Description
Activity
mr-nfamous commented
on Mar 30, 2017 mr-nfamousmannequinMannequinAuthorThe simplest example:
def mydec(cls): return type(cls.__name__, cls.__bases__, dict(cls.__dict__)) @mydec class MyList(list): def extend(self, item): super(MyList, self).extend(item) def insert(self, index, object): super().insert(index, object)
>>> lst = MyList() >>> lst.extend([2,3]) >>> lst.insert(0, 1) TypeError: super(type, obj): obj must be an instance or subtype of type >>> lst [2, 3]
If this is intended behavior, at least the error message could be fixed.
Reacted by Ryan SiemensInterestingly, even
types.new_classmisbehaves in this case:>>> def mydec(cls): ... return types.new_class(cls.__name__, cls.__bases__, exec_body=lambda ns: ns.update(cls.__dict__)) ... >>> @mydec ... class MyList(list): ... def insert(self, idx, obj): ... super().insert(idx, obj) ... >>> MyList().insert(0, 1) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in insert TypeError: super(type, obj): obj must be an instance or subtype of typeThe error message is confusing here because it's using 'type' as a metavariable to refer to the first argument of super(), *not* to the builtin called "type".
If we poke around in the MyList.insert closure, we can see the origin of the problem:
>>> MyList.insert.__closure__[0].cell_contents <class '__main__.MyList'> >>> MyList.insert.__closure__[0].cell_contents is MyList FalseThe class cell is still bound to the originally created class object, but the decorator threw that away and returned a completely different object. As a result, the "self" passed to the bound method is *not* an instance of the type stored in the "__class__" cell, and super() complains.
Given that super() doesn't take keyword arguments, perhaps we should propose on python-dev that "type" be changed to "starting_type" in the super() error messages and documentation?
Reacted by Ryan SiemensMojoVampire commented
on Mar 30, 2017 MojoVampiremannequinMannequinIt looks like this is a general problem caused by the fact that a function that is:
- Defined in a class
- References the name "super" (even if it's a local variable name, even if it's never called)
isn't "just" a plain function. Even though Python 3 officially removed the concept of unbound methods, there is still some unbound method-like behavior going on, which makes that sort of function intrinsically tied to where it executes.
Specifically, a function of that sort is effectively a closure, where the closure scope defines __class__ (insert.__code__.co_freevars will be non-empty with the string '__class__' in it); the value of that closure variable cell is set when the class finishes being defined (that is, the closure cell is empty when the def block finishes, but when the class block itself completes, the cell is populated with a reference to the class).
No argument super (specifically, super_init) relies on this assistance, as it looks up the __class__ cell in the caller's frame when not provided a class explicitly. In your repro, the problem is that __class__ is set to the original MyList (it was set at the moment the class MyList(list): block finished, before the decorator was invoked), but at call time, the self instance is of the new type you created (which has the same name, but the name isn't looked up, it has a cached reference to the class).
The same problem applies if you do:
class MyList(list): def insert(self, index, object): super().insert(index, object) class MyOtherList(list): insert = MyList.insert
because MyList.insert is permanently tied to MyList, it doesn't get "reclosured" as a result of MyOtherList using it. Similarly:
class MyList(list): pass def insert(self, index, object): super().insert(index, object) MyList.insert = insert
fails when insert is called with error: "RuntimeError: super(): __class__ cell not found", because functions def-ed outside a class block entirely don't have __class__ as "virtual closure" scope.
It's arguable whether this should be changed: If you're using super() without arguments, the implicit behavior is that it works with "whatever class I'm currently defining", so trying to reuse it with some other class is trying to extend that implicit behavior to "whatever class I was eventually assigned to". Making it work would mean that already "class bound" methods must be rebound when assigned to a new class, and non-bound methods must be bound (in both cases, you'd need to handle this when implicitly assigned in the class body, or explicitly assigned later as an attribute on the constructed class).
Doing so would break at least one pattern someone might be using already, where they have multiple inheritance, and while most methods should prioritize the first class in the MRO, they want a specific method to bypass one or more classes in the MRO without calling it. For example:
class Stuff: # Top of diamond, just so super().stuff() valid in all children def stuff(self): print("Stuff") class Spam: def stuff(self): print("Spam") super().stuff() class Eggs: def stuff(self): print("Eggs") super().stuff() class SpamAndEggsWithoutSpam(Spam, Eggs): stuff = Eggs.stuff # I'd like my stuff without Spam
Under the current design, Eggs.stuff is already bound to Eggs, so this works; you see:
Eggs
Stuffprinted, but not Spam. If rebinding were made a thing, you'd end up with:
Eggs
Spam
Eggs
Stuffwith the first Eggs being from the rebound method.
MojoVampire commented
on Mar 30, 2017 MojoVampiremannequinMannequinThis is what I get for leaving a response up and working on it intermittently for three hours, and not rechecking: I found the same basic problem.
For the record, I also found a terribly hacky way of doing this sort of rebinding manually, should you actually need to do so under the current design:
def make_class_closure(__class__): '''Makes a one-tuple closure with a cell for the provided class''' return (lambda: super).__closure__ class MyList(list): def insert(self, index, object): super().insert(index, object) class MyList2(list): pass MyList2.insert = types.FunctionType(MyList.insert.__code__, globals(), closure=make_class_closure(MyList2))
It would need further improvements to recognize when it needs to run, deal with existing closures, handle top-level functions without __class__ in their closure co_freevars, etc., and I won't even pretend that's a sane workaround for the problem, but it does demonstrate what is happening/what would need to be changed to make it work, if indeed that's desirable.
Right, there's a very similar instance-method-reuse related problem described in http://bugs.python.org/issue29270, where ctypes re-uses a class namespace is define a swapped-endianness version of the originally defined class.
The easiest place to deflect conflicts is at the point where bound instance methods are created:
>>> bound_method = MyList().insert >>> bound_method.__self__.__class__ <class '__main__.MyList'> >>> bound_method.__func__.__closure__[0].cell_contents <class '__main__.MyList'> >>> bound_method.__self__.__class__ is bound_method.__func__.__closure__[0].cell_contents False
However, that method of detection only works for plain instance methods that only close over the
__class__variable: as soon as you wrap them in a decorator, you may not have easy access to the__closure__attribute any more, and if the method has closure references to more than just__class__, then it's a bit more work to find the right closure index from outside the function.- changed the title
[-]Argumentless super() calls do not work in classes constructed with type()[/-][+]Argumentless super() fails in classes constructed with type()[/+]on Apr 1, 2017 mr-nfamous commented
on Apr 9, 2017 mr-nfamousmannequinMannequinAuthorJust wanted to add that I found this when I was trying to find a way to decorate all methods in a class without using a metaclass or modifying __getattr__.
Using Josh's workaround, here's a simple demonstration that (at least at first glance) prints a message when a method is called and when it is finished:
def mydec(*, enter=True, exit=True): def make_closure(__class__): return (lambda: super).__closure__ def outer(cls): def wrapper(method): @functools.wraps(method) def inner(*args, **kwargs): if enter: print('entering', method.__qualname__) r = method(*args, **kwargs) if exit: print('exiting', method.__qualname__) return method(*args, **kwargs) return inner for name, value in cls.__dict__.items(): if isinstance(value, types.FunctionType): method = types.FunctionType(getattr(cls, name).__code__, globals(), closure=make_closure(cls)) setattr(cls, name, wrapper(method)) return cls return outer
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: