Zero argument super is broken in 3.6 for methods with a hacked __class__ cell #76357

Activity

1. mr-nfamous commented on Nov 30, 2017

   

   mr-nfamousmannequin

   on Nov 30, 2017

   MannequinAuthor

   More actions

   The following code works in 3.3, 3.4, and 3.5, but in 3.6 it throws RuntimeError: super(): bad __class__ cell.

   from types import FunctionType, CodeType
   
   def create_closure(__class__):
       return (lambda: __class__).__closure__
   
   def new_code(c_or_f):
       '''A new code object with a __class__ cell added to freevars'''
       c = c_or_f.__code__ if isinstance(c_or_f, FunctionType) else c_or_f
       return CodeType(
           c.co_argcount, c.co_kwonlyargcount, c.co_nlocals,
           c.co_stacksize, c.co_flags, c.co_code, c.co_consts, c.co_names,
           c.co_varnames, c.co_filename, c.co_name, c.co_firstlineno,
           c.co_lnotab, c.co_freevars + ('__class__',), c.co_cellvars)
   
   def add_foreign_method(cls, f):
       code = new_code(f.__code__)
       name = f.__name__
       defaults = f.__defaults__
       closure = (f.__closure__ or ()) + create_closure(cls)
       setattr(cls, name, FunctionType(code, globals(), name, defaults, closure))
   
   class List(list):
       def append(self, elem):
           super().append(elem)
       def extend(self, elems):
           super().extend(elems)
   def __getitem__(self, i):
       print('foreign getitem')
       return super().__getitem__(i)
   
   add_foreign_method(List, __getitem__)
   
   self = List([1,2,3])
   self[0]
2. mr-nfamous

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Nov 30, 2017
3. mr-nfamous commented on Nov 30, 2017

   

   mr-nfamousmannequin

   on Nov 30, 2017

   MannequinAuthor

   More actions

   The hacked cell object using this method appears to be changed to NULL when accessed by frame.f_localsplus. I don't know C well enough to find out what's happening because nothing looks different to me in PyFrame_FastToLocalsWithError.

   Also creating a closure with:
   from ctypes import pythonapi, py_object

   new_cell = pythonapi.PyCell_New
   new_cell.argtypes = (py_object, )
   new_cell.restype = py_object

   doesn't solve this either.
4. mr-nfamous commented on Dec 1, 2017

   

   mr-nfamousmannequin

   on Dec 1, 2017

   MannequinAuthor

   More actions

   So while CO_NOFREE is set in all versions with the example code, it appears only python 3.6 recognizes that flag and disallows the accessing of the __class__ cell. In this case the error message is bad because it implies that the __class__ cell is the wrong type. Disabling the flag when creating the code objects allows the above code to work.
5. ncoghlan commented on Dec 2, 2017

   

   ncoghlan

   on Dec 2, 2017

   Contributor

   More actions

   Given that, I'd say the way to cleanest way to fix this would be to remove these lines from "compute_code_flags" in compile.c:

   if (!PyDict_GET_SIZE(c->u->u_freevars) &&
       !PyDict_GET_SIZE(c->u->u_cellvars)) {
       flags |= CO_NOFREE;
   }
   

   and replace them with a check like the following in PyCode_New just after we ensure the Unicode string for the filename is ready:

   if (!PyTuple_GET_SIZE(freevars) &&
       !PyTuple_GET_SIZE(cellvars)) {
       flags |= CO_NOFREE;
   }
   

   That way CO_NOFREE will be set only when appropriate regardless of how the code object is created, rather than relying on the caller to set it correctly.
6. ncoghlan commented on Dec 3, 2017

   

   ncoghlan

   on Dec 3, 2017

   Contributor

   More actions

   New changeset 078f181 by Nick Coghlan in branch 'master':
   bpo-32176: Set CO_NOFREE in the code object constructor (GH-4675)
   078f181
7. ncoghlan commented on Dec 3, 2017

   

   ncoghlan

   on Dec 3, 2017

   Contributor

   More actions

   New changeset c8f32aa by Nick Coghlan in branch '3.6':
   [3.6] bpo-32176: Set CO_NOFREE in the code object constructor (GH-4684)
   c8f32aa
8. ncoghlan commented on Dec 3, 2017

   

   ncoghlan

   on Dec 3, 2017

   Contributor

   More actions

   Thanks for the issue report! The fix will be released in 3.6.4 and 3.7.0a3 (both of which are expected to be later this month).
9. ncoghlan

   added

   3.7 (EOL)end of lifeend of life

   on Dec 3, 2017
10. ncoghlan

    closed this as completedon Dec 3, 2017
11. ezio-melotti

    transferred this issue fromon Apr 10, 2022