Repository navigation
crash appending list and namedtuple #53093
Description
Activity
benrg commented
on May 28, 2010 benrgmannequinMannequinAuthorc:\>python Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from collections import namedtuple >>> foo = namedtuple('foo', '') >>> [1] + foo()At this point the interpreter crashes. Also happens when foo has named arguments, and in batch scripts. foo() + [1] throws a TypeError as expected. [] + foo() returns (). The immediate cause of the crash is the CALL instruction at 1E031D5A in python31.dll jumping into uninitialized memory.
I can't reproduce this on either 3.1.2 or py3k trunk.
Running the exact same binary on winxp with an amd athlon processor,
I *did* get a crash after about 5 seconds. "python.exe has encountered a problem and needs to close. We are sorry for the inconvenience."
Trying again with IDLE instead of the command window, I get the same message (for pythonw instead of python) and the shell restarts when I close the message.Even though foo()+[1] correctly raises a TypeError, the reverse [1] + foo() very bizarrely produces a length 1 tuple whose bizarre member is supposedly an instance of a empty list [] with length 1
from collections import namedtuple foo = namedtuple('foo', '') a = [1] + foo() b=a[0] print (type(a), len(a), type(b), len(type(b)), type(type(b)))
# <class 'tuple'> 1 [] 1 <class 'list'>
([2]+foo() produces same output)
Other than the above, any attempt to do anything with b or type(b) that I tried crashes. I presume that this is due to attribute lookups on the invalid type(b) (or something like that), which type(b) must bypass. In particular, the OP's crash is due to trying to print the tuple which tries to print its member which looks for type(b).__str__ which crashes.
To anyone: investigating crashers like this is a lot easier with IDLE and an edit window than with the interactive command window.
More experiments
from collections import namedtuple foo = namedtuple('foo', '') a = [] + foo() print (a, type(a), len(a)) # () <class 'tuple'> 0
ie, a standard empty tuple, whereas
a = [1,1] + foo()
crashes immediately. So the behavior of list()+namedtuple depends on the length of the list.There are also some funny interactions. Adding
try:
a = foo()+[]
except TypeError:
print("correct TypeError")after the 'foo = ' line in my original 5 line example causes the final print to crash, whereas adding the same 4 lines to the 4 line example at the beginning of this message does not obviously change anything.
David, since you omitted all details, I wonder if you tested in batch mode, as I did, and hence never tried to print the malformed object, or used different OS or hardware.
I can't reproduce on 3k trunk with Ubuntu 10.04, gcc 4.4.3
namedtuples are just a subclass of tuple with only two dunder methods defined (a plain __new__ with empty __slots__). Can you provoke the same behavior with plain tuples, or a subclass of tuple that looks like one of these?
class Crasher(tuple): pass class Crasher(tuple): __slots__ = () class Crasher(tuple): def __new__(cls,): return tuple.__new__(cls,) __slots__ = ()
Substituting
foo = tuple()
TypeError: can only concatenate list (not "tuple") to listclass Crasher(tuple): pass foo = Crasher() a = [1] + foo b=a[0] print (type(a), len(a), type(b), len(type(b)), type(type(b)))
<class 'tuple'> 1 [] 1 <class 'list'>
as before, so namedtuple is not, in particular, the culprit.
Other two Crasher versions do the same. I also get a delayed pythonw error message after the print that does not cause the shell to restart. This may partly be an IDLE artifact.Two more probes:
-
does it also have the same strange/crashy behavior when you subclass list and concat that to a tuple?
-
does dropping the optimization level down to -O help? This has "compiler quirk" written all over it. The C-code for list and tuple concat are almost identical, and both start with a type check.
-
"can't reproduce" does not inform as to what *did* happen with which code.
More experiments:
foo = str()
TypeError: can only concatenate list (not "str") to listclass s(str): pass foo = s() TypeError: Can't convert 'list' object to str implicitly
Why is it trying to do that? Of course, the interpreter can (implicitly) convert list to tuple, which must be what happens in OP example.
The subclasses of tuple and str do not gain an __radd__ method. If we add one
class s(str): def __radd__(s,o): print('in radd, type(o)') foo = s() a = [1] + foo # prints "in radd <class 'list'>"
no implicit conversion is done.
Reversing tuple and list
class Crasher(list): pass a = () + Crasher() # or Crasher([1]) print(a, type(a), len(a)) #[] <class 'list'> 0 # or [1] <class 'list'> 1
whereas
a = (1,) + Crasher()
crashes, so not completely symmetrical- answered before you asked (yes, similar)
- (same thought) I am using PSF windows installer, however that was prepared. Martin?
if the id() of the left operand is identical to the id() of the first element in the result it would strongly support compiler skulldugerry.
class Crasher(tuple): pass foo = Crasher() x = [1] a = x + foo b=a[0] if id(b) == id(x): raise Exception("It's the C compiler what did it!")
The only way I can think of this coming about is the right_op getting new'd and then .extend'ing(left_op). That extend() must be going batsh*t and inserting the left_op instead of it's contained items. The C-code for extend is more fiddly than the code for concatenation so there is more room for the compiler to generate bad code.
Good try, but for one run, the ids of foo, x, a, and b are >>> 15719440 15717880 15273104 12266976
21 remaining items
Antoine Pitrou <report@bugs.python.org> wrote:
> I think we want to add those tests to the test suite as well.What's a good place? Shall we just add one of the tests to test_tuple?
Sounds good. And another of them to test_list perhaps as well :)
Also, the only person to run the tests with the PGO build will probably
be Martin just before the releases. :)True, but other people may still run them *after* the release.
skrah commented
on Jul 31, 2012 skrahmannequinMannequinNew patches with tests for 3.2 and 3.3. For 3.2 I determined empirically
that EnableCOMDATFolding="1" (and not "0") turns on NOICF.If anyone can confirm that this is the case or has a pointer to
the relevant vcproj docs, I'd be thrilled.loewis commented
on Jul 31, 2012 loewismannequinMannequinAlso, the only person to run the tests with the PGO build will probably
be Martin just before the releases. :)We could set up a buildbot slave which does PGO builds, provided somebody
volunteered an installation (including VS Pro), and somebody contributed
a build script that deviates from the regular build (perhaps including some
training on the PGI). This is a separate issue, of course.loewis commented
on Jul 31, 2012 loewismannequinMannequinIf anyone can confirm that this is the case or has a pointer to
the relevant vcproj docs, I'd be thrilled.While the actual values for the XML schema aren't documented, I expect that
they have the numeric values that they have in C++ (i.e. optFoldingDefault=0,
optNoFolding=1, optFolding=2)To confirm, just look up the setting in the UI.
python-dev commented
on Aug 1, 2012 python-devmannequinMannequinNew changeset 5a8c5631463f by Martin v. Löwis in branch '2.7':
Issue bpo-8847: Disable COMDAT folding in Windows PGO builds.
http://hg.python.org/cpython/rev/5a8c5631463fpython-dev commented
on Aug 1, 2012 python-devmannequinMannequinNew changeset 2638ce032151 by Martin v. Löwis in branch '3.2':
Issue bpo-8847: Disable COMDAT folding in Windows PGO builds.
http://hg.python.org/cpython/rev/2638ce032151New changeset 029cde4e58c5 by Martin v. Löwis in branch 'default':
Issue bpo-8847: Disable COMDAT folding in Windows PGO builds.
http://hg.python.org/cpython/rev/029cde4e58c5New changeset d3afe5d8a4da by Martin v. Löwis in branch 'default':
Issue bpo-8847: Merge with 3.2
http://hg.python.org/cpython/rev/d3afe5d8a4daloewis commented
on Aug 1, 2012 loewismannequinMannequinThanks for the research and the fix!
You didn't add any test for non regression??
Reacted by Gastón Avilaloewis commented
on Aug 1, 2012 loewismannequinMannequinYou didn't add any test for non regression??
Please rephrase your question: what tests did I not add?
I did add the tests that Stefan proposed.Please rephrase your question: what tests did I not add?
I did add the tests that Stefan proposed.Ah yes, you added new tests to Python 3.2 and 3.3, but no to Python
2.7. Why not adding these new tests to Python 2.7?loewis commented
on Aug 1, 2012 loewismannequinMannequinAh yes, you added new tests to Python 3.2 and 3.3, but no to Python
2.7. Why not adding these new tests to Python 2.7?The tests don't crash Python 2.7. So they are not useful as a test whether
the bug has been worked-around. I actually don't know how to test this
compiler
bug in Python 2.7 (except for writing specific C code that tries to trigger
the bug).- added 2 commits that reference this issue
on May 25, 2025
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: