Repository navigation
Stack overflow exception caused by test_marshal on Windows x64 #46539
Description
| BPO | 2286 |
|---|---|
| Nosy | @terryjreedy, @amauryfa, @tpn, @briancurtin |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/tpn'
closed_at = <Date 2013-07-02.17:45:05.069>
created_at = <Date 2008-03-14.12:57:55.190>
labels = ['tests', 'OS-windows', 'type-crash']
title = 'Stack overflow exception caused by test_marshal on Windows x64'
updated_at = <Date 2013-07-02.17:45:05.068>
user = 'https://github.com/tpn'bugs.python.org fields:
activity = <Date 2013-07-02.17:45:05.068>
actor = 'sbt'
assignee = 'trent'
closed = True
closed_date = <Date 2013-07-02.17:45:05.069>
closer = 'sbt'
components = ['Tests', 'Windows']
creation = <Date 2008-03-14.12:57:55.190>
creator = 'trent'
dependencies = []
files = []
hgrepos = []
issue_num = 2286
keywords = ['64bit']
message_count = 11.0
messages = ['63523', '63536', '63540', '63902', '147900', '147907', '148367', '148470', '170651', '192211', '192213']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'amaury.forgeotdarc', 'sable', 'trent', 'brian.curtin', 'sbt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue2286'
versions = ['Python 2.6', 'Python 2.7', 'Python 3.4']
Activity
tpn commentedon Mar 14, 2008
S:\src\svn\svn.python.org\projects\python\trunk.x64\PCbuild>amd64\python_d
..\lib\test\test_marshal.py
test_bool (main.IntTestCase) ... ok
test_int64 (main.IntTestCase) ... ok
test_ints (main.IntTestCase) ... ok
test_floats (main.FloatTestCase) ... ok
test_buffer (main.StringTestCase) ... ok
test_string (main.StringTestCase) ... ok
test_unicode (main.StringTestCase) ... ok
test_code (main.CodeTestCase) ... ok
test_dict (main.ContainerTestCase) ... ok
test_list (main.ContainerTestCase) ... ok
test_sets (main.ContainerTestCase) ... ok
test_tuple (main.ContainerTestCase) ... ok
test_exceptions (main.ExceptionTestCase) ... ok
test_bug_5888452 (main.BugsTestCase) ... ok
test_exact_type_match (main.BugsTestCase) ... ok
test_fuzz (main.BugsTestCase) ... ok
test_loads_recursion (main.BugsTestCase) ...
^^^
python_d.exe crashes at this point with a stack overflow. Just
capturing the issue for now. Will investigate shortly and hopefully
provide a patch.
tpn commentedon Mar 14, 2008
Traced the problem down to the following minimal code snippet:
When Python/marshal.c:18 MAX_MARSHAL_STACK_DEPTH is 2000 (which is what
it is currently), marshal.loads() eventually overflows the stack in
r_object(). There is a check in r_object() to avoid this though:
On Windows x64, a value of 1964 raises the recursion limit exception
above (which is what test_marshal is expecting). With a value of 1965,
a C stack overflow exception is raised.
So, MAX_MARSHAL_STACK_DEPTH needs to be <= 1964 in order to prevent this
particular code from overflowing the stack on Win64 before we can raise
a Python recursion limit exception.
Was there any science behind choosing 2000 as the current value? Should
a new value (e.g. 1500) be provided for only Win64, leaving everyone
else at 2000? Interesting that Linux/BSD etc on AMD64 don't run into this.
amauryfa commentedon Mar 14, 2008
The situation was the same on win32 with the VC2005 debug build, some
months ago.
I think this is because of the extra stack checks added with "recent"
versions of Visual Studio, together with the fact that local variables
are not shared when optimizations are disabled.
Can you try to raise the stack size on x64 builds? If 2Mb is enough for
32bit, 4Mb should be good in your case.
tpn commentedon Mar 18, 2008
Fixed in r61509; bumped the stack size from 200000 to 210000, which
stops the stack overflow from being triggered.
sable commentedon Nov 18, 2011
I had the exact same problem when compiling the Python 2.7 branch with Visual Studio 2010.
In debug mode and 64 bits, python will crash on test_loads_recursion in test_marshal and on the code provided in msg63536.
Could you please reopen this issue?
sable commentedon Nov 18, 2011
I don't have the issue anymore when bumping the stack size to 2300000.
terryjreedy commentedon Nov 25, 2011
The 'official' PSF build for 2.7 (and 3.x) is done with VS2008, so this may not be the last problem you have. I am pretty sure this will never change for 2.7, but that is not to say we cannot make a simple change to accommodate VS2010. I am sure we will eventually update VS for 3.x. Can you test 3.2 or even better 3.3?
sable commentedon Nov 28, 2011
Thanks Terry, I am aware of that.
We are working on making Python work with VS2010 in bpo-13210.
I will check with the py3k branch soon and report here if the same problem applies.
tpn commentedon Sep 18, 2012
Closing issue; this has been fixed.
sbt commentedon Jul 2, 2013
Reopening because I think this is again a problem for Win64 and 3.x. The Win64 buildbots always seem to crash on test_marshal (and I do too).
It appears to be BugsTestCase.test_loads_2x_code() which crashes, which is virtually the same as test_loads_recursion().
sbt commentedon Jul 2, 2013
Closing because this is caused by bpo-17206 and is already discussed there.