Repository navigation
test_marshal: crash in Python 3.7b5 on Windows 10 #77901
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/zooba'
closed_at = <Date 2018-07-24.08:07:53.063>
created_at = <Date 2018-05-31.15:13:35.091>
labels = ['3.8', '3.7', 'tests', 'OS-windows', 'type-crash']
title = 'test_marshal: crash in Python 3.7b5 on Windows 10'
updated_at = <Date 2018-07-24.08:07:53.062>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2018-07-24.08:07:53.062>
actor = 'serhiy.storchaka'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2018-07-24.08:07:53.063>
closer = 'serhiy.storchaka'
components = ['Tests', 'Windows']
creation = <Date 2018-05-31.15:13:35.091>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 33720
keywords = ['patch']
message_count = 29.0
messages = ['318323', '318325', '318326', '318327', '318328', '318329', '318330', '318331', '318442', '318497', '318689', '318690', '318693', '318695', '318696', '318698', '318714', '318771', '320053', '320221', '321012', '321054', '321055', '321059', '321084', '321087', '321089', '321092', '322279']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ned.deily', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'miss-islington']
pr_nums = ['7336', '7401', '7405', '8071', '8104', '8105', '8107']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue33720'
versions = ['Python 3.7', 'Python 3.8']
Activity
vstinner commentedon May 31, 2018
Follow-up of bpo-33719.
C:\Users\vstinner\AppData\Local\Programs\Python\Python37>python.exe -m test test_marshal -v
== CPython 3.7.0b5 (v3.7.0b5:abb8802389, May 31 2018, 01:54:01) [MSC v.1913 64 bit (AMD64)]
== Windows-10-10.0.16299-SP0 little-endian
== cwd: C:\Users\vstinner\AppData\Local\Temp\test_python_3836
== CPU count: 2
== encodings: locale=cp1252, FS=utf-8
Run tests sequentially
0:00:00 [1/1] test_marshal
(...)
test_loads_2x_code (test.test_marshal.BugsTestCase) ... Windows fatal exception: stack overflow
Current thread 0x000003a0 (most recent call first):
File "C:\Users\vstinner\AppData\Local\Programs\Python\Python37\lib\unittest\case.py", line 178 in handle
File "C:\Users\vstinner\AppData\Local\Programs\Python\Python37\lib\unittest\case.py", line 743 in assertRaises
File "C:\Users\vstinner\AppData\Local\Programs\Python\Python37\lib\test\test_marshal.py", line 215 in test_loads_2x_code
(...)
Crashes in test_marshal is on old topic:
Current stack size: 2 million bytes (1.9 MiB)
PCbuild/python.vcxproj: <StackReserveSize>2000000</StackReserveSize>
PCbuild/pythonw.vcxproj: <StackReserveSize>2000000</StackReserveSize>
vstinner commentedon May 31, 2018
I compiled the master branch of Python in release mode using VS2015 (MSC v.1912 64 bit) and I failed to reproduce the crash:
vstinner commentedon May 31, 2018
I also failed to reproduce the crash in the 3.7 branch.
I guess that the python.org binary has been compiled differently.
zooba commentedon May 31, 2018
The uploaded binary is compiled with PGO enabled (and trained on most of the test suite). I'll check it out - hopefully we don't need to do anything drastic and can get away with either a compiler update or disabling optimizations on a single function.
zooba commentedon May 31, 2018
Ned, FYI
vstinner commentedon May 31, 2018
I don't think that it's a release blocker. test_marshal does only crash on corner cases which should not occur on usual "valid" data.
11 remaining items
zooba commentedon Jun 5, 2018
Thanks. Agreed this should stay open, but it is blocked on a compiler fix.
vstinner commentedon Jun 20, 2018
Do you want to have a different limit depending on the compiler and the compiler version? I'm not sure that it's worth it. I'm happy with the heuristic "use lower limit on Windows". By design, the test is fragile.
I would suggest to close the issue, but you can keep it open if you prefer to increase the limit on a fixed compiler.
serhiy-storchaka commentedon Jun 22, 2018
What are your thoughts about rewriting the test (PR 7336)?
serhiy-storchaka commentedon Jul 4, 2018
The alternate solution is provided by PR 8071. The culprit of the main stack consumption is the code for unmarshalling float and complex in protocols 0 and 1 which uses 256-bytes buffers on the stack. Moving it to the separate function allows to decrease the stack consumption and increase the maximum marshal recursion depth on Windows.
zooba commentedon Jul 4, 2018
Thanks for writing that second PR, Serhiy! I'd already spotted that as the likely fix on our side, though I do want to complete the investigation into why MSVC generated unnecessarily sparse frames. But that is basically an internal discussion for me to follow up on at work.
zooba commentedon Jul 4, 2018
I added the backport to 3.7 label to your PR, as I want to restore the marshal stack depth limit in 3.7.1.
serhiy-storchaka commentedon Jul 4, 2018
Unfortunately PR 8071 doesn't fix the crash on PGO builds. :-( Initially I tested it only on the debug build, and now have tested on the PGO build.
serhiy-storchaka commentedon Jul 5, 2018
New changeset fc05e68 by Serhiy Storchaka in branch 'master':
bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336)
fc05e68
miss-islington commentedon Jul 5, 2018
New changeset 3f121a4 by Miss Islington (bot) in branch '3.7':
bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336)
3f121a4
miss-islington commentedon Jul 5, 2018
New changeset 878c4fe by Miss Islington (bot) in branch '3.6':
bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336)
878c4fe
serhiy-storchaka commentedon Jul 5, 2018
New changeset 9720f60 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336). (GH-8107)
9720f60
serhiy-storchaka commentedon Jul 24, 2018
New changeset c573499 by Serhiy Storchaka in branch 'master':
bpo-33720: Refactor marshalling/unmarshalling floats. (GH-8071)
c573499