Fixes crashes found by pyfailmalloc

@gpshead

BPO	18408
Nosy	@gpshead, @pitrou, @vstinner, @serhiy-storchaka
Files	use_failmalloc.patch call_exc_trace.patch type_slot_tp_str.patch misc_fixes.patch listpop.patch frame_fasttolocals.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2013-10-29.00:20:51.253>
created_at = <Date 2013-07-08.20:10:43.005>
labels = ['type-crash']
title = 'Fixes crashes found by pyfailmalloc'
updated_at = <Date 2015-03-26.08:49:25.945>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-03-26.08:49:25.945>
actor = 'python-dev'
assignee = 'vstinner'
closed = True
closed_date = <Date 2013-10-29.00:20:51.253>
closer = 'vstinner'
components = []
creation = <Date 2013-07-08.20:10:43.005>
creator = 'vstinner'
dependencies = []
files = ['30866', '30867', '30868', '30869', '30932', '30933']
hgrepos = []
issue_num = 18408
keywords = ['patch']
message_count = 69.0
messages = ['192691', '192693', '192694', '192701', '192703', '192704', '192705', '192706', '192797', '192902', '192908', '192913', '192917', '192918', '192920', '193130', '193131', '193132', '193138', '193140', '193142', '193144', '193192', '193196', '193198', '193199', '193200', '193202', '193209', '193211', '193213', '193218', '193219', '193224', '193225', '193245', '193250', '193258', '193261', '193262', '193263', '193264', '193265', '193316', '193324', '193327', '193329', '193337', '193350', '193374', '193378', '193382', '193435', '193438', '194129', '195112', '195113', '196029', '196030', '196041', '196042', '196199', '201593', '201594', '201596', '201599', '201601', '204000', '239316']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'pitrou', 'vstinner', 'neologix', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue18408'
versions = ['Python 3.4']

I created a first toy based on the PEP-445 to test of Python handles MemoryError: in short, Python handles them badly...

Home page of the project:
https://pypi.python.org/pypi/pyfailmalloc

I will use this issue to track crashes found by this tool.

New changeset 111c2a070f28 by Victor Stinner in branch 'default':
Issue bpo-18408: PyObject_GC_NewVar() now raises SystemError exception if nitems
http://hg.python.org/cpython/rev/111c2a070f28

New changeset ba766323b53a by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyDict_New() to handle correctly new_keys_object() failure
http://hg.python.org/cpython/rev/ba766323b53a

New changeset 68887e177dd4 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix list.pop() to handle list_resize() failure (MemoryError).
http://hg.python.org/cpython/rev/68887e177dd4

New changeset 697d722d97f9 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix marshal reader for Unicode strings: handle
http://hg.python.org/cpython/rev/697d722d97f9

New changeset de1473f4503b by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyType_Ready() and type.__bases__ setter to handle
http://hg.python.org/cpython/rev/de1473f4503b

New changeset f4311870e329 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix call_function() of ceval.c to handle PyTuple_New() failure
http://hg.python.org/cpython/rev/f4311870e329

New changeset df8b40593a08 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _PyUnicodeWriter_Finish(): clear writer->buffer,
http://hg.python.org/cpython/rev/df8b40593a08

"I created a first toy based on the PEP-445 to test of Python handles MemoryError: in short, Python handles them badly..."

Ooops: to test *how* Python handles..

New changeset ba79f6a86300 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix ConvParam() of the ctypes module to handle paramfunc failure
http://hg.python.org/cpython/rev/ba79f6a86300

New changeset 2ef2edfd1a4c by Victor Stinner in branch 'default':
Issue bpo-18408: Fix zlib.compressobj() to handle PyThread_allocate_lock() failure
http://hg.python.org/cpython/rev/2ef2edfd1a4c

New changeset a45407fa4a5b by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyCode_Optimize(): raise a MemoryError on memory allocation
http://hg.python.org/cpython/rev/a45407fa4a5b

New changeset ed0c9d77e179 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix usage of _PyBytes_Resize()
http://hg.python.org/cpython/rev/ed0c9d77e179

New changeset d605c6b8095c by Victor Stinner in branch 'default':
Issue bpo-18408: _PyUnicodeWriter_Finish() now clears its buffer attribute in all
http://hg.python.org/cpython/rev/d605c6b8095c

New changeset 549d8d3297f2 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _PyMem_DebugRealloc()
http://hg.python.org/cpython/rev/549d8d3297f2

New changeset c91e7f707562 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix select.select() to handle PyList_New() failure (MemoryError)
http://hg.python.org/cpython/rev/c91e7f707562

use_failmalloc.patch: Patch to use pyfailmalloc on the Python test suite. Command to run test suite: "gdb -args ./python -m test -F".

call_exc_trace.patch: Fix for the call_exc_trace() function when the traceback is not set.

type_slot_tp_str.patch: Fix for the __str__ slot of types (slot_tp_str() in typeobject.c).

misc_fixes.patch: Misc bugfixes related to MemoryError, this patch should be read carefully. The changes in unicodeobject.c may be written differently. I wrote them to fix a failing assertion ("assert(data != NULL);") in _PyUnicode_CheckConsistency(), called when the creation of a new string failed.

New changeset 4f730c045f5f by Victor Stinner in branch 'default':
Issue bpo-18408: Fix call_exc_trace(): if the traceback is NULL, use None when
http://hg.python.org/cpython/rev/4f730c045f5f

New changeset 01a46dc00fc8 by Victor Stinner in branch 'default':
Issue bpo-18408: slot_tp_str() must not fallback on slot_tp_repr() on error
http://hg.python.org/cpython/rev/01a46dc00fc8

New changeset 39eb1ce5f377 by Victor Stinner in branch 'default':
Issue bpo-18408: ste_new() initialize all attributes before handling error
http://hg.python.org/cpython/rev/39eb1ce5f377

New changeset aaa6e8b8a5c9 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix compiler_import() to handle PyUnicode_Substring() failure properly
http://hg.python.org/cpython/rev/aaa6e8b8a5c9

New changeset ff93930a53c0 by Victor Stinner in branch 'default':
Issue bpo-18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/ff93930a53c0

New changeset 1a1869baec4c by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _Pickler_New() and _Unpickler_New(): initialize all
http://hg.python.org/cpython/rev/1a1869baec4c

New changeset e11121b9bd09 by Victor Stinner in branch 'default':
Issue bpo-18408: _elementtree.c now handles create_extra() failure
http://hg.python.org/cpython/rev/e11121b9bd09

New changeset 0877e17fa25e by Victor Stinner in branch 'default':
Issue bpo-18408: Different fixes in _elementtree.c to handle correctly MemoryError
http://hg.python.org/cpython/rev/0877e17fa25e

New changeset 51eddca9dd6f by Victor Stinner in branch 'default':
Issue bpo-18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/51eddca9dd6f

New changeset 5e50f1a0c985 by Victor Stinner in branch 'default':
Issue bpo-18408: In debug mode, PyCFunction_Call() now checks if an exception was
http://hg.python.org/cpython/rev/5e50f1a0c985

New changeset 4975bcd67aa7 by Victor Stinner in branch 'default':
Issue bpo-18408: normalizestring() now raises MemoryError on memory allocation failure
http://hg.python.org/cpython/rev/4975bcd67aa7

New changeset 1eac89af9acf by Victor Stinner in branch 'default':
Issue bpo-18408: Oh, I was wrong: Pickler_New() must call Py_DECREF() to destroy
http://hg.python.org/cpython/rev/1eac89af9acf

New changeset d9446c2a2fd4 by Victor Stinner in branch 'default':
Issue bpo-18408: ceval.c: in debug mode, convert the PyErr_Occurred() check on
http://hg.python.org/cpython/rev/d9446c2a2fd4

New changeset 2f7c4df5cc46 by Victor Stinner in branch 'default':
Issue bpo-18408: errors.c: in debug mode, calling PyErr_BadInternalCall() now
http://hg.python.org/cpython/rev/2f7c4df5cc46

New changeset affb87b1d7ec by Victor Stinner in branch 'default':
Issue bpo-18408: pmerge() help of mro_implementation() now raises MemoryError on
http://hg.python.org/cpython/rev/affb87b1d7ec

New changeset f85fcbbbe8de by Victor Stinner in branch 'default':
Issue bpo-18408: _PyMemoTable_ResizeTable() now restores the old table if
http://hg.python.org/cpython/rev/f85fcbbbe8de

New changeset bb5da6e795a1 by Victor Stinner in branch 'default':
Issue bpo-18408: _pickle.c: Add missing PyErr_NoMemory() on memory allocation failures
http://hg.python.org/cpython/rev/bb5da6e795a1

New changeset 89c6495d1ff2 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix parser.sequence2st() and parser.tuple2st(): raise MemoryError
http://hg.python.org/cpython/rev/89c6495d1ff2

New changeset c80a9705803a by Victor Stinner in branch 'default':
Issue bpo-18408: parser module: fix error handling in node2tuple()
http://hg.python.org/cpython/rev/c80a9705803a

TODO:

review and integrate frame_fasttolocals.patch
bpo-18507: import_init() should not use Py_FatalError() but return an error
bpo-18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter error
fix tests hang when an exception occurs in a thread

Most bugs have been fixed.

New changeset 42aecc601a41 by Antoine Pitrou in branch 'default':
Back out 5bd9db528aed (issue bpo-18408). It caused unsolved buildbot failures.
http://hg.python.org/cpython/rev/42aecc601a41

Victor, I'll let you find out how to improve your debug checks, but in the meantime I've reverted the commit that randomly broke some Windows buildbots.

I didn't know that buildbots are failing. Do you have at least an example?
An error message, anything?

It's reported in bpo-18664 (you're nosied there :-)).

New changeset 79ce25c70795 by Victor Stinner in branch 'default':
Issue bpo-18408: _PyObject_Dump() now saves/restores the current exception
http://hg.python.org/cpython/rev/79ce25c70795

New changeset e63f19d0a651 by Victor Stinner in branch 'default':
Issue bpo-18664, bpo-18408: Rewrite PyErr_WriteUnraisable() to handle errors
http://hg.python.org/cpython/rev/e63f19d0a651

New changeset 8fb3a6f9b0a4 by Victor Stinner in branch 'default':
Restore changeset 5bd9db528aed (issue bpo-18408)
http://hg.python.org/cpython/rev/8fb3a6f9b0a4

New changeset 4ef4578db38a by Victor Stinner in branch 'default':
Issue bpo-18408: Add a new PyFrame_FastToLocalsWithError() function to handle
http://hg.python.org/cpython/rev/4ef4578db38a

Ok, most critical bugs have been fixed. If I find new bugs, I will open new issues.

New changeset e1d51c42e5a1 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
http://hg.python.org/cpython/rev/e1d51c42e5a1

New changeset 7f4a976829f1 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix zipimport, handle PyUnicode_Substring() and get_subname() failures
http://hg.python.org/cpython/rev/7f4a976829f1

New changeset 4749c3ac0654 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix iobase_readline(), handle PyByteArray_Resize() failure
http://hg.python.org/cpython/rev/4749c3ac0654

New changeset 549cd815bf8d by Victor Stinner in branch 'default':
Issue bpo-18408: Fix error handling in PyBytes_FromObject()
http://hg.python.org/cpython/rev/549cd815bf8d

New changeset 9d9371d4ff7b by Victor Stinner in branch 'default':
Issue bpo-18408: Fix bytearrayiter.partition()/rpartition(), handle
http://hg.python.org/cpython/rev/9d9371d4ff7b

New changeset 20cd15a28ad3 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _ctypes_alloc_format_string(), raise MemoryError on memory
http://hg.python.org/cpython/rev/20cd15a28ad3

New changeset 1386fb31c0ed by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyCStructUnionType_update_stgdict(), handle
http://hg.python.org/cpython/rev/1386fb31c0ed

New changeset 8f556ee0f6ba by Antoine Pitrou in branch 'default':
Fix refleak introduced by 4f730c045f5f (issue bpo-18408) and unveiled by 95eea8624d05 (issue bpo-16596).
http://hg.python.org/cpython/rev/8f556ee0f6ba

New changeset b742c1c5c0bf by Victor Stinner in branch 'default':
PEP-490: add bpo-18408
https://hg.python.org/peps/rev/b742c1c5c0bf

Navigation Menu

Repository navigation

Fixes crashes found by pyfailmalloc #62608

43 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Fixes crashes found by pyfailmalloc #62608

Description

Activity

vstinner commented on Jul 8, 2013

python-dev commented on Jul 8, 2013

vstinner commented on Jul 8, 2013

python-dev commented on Jul 8, 2013

vstinner commented on Jul 8, 2013

vstinner commented on Jul 9, 2013

vstinner commented on Jul 9, 2013

vstinner commented on Jul 9, 2013

python-dev commented on Jul 10, 2013

python-dev commented on Jul 11, 2013

python-dev commented on Jul 11, 2013

python-dev commented on Jul 11, 2013

python-dev commented on Jul 11, 2013

python-dev commented on Jul 11, 2013

43 remaining items

vstinner commented on Aug 14, 2013

python-dev commented on Aug 23, 2013

pitrou commented on Aug 23, 2013

vstinner commented on Aug 23, 2013

pitrou commented on Aug 23, 2013

python-dev commented on Aug 26, 2013

python-dev commented on Oct 29, 2013

vstinner commented on Oct 29, 2013

python-dev commented on Oct 29, 2013

python-dev commented on Oct 29, 2013

python-dev commented on Oct 29, 2013

python-dev commented on Nov 23, 2013

python-dev commented on Mar 26, 2015

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions