Repository navigation
Replace direct calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc() #62403
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2013-07-07.21:33:47.401>
created_at = <Date 2013-06-13.10:52:39.545>
labels = ['type-feature']
title = 'Replace direct calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()'
updated_at = <Date 2013-07-27.00:39:51.377>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2013-07-27.00:39:51.377>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2013-07-07.21:33:47.401>
closer = 'vstinner'
components = []
creation = <Date 2013-06-13.10:52:39.545>
creator = 'vstinner'
dependencies = []
files = ['30581', '30590', '30591', '30592', '30594', '30595']
hgrepos = []
issue_num = 18203
keywords = ['patch']
message_count = 23.0
messages = ['191076', '191078', '191080', '191082', '191086', '191093', '191111', '191169', '191174', '191175', '191176', '191179', '191180', '191182', '191183', '191195', '192559', '192563', '192572', '192591', '192596', '192597', '193766']
nosy_count = 10.0
nosy_names = ['georg.brandl', 'amaury.forgeotdarc', 'ncoghlan', 'kristjan.jonsson', 'vstinner', 'christian.heimes', 'trent', 'python-dev', 'aliles', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18203'
versions = ['Python 3.4']
Activity
vstinner commentedon Jun 13, 2013
The issue bpo-3329 proposes an API to replace memory allocator functions. But Python calls directly malloc(), realloc() and free() in some functions, so custom allocators would not be used there.
Examples of functions calling malloc/realloc/free directly: _PySequence_BytesToCharpArray(), block_new() (of pyarena.c), find_key() (of thread.c), PyInterpreterState_New(), win32_wchdir(), posix_getcwd(), Py_Main(), etc.
We have to be careful with the GIL: PyMem_*() functions can only be called when holding the GIL.
serhiy-storchaka commentedon Jun 13, 2013
Be aware about external code which allocate memory itself (i.e. expat).
vstinner commentedon Jun 13, 2013
Well, I know that it will hard to cover 100% of the stdlib. I just want to replace the most obvious calls.
Some libraries can be configured to use a custom memory allocators:
We should probably uses these functions to reuse Python allocators (PyMem_Malloc()), or maybe only if PyMem_SetAllocators() has been called? (if Python does not use system allocators, aka malloc)
See also bpo-18178 for libffi, it may be related.
The _decimal module configures libmpdec to use PyMem_Malloc:
tiran commentedon Jun 13, 2013
expat has a XML_Memory_Handling_Suite. You just have to replace XML_ParserCreate() and XML_ParserCreateNS with XML_ParserCreate_MM().
birkenfeld commentedon Jun 13, 2013
I think there's a potential problem here :)
vstinner commentedon Jun 13, 2013
I didn't understand the motivation to require the GIL held for PyMem_Malloc(). I searched in the source code history and on the Internet (archives of python-dev). In my opinion, the restiction is motivated by a bug: PyMem_Malloc() calls (indirectly) PyObject_Malloc() in debug mode, and PyObject_Malloc() is not thread-safe.
I opened a thread on python-dev to discuss this point.
kristjanvalur commentedon Jun 14, 2013
Keeping the GIL requirement is _very_ useful for PyMem_MALLOC et al. It allows applications to hook in their own monitoring code, accessible from python itself, without having to worry about conflicts with python.
even if it were not for the GIL itself, PyMem_Malloc() may have all sorts of side effects.
Because of this, and to allow ourselves the flexibility to do all sorts of things inside PyMem_Malloc(), at CCP we added a parallel api, PyMem_MALLOC_RAW() etc.
This api is guaranteed to delegate directly to the external allocator (malloc by default, or an embedding application's supplied allocastor)
We have patched pythoncore in 2.7 in all places there were using malloc directly using the file attached to the defect. Notice how it can patch "malloc" in two different ways, using either regular malloc (in non-sensitive areas) and using the raw malloc (in sensitive areas.)
vstinner commentedon Jun 14, 2013
I commited my new API to customize memory allocators:
New changeset 6661a8154eb3 by Victor Stinner in branch 'default':
Issue bpo-3329: Add new APIs to customize memory allocators
http://hg.python.org/cpython/rev/6661a8154eb3
I added PyMem_RawMalloc(), PyMem_RawRealloc() and PyMem_RawFree() in the same commit. These functions are wrappers to malloc/realloc/free which can be called without the GIL held. Using these new functions instead of malloc/realloc/free is interesting because the internal functions can be replaced with PyMem_SetRawAllocators() and many checks are added in debug mode (ex: check for buffer under- and overflow).
vstinner commentedon Jun 15, 2013
pymem_debugcheckgil.patch: Patch adding check in debug mode to ensure that PyMem_Malloc() and PyObject_Malloc() are called with the GIL held.
vstinner commentedon Jun 15, 2013
py_finalize.patch: modify Py_Finalize() to destroy the GIL after the last call to PyMem_Malloc() or PyObject_Malloc(). For example,
PyCFunction_Fini() calls PyObject_GC_Del() which calls PyObject_FREE().
vstinner commentedon Jun 15, 2013
pymem_debugcheckgil.patch: oops, a forgot a change in pgenmain.c. New fixed patch attached.
vstinner commentedon Jun 15, 2013
See also issue bpo-16742: "PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe".
vstinner commentedon Jun 15, 2013
malloc_init.patch: Patch for functions called at Python initialization.
This patch is not complete: to parse "-X" option (PySys_AddXOption) and "-W" (PySys_AddWarnOption), PyMem_Malloc() and PyObject_Malloc() are still called indirectly. Fixing this issue may need to reorganize completly how Python is initialized, because we need to have basic Python types (ex: Unicode) to be ready to be able to parse the command line. But we cannot initialize too much because the Python initialization also depends on options from the command line...
vstinner commentedon Jun 15, 2013
malloc_modules.patch: Patch for different Python modules.
vstinner commentedon Jun 15, 2013
Ok, initial patches are attached. Let describe them a little bit.
pymem_debugcheckgil-2.patch: I don't think that this patch can be commited before the "bootstrap" issue is solved (Python doesn't start in debug mode with this patch when -W or -X command line option is used). I wrote it to check that other patches are correct (check if the GIL is held when PyMem_Malloc is called).
py_finalize.patch: should be safe
malloc_init.patch: change _Py_char2wchar() API, the result must now be freed by PyMem_RawFree() instead of PyMem_Free(). the change does not hurt in release mode (both functions are just wrapper to free()), but may break in debug mode (because python checks that PyMem_RawFree() is called on a buffer allocated by PyMem_RawMalloc()) in extension modules using _Py_char2wchar(). I would like to _Py_char2wchar() API (not to solve this issue, just because it is very useful for applications embedding Python), so changing its name would avoid a crash (applications would get a compilation or link error instead). Other change of the patch: replace free() and PyMem_Free() with PyMem_RawFree(), should be safe.
malloc_modules.patch: I replaced many malloc() with PyMem_Malloc(), which is not safe.
All these patches must be reviewed carefully to check if the GIL is held or not, and tested with pymem_debugcheckgil-2.patch (on Windows too! some patched functions os posixmodule.c are only compiled on Windows).
ncoghlan commentedon Jun 15, 2013
Note that CPython's main function accesses the C API before calling Py_Initialize(). This is insane, but fixing it is hard (and one of the main goals of PEP-432).
I suggest using Py_IsInitialized() to exclude those from your debug checks for now.
[-]Replace calls to malloc() with PyMem_Malloc()[/-][+]Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()[/+][-]Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()[/-][+]Replace direct calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()[/+]python-dev commentedon Jul 7, 2013
New changeset 213d6d7f5979 by Victor Stinner in branch 'default':
Issue bpo-18203: Fix Py_Finalize(): destroy the GIL after the last call to
http://hg.python.org/cpython/rev/213d6d7f5979
New changeset 18bb92b0c458 by Victor Stinner in branch 'default':
Issue bpo-18203: Replace malloc() with PyMem_RawMalloc() at Python initialization
http://hg.python.org/cpython/rev/18bb92b0c458
New changeset 41ef797e6639 by Victor Stinner in branch 'default':
Issue bpo-18203: Replace malloc() with PyMem_Malloc() in Python modules
http://hg.python.org/cpython/rev/41ef797e6639
python-dev commentedon Jul 7, 2013
New changeset fc01f9497da7 by Victor Stinner in branch 'default':
Issue bpo-18203: Fix decode_ascii_surrogateescape(), use PyMem_RawMalloc() as _Py_char2wchar()
http://hg.python.org/cpython/rev/fc01f9497da7
python-dev commentedon Jul 7, 2013
New changeset 638d43665356 by Victor Stinner in branch 'default':
Issue bpo-18203: Replace malloc() with PyMem_Malloc() in _ssl for the password
http://hg.python.org/cpython/rev/638d43665356
New changeset 9af1905f20af by Victor Stinner in branch 'default':
Issue bpo-18203: Replace malloc() with PyMem_RawMalloc() to allocate thread locks
http://hg.python.org/cpython/rev/9af1905f20af
New changeset fb7d346b45fa by Victor Stinner in branch 'default':
Issue bpo-18203: Replace malloc() with PyMem_Malloc() to allocate arena objects
http://hg.python.org/cpython/rev/fb7d346b45fa
New changeset 10db0c67fc72 by Victor Stinner in branch 'default':
Issue bpo-18203: Replace malloc() with PyMem_Malloc() in _PySequence_BytesToCharpArray()
http://hg.python.org/cpython/rev/10db0c67fc72
python-dev commentedon Jul 7, 2013
New changeset 31a635303e55 by Victor Stinner in branch 'default':
Issue bpo-18203: Fix _Py_DecodeUTF8_surrogateescape(), use PyMem_RawMalloc() as _Py_char2wchar()
http://hg.python.org/cpython/rev/31a635303e55
python-dev commentedon Jul 7, 2013
New changeset 65f2c92ed079 by Victor Stinner in branch 'default':
Issue bpo-18203: Add _PyMem_RawStrdup() and _PyMem_Strdup()
http://hg.python.org/cpython/rev/65f2c92ed079
vstinner commentedon Jul 7, 2013
Ok, with the last changeset, I consider the issue has fixed. There is no more direct call to malloc() in the Python source code.
python-dev commentedon Jul 27, 2013
New changeset 1cba6687993e by Victor Stinner in branch 'default':
Issue bpo-18203: Replace PyMem_Malloc() with PyMem_RawMalloc() at Python initialization
http://hg.python.org/cpython/rev/1cba6687993e