Repository navigation
PyMem_Malloc(): check that the GIL is hold in debug hooks #70750
Description
| BPO | 26563 |
|---|---|
| Nosy | @vstinner |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2016-03-17.08:20:02.723>
created_at = <Date 2016-03-14.22:52:53.562>
labels = []
title = 'PyMem_Malloc(): check that the GIL is hold in debug hooks'
updated_at = <Date 2016-03-18.10:04:44.032>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2016-03-18.10:04:44.032>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2016-03-17.08:20:02.723>
closer = 'vstinner'
components = []
creation = <Date 2016-03-14.22:52:53.562>
creator = 'vstinner'
dependencies = []
files = ['42165']
hgrepos = []
issue_num = 26563
keywords = ['patch']
message_count = 11.0
messages = ['261787', '261850', '261851', '261854', '261865', '261866', '261867', '261874', '261875', '261902', '261954']
nosy_count = 2.0
nosy_names = ['vstinner', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue26563'
versions = ['Python 3.6']
Activity
vstinner commentedon Mar 14, 2016
With the issue bpo-26558, debug hooks of PyObject_Malloc() now checks that the GIL is hold.
I left PyMem_Malloc() unchanged, because I am not 100% sure that it's ok yet to implement the same check in PyMem_Malloc(). So I opened this issue.
Python 3 doc explicitly asks that the GIL is hold to call PyMem_Malloc():
https://docs.python.org/dev/c-api/memory.html#memory-interface
Python 2 doc doesn't say anything about the GIL:
https://docs.python.org/2/c-api/memory.html#memory-interface
Usually, functions prefixed by "Py" require the GIL to be hold.
In practice, currently PyMem_Malloc() is implemented with malloc() which is thread-safe.
In the issue bpo-26249, I tested numpy, lxml, Pillow and cryptography with all debug hooks enabled, including GIL checks in PyMem_Malloc() and PyObject_Malloc(). I only found one bug in numpy: numpy/numpy#7404
Attached patch changes debug hooks on PyMem_Malloc() to ensure that the GIL is hold.
[-]PyMem_Malloc(): check that the GIL is held in debug hooks[/-][+]PyMem_Malloc(): check that the GIL is hold in debug hooks[/+]python-dev commentedon Mar 16, 2016
New changeset 58644086c195 by Victor Stinner in branch 'default':
Fail if PyMem_Malloc() is called without holding the GIL
https://hg.python.org/cpython/rev/58644086c195
vstinner commentedon Mar 16, 2016
I pushed the patch (I added an unit test).
If too many users complain, we can revert the change since it's now a small change.
python-dev commentedon Mar 16, 2016
New changeset ae76a1046bb9 by Victor Stinner in branch 'default':
Fix usage of PyMem_Malloc() in os.stat()
https://hg.python.org/cpython/rev/ae76a1046bb9
python-dev commentedon Mar 16, 2016
New changeset 5b4f173f0690 by Victor Stinner in branch 'default':
faulthandler now works in non-Python threads
https://hg.python.org/cpython/rev/5b4f173f0690
New changeset d6d64168ee8c by Victor Stinner in branch 'default':
Py_FatalError: disable faulthandler earlier
https://hg.python.org/cpython/rev/d6d64168ee8c
python-dev commentedon Mar 16, 2016
New changeset 959e58cfbde9 by Victor Stinner in branch 'default':
Fix usage of PyMem_Malloc() in overlapped.c
https://hg.python.org/cpython/rev/959e58cfbde9
python-dev commentedon Mar 16, 2016
New changeset 73d8adc0d5ea by Victor Stinner in branch '3.5':
Fix usage of PyMem_Malloc() in overlapped.c
https://hg.python.org/cpython/rev/73d8adc0d5ea
vstinner commentedon Mar 17, 2016
Reopen. I found and fixed two bugs in os and _overlapped modules (_overlapped is used by asyncio on Windows).
python-dev commentedon Mar 17, 2016
New changeset ce62df22f3bf by Victor Stinner in branch '3.5':
Py_FatalError: disable faulthandler earlier
https://hg.python.org/cpython/rev/ce62df22f3bf
vstinner commentedon Mar 17, 2016
Hum ok, all buildbots look to be happy again :-) I close the issue (again).
python-dev commentedon Mar 18, 2016
New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774