Repository navigation
C APIs called without GIL in PyOS_Readline #75463
Description
| BPO | 31282 |
|---|---|
| Nosy | @vstinner, @serhiy-storchaka, @zhangyangyu |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-06-25.09:45:27.273>
created_at = <Date 2017-08-26.04:51:13.114>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10']
title = 'C APIs called without GIL in PyOS_Readline'
updated_at = <Date 2020-06-25.09:45:27.271>
user = 'https://github.com/zhangyangyu'bugs.python.org fields:
activity = <Date 2020-06-25.09:45:27.271>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-25.09:45:27.273>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2017-08-26.04:51:13.114>
creator = 'xiang.zhang'
dependencies = []
files = []
hgrepos = []
issue_num = 31282
keywords = []
message_count = 6.0
messages = ['300862', '300935', '300936', '300945', '300948', '372329']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'xiang.zhang']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31282'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
zhangyangyu commentedon Aug 26, 2017
When debugging our project I find something interesting. In PyOS_Readline it releases the GIL and delegate its job to PyOS_ReadlineFunctionPointer, which could be call_readline or PyOS_StdioReadline(I don't find where vms__StdioReadline is defined). But in the two functions, they use some C APIs like PyMem_Malloc/MALLOC, PyErr_SetString which need guarded by GIL. I don't understand why not doing find-grained lock control in call_readline or PyOS_StdioReadline since the code is ancient. :-(
I find this because our project makes test_cmd_line fail with
echo "Timer\n" | python -i -m timeit -n 1.serhiy-storchaka commentedon Aug 28, 2017
AFAIK PyMem_Malloc/MALLOC can be used with released GIL.
zhangyangyu commentedon Aug 28, 2017
In pymem.h, it's stated that "The GIL must be held when using these APIs". I think the reason is when PYMALLOC_DEBUG is defined, they finally delegate to PyObject_Malloc.
Victor must be familiar with them since once he proposed to remote the restriction: https://mail.python.org/pipermail/python-dev/2013-June/126822.html.
vstinner commentedon Aug 28, 2017
You must hold the GIL to call PyMem_Malloc(). A debug assert should be
raised if you don't hold the GIL since Python 3.6 with PYTHONMALLOC=debug.
Call PyMem_RawMalloc().
I fixed Python 3, no?
For Python 2, in practice you can call PyMem_Malloc() without holding the
GIL, it's just malloc() which is thread safe.
zhangyangyu commentedon Aug 28, 2017
Yes. In Python3 they are replaced by PyMem_RawMalloc. But it's not only PyMem_Malloc, there are also PyErr_SetString, PyErr_NoMemory, even in Python3.
BTW, even in Python3, when memory allocators are in debug mode, it finally calls bumpserialno, which IIUC, is not thread safe. But of course it's another issue.
Hmm, I know it. But it's not stated in the doc they are thread safe, I am not sure assuming this is suitable. An example is psycopg/psycopg2#110.
vstinner commentedon Jun 25, 2020
I fixed this issue in bpo-40826, especially with this change:
commit c353764
Author: Victor Stinner <vstinner@python.org>
Date: Mon Jun 1 20:59:35 2020 +0200