Repository navigation
Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode #79380
Description
| BPO | 35199 |
|---|---|
| Nosy | @rhettinger, @vstinner |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-11-10.00:36:49.655>
created_at = <Date 2018-11-09.15:32:02.819>
labels = ['interpreter-core', '3.8']
title = 'Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode'
updated_at = <Date 2018-11-10.01:43:18.485>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2018-11-10.01:43:18.485>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-10.00:36:49.655>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-11-09.15:32:02.819>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35199
keywords = ['patch']
message_count = 9.0
messages = ['329523', '329525', '329529', '329530', '329536', '329570', '329572', '329577', '329585']
nosy_count = 2.0
nosy_names = ['rhettinger', 'vstinner']
pr_nums = ['10434', '10435']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35199'
versions = ['Python 3.8']
Activity
vstinner commentedon Nov 9, 2018
Currently, even when Python is compiled in debug mode, PyTuple_GET_ITEM() doesn't check that the first argument is a tuple objet and that the second argument is valid index. It can lead to a crash and Python doesn't help debugging.
I propose to convert the macro to a function call and use regular assertions to abort Python if the C API is misused.
I propose to use a function call rather than abusing the preprocessor syntax like (assert(...),expr) syntax used in unicodeobject.h, because I the preprocessor causes complex bugs (difficult to understand and to fix/work around) and because later I would like to experiment to be able to compile C extensions to always use function calls, but get a different implementation depending on the "Python runtime".
I elaborated this idea on this website:
I am working on an implementation.
vstinner commentedon Nov 9, 2018
Currently, PyTuple_GET_ITEM() can be used with &PyTuple_GET_ITEM(). If we convert PyTuple_GET_ITEM() to a function returning PyObject*, it's no longer possible. My PR 10434 prepares the code for that.
vstinner commentedon Nov 9, 2018
New changeset d17a693 by Victor Stinner in branch 'master':
bpo-35199: Add an internal _PyTuple_ITEMS() macro (GH-10434)
d17a693
vstinner commentedon Nov 9, 2018
Copy of my comment:
#10435 (comment)
This change breaks the backward compatibility when a C extension is compiled with Py_DEBUG. So I'm not sure that it should be merged into Python 3.8. Maybe we should add a new opt-in experimental option to enable it, instead of Py_DEBUG?
rhettinger commentedon Nov 9, 2018
I don't think this should be merged. A lot of code of uses PyTuple_GET_ITEM(). Also, the "problem" your solving doesn't seem to exist in practice. Further, I worry that a compiler may choose not to inline on occasion, leading to performance regressions in code that has already been finely tuned.
vstinner commentedon Nov 9, 2018
Sorry my comment on the PR, a friend contacted me because (...) he misused PyTuple_GET_ITEM() :-)
Oh, maybe I didn't explain properly my change. I don't want to change the "default" API nor the "default" implementation. My change is enabled enabled if you compile Python in debug mode, if Py_DEBUG is defined.
In fact, it's an early attempt to see how we can introduce a new C API in Python. It seems like abusing Py_DEBUG was a bad idea, so I close my PR and I will close this issue as well. I will start a discussion on python-dev later, when I will have a plan.
See also the discussion on the PR:
#10435 (comment)
rhettinger commentedon Nov 10, 2018
Thanks. I don't really know enough about compilers and required-vs-optional C99 semantics to know whether these changes are safe. These are very old APIs and are used throughout the entire Python ecosystem. It would pay to be somewhat cautious.
The whole C-API project is predicated on giving a 2x speed-up in exchange for destabilizing changes. These cross-module inline function-for-macro swaps aren't aligned with that goal. At the very best, the swaps will be performance neutral. At worst, they will invisibly degrade a huge swath of code, someone of which is finely-tuned (people generally only use PyTuple_GET_ITEM() in places where they want better performance than the existing function based alternative)
vstinner commentedon Nov 10, 2018
Oops, I forgot to close the issue.
This change was an early attempt (tagged as WIP/DO-NOT-MERGE) to expriment changing the implementation without touching the API. But I was wrong, it does change the API (&PyTuple_GET_ITEM(ob, i) causes a compilation error) and so I closed my PR.
Instead, I opened a thread on python-dev to better explain the whole context and my intent:
https://mail.python.org/pipermail/python-dev/2018-November/155702.html
I invite everyone to discuss there instead :-)
vstinner commentedon Nov 10, 2018
I created bpo-35206 "Add a new experimental _Py_CAPI2 API".