Repository navigation
Remove cross-version binary compatibility #76569
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-05-29.20:13:09.708>
created_at = <Date 2017-12-20.17:27:58.959>
labels = ['interpreter-core', 'type-bug', '3.8']
title = 'Remove cross-version binary compatibility'
updated_at = <Date 2019-06-12.13:11:44.251>
user = 'https://github.com/pitrou'bugs.python.org fields:
activity = <Date 2019-06-12.13:11:44.251>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-29.20:13:09.708>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2017-12-20.17:27:58.959>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 32388
keywords = ['patch']
message_count = 12.0
messages = ['308753', '308881', '308882', '308884', '313182', '313351', '319944', '319947', '341560', '343918', '343919', '345339']
nosy_count = 12.0
nosy_names = ['barry', 'paul.moore', 'ncoghlan', 'pitrou', 'scoder', 'vstinner', 'tim.golden', 'ned.deily', 'njs', 'zach.ware', 'serhiy.storchaka', 'steve.dower']
pr_nums = ['4944']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32388'
versions = ['Python 3.8']
Activity
pitrou commentedon Dec 20, 2017
See original discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2017-December/151328.html
Summary: binary compatibility for C extensions which don't use the stable ABI is currently "best effort" and uses distinguished flags in the tp_flags field of type objects to indicate whether a field is physically present or not. Unfortunately, tp_flags is 32 bits and therefore a scarce resource. Also, binary compatibility is 1) not guaranteed anyway 2) of less use nowadays.
So we propose to remove the binary compatibility requirement when creating static type objects (i.e. not using the stable ABI).
zooba commentedon Dec 21, 2017
Totally okay with this in theory, but wanted to clarify that the linked PR is really just taking advantage of removing the guarantee?
That is, the change isn't necessary to remove the requirement.
pitrou commentedon Dec 21, 2017
That's basically exact. Actually, there is no ABI guarantee at all (when not using the stable ABI), just a best effort for advanced users who know what they're doing.
Actually, this PR stems from Nathaniel's message here:
https://mail.python.org/pipermail/python-dev/2017-December/151345.html
zooba commentedon Dec 21, 2017
Good to know. As I said on the other issue, Windows won't load extension modules built for a different version anyway unless they use the stable ABI, so there's no real reason these flags need to be binary compatible between 3.x versions.
scoder commentedon Mar 3, 2018
I'm ok with "removing" this "guarantee" (although it seems too late to apply this specific change to 3.7 now).
While Cython users do ask for a way to build cross-version binaries from time to time, it's neither supported nor planned, and the underlying problem is IMHO solved better via tooling like the manylinux wheel build.
I also generally consider it good to let user code take advantage of new features in new CPython releases, even if it means that they need to retranslate their Cython code from time to time. The 1.5 years release cycle of CPython should exceed that of most "still in maintenance" extension modules out there, and it's long enough to not impose a real burden on their maintainers.
pitrou commentedon Mar 6, 2018
Yes, let's retarget to 3.8.
vstinner commentedon Jun 19, 2018
In which version of Python tp_finalize slot has been added? Does it mean that Python 3.8 wil crash on loading C extensions compiled on Python older than this version?
pitrou commentedon Jun 19, 2018
What makes you think that it would be otherwise be able to load and execute such C extensions without any bugs?
There is no ABI except the stable ABI, and tp_flags isn't part of the stable ABI.
PS : tp_finalize is in Python 3.4, but that doesn't really matter.
pitrou commentedon May 6, 2019
I updated the PR (sorry to let this sleep).
pitrou commentedon May 29, 2019
New changeset ada319b by Antoine Pitrou in branch 'master':
bpo-32388: Remove cross-version binary compatibility requirement in tp_flags (GH-4944)
ada319b
pitrou commentedon May 29, 2019
PR is merged now!
vstinner commentedon Jun 12, 2019
About ABI "foward compatibility", Python 3.8 introduced another backward incompatible change: bpo-37250 "C files generated by Cython set tp_print to NULL: PyTypeObject.tp_print removed, replaced with tp_vectorcall_offset".