Repository navigation
[subinterpreters] Make free lists and unicode caches per-interpreter #84701
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2020-05-05.15:48:02.489>
labels = ['expert-subinterpreters', '3.10']
title = '[subinterpreters] Make free lists and unicode caches per-interpreter'
updated_at = <Date 2022-01-06.15:24:07.868>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2022-01-06.15:24:07.868>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2020-05-05.15:48:02.489>
creator = 'vstinner'
dependencies = []
files = ['49212', '49213', '49216', '49699', '49899']
hgrepos = []
issue_num = 40521
keywords = ['patch']
message_count = 56.0
messages = ['368175', '368177', '368187', '368278', '368283', '368807', '368808', '369407', '370636', '370733', '370734', '370735', '370737', '370740', '370741', '370742', '370754', '370755', '370756', '370757', '370771', '370928', '370969', '372146', '372148', '372161', '372168', '372169', '372176', '372181', '372207', '372209', '372216', '372220', '372223', '372250', '372357', '372795', '377368', '383789', '383790', '383829', '385950', '388492', '388493', '388617', '389226', '389294', '389305', '389525', '389527', '393787', '395860', '409819', '409856', '409862']
nosy_count = 7.0
nosy_names = ['rhettinger', 'mark.dickinson', 'vstinner', 'Mark.Shannon', 'corona10', 'shihai1991', 'JunyiXie']
pr_nums = ['19933', '19937', '19959', '19960', '20081', '20082', '20085', '20246', '20247', '20636', '20637', '20638', '20642', '20643', '20644', '20645', '21068', '21073', '21074', '21082', '21085', '21086', '21096', '21099', '21101', '21103', '21116', '21142', '21265', '22376', '24821', '24944', '24964', '25906', '30422', '30425', '30433']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue40521'
versions = ['Python 3.10']
Activity
vstinner commentedon May 5, 2020
tuple, dict and frame use free lists to optimize the creation of objects.
Unicode uses "interned" strings to reduce the Python memory footprint and speedup dictionary lookups.
Unicode also uses singletons for single letter Latin1 characters ([U+0000; U+00FF] range).
All these optimizations are incompatible with isolated subinterpreters, since caches are currently shared by all inteprepreters. These caches should be made per-intepreter. See bpo-40512 "Meta issue: per-interpreter GIL" for the rationale.
I already made small integer singletons per interpreter in bpo-38858:
vstinner commentedon May 5, 2020
New changeset 607b102 by Victor Stinner in branch 'master':
bpo-40521: Disable Unicode caches in isolated subinterpreters (GH-19933)
607b102
vstinner commentedon May 5, 2020
New changeset b4b5386 by Victor Stinner in branch 'master':
bpo-40521: Disable free lists in subinterpreters (GH-19937)
b4b5386
vstinner commentedon May 6, 2020
New changeset 89fc4a3 by Victor Stinner in branch 'master':
bpo-40521: Disable method cache in subinterpreters (GH-19960)
89fc4a3
vstinner commentedon May 6, 2020
New changeset b7aa23d by Victor Stinner in branch 'master':
bpo-40521: Disable list free list in subinterpreters (GH-19959)
b7aa23d
vstinner commentedon May 13, 2020
I wrote a draft PR to make interned strings per-interpreter. It does crash because it requires to make method cache and _PyUnicode_FromId() (bpo-39465) compatible with subinterpreters.
vstinner commentedon May 13, 2020
New changeset 3d17c04 by Victor Stinner in branch 'master':
bpo-40521: Add PyInterpreterState.unicode (GH-20081)
3d17c04
[-]Make tuple, dict, frame free lists, unicode interned strings, unicode latin1 singletons per-interpreter[/-][+][subinterpreters] Make free lists and unicode caches per-interpreter[/+]55 remaining items
vstinner commentedon Mar 25, 2021
I reopen the issue to remind me that collections.deque() freelist is shared by all interpreters.
vstinner commentedon May 17, 2021
Each deque instance now has its own free list.
But dtoa.c still has a per-process cache, shared by all interpreters.
rhettinger commentedon Jun 15, 2021
[Victor Stinner]
FWIW, PyFloat_FromDouble() is the most performance critical function in floatobject.c.
vstinner commentedon Jan 6, 2022
New changeset 35d6540 by Victor Stinner in branch 'main':
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (GH-20085)" (GH-30422)
35d6540
vstinner commentedon Jan 6, 2022
New changeset 72c260c by Victor Stinner in branch '3.10':
[3.10] bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (GH-20085)" (GH-30422) (GH-30425)
72c260c
vstinner commentedon Jan 6, 2022
My commit ea25180 (interned strings) introduced bpo-46006 "[subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters" regression.
To unblock the Python 3.11.0a4 release, I just reverted the change. It reintroduces the issue, so I created bpo-46283: "[subinterpreters] Unicode interned strings must not be shared between interpreters".
vstinner commentedon Nov 3, 2022
I don't have the bandwidth to fix the remaining issues, so I just close again the issue.
Use mortal strings in type_setattro