Repository navigation
Potentially serious regression in garbage collector performance in Python 3.14 #139951
Description
Activity
- added a commit that references this issue
on Oct 13, 2025 @picnixz It could be that it is some hash collision/PGO stuff, but by bisecting, I found that the first commit that makes the following code run in more than just a couple of seconds, is e28477f (from #117120 --- Change the size of the GC increment to about 1% of the total heap size).
n = 2000 d = {} for i in range(n): d[(i, i)] = d[(i, 0)] = d[(0, i)] = 0 for i in range(1, n): for j in range(1, n): d[(i, j)] = ( i + j + d[(i - 1, j)] + d[(i, j - 1)] + d[(i - 1, j - 1)] ) % ((2**20) - 1) print((d[(n - 1, n - 1)] % 2) != 0)
That said, the code base today runs the code in around 20 seconds on my computer. Which means that Python went from 4 seconds to 60 seconds down to 20 seconds.
Appendix: bisection log
git bisect start # status: waiting for both good and bad commits # bad: [52996aaa78706e0f2a70953feaf97b474eb438e7] gh-107073: fix relevant typo in `PyObject_ClearManagedDict` (#140032) git bisect bad 52996aaa78706e0f2a70953feaf97b474eb438e7 # status: waiting for good commit(s), bad commit known # good: [357bed0bcd3c5d7c4a8caad451754a9a172aca3e] GH-104668: Don't call PyOS_* hooks in subinterpreters (GH-104674) git bisect good 357bed0bcd3c5d7c4a8caad451754a9a172aca3e # bad: [422855ad21f09b82c0bfa891dfb8fb48182c6d2b] gh-120823: Fix doc for ftplib.FTP.retrbinary() (GH-121697) git bisect bad 422855ad21f09b82c0bfa891dfb8fb48182c6d2b # good: [cfa25fe3e39e09612a6ba8409c46cf35a091de3c] gh-113149: Improve error message when JSON has trailing comma (GH-113227) git bisect good cfa25fe3e39e09612a6ba8409c46cf35a091de3c # bad: [26d328b2ba26374fb8d9ffe8215ecef7c5e3f7a2] GH-117121: Add pystats to JIT builds (GH-117346) git bisect bad 26d328b2ba26374fb8d9ffe8215ecef7c5e3f7a2 # good: [72340d15cdfdfa4796fdd7c702094c852c2b32d2] gh-114563: C decimal falls back to pydecimal for unsupported format strings (GH-114879) git bisect good 72340d15cdfdfa4796fdd7c702094c852c2b32d2 # good: [eb22e2b251002b65f3b93e67c990c21e1151b25d] gh-115490: Make the interpreter.channels and interpreter.queues Modules Handle Reloading Properly (gh-115493) git bisect good eb22e2b251002b65f3b93e67c990c21e1151b25d # good: [ce2c996b2f645cd886d05f6ef8f1ba60ced7d4b7] gh-111926: Simplify proxy creation logic (#116844) git bisect good ce2c996b2f645cd886d05f6ef8f1ba60ced7d4b7 # good: [f4cc77d494ee0e10ed84ce369f0910c70a2f6d44] gh-116869: Enable -Werror in test_cext for Free Threading (#117106) git bisect good f4cc77d494ee0e10ed84ce369f0910c70a2f6d44 # bad: [507896d97dcff2d7999efa264b29d9003c525c49] gh-116936: Add PyType_GetModuleByDef() to the limited C API (#116937) git bisect bad 507896d97dcff2d7999efa264b29d9003c525c49 # good: [42ae924d278c48a719fb0ab86357f3235a9f7ab9] gh-117127: glob tests: Reopen dir_fd to pick up directory changes (GH-117128) git bisect good 42ae924d278c48a719fb0ab86357f3235a9f7ab9 # bad: [f11d0d8be8af28e1368c3c7c116218cf65ddf93e] gh-91227: Ignore ERROR_PORT_UNREACHABLE in proactor recvfrom() (#32011) git bisect bad f11d0d8be8af28e1368c3c7c116218cf65ddf93e # good: [5a78f6e798d5c2af1dba2df6c9f1f1e5aac02a86] gh-117134: Microoptimize glob() for include_hidden=True (GH-117135) git bisect good 5a78f6e798d5c2af1dba2df6c9f1f1e5aac02a86 # bad: [e28477f214276db941e715eebc8cdfb96c1207d9] GH-117108: Change the size of the GC increment to about 1% of the total heap size. (GH-117120) git bisect bad e28477f214276db941e715eebc8cdfb96c1207d9 # good: [e2e0b4b4b92694ba894e02b4a66fd87c166ed10f] gh-113024: C API: Add PyObject_GenericHash() function (GH-113025) git bisect good e2e0b4b4b92694ba894e02b4a66fd87c166ed10f # first bad commit: [e28477f214276db941e715eebc8cdfb96c1207d9] GH-117108: Change the size of the GC increment to about 1% of the total heap size. (GH-117120)
First bad commit
e28477f is the first bad commit
commit e28477f214276db941e715eebc8cdfb96c1207d9 Author: Mark Shannon <mark@hotpy.org> Date: Fri Mar 22 18:43:25 2024 +0000 GH-117108: Change the size of the GC increment to about 1% of the total heap size. (GH-117120) Include/internal/pycore_gc.h | 3 +- Lib/test/test_gc.py | 35 +++++++++++++++------- .../2024-03-21-12-10-11.gh-issue-117108._6jIrB.rst | 3 ++ Modules/gcmodule.c | 2 +- Python/gc.c | 30 +++++++++---------- Python/gc_free_threading.c | 2 +- 6 files changed, 47 insertions(+), 28 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2024-03-21-12-10-11.gh-issue-117108._6jIrB.rstReacted by Erik Parmann and yihongcc @markshannon
Perhaps interestingly, we see the exact same behavior if we use
functools.cacheand recursion:from functools import cache n = 2000 @cache def d(i, j): if i == 0 or j == 0 or i == j: return 0 return (i + j + d(i - 1, j) + d(i, j - 1) + d(i - 1, j - 1)) % ((2**20) - 1) count = 0 for i in range(n): for j in range(n): count += (d(i, j) % 2) != 0 print(count)
This probably just shows that
cacheusesdictto memoize.[~]$ time pypy3 memo.py real 0m4,820s [~]$ time python memo.py real 0m4,476s [~]$ time cpython/python memo.py real 0m16,431sIt seems that your example abuses the GC.
If you setgc.set_threshold(2000*2000), for example, you will get a better result.After some digging, I found the following.
- The default value for the first GC threshold is 2000.
- Each innermost cycle creates a few tuples.
- These tuples are short-living and will be destroyed in either
BINARY_OPorSTORE_SUBSCR - Creating each tuple increments the value of
gc.young.count. - And this is a problem because
gc.young.countgrows until it reachesgc.threshold[0]value, at which point the GC will be scheduled - At the end of the loop iteration (after
JUMP_BACKWARD) the scheduled GC will be run. - In your example, you get a lot of GC runs despite the short-lived tuples.
Where is the problem?
In the
tuple_dealloc:
Lines 215 to 218 in 13e9c2d
There is a condition that checks the freelist and calls
tp_freeunder certain conditions.tp_freefor PyTuple is aPyObject_GC_Delwhich decrements the value ofgc.young.count.For some reason, the condition is false in our case, so we don't call
PyObject_GC_Deland don't decrementgc.young.count.I don't yet know how to fix this. However, I will investigate further.
Reacted by Mikhail Efimov and yihongOk, I studied a bit further how freelists work.
Sotuple_deallocworks as expected.What is going on:
Runtime has a set of freelists for tuples (for different sizes - from 1 to 10), size of the each freelist for tuples are 2000 elements.When we create new tuple, we call
tuple_allocand if freelist has available items - we get one and use it.In our case we create 4 tuples on each iteration. One of them we store as a key in our dict. Other 3 tuples are short-living and
tuple_dealloccalled for them and they returned to freelist. But we took 4 and return 3. This leads to the situation when after some iteration there are no available items in freelist and we create new one withPyObject_GC_NewVarand increfgc.young.count.Created tuple then will be returned to the freelist. And after some repetitions of this situation, we get
gc.young.count == 2000and GC will be scheduled and run.When we run the schedule GC then the incremental version is called. The incremental version has to check local variables/frames and mark live objects and exclude them from the list to process.
So, we run the GC in our cycles may times. Each run will check the local
ddict and all keys and values of it. On each run we have to check more and more keys.This is a problem as I see it. And @pgdr right about bisected commit.
There are two approaches to work around the problem for now - disable the GC before and enable after filling the local dictionary. Or change the value of the first threshold for more appropriate value.
Reacted by Sergey B Kirpichev, Mikhail Efimov and yihongI can confirm that the following code
import gc #gc.disable() n = 2000 d = {} for i in range(1, n): for j in range(1, n): d[(i, j)] = i
takes Python 3.14 from 10 seconds to 1.4 seconds by toggling disable.
Apologies for the noise.
With
test.pyas:import sys, gc if len(sys.argv) == 2 and sys.argv[1].strip().lower() == "disable": gc.disable() n = 2000 d = {} for i in range(1, n): for j in range(1, n): d[(i, j)] = i
[...]$ hyperfine "python3.13 test.py" "python3.13 test.py disable" "python3.14 test.py" "python3.14 test.py disable" Benchmark 1: python3.13 test.py Time (mean ± σ): 1.488 s ± 0.061 s [User: 1.217 s, System: 0.257 s] Range (min … max): 1.441 s … 1.643 s 10 runs Benchmark 2: python3.13 test.py disable Time (mean ± σ): 1.287 s ± 0.015 s [User: 1.024 s, System: 0.251 s] Range (min … max): 1.270 s … 1.311 s 10 runs Benchmark 3: python3.14 test.py Time (mean ± σ): 10.982 s ± 0.033 s [User: 10.678 s, System: 0.267 s] Range (min … max): 10.955 s … 11.064 s 10 runs Benchmark 4: python3.14 test.py disable Time (mean ± σ): 1.428 s ± 0.023 s [User: 1.147 s, System: 0.264 s] Range (min … max): 1.409 s … 1.489 s 10 runs Summary 'python3.13 test.py disable' ran 1.11 ± 0.02 times faster than 'python3.14 test.py disable' 1.16 ± 0.05 times faster than 'python3.13 test.py' 8.53 ± 0.10 times faster than 'python3.14 test.py'Reacted by Sergey Miryanov and Mikhail Efimov30 remaining items
- added a commit that references this issue
on Oct 28, 2025 However, I'd like to move some tests from #140204 to the new PR and link it here.
I propose closing this ticket once this work is completed.Tests have been added. Can we close the issue?
If someone wants to work on a GC-specific benchmark, I suggest opening an issue in the pyperformance project. But it seems like @nascheme already wrote such benchmark (even if it doesn't cover this specific issue).
Reacted by Cody MaloneyTests have been added. Can we close the issue?
I think so.
- added a commit that references this issue
on Oct 29, 2025 - added a commit that references this issue
on Nov 27, 2025 - added 2 commits that reference this issue
on Jan 24, 2026
Bug report
Bug description:
The following code, consisting of a bunch of
dictlookups and writes, runs inTests are run on a Dell Precision 5490, running Ubuntu:
The cpython is from python-build-standalone
cpython-3.14.0+20251010-x86_64-unknown-linux-gnu-install_only.tar.gzcpython-3.13.8+20251010-x86_64-unknown-linux-gnu-install_only.tar.gzLog
It seems to scale very poorly too. With the same code, but
n = 2000(instead ofn = 1000), leads to 3.3 sec (Python 3.13) vs 20 seconds (Python 3.14)! I.e., Python 3.13 being 5 times faster.Plotting time (in seconds) of the script run with
n = 0 ... 2000.CPython versions tested on:
3.14
Operating systems tested on:
Linux
Linked PRs