Repository navigation
Memory keeps increasing with fixed-size dict during multi-threaded set/delete in 3.13.3t #133136
Description
Activity
This bug can be reproduced on my Linux environment. I think may need some time to figure it out
For now, I think this may related with the gc timing
import random import time from threading import Lock, Thread import resource import gc class Cache: def __init__(self, cap): self.m = {} self.mu = Lock() self.cap = cap def get(self, key): with self.mu: try: v = self.m[key] except KeyError: return 0, False return v, True def set(self, key, value): with self.mu: self.m[key] = value if len(self.m) > self.cap: self.m.popitem() def set_key(cache): c = 0 while True: key = random.randint(0, 1000000000) cache.set(key, key) c += 1 if c == 100000: c = 0 print(".", end="", flush=True) def run(): cache = Cache(500000) for _ in range(4): thread = Thread(target=set_key, args=[cache]) thread.start() while True: time.sleep(1) with cache.mu: usage = resource.getrusage(resource.RUSAGE_SELF) rss_mb = usage.ru_maxrss print(f"Process RSS: {rss_mb:.2f} KB") # gc.collect() if __name__ == "__main__": run()
if we run the gc.collect manually, the memory size will keep normal. Otherwise, the memory size will increase in 40MB in regular peroid
It seems to run normally in v3.13.0a3.
After 10-minute loops, the memory size remains at less than 200MB.
This is a good signal. I think I can bisect for the bad commit.
Reacted by Yiling-J@Zheaoli Thanks for investigating the issue. I tested by manually triggering GC every 10 seconds and every 1 second. For the 10-second interval, the RSS stabilized around
rss: 893140992, dict len: 500000. For the 1-second interval, the RSS stabilized aroundrss: 490344448, dict len: 500000. I also tested 3.13.0t without manually triggering GC, the memory usage still increases, though slower compared to 3.13.3t.Got it. I think this bug is introduced into the codebase in #116336
But I don't have enough idea about how to fix this. Would you mind taking a little bit of help? @colesbury
@colesbury PTAL when you got time
Reacted by Yiling-JI think this issue is still related with the GC. I count the
free_keys_objecttime. The normal version is much more than the free thread version.Reacted by Yiling-JI think this issue is still related with the GC. I count the
free_keys_objecttime. The normal version is much more than the free thread version.This seems like a reasonable explanation for the issue. Hopefully, @colesbury can take a look when he has time, or if he's too busy, perhaps someone else familiar with free-threading could also help investigate.
I think we need better heuristics for the QSBR here. The repeated insertions and
popitem()calls lead to the dictionary keys object getting reallocated every so often. The PyDictKeysObjects here are large here so keeping a bunch of them means a lot of extra memory usage in the example.We "buffer" QSBR frees in two ways:
- We only advance the global QSBR counter every 10 calls to
_Py_qsbr_deferred_advance. This is to reduce contention on the shared counter for frequent calls, but that's probably too infrequent here given the size of the keys objects
Line 1203 in 1ffe913
- We only process the delayed free requests every
WORK_ITEMS_PER_CHUNK(254) calls, again to limit work for frequent calls.
Lines 1208 to 1210 in 1ffe913
In addition to periodic processing every 10 or 254 calls, I think we probably want to take into account the amount of memory that the QSBR subsystem is holding.
Reacted by Yiling-J- We only advance the global QSBR counter every 10 calls to
- added a commit that references this issue
on Jun 3, 2025 An update on the status: fix is in main and 3.14. We are waiting for the 3.13 backport to be approved and then I think this bug can be closed.
Reacted by Yiling-J
Bug report
Bug description:
Platform: macOS-14.6.1-x86_64-i386-64bit-Mach-O
Python 3.13.3t
Python 3.13.3
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Linked PRs