Repository navigation
ElementTree.iterparse "leaks" file descriptor when not exhausted #101438
Description
Activity
I think this would require something like PEP 533 “Deterministic cleanup for iterators“ to solve.
In this case, I think you can fix it just by avoiding reference cycles that involve the generator. Below is an example refactoring of
iterparse.def iterparse(source, events=None, parser=None): # Use the internal, undocumented _parser argument for now; When the # parser argument of iterparse is removed, this can be killed. pullparser = XMLPullParser(events=events, _parser=parser) _root = None def iterator(source): nonlocal _root close_source = False try: if not hasattr(source, "read"): source = open(source, "rb") close_source = True yield None while True: yield from pullparser.read_events() # load event buffer data = source.read(16 * 1024) if not data: break pullparser.feed(data) root = pullparser._close_and_return_root() yield from pullparser.read_events() _root = root finally: if close_source: source.close() class IterParseIterator(collections.abc.Iterator): def __init__(self, source): self.it = iterator(source) def __next__(self): return next(self.it) @property def root(self): nonlocal _root return _root it = IterParseIterator(source) del iterator, IterParseIterator next(it) return it
This avoids two reference cycles that captured the generator created by
iterator(source):- Classes naturally form reference cycles, so move
iterator(source)to the instance instead of a class attribute - Move
_rootup so thatiterator(source)doesn't refer to the instanceit.
I'm not sure about a few things including:
- How do you test this?
- Is
it.rootimportant? It's not documented. Is it only for testing?
EDIT: the problem is not capturing of the file descriptor, but the generator
iterator(source).- Classes naturally form reference cycles, so move
@colesbury Doe the issue persist if you add
gc.collect()after thedel it?@rhettinger everything is cleaned up properly once the garbage collector runs. So adding a
gc.collect()after thedel itwould collect the generator and close the file descriptor.- added a commit that references this issue
on Apr 19, 2023
The PR #31696 attempts to fix the "leak" of file descriptors when the iterator is not exhausted. That PR fixes the warning, but not the underlying issue that the files aren't closed until the next tracing garbage collection cycle.
Note that there isn't truly a leak of file descriptors. The file descriptors are eventually closed when the file object is finalized (at cyclic garbage collection). The point of the
ResourceWarning(in my understanding) is that waiting until the next garbage collection cycle means that you may temporarily have a lot of unwanted open file descriptors, which could exhaust the global limit or prevent successful writes to those files on Windows.On my system, after lowering the file descriptor limit to 1000 (via
ulimit -Sn 1000) I get:Linked PRs