Repository navigation
Pathlib.iterdir semantics change dramatically under Python 3.13聽#129871
Description
Activity
Silly me. I just realized this issue probably isn't about a change to the generator expression but a change to pathlib.
- changed the title
[-]Generator semantics change dramatically under Python 3.13[/-][+]Pathlib.iterdir semantics change dramatically under Python 3.13[/+]on Feb 8, 2025 Indeed, the generator expression is not relevant:
馃悮 py -3.12 -c "import pathlib; pathlib.Path('does-not-exist').iterdir()" 馃悮 py -3.13 -c "import pathlib; pathlib.Path('does-not-exist').iterdir()" Traceback (most recent call last): File "<string>", line 1, in <module> import pathlib; pathlib.Path('does-not-exist').iterdir() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pathlib/_local.py", line 575, in iterdir with os.scandir(root_dir) as scandir_it: ~~~~~~~~~~^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: 'does-not-exist'I was even involved in that issue and had completely forgotten.
So it was an intentional change, even though it doesn't appear in the "what's new". At the very least, we should add a note to "what's new", as this behavior might be surprising.
I'm also thinking we should revisit our assumptions from #78722 in light of this new experience. It demonstrates that the change in behavior isn't necessarily beneficial and there may be other use-cases out there that are relying on the late evaluation of the error condition.
I don't feel strongly about it, and I'm happy to proceed with this new behavior, but we should ask ourselves if a rollback is warranted to retain consistency across Python versions.
I'll get a PR up for adding it to the whatsnew, and perhaps to the
Path.iterdir()docsI still prefer the new behaviour - it meets most users expectations better, it makes exception handling much more straightforward, and it matches how
os.scandir()raises exceptions too.Reacted by Jason R. Coombs and Petr ViktorinThis is indeed a big change.
Some libs such asanyioare relying on the fact thatiterdir()doesn't do blocking syscalls and defer to a thread only when it's iterated.Maybe we should revert the
iterdir()change then, despite it being a little awkward to handle exceptions previously.I'm so conflicted on this, I can't even advocate for one approach over the other. I'm inclined to say we should raise the issue with a larger group. I do think it would be worthwhile to make a decision quickly to reduce the duration of exposure. Maybe just a post in Core Dev discuss? I'm happy to defer to your judgment Barney, but let me know if you'd like my help garnering a wider consensus.
I think I would expect an iterator to lazily surface exceptions as it does work. If a user wants to surface them sooner they can always wrap the result in a tuple or list themselves to drive the iterator.
Given that this change broke anyio, I think it should be reverted. Since there is a not-too-complicated alternative (wrap Path.iterdir() in
list()if you want eager exceptions to be raised), the motivation does not seem to justify breaking behavior of downstream users to me.I think that raising an exception for non-existing directory (and for non-directory) is right. If you want to make this exception lazy, just wrap the creation of the iterator in a generator function:
def path_iterdir(p): yield from p.iterdir()
Bug report
Bug description:
This code behaves very differently on Python 3.13 than 3.12:
In Python 3.12, the generator expression was evaluated lazily, not invoking
path.iterdir()until the generator was consumed. On 3.13, at least a portion of the generator is evaluated, triggering the exception when the dir does not exist.I discovered this issue in jaraco/jaraco.develop#26.
I looked at What's New for Python 3.13, and there's only one mention of generator expressions regarding mutation of locals in generator expressions, which doesn't seem to be relevant here.
That change mentions #74929, so maybe that change is also implicated in the change in execution order.
CPython versions tested on:
3.13
Operating systems tested on:
macOS