Repository navigation
Generator expression behavior changed in 3.13.4 - it does not throw exception anymore #135171
Description
Activity
We should restore an old behavior which was intentional: GET_ITER should be called before passing argument to generator expression, not after.
Sounds like the change should be reverted. We're doing an expedited 3.13.5 for other reasons, and the rollback can go in there.
We need two rollbacks -- one for the change that removed GET_ITER before calling the generator expression, and other for the change that added GET_ITER in a generator expression (that may not be trivial).
What is the bug here?
bool(_ for item in False)should beTrue. All generators are truthy.If the intent was to write
any(_ for item in False)orall(_ for item in False)then the expectedTypeErroris raised.Consider the equivalent generator function:
def gen(): for item in False: yield _
creating the generator does not raise an exception, but iterating over it does.
>>> bool(gen()) True
Iterating over an iterable is a two step process.:
- Convert the iterable into an iterator (using the
GET_ITERinstruction) - Repeatedly get the next item from the iterator (using the
FOR_ITERinstruction)
For
forloops and generators, theFOR_ITERandGET_ITERare together and everything works fine.Historically, for reasons I am unaware of, in generator expressions the conversion from iterable to iterator occurred when the generator was created, not when used. This left a gap between conversion and the (unchecked) iteration which could mean that iteration could crash.
#125038.We attempted to fix that by putting a
GET_ITERin the generator expression as well, but that breaks iterators without__iter__methods, or where the__iter__has side effects. #127682
It also introduced a crash, but in SWIG generated code: rpm-software-management/libdnf#1682So we must do one of the following:
- Allow a crash. I'm strongly opposed to this option
- Break working iterators that do not implement
__iter__, which have always worked. - Allow the creation of some generator expressions that was previously not allowed, but will still fail as expected when iterated over.
Option 3 is what we now have and seems to me to be easily the best option. It breaks no working code, and produces the same exceptions for faulty code in a way that is largely indistinguishable from the behavior in 3.12.
That generator expressions and generator functions behaved differently could also be considered a bug.
See also #127682 (comment)
Reacted by Kirill Podoprigora- Convert the iterable into an iterator (using the
It should be a TypeError, because False is not iterable. It's been that way since the beginning, and I believe it was intentional, and that the existing code depends on it. It helps to detect bugs. Even if we want to change the behavior, it should be preceded by a general discussion, and we can only do that in a new Python version, not in a bugfix release.
We just have to return the old behavior. The crash you mentioned is only possible after low level hacking. The behavior change is visible to common users.
Reacted by Gregory P. Smith, Barry Warsaw and jtaub@markshannon I do not insist that it must be fixed, but I thought it could be considered for a major release. I encountered this issue in legacy code, and there’s no question it should be refactored to avoid this. The exception is used there to exit recursion while parsing nested dict-like structures, if you need a real-world scenario.
Casting toboolis not important, a more generic example would be something likex = (_ for _ in None). If the examples in the bug description seem confusing, I can replace them.@Roman513 the problem is not that your examples are hard to use, but they appear contrived. What was the original issue you found (in actual code)?
@serhiy-storchaka What about the other issues I listed, do they do not deserve to be fixed? Why this issue and not those?
One other thing. what about generator expressions with multiple iterables e.g.
((i,j) for i in range(3) for j in range(n))?Python 3.12 and earlier were already inconsistent here:
Python 3.12 >>> ((i,j) for i in range(3) for j in False) <generator object <genexpr> at 0x7447eaba43c0> >>> ((i,j) for i in False for j in range(n)) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'bool' object is not iterable
Reacted by Mikhail EfimovOverall, I believe the current behavior is the best. It is the robust, efficient and is consistent with generator functions. So I definitely want to keep it for 3.15+.
Regarding 3.13 and 3.14 there is one more thing we could attempt:
- Move
GET_ITERback to the creation of the generator expression for 3.13 and 3.14 - Add a
tp_iternextfunction toobjectso thatFOR_ITERnever crashes, just throws the expected exception.
The downside of this is that any C extensions doing
Py_TYPE(op)->tp_iternext != NULLto determine if an object is an iterator will be broken. Hyrum's law states that someone will be doing this 🙁Reacted by Mikhail Efimov- Move
32 remaining items
Should we close this issue? PRs for 3.13 and 3.14 are being merged.
But for Python 3.15+ we have to decide what to do.
I'll create a d.p.o topic to discuss this decision (see #135352 (comment)).Reacted by Hugo van KemenadeYes, let's. Thanks all!
Reacted by Just van Rossum and Mikhail Efimov- moved this from Todo to Done in Release and Deferred blockers 🚫
on Jun 17, 2025 If close this issue, we should open a new issue for
main. It is better to solve it earlier, because the more time will pass, the more changes we will need to roll back, and it may conflict with other code.Reacted by Mikhail EfimovI've started related d.p.o. thread here: https://discuss.python.org/t/generator-expressions-behavior-with-non-iterable-objects/96329.
Reacted by Hugo van Kemenade
Bug report
Bug description:
In 3.13.3 and before
Now in 3.13.4
This is very fundamental change. Is it expected?
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs