Repository navigation
Regression in python3.10 with traceback frame having lineno of -1 #88099
Description
Activity
asottile commented
on Apr 24, 2021 asottilemannequinMannequinAuthorThis is breaking pytest for failed assertions: pytest-dev/pytest#8227
It also breaks the traceback in the output below
Here's a minimal example:
class Boom: def __enter__(self): return self def __exit__(self, *_): raise AssertionError('boom!') def main() -> int: with Boom(): raise AssertionError('hi') if __name__ == '__main__': exit(main())
On python3.9 you get this:
Traceback (most recent call last): File "/home/asottile/workspace/cpython/t.py", line 10, in main raise AssertionError('hi') AssertionError: hi During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/asottile/workspace/cpython/t.py", line 14, in <module> exit(main()) File "/home/asottile/workspace/cpython/t.py", line 10, in main raise AssertionError('hi') File "/home/asottile/workspace/cpython/t.py", line 5, in __exit__ raise AssertionError('boom!') AssertionError: boom!output in python3.10:
Traceback (most recent call last): File "/home/asottile/workspace/cpython/t.py", line 10, in main raise AssertionError('hi') AssertionError: hi During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/asottile/workspace/cpython/t.py", line 14, in <module> exit(main()) File "/home/asottile/workspace/cpython/t.py", line -1, in main File "/home/asottile/workspace/cpython/t.py", line 5, in __exit__ raise AssertionError('boom!') AssertionError: boom!Notice the second to last frame is now missing and invalid (line -1)
I bisected this and found this is the culprit:
3bd6035b6baf1a7d51b7cc2c6bb2c81886236b67 is the first bad commit commit 3bd6035b6baf1a7d51b7cc2c6bb2c81886236b67 Author: Mark Shannon <mark@hotpy.org> Date: Wed Jan 13 12:05:43 2021 +0000 bpo-42908: Mark cleanup code at end of try-except and with artificial (#24202) * Mark bytecodes at end of try-except as artificial. * Make sure that the CFG is consistent throughout optimiization. * Extend line-number propagation logic so that implicit returns after 'try-except' or 'with' have the correct line numbers. * Update importlib Lib/test/test_dis.py | 2 +- Lib/test/test_sys_settrace.py | 40 + Python/compile.c | 135 +- Python/importlib.h | 3153 ++++++++++++++--------------- Python/importlib_external.h | 4489 ++++++++++++++++++++--------------------- Python/importlib_zipimport.h | 1013 +++++----- 6 files changed, 4473 insertions(+), 4359 deletions(-) bisect run successwhich appears to be due to bpo-42908
Marking as release blocker since is breaking pytest
Thanks for the bisection, Anthony
What exactly is the issue here?
That the line number is -1, or that is different from 3.9?The
-1should beNone. I can fix that, but there is the issue of whether the cleanup code in a with statement should have a line number or not.The traceback for 3.9 is incorrect, the program is not on line 10 when __exit__ is called.
Currently we mark the cleanup code for the with statement as artificial meaning that it has no line number.
We could give it the line number of thewithkeyword, but that might break coverage and profiling tools.PEP-626 says that the "with" keyword is executable, and should be traced, but nothing about whether that includes the cleanup call to __exit__.
Mark, you say:
The traceback for 3.9 is incorrect, the program is not on line 10 when __exit__ is called.
It's kind of not on any line in main(), but line 10 is the most useful line to indicate. It's the last bit of the user's code that was running, and it is the reason for leaving the with statement.
In this case, if happens that line 10 was the last line of code executing in the body of the with statement.
But the reason it was listed, was that it was statically the last line of code in the body.If the body is something like:
- if TRUE:
-
... - else:
-
...
Then the reported last line will be 4, even though it is never executed.
We can fix this, at the cost of extra compiler complexity and increased bytecode size for the non-exceptional case, but it is very difficult for the exceptional case, because we don't know statically where the exception has come from.This test case was added for 3.10.
In 3.9 it reports that the pass statement executes, even though it is unreachable.https://github.com/python/cpython/blob/master/Lib/test/test_sys_settrace.py#L919
8 remaining items
asottile commented
on Apr 30, 2021 asottilemannequinMannequinAuthorcan confirm that pytest no longer crashes
I'm trying to see if these changes affect coverage.py, but pytest 6.2.3 fails with:
=== CPython 3.10.0a7+ (rev 558df90) with Python tracer (.tox/anypy/bin/python) ===
ImportError while loading conftest '/Users/nedbatchelder/coverage/trunk/tests/conftest.py'.
TypeError: required field "lineno" missing from aliasIs this something I am doing wrong?
asottile commented
on May 1, 2021 asottilemannequinMannequinAuthorah yeah that's the
asnameast change -- if you use the unreleased main branch it has a fix for thatThanks, tip of pytest works for me. Is this error going to happen for everyone testing with 3.10b1? If so, can we get a release of pytest?
Is this error going to happen for everyone testing with 3.10b1?
3.10.0b1 includes both of these PRs so people testing the beta should not get affected. That's why I marked this as release blocker.
The pytest failure still happens with 3.10.0b1 and pytest 6.2.3. I'm asking if pytest can make a release to update.
The pytest failure still happens with 3.10.0b1 and pytest 6.2.3.
What I am missing? 3.10.0b1 includes the fixes for this issue, no? Is it because the fixes still require some changes in the source? If that's the case, we should mention it in the What's New
I'm asking if pytest can make a release to update.
In any case, shouldn't that be requested in the pytest issue tracker?
asottile commented
on May 4, 2021 asottilemannequinMannequinAuthorI've released pytest 6.2.4 for the other breakage (which is unrelated to this bpo -- it was the asname source position issue)
Mark, I'm not clear if the line number can still be None in some circumstances.
With this code (from site.py in the stdlib):
545: def execsitecustomize():
546: """Run custom site specific code, if available."""
547: try:
548: try:
549: import sitecustomize
550: except ImportError as exc:
551: if exc.name == 'sitecustomize':
552: pass
553: else:
554: raise
555: except Exception as err:
556: if sys.flags.verbose:
557: sys.excepthook(*sys.exc_info())
558: else:
559: sys.stderr.write(
560: "Error in sitecustomize; set PYTHONVERBOSE for traceback:\n"
561: "%s: %s\n" %
562: (err.__class__.__name__, err))I get traces with these events and line numbers:
exception 549 (ModuleNotFoundError("No module named 'sitecustomize'"))
line 550
line 551
line 552
return NoneIs this what you expected?
Ned, no line numbers should never be None.
(Except under very unusual circumstances, like the trace function raising an error)Taking a look at the disassembly of execsitecustomize, there is a return with no line number, which shouldn't happen.
BTW, I created a separate issue about this: https://bugs.python.org/issue44088 because I wasn't sure if I should continue on this one, sorry. Feel free to close either one.
Thanks, this fixes my issue.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: