Repository navigation
Weird test failures related to tracebacks #95818
Description
Bug report
I've recently been getting test failures, including segmentation faults, on different arcs (MacOS, Docker images) with Python 3.11 beta. The test itself is not at fault so I guess it's some kind of runtime issue.
Here's the traceback:
________________________ test_read_single_cell_formula[False-='Sheet2 - Numbers'!D5] ________________________
exc = <class 'ResourceWarning'>
value = ResourceWarning("unclosed file <_io.BufferedReader name='sample.xlsx'>")
tb = <traceback object at 0x10e3f4b00>, limit = None, chain = True
def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, \
chain=True):
"""Format a stack trace and the exception information.
The arguments have the same meaning as the corresponding arguments
to print_exception(). The return value is a list of strings, each
ending in a newline and some containing internal newlines. When
these lines are concatenated and printed, exactly the same text is
printed as does print_exception().
"""
value, tb = _parse_value_tb(exc, value, tb)
> te = TracebackException(type(value), value, tb, limit=limit, compact=True)
chain = True
exc = <class 'ResourceWarning'>
limit = None
tb = <traceback object at 0x10e3f4b00>
value = ResourceWarning("unclosed file <_io.BufferedReader name='sample.xlsx'>")
/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:139:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:688: in __init__
self.stack = StackSummary._extract_from_extended_frame_gen(
_seen = {4539126048}
capture_locals = False
compact = True
exc_traceback = <traceback object at 0x10e3f4b00>
exc_type = <class 'ResourceWarning'>
exc_value = ResourceWarning("unclosed file <_io.BufferedReader name='sample.xlsx'>")
is_recursive_call = False
limit = None
lookup_lines = True
max_group_depth = 10
max_group_width = 15
self = <traceback.TracebackException object at 0x10e4029d0>
/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:416: in _extract_from_extended_frame_gen
for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
capture_locals = False
fnames = set()
frame_gen = <generator object _walk_tb_with_full_positions at 0x10d50c3c0>
klass = <class 'traceback.StackSummary'>
limit = None
lookup_lines = True
result = []
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tb = <traceback object at 0x10e3f4b00>
def _walk_tb_with_full_positions(tb):
# Internal version of walk_tb that yields full code positions including
# end line and column information.
while tb is not None:
> positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
E AttributeError: 'frozenset' object has no attribute 'f_code'
tb = <traceback object at 0x10e3f4b00>
/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:353: AttributeError
========================================== short test summary info ==========================================
FAILED openpyxl/tests/test_iter.py::test_read_single_cell_formula[False-='Sheet2 - Numbers'!D5] - Attribut...
=========================== 1 failed, 2467 passed, 16 skipped, 6 xfailed in 8.24s ===========================
ERROR: InvocationError for command /Users/charlieclark/Projects/openpyxl/.tox/py311/bin/pytest openpyxl (exited with code 1)
__________________________________________________ summary __________________________________________________
ERROR: py311: commands failedAnd an example from the Docker image at https://foss.heptapod.net/openpyxl/openpyxl/-/jobs/618571
Your environment
MacOS 10.15, official docker image (slim)
Python 3.11 beta 4 & 5
I suspect the issue is somewhere between pytest and Python3.11
Activity
mdboom commentedon Aug 9, 2022
Are you able to provide a standalone test case that reproduces the issue? Still inside pytest is probably fine, but it would be helpful to disentangle it from the rest of your test suite.
What version of pytest are you using?
Themanwithoutaplan commentedon Aug 9, 2022
iritkatriel commentedon Aug 9, 2022
Any idea how you ended up with
tb.tb_framewhich is a frozenset?iritkatriel commentedon Aug 9, 2022
What is "everything" exactly?
iritkatriel commentedon Aug 9, 2022
Since I can't reproduce on my end, could you try debugging with a monkey-patch of the traceback display code, along these lines?
Themanwithoutaplan commentedon Aug 10, 2022
pytest -rf openpyxl or tox -e py311 openpyxl
Themanwithoutaplan commentedon Aug 10, 2022
No. This is obviously a bug, which is why I'm reporting it but it's not in any of my code. I suspect this is coming from some kind of interaction between pytest and Python 3.11 in a race condition when lots of tests are run, which is the only way I can reproduce it.
iritkatriel commentedon Aug 10, 2022
It could be a bug in pytest, openpyxl, tox or python.
Please report it to the other projects so that they can investigate and, if the bug is in python, narrow it down to a small reproducer that we can look into.
Themanwithoutaplan commentedon Aug 10, 2022
It's obviously not a bug in openpyxl. The same code relevant to the test hasn't been touched for years and the seg fault only appeared on CI last week and only with Python 3.11 which is why I submitted the bug. We1 test against Python ≥ 3.6, including PyPy, and the problem only occurs with Python 3.11 but can be reliably reproduced.
iritkatriel commentedon Aug 10, 2022
It's not a segfault, it's an exception.
If you can't reduce this to something that doesn't involve 3 third-party programs, or debug it like I suggested, then the next option is to report it to pytest and see if they can.
(they seem to be using frozensets for paths, so maybe they will know what happened).
Themanwithoutaplan commentedon Aug 10, 2022
As the problem does not stem directly from the code being tested, or even from an individual test, see the extensive multiversion test history, I'm limited in what I can debug directly. I will report the problem to pytest but there's definitely some kind of combination with Python 3.11 and this is recent. I've been running CI against 3.11 since before the first beta. I do this so that the library can be used with Python 3.11 as soon as this is released but with failures like this I'll have to mark openpyxl as not suitable for use with Python 3.11.
From a relevant segfault
iritkatriel commentedon Aug 10, 2022
Your original report was about an exception being raised, and you twice referred to it as a segfault in followup comments. Is there a segfault?
I'm finding this bug report very confusing.
I believe everything you say (I believed you the first time), but none of this is helping me fix anything.
ericsnowcurrently commentedon Aug 10, 2022
Were all the loaded extension modules built against the same (beta) version of Python that you ran when the failure happened?
36 remaining items
brandtbucher commentedon Aug 11, 2022
Nice debugging! I'm still not sure how this new bug relates to the weird
tb_framebug, though. Does the interpreter state just get messed up that badly if the assert is turned off?pablogsal commentedon Aug 11, 2022
I think there is some ownership problems with they frame and it gets deleted while someone is holding to it and then you get garbage.
brandtbucher commentedon Aug 11, 2022
@markshannon, looks like
assert(!_PyFrame_IsIncomplete(frame))in_PyFrame_GetFrameObjectis invalid when generator creation kicks off a GC run and one of the finalizers grabs the frame one level up.markshannon commentedon Aug 11, 2022
@pablogsal Thanks for the reproducer.
In theory, incomplete frames should be skipped in tracebacks and the like. But it looks like we don't do that in
PyThreadState_GetFrame(). Let me fix that.pythonGH-95818: Skip incomplete frames in `PyThreadState_GetFrame` (p…
GH-95818: Skip incomplete frames in `PyThreadState_GetFrame` (GH-95886)
GH-95818: Skip incomplete frames in `PyThreadState_GetFrame` (GH-95886)…
markshannon commentedon Aug 24, 2022
Is this fixed now?
Themanwithoutaplan commentedon Aug 24, 2022
pablogsal commentedon Sep 5, 2022
I'm closing this as the underlying issue seems to be fixed. Please reopen if we are missing something.
sweeneyde commentedon Sep 8, 2022
The added test case looks delicate, so I don't want to disrupt it too much, but it looks like it's emitting the following warning on some buildbots:
Should this be run in a separate process or have the warnings suppressed or something?
sweeneyde commentedon Sep 8, 2022
I opened #96696