Repository navigation
Traceback parsing changes in 2.15.0 error on valid pycon code blocks #2407
Description
(Sorry, accidentally hit the submit shortcut before I was done writing it)
Unfortunately, the traceback parsing changes to fix #2226 and #2329 implemented in fe42aac (I can't seem to find the associated pull request) appears to result in Pygments treating certain valid code blocks as invalid, issuing a warning and failing Sphinx builds (with -W). In particular, this occurred on the following code block in the CPython's multiprocessing module docs:
.. note::
Functionality within this package requires that the ``__main__`` module be
importable by the children. This is covered in :ref:`multiprocessing-programming`
however it is worth pointing out here. This means that some examples, such
as the :class:`multiprocessing.pool.Pool` examples will not work in the
interactive interpreter. For example::
>>> from multiprocessing import Pool
>>> p = Pool(5)
>>> def f(x):
... return x*x
...
>>> with p:
... p.map(f, [1,2,3])
Process PoolWorker-1:
Process PoolWorker-2:
Process PoolWorker-3:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>that worked in 2.14.9, which when 2.15.0 was released suddenly broke the CPython Docs CI job with the error:
WARNING: Could not lex literal_block as "pycon". Highlighting skipped.
Here's a more minimal example that also doesn't parse:
Here's a code block::
>>> print("\n".join(["Traceback (most recent call last):"] * 2))
Traceback (most recent call last):
Traceback (most recent call last):
This also occurs with an explicit pycon lexer specified:
Here's another code block:
.. code-block:: pycon
>>> print("\n".join(["Traceback (most recent call last):"] * 2))
Traceback (most recent call last):
Traceback (most recent call last):and also with arbitrary text after the traceback line:
Here's a code block::
>>> print("\n".join(["Traceback (most recent call last):", "blah blah"])
Traceback (most recent call last):
blah blahWhile this example parses:
Here's a code block::
>>> print("\n".join(["Traceback (most recent call last):"] * 1))
Traceback (most recent call last):
Activity
[-]raceback parsing changes in 2.15.0 rejects valid pycon code blocks[/-][+]Traceback parsing changes in 2.15.0 rejects valid pycon code blocks[/+][-]Traceback parsing changes in 2.15.0 rejects valid pycon code blocks[/-][+]Traceback parsing changes in 2.15.0 error on valid pycon code blocks[/+]jeanas commentedon Apr 10, 2023
Thanks for reporting! On a note, though, does CPython not pin Pygments and other dependencies in its CI? I would have expected it, for such a large project where keeping a working CI is critical...
hugovk commentedon Apr 10, 2023
The direct dependencies are mostly pinned (https://github.com/python/cpython/blob/c3cd3d10788769558a4668e9e5cfff42fe377852/Doc/requirements.txt) but pygments is an indirect dependency, of Sphinx (https://github.com/sphinx-doc/sphinx/blob/2c83af0aab7080e0b78d4a16981eed878b2cac4c/pyproject.toml#L65).
jeanas commentedon Apr 10, 2023
OK, although it could use a lock file, or the sort-of-lockfile substitute given by
pip freeze.CAM-Gerlach commentedon Apr 11, 2023
Yeah, we definitely keep the direct deps pinned; this is just the first time we've seen this issue with the transitive deps and it wouldn't necessarily work cross platform if any of the transitive deps are conditional, and it also work would with other package managers like Conda. AFAIK this is the first time I can recall that we've seen this sort of breakage before from a non-direct dep. Also, the docs build server runs on a whole separate system (AFAIK) that I think is all pinned down, so not sure it would help the actual production builds right now anyway.
Usar Pygments < 2.15
Usar Pygments < 2.15 (#2373)
Python console: do not require output that looks like a traceback to …
birkenfeld commentedon Apr 12, 2023
Fix in #2410 - please try.
In general, 2.15 is still better behaved than older versions, which silently swallowed part of the output, as you can see e.g. here:
https://docs.python.org/3.10/library/multiprocessing.html
Note the single "Traceback" line.
Python traceback: do not require output to be valid
jeanas commentedon Apr 12, 2023
See also #2411
CAM-Gerlach commentedon Apr 13, 2023
Thanks @birkenfeld & @jeanas for the quick fix! Looks like that at least fixes the error and restores syntax highlighting on the block. It does look a bit odd that all three
Tracebacklines are highlighted, but only the firstAttributeErrorline is, but this is a pretty esoteric case....Any idea when the next release containing this fix will be released? If its soon, we can skip backporting the change to the older security-only CPython branches and just wait for the new version to be released. A lockfile would be nice longer term, but unfortunately, its not quite as simple as just having our GH Actions jobs use it, as the actual production docs site and other docs formats are currently produced by a separately managed build server with its own maintainers, scripts and configuration that would need to be investigated, modified, tested and deployed.
Temporarily install latest pygments to workaround warnings coming from …
Temporarily install latest pygments to workaround warnings coming from …
Temporarily install latest pygments to workaround warnings coming from …
jeanas commentedon Apr 18, 2023
Pygments 2.15.1 has been released with #2410 and #2412 (thanks @Anteru for the release work!).