Decide what should we do with doctests in CI

@AlexWaygood

Bug report

Initial PR and discussion: #111682

So, here's the problem:

Example 1::

>>> 1 + 2
3

Example 2:

.. doctest::

>>> 1 + 2
3

We have two tools: doctest (builtin module) and sphinx (3rd party) that can run these samples as doctests. The problem is that they define "doctest" term differently:

doctest module says that both of these examples are doctests
sphinx says that only example with .. doctest:: directive is a doctest

This is problematic, because not all example are actually executed in CI, right now we have a CI job that uses Sphinx. So, in the result of this we have broken doctest (1st meaning) examples in the docs.

Examples:

What can we do?

Options:

Add .. doctest:: directives everywhere, but I think it is too much work and people will forget to do this
Try to use doctest_test_doctest_blocks, see docs https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html This is what I plan to do locally and I will send a PR with the results. I fear that quite a lot of doctests might be broken
Add doctests to test_* files using something like

def load_tests(loader, tests, pattern):
    tests.addTests(doctest.DocFileSuite(
        os.path.join(REPO_ROOT, 'Doc/library/LIBNAME.rst'),
        module_relative=False,
    ))
    return tests

The last option is also problematic, because right now we can only run doctests in CI, when PR only touches Doc/ folder, I am not sure that it will be easy with this setup.

CC @AlexWaygood @AA-Turner

Linked PRs

I hope that I will send the first PR today / tomorrow, but I am a bit sick :(

@AA-Turner

Option (2) is definitely attractive. Just two concerns from me:

How long will this make the Doctest CI job? It already takes 10 minutes to run all CPython's doctests via Sphinx; I worry that this option could make it significantly slower, by dramatically increasing the number of doctests run in CI. This is hypothetical, though; I might be wrong!
I agree with you that there'll probably be quite a few that fail at first. If we could figure out a way to tackle the errors incrementally, like what we've been doing with the Sphinx nitpicks, that would be ideal. Would it be possible to apply doctest_test_doctest_blocks only to certain files? (Cc. @AA-Turner again as the Sphinx expert here!)

Also, rest up and get well soon!

Option 4: Modify Sphinx to run both types?

But I'm also concerned about the doctest taking longer. Of the three docs jobs, doctest is the longest at 10 minutes (= 3m setup/building + 7m doctest) compared to 2.5 minutes each for the other two:

Can we do them in parallel instead of sequential? Can the doctest module run them quicker?

Modify Sphinx to run both types?

This is option 2 :)

pythongh-111704: Add `doctest_test_doctest_blocks` option to Sphinx

After working with make doctest for a while, here are my thoughts:

It does not allow to even run a single file (or maybe I cannot find the docs about it). This causes a lot of problems when debugging tests
It uses stuff that is fundamentally incompatible with -m doctest, like testsetup
It is hard to clean things up after usage (see gh-111726: Explicitly close database connections in sqlite3 doctests #111730), because you have to separate cleanup from the test itself (to not show it to user):

.. testcleanup::

   import os
   os.remove("tutorial.db")

If you want to skip some file, you have to:

Change conf.py and add a test logic there
Add :skipIf: TO ALL DOCTESTS in a file (see Doc/library/turtle.rst doctests are failing when tkinter is not present #111735)

I didn't like this tool at all.
It might be useful for small projects, but not for big ones.

Add :skipIf: TO ALL DOCTESTS in a file (see

Can't you just not add .. doctest to the REPL snippets in the documentation that you don't want to be treated as doctests? (Though I know we'd need to change that if we were to switch to a system where sphinx automatically treated all REPL snippets as doctests, regardless of whether they have the .. doctest directive above them.)

Can't you just not add .. doctest

Turns out that the answer is "no".
See #111735
It is a bit of a mess.

I propose adding more text files to doctests, we can start with important ones and move slowly to increase the coverage.

There are two options:

.. doctest:: directive in the file itself. This allows us to add only some parts of the file to the doctest. Like we do here:

cpython/Doc/library/typing.rst

Lines 1473 to 1481 in c176543

    
           .. doctest:: 
        
              >>> from typing import Annotated, get_type_hints 
        
              >>> def func(x: Annotated[int, "metadata"]) -> None: pass 
        
              ... 
        
              >>> get_type_hints(func) 
        
              {'x': <class 'int'>, 'return': <class 'NoneType'>} 
        
              >>> get_type_hints(func, include_extras=True) 
        
              {'x': typing.Annotated[int, 'metadata'], 'return': <class 'NoneType'>}

This way we can also create test setups and teardowns

Using def load_test in some related test file with passing filename to load. Like we do here:

cpython/Lib/test/test_enum.py

Lines 28 to 45 in c176543

    
           def load_tests(loader, tests, ignore): 
        
               tests.addTests(doctest.DocTestSuite(enum)) 
        
               lib_tests = os.path.join(REPO_ROOT, 'Doc/library/enum.rst') 
        
               if os.path.exists(lib_tests): 
        
                   tests.addTests(doctest.DocFileSuite( 
        
                           lib_tests, 
        
                           module_relative=False, 
        
                           optionflags=doctest.ELLIPSIS|doctest.NORMALIZE_WHITESPACE, 
        
                           )) 
        
               howto_tests = os.path.join(REPO_ROOT, 'Doc/howto/enum.rst') 
        
               if os.path.exists(howto_tests) and sys.float_repr_style == 'short': 
        
                   tests.addTests(doctest.DocFileSuite( 
        
                           howto_tests, 
        
                           module_relative=False, 
        
                           optionflags=doctest.ELLIPSIS|doctest.NORMALIZE_WHITESPACE, 
        
                           )) 
        
               return tests

Important things to keep in mind:

There are builds that do not have docs nearby
There are different float formatting options
There are different optional dependencies
There are different OSes

pythongh-111704: Add `stdtypes.rst` to doctests

Navigation Menu

Repository navigation

Decide what should we do with doctests in CI #111704

Bug report

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	.. doctest::

	>>> from typing import Annotated, get_type_hints
	>>> def func(x: Annotated[int, "metadata"]) -> None: pass
	...
	>>> get_type_hints(func)
	{'x': <class 'int'>, 'return': <class 'NoneType'>}
	>>> get_type_hints(func, include_extras=True)
	{'x': typing.Annotated[int, 'metadata'], 'return': <class 'NoneType'>}

	def load_tests(loader, tests, ignore):
	tests.addTests(doctest.DocTestSuite(enum))

	lib_tests = os.path.join(REPO_ROOT, 'Doc/library/enum.rst')
	if os.path.exists(lib_tests):
	tests.addTests(doctest.DocFileSuite(
	lib_tests,
	module_relative=False,
	optionflags=doctest.ELLIPSIS\|doctest.NORMALIZE_WHITESPACE,
	))
	howto_tests = os.path.join(REPO_ROOT, 'Doc/howto/enum.rst')
	if os.path.exists(howto_tests) and sys.float_repr_style == 'short':
	tests.addTests(doctest.DocFileSuite(
	howto_tests,
	module_relative=False,
	optionflags=doctest.ELLIPSIS\|doctest.NORMALIZE_WHITESPACE,
	))
	return tests

Decide what should we do with doctests in CI #111704

Description

Bug report

Linked PRs

Activity

sobolevn commented on Nov 3, 2023

AlexWaygood commented on Nov 3, 2023

AlexWaygood commented on Nov 3, 2023

hugovk commented on Nov 3, 2023

sobolevn commented on Nov 3, 2023

sobolevn commented on Nov 7, 2023

AlexWaygood commented on Nov 7, 2023

sobolevn commented on Nov 7, 2023

sobolevn commented on Jul 10, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions