pytest --doctest-modules crashes on file-level __test__ = False

Bug report

Bug description:

First opened this issue at pytest-dev but was directed here. pytest-dev/pytest#11730

A file level __test__ = False succesfully stops pytest from collecting any test from that file, however when running pytest --doctest- modules on the same file, this error appears:

/sw/Python/Ubuntu-22.04/3.10.5/lib/python3.10/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/sw/Python/Ubuntu-22.04/3.10.5/lib/python3.10/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/sw/Python/Ubuntu-22.04/3.10.5/lib/python3.10/site-packages/_pytest/doctest.py:587: in collect
    for test in finder.find(module, module.__name__):
/sw/Python/Ubuntu-22.04/3.10.5/lib/python3.10/doctest.py:940: in find
    self._find(tests, obj, name, module, source_lines, globs, {})
/sw/Python/Ubuntu-22.04/3.10.5/lib/python3.10/site-packages/_pytest/doctest.py:536: in _find
    super()._find(  # type:ignore[misc]
/sw/Python/Ubuntu-22.04/3.10.5/lib/python3.10/doctest.py:1018: in _find
    for valname, val in getattr(obj, '__test__', {}).items():
E   AttributeError: 'bool' object has no attribute 'items'

It's enoough to have a file with only this content to see the behavior:

__test__ = False

Version:

Python 3.10.5
pytest 7.4.3

CPython versions tested on:

3.10

Operating systems tested on:

Linux

Linked PRs

The following is a self-contained reproducer for the exception above.

__test__ = False
import doctest
doctest.testmod()

doctest doc says, with 'M' a module, "If M.test exists and is truthy, it must be a dict". It says nothing about 'falsey' values, which to me, implies that they will be ignored. AFAIK, the _find code has never checked the boolean value of __test__. In 2004, why would anyone not make it a dict or mapping?

I think the doc should have always said that __test__ should be a mapping, ~~but too late now~~.
<EDIT: strikeout wrong conclusion. I agree with just changing the doc.>
Instead, add a new unittest that currently fails. (This may or may not be easy -- a beginner should try this first.) Next replace

            for valname, val in getattr(obj, '__test__', {}).items():

with

            if (tests = getattr(obj, '__test__', {})):
                for valname, val in tests.items():

and indent the suite under the 'for' statement.

In one place the documentation says:

If ``M.__test__`` exists and is truthy, it must be a dict, and each
entry maps a (string) name to a function object, class object, or string.
Function and class object docstrings found from ``M.__test__`` are searched, and
strings are treated as if they were docstrings.

So it can be a dict or a false value.

In other place it says:

   Also test examples reachable from dict ``m.__test__``, if it exists and is not
   ``None``.  ``m.__test__`` maps names (strings) to functions, classes and
   strings; function and class docstrings are searched for examples; strings are
   searched directly, as if they were docstrings.

So it can be a dict or None.

A small contradiction is that the first description implies any false value, wile the second one implies only None.

But the code implies that it should always be a dict.

            for valname, val in (getattr(obj, '__test__', None) or {}).items():

The documentation matches the behavior of the old, pre-2.4, code, which was changed in 8485b56. The last 19 years the documentation does not match implementation.

pythongh-113360: Fix `doctest` failure when `__test__` is set to fals…

pythongh-113360: Fix the documentation of module's attribute __test__

gh-113360: Fix the documentation of module's attribute __test__ (GH-1…

pythongh-113360: Fix the documentation of module's attribute __test__ (…

[3.11] gh-113360: Fix the documentation of module's attribute __test__ (

[3.12] gh-113360: Fix the documentation of module's attribute __test__ (

The documentation was updated. __test__ = False is not valid since Python 2.4.

pythongh-113360: Fix the documentation of module's attribute __test__ (…

Where can the updated documentation be read and which values are now valid?

It is in the doctest documentation: "If M.__test__ exists, it must be a dict, ..."

It is even in index. Although the index entry was incorrect, and the reference had a weird anchor. #136674 has been created to fix this.

Also, is it actually clear that __test__ can be used to ignore/skip tests rather than just add tests to the discovery process?

AFAIK it is only used to add tests in doctest.

I see, what has been confusing me is that pytest uses the same attribute __test__ and can hide tests by setting it to empty dict.

with test_foo.py:

def test_a():
    """">>>5
    1
    """

the command pytest --doctest-modules would have 2 tests:

test_tmp.py::test_tmp.test_a          # this is the doctest
test_tmp.py::test_a

But if __test__ = {} is added, the same command only finds the doctest.

Adding the below lines and running it as python3 test_foo.py

import doctest
doctest.testmod()

indeed always finds just 1 test, the doctest, and is unaffected by an empty __test__ attribute.

IMHO the doctest module doesn't have established ownership of the dunder test attribute

If it was dunder doctest_test there would be no room to argue

However nose and pytest and others use dunder test being falsy as indicator not to collect something

@tim-one

All dunder names are reserved for Python interpreter and the stdlib.

Maybe nose used __test__ before Python 2.4 (and pytest inherited this from nose). In Python 2.1-2.3 it must only be a dict if it has a true boolean value. It was broken in 2.4 (8485b56).

Maybe @tim-one will shed some light on the origin of __test__. Did it originate in doctest or was borrowed from nose (if it existed at that time)? Was the change in handling false __test__ values intentional (it removed also a large part of the module docstring which described this behavior)?

In that case it seems necessary to either doa pep to elevate the dunder test false value from deviant use to recognized pattern or to deprecate it and replace it

I vaguely recall it coming from nose somewhere around 2005 and pytest adopting it from compatibility code

Ill verify if memory serves me right

In that case it seems necessary to either doa pep to elevate the dunder test false value from deviant use to recognized pattern or to deprecate it and replace it

I vaguely recall it coming from nose somewhere around 2005 and pytest adopting it from compatibility code

Ill verify if memory serves me right

Pytest took it to be compatible with nose (according to docs). Though whether it first came from nose or doctest, I don't know.

Any updates on this issue?

Its a wontfix

In that case it seems necessary to either doa pep to elevate the dunder test false value from deviant use to recognized pattern or to deprecate it and replace it

In this post it seemed like you thought something needed to be done, now not? What changed?
Or does "wontfix" just mean not big enough issue to care about? (Is it cpython only related)

The ideal outcome would be for there to be 1 definition for how __test__ is supposed to be used so the developers can know what to type and that all tools can be made to agree.

If cpython is going to keep using __test__ as is, I suppose at least a change in pytest should be done.

A change in pytest is necessary

Support for __test__ in the doctest module was from the beginning, since adding the module in 2001 (eccd02a).

Support for __test__ in the nose module was added in 2007 (nose-devs/nose@08f788b).

In nose2 it was added in 2010 (nose-devs/nose2@a2b93af) and in 2014 (nose-devs/nose2@317b93a).

In pytest it was added in 2014 (pytest-dev/pytest@494be73).

All uses outside of doctest are not compatible with the use in doctest. In doctest, __test__ specifies additional doctests. Setting it to an empty dict does not disable testing, and setting it to anything that is not a dict causes error since 2004 (8485b56). The documentation was outdated, it has been fixed.

So, this is not a Python issue. All other testing packages are wrong. They should use other way to omit all tests in the module.

Navigation Menu

Repository navigation

pytest --doctest-modules crashes on file-level test = False #113360

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

pytest --doctest-modules crashes on file-level __test__ = False #113360

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

terryjreedy commented on Dec 21, 2023

serhiy-storchaka commented on Dec 21, 2023

serhiy-storchaka commented on Jan 6, 2024

5 remaining items

serhiy-storchaka commented on Jul 15, 2025

serhiy-storchaka commented on Jul 15, 2025

kaddkaka commented on Jul 15, 2025

RonnyPfannschmidt commented on Jul 15, 2025

serhiy-storchaka commented on Jul 15, 2025

RonnyPfannschmidt commented on Jul 15, 2025

kaddkaka commented on Jul 15, 2025

kaddkaka commented on Sep 1, 2025

RonnyPfannschmidt commented on Sep 1, 2025

kaddkaka commented on Sep 1, 2025

RonnyPfannschmidt commented on Sep 1, 2025

serhiy-storchaka commented on Sep 1, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

pytest --doctest-modules crashes on file-level test = False #113360