IDLE test-suite integrity

idle_test/, CPython main 993a0c65a7b.

Tracked in the umbrella issue python/cpython#154760 — one of 69 idlelib findings from a two-pass review with code-review-toolkit.

Each numbered heading below is one finding and has its own anchor, so the umbrella table links straight to it. The numbers are row numbers in that table, not identifiers — when referring to one of these, cite #154760 and the row, not the bare number.

Do not start work on a finding until a maintainer has cleared it in #154760. Every row there starts at Under analysis, which means not yet cleared.

AI disclaimer: these reports were drafted with Claude Code. Every cited line was re-read in the tree, every "Verified" block was produced by running the code, and the prior-art search was run against the full python/cpython tracker before publishing.

How to read a finding. Where is the line as of the commit above and will drift. Guarded twin is a place in idlelib that already handles the same situation correctly — it is both the evidence that this is a defect by the project's own standard and the shape of the fix. Prior art records what the tracker already says.

26 — Five tests in the suite cannot fail

CONSIDER · confirmed by reading · shape test-cannot-fail

Where: idle_test/test_autocomplete.py:241, idle_test/test_autocomplete.py:242, idle_test/test_editor.py:236, idle_test/test_configdialog.py:55, idle_test/template.py:25

What happens. assertTrue(all(filter(lambda x: x.startswith('_'), s))) is true whether s has none or all such names — filter already dropped the rest, so the predicate is never tested; per the twin on the next line the intent was assertFalse(any(...)). RMenuTest.test_rclick and test_deactivate_current_config have pass bodies. Each reports as coverage while verifying nothing.

Guarded twin. The correct form is all(pred(x) for x in xs).

Suggested fix. Rewrite the filter assertions as generator expressions; write or skip the empty tests.

Prior art — no matching report found. Searched the cached python/cpython tracker corpus (1626 topic-IDLE issues + 281 unlabelled idlelib issues + 573 PRs + 86 idle.rst items; 2503 unique) for this defect's terms and for its file/function names. No matching report.

Note. Found independently by an agent and by the scanner's test-cannot-fail check — the two methods converged on the same five sites.

71 — The shared test stubs cannot express the conditions the guards under test exist for

CONSIDER · confirmed by reading · shape assertion-against-a-stub-that-cannot-fail

Where: idle_test/mock_idle.py:48, idle_test/mock_tk.py

What happens. mock_idle.Editor.get_selection_indices returns ('1.0', 'end') unconditionally, so the no-selection branch of all five production consumers is unreachable under it. The same shape hides finding 25: both Idb.user_line (guarded) and Idb.user_exception (unguarded) are tested with self.gui = Mock(), and a Mock never raises TclError, so the two tests pass identically whether or not the guard is there while line coverage reads 100% on both. Of 613 test methods, 43 assert only against a mock recorder and 11 assert nothing at all; 70% live in files importing mock_tk/mock_idle, and mock_idle.py itself — which defines the Func recorder used by 315 test methods — has no test of its own.

Guarded twin. The GUI-requiring tests, which use a real Tk widget and can therefore express TclError. Headless runs 296 of 623 tests, so a developer running the suite bare exercises less than half of it.

Suggested fix. Give the stubs a way to produce the failure — Mock(side_effect=TclError), a no-selection return — and test mock_idle itself.

72 — A test class freezes text.index to a constant, leaving seven mutations of live code undetected

CONSIDER · confirmed by reading · shape assertion-against-a-stub-that-cannot-fail

Where: idle_test/test_searchengine.py:279, searchengine.py:182, searchengine.py:185, undo.py:218, undo.py:279, sidebar.py:193

What happens. cls.text.index = lambda index: '4.0' freezes every index expression to one constant, so nothing the backward search computes from an index is constrained. Mutation testing against the full 623-test GUI suite: searchengine.py:182 'end-1c'->'end' SURVIVES, searchengine.py:185 col=len(chars)-1 -> 0 SURVIVES, both undo.py 'end-1c' clamps dropped SURVIVES, all four sidebar.py drag/click mutations SURVIVE, and mock_tk.mark_set made functional SURVIVES. The forward/backward asymmetry is the sharp one — search_forward's wrap mutation DIES with 5 failures while search_backward's is unconstrained, in the same file and with the same asymmetry as finding 5.

Verified:

# Mutation testing against the full 623-test GUI suite:
searchengine.py:182  'end-1c' -> 'end'          SURVIVES
searchengine.py:185  col = len(chars)-1 -> 0    SURVIVES
undo.py:218,279      drop both 'end-1c' clamps  SURVIVES
sidebar.py:193-202   four drag/click mutations  ALL SURVIVE
mock_tk.mark_set     made functional            SURVIVES
searchengine.py:161  forward-wrap restart line  dies (5 failures)   <- the one that IS covered

Guarded twin. searchengine.py:161, whose forward-wrap restart line is genuinely constrained — the same suite kills that mutant.

Suggested fix. Use a real Tk text or a stub whose index() actually computes; then re-run the mutants.

Note. Mutation results are the agent's, run in the live checkout and restored; the tree was verified clean by SHA afterwards.

73 — Four test files exit 0 even when their tests fail

CONSIDER · confirmed by reading · shape test-cannot-fail

Where: idle_test/test_undo.py, idle_test/test_text.py, idle_test/test_pathbrowser.py, idle_test/test_query.py

What happens. These four call unittest.main(verbosity=2, exit=False), so running the file directly returns 0 regardless of the result — any wrapper that shells out to python test_undo.py and checks the exit status sees a pass. 51 sibling files use the plain unittest.main(verbosity=2), which exits non-zero correctly.

Guarded twin. The 51 files using unittest.main(verbosity=2).

Suggested fix. Drop exit=False.

Note. CORRECTION to reports/idlelib_v4/FINDINGS.md, which recorded 'exit=2 at 5 sites vs exit=False at 42' and claimed 42 files return 0 on failure. Measured counts on this commit: 51 plain, 5 with exit=2, 4 with exit=False. exit=2 is truthy and behaves as exit=True, so it is a harmless oddity, not a defect. Only the four exit=False files are affected.

devdanzin/idlelib-tests.md

Select an option

No results found