pyparse.py, hyperparser.py, codecontext.py, colorizer.py, CPython main 993a0c65a7b.
Tracked in the umbrella issue python/cpython#154760 — one of 69 idlelib findings from a two-pass review with code-review-toolkit.
Each numbered heading below is one finding and has its own anchor, so the umbrella table links straight to it. The numbers are row numbers in that table, not identifiers — when referring to one of these, cite #154760 and the row, not the bare number.
Do not start work on a finding until a maintainer has cleared it in #154760. Every row there starts at Under analysis, which means not yet cleared.
AI disclaimer: these reports were drafted with Claude Code. Every cited line was re-read in the tree, every "Verified" block was produced by running the code, and the prior-art search was run against the full python/cpython tracker before publishing.
How to read a finding. Where is the line as of the commit above and will drift. Guarded twin is a place in idlelib that already handles the same situation correctly — it is both the evidence that this is a defect by the project's own standard and the shape of the fix. Prior art records what the tracker already says.
CONSIDER · confirmed by reading · shape one-predicate-two-implementations
Where: hyperparser.py:298, codecontext.py:22, pyparse.py:21, pyparse.py:44, colorizer.py:55
What happens. idlelib answers 'is this Python syntax' in five places — pyparse._synchre, pyparse._junkre, HyperParser's prefix scan, codecontext.BLOCKOPENERS and colorizer's regexes — with no shared definition. New syntax is taught to some and not others, and colorizer is consistently the one that gets updated. Confirmed drift, each with a twin inside the package: hyperparser.py:298 missing f/t (30); codecontext.py:22 missing match/case (47); _synchre matching def but not async def (58); _junkre \#\S (17). This is the systemic root, not a fourth instance.
Guarded twin. colorizer.py is the twin for all four: it is the recognizer the project actually maintains.
Suggested fix. Hoist the shared vocabulary (string prefixes, block-opening keywords, statement keywords) into one module the five consume, so a new keyword is added once.
Prior art — no matching report found. Searched the cached python/cpython tracker corpus (1626 topic-IDLE issues + 281 unlabelled idlelib issues + 573 PRs + 86 idle.rst items; 2503 unique) for this defect's terms and for its file/function names. No matching report.
CONSIDER · reproduced · shape recognizer-rejects-a-legal-variant-spelling
Where: pyparse.py:44-48
What happens. _junkre treats #foo (but not # foo) as junk, so _study2 drops the line and is_in_code() returns True inside the comment. With x = 1\n#os.pa and the cursor at end, Tab pops up 400+ os attributes inside a comment; one extra space and IDLE correctly declines. The same root cause makes parenmatch ring a spurious bell on ) at the end of a #...) line.
Guarded twin. None in idlelib — no sibling does this correctly.
Suggested fix. Treat # followed by any character as a comment in _junkre.
Prior art — no matching report found. Searched the cached python/cpython tracker corpus (1626 topic-IDLE issues + 281 unlabelled idlelib issues + 573 PRs + 86 idle.rst items; 2503 unique) for this defect's terms and for its file/function names. No matching report.
Note. test_pyparse.py:228-232 records the divergence with the comment 'A comment without a space is a special case'. The tests that 'prove' comments are not code both use '# ' with a space.
FIX · confirmed by reading · shape recognizer-rejects-a-legal-variant-spelling
Where: hyperparser.py:298
What happens. while pos > 0 and rawtext[pos - 1] in "rRbBuU" omits f/F (3.6) and t/T (3.14), so the backward scan stops inside the prefix. For t'...'.strip the completion machinery evals a truncated expression, gets a str, and offers str methods — but the real object is string.templatelib.Template, on which none of them exist.
Guarded twin. colorizer.py:50 — stringprefix = r"(?i:r|u|f|fr|rf|b|br|rb|t|rt|tr)?", complete. b was added to BOTH sites in 2013 (#16819); the colorizer got f in 2017 and t on 2025-10-09 (gh-139742). hyperparser.py:298 has not been touched since 2013.
Suggested fix. Add fFtT to the prefix character set.
Prior art — novel; the change that introduced it is identified. #139742 [CLOSED] 'IDLE - support t-string prefix in colorizer' (2025-10-09) is the fix that landed in colorizer.py only. b was added to BOTH sites in 2013 (#16819); colorizer got f in 2017 and t in 2025; hyperparser.py:298 has not been touched since 2013. This is the un-propagated half.
python/cpython#139742 python/cpython#139756
FIX · confirmed by reading · shape recognizer-rejects-a-legal-variant-spelling
Where: codecontext.py:22
What happens. BLOCKOPENERS is {'class','def','if','elif','else','while','for','try','except','finally','with','async'} — no 'match', no 'case'. Code Context therefore never shows the enclosing match statement, so a user scrolled deep inside a large match block sees the wrong enclosing context rather than none, which is the worse failure.
Guarded twin. colorizer got match/case in 2021, and autocomplete.py:15 and editor.py:1595 both have them. BLOCKOPENERS itself was extended for 'async' in 2018, so the practice of updating it exists.
Suggested fix. Add 'match' and 'case'.
Prior art — novel; the change that introduced it is identified. Both halves of the argument are on the tracker: #77745 [CLOSED] 'IDLE: add async to codecontext block openers' (2018) shows BLOCKOPENERS is maintained, and PR #25851 (bpo-44010, 2021) 'IDLE: colorize pattern-matching soft keywords' gave match/case to the colorizer and not to codecontext. #95352 [OPEN] 'More code_context improvements' has a live checklist and does not list this.
python/cpython#77745 python/cpython#25851
Note. An instance of the systemic root recorded as 0031.
58 — _synchre matches def but cannot match async def, making indent analysis quadratic in async modules
CONSIDER · confirmed by reading · shape recognizer-rejects-a-legal-variant-spelling
Where: pyparse.py:21
What happens. The _synchre alternation is anchored with ^[ \t]* and lists def but not async def, so in a module of async functions the parser finds no synchronisation point and rescans from further and further back. Agent-measured: 2.61 ms per call versus 0.00 ms on a 2000-function async module — and this runs per keystroke.
Guarded twin. codecontext.BLOCKOPENERS was extended for 'async' in 2018; the same word was never added here.
Suggested fix. Add async[ \t]+def to the alternation.
Prior art — an open issue covers the area, not this defect. #77061 [OPEN] 'IDLE: Fix and update and cleanup pyparse' (bpo-32880) already records that _synchre is missing 'if'. It does not mention 'async def', and carries no cost measurement. Best filed as a comment on #77061.
Note. An instance of the systemic root recorded as 0031. The timing figure is the agent's.
FIX · reproduced · shape recognizer-rejects-a-legal-variant-spelling
Where: pyparse.py:245
What happens. Reproduced differentially against this tree's own build. For the four-line sources x = (\n a + b\n)\ny = 1 and x = """\n a + b\n"""\ny = 1, _study1 correctly yields goodlines [0, 3, 4] — lines 1 and 2 are not statement starts. For the PEP 701 equivalent x = f"{\n a + b\n}"\ny = 1 it yields [0, 1, 2, 3, 4]: the f-string body is treated as two new statements. Continuation lines inside a legal multi-line f-string therefore indent to column 0, and every consumer of goodlines (auto-indent, Format Paragraph, Code Context) reasons from the wrong statement boundaries.
Verified:
# _study1 goodlines: which lines it believes start a statement.
plain multiline paren x = (\n a + b\n)\ny = 1 -> [0, 3, 4] correct
triple-quoted string x = """\n a + b\n"""\ny = 1 -> [0, 3, 4] correct
PEP 701 multiline f-string x = f"{\n a + b\n}"\ny = 1 -> [0, 1, 2, 3, 4] WRONG
Guarded twin. The triple-quoted-string case in the same scanner, which is handled correctly — shown by the differential above.
Suggested fix. Give the scanner an f-string brace state, as the tokenizer has had since 3.12.
Prior art — an open issue covers the area, not this defect. Same open pyparse cleanup issue (#77061). f-strings are not mentioned in it; PEP 701 postdates it by five years. The differential against the triple-quoted case is new.
FIX · confirmed by reading · shape mirrored-direction-handles-fewer-cases
Where: hyperparser.py:137-140
What happens. The backward loop tests self.rawtext[self.bracketing[before][0]] not in openers; the forward loop at :138-140 tests only the nesting level. Reproduced by the agent: x = (a] reports ( as matched with ]. pyparse.py:120 deliberately erases bracket identity, and only one of the two scans compensates for that.
Guarded twin. The backward scan at hyperparser.py:130-132, which does check the type.
Suggested fix. Check the closing bracket's type in the forward scan too.
Prior art — an open issue covers the area, not this defect. #124451 [OPEN] 'IDLE doesn't know where to start parsing' reports the same user-visible symptom (a wrong paren highlighted) from a different cause — the parser not knowing where the prompt begins. The type-asymmetry between the backward and forward scans is a second, independent cause of that symptom.