Skip to content

IDLE fails to detect corresponding opening parenthesis #85560

Closed
Closed
IDLE fails to detect corresponding opening parenthesis#85560
Assignees
terryjreedy
Labels
topic-IDLEtype-bugAn unexpected behavior, bug, or error

Description

@AlexeyBurdin
AlexeyBurdin
mannequin
opened on Jul 24, 2020
BPO 41388
Nosy @terryjreedy, @taleinat, @lrjball

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2020-07-24.19:10:41.733>
labels = ['expert-IDLE', 'type-bug', '3.8', '3.9', '3.10']
title = 'IDLE fails to detect corresponding opening parenthesis'
updated_at = <Date 2020-07-26.23:35:02.207>
user = 'https://bugs.python.org/AlexeyBurdin'

bugs.python.org fields:

activity = <Date 2020-07-26.23:35:02.207>
actor = 'Lewis Ball'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2020-07-24.19:10:41.733>
creator = 'Alexey Burdin'
dependencies = []
files = []
hgrepos = []
issue_num = 41388
keywords = []
message_count = 9.0
messages = ['374205', '374218', '374224', '374225', '374226', '374274', '374298', '374301', '374350']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'taleinat', 'Alexey Burdin', 'Lewis Ball']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41388'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

Activity

  1. AlexeyBurdin commented on Jul 24, 2020

    @AlexeyBurdin
    AlexeyBurdinmannequin
    on Jul 24, 2020
    MannequinAuthor
    answers_field_order=sorted(
    set(j for i in data['items'] for j in i),
    key=cmp_to_key(lambda x,y:(
        -1 if (x,y) in answer_order 
        else (0 if x==y else 1)))
        )

    when the cursor is placed in line 5 col 31 (between ) and )) ) hitting Ctrl-0 (Show surrounding parens) produces an error sound, though there's no error, the script works fine.

    $ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
$ uname -a
Linux odd-one 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ python3.8
Python 3.8.0 (default, Oct 28 2019, 16:14:01) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec('idlelib.pyshell')
ModuleSpec(name='idlelib.pyshell', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f1ea8ee14c0>, origin='/usr/lib/python3.8/idlelib/pyshell.py')
>>> exit()
$ dpkg-query -S /usr/lib/python3.8/idlelib/pyshell.py
idle-python3.8: /usr/lib/python3.8/idlelib/pyshell.py
$ dpkg -l | grep idle-python3\.8
ii  idle-python3.8                             3.8.0-3~18.04                                    all          IDE for Python (v3.8) using Tkinter
  2. AlexeyBurdin
    added
    3.8 (EOL)end of life
    on Jul 24, 2020
  3. AlexeyBurdin
    assigned
    terryjreedy
    on Jul 24, 2020
  4. AlexeyBurdin
    added
    topic-IDLE
    type-bugAn unexpected behavior, bug, or error
    3.8 (EOL)end of life
    on Jul 24, 2020
  5. AlexeyBurdin
    assigned
    terryjreedy
    on Jul 24, 2020
  6. AlexeyBurdin
    added
    topic-IDLE
    type-bugAn unexpected behavior, bug, or error
    on Jul 24, 2020

  7. lrjball commented on Jul 25, 2020

    @lrjball
    lrjballmannequin
    on Jul 25, 2020
    Mannequin

    A minimal example of the same issue seems to be:

    (1 if x
 else 0)

    In this case the parentheses matching also fails. It only ever seems to fail when else is the first word on a newline inside brackets, and adding the line break elsewhere in the above statement will not cause the matching to fail.

    I imagine it stops looking for the matching parentheses when it gets to the else statement, as an else when part of an if/else cannot be inside parentheses. Obviously the exception to this is the ternary operator.

    Bizarrely, it only seems to fail the first time it encounters this issue. Any subsequent code with the same pattern seems to be matched fine. So in the following example, the second brackets get matched without an issue:

    (1 if x
 else 0)

(1 if x
 else 0)

    Hopefully someone more familiar with the source code can shed some light on this!

  8. terryjreedy commented on Jul 25, 2020

    @terryjreedy
    terryjreedy
    on Jul 25, 2020
    Member

    (While I wrote the following, Lewis Ball posted some of the same conclusions.)

    1. I verified the failure with 3.8.5 and current master.

    2. I reduced the failing example to a minimum of 7 chars on 2 lines.
      (
      else)

    1a. Deleting '\n' or any letter of 'else' fixes the problem.
    1b. Adding 'else' before '(' fixes the problem.
    1a is true for the initial 'else' on line 4 of the original example.
    1b is true for 'else' on a new line before the first.

    1. I tried multiple variations
      2a. Matching also fails when typing ')'.
      2b. Replacing '()' with '[]' or '{}' makes no difference.

    2c. Replacing 'else' with 'elif', 'while', 'def', or 'class' makes no difference. This include the addition for 1b above. Replacing 'else' with 'if', 'for', or 'with' fixes the bug.

    Bingo!!! Matching uses idlelib.hyperparser, which uses idlelib.pyparse, which has regex _synchre, which finds "what looks like the start of a popular statement". It matches 'elif', etcetera, but not 'if', etcetera. I noticed previously that these 'popular' line beginners were missing and have planned to add them.* But we need to make matched words not disable fence matching.

    • Keyword 'with' was likely added to python after this re was last revised. Keywords 'if' and 'for' might have been removed when comprehensions were added. If so, 'else' should have been removed when conditional expressions were added and 'yield' when yield expressions were added. These latter two are the only keywords matched by _synchre that can properly appear in the middle of a statement but at the beginning of a continuation line.

    2d. In the original and minimal examples, moving the offending 'else' to the end of the previous line fixes the problem. So I believe that 'else' at the beginning of the line is seen marking the beginning of the statememt, and matching does not look back further. If 'else' and 'yield' were left in _synchre and 'if' and 'for' were added back, the matcher would have to determine whether the line is a continuation or not. (And I believe that not having 'if' and 'for' degrades the use of pyparse at least for other uses.)

    2e. The minimal failing example works in the shell. Why are _synchre keywords not poisonous here? Is it just because the current statement is known to begin just after the '>>> ' prompt, so that else beginning the second line is somehow not seen as beginning a new statement? We need to check if and how fence matching in shell is different other than the presence of the marker where code input begins.

    Alexey: in (#65955 (comment)) I claimed that there is no issue with typing closers or ^0 on the same line just before the closer. Thank you for reporting this counter-example. I imagine that you are not the first IDLE user to write a conditional expression on 2 or more lines with else at the beginning of a line. Such expressions always need a closer.

  9. terryjreedy
    added
    3.9 (EOL)end of life
    3.10 (EOL)end of life
    on Jul 25, 2020

  10. terryjreedy commented on Jul 25, 2020

    @terryjreedy
    terryjreedy
    on Jul 25, 2020
    Member

    Without rereading hyperparser and pyparse, I don't understand why a previous _synchre match prevents subsequent failures. It does suggest that when matching parentheses, a fix would be to temporarily prepend 'else\n' to the text, or otherwise simulating the effect of that.

    But the previous 'else' might only work before the 'current' statement. And we should determine whether any other uses of these modules have related bugs. Also, removing 'else' from the _synchre regex did not fix the issue, so there is something else I am missing.

  11. 4 remaining items

  12. lrjball commented on Jul 26, 2020

    @lrjball
    lrjballmannequin
    on Jul 26, 2020
    Mannequin

    Okay that makes sense. Removing things from _synchre would slow down the matching slightly, although the matching still seems plenty fast enough when done inside an if clause at the moment, so I'm not sure how noticeable the removal of `else` would be. One thing could be to search for `else:` instead of just `else` in _synchre, as only the former indicates the start of a new statement (although I guess it would actually have to check for `else[\s\\]*:` or something).

    Like you say though, if there are other more pressing issues with the matching then maybe it is worth fixing them all at the same time. Happy to help with that if needed.

  13. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  14. terryjreedy
    added this to IDLE Issueson Jul 23, 2022
  15. terryjreedy
    removed
    3.10 (EOL)end of life
    3.9 (EOL)end of life
    3.8 (EOL)end of life
    on Aug 6, 2024

  16. terryjreedy commented on Sep 24, 2024

    @terryjreedy
    terryjreedy
    on Sep 24, 2024
    Member

    I reproduced the issue with 3.13.0rc2 on macOS and main on Windows. (\nelse) (with auto indent added) must be the first pair typed since opening the window. Previous text loaded on opening a existing file does not count.

  17. serhiy-storchaka
    added a commit that references this issue on Sep 17, 2026

    pythongh-85560: Fix IDLE paren matching with "else" or "yield" at the…

    12f8dd8

  18. serhiy-storchaka commented on Sep 17, 2026

    @serhiy-storchaka
    serhiy-storchaka
    on Sep 17, 2026
    Member

    #157693 makes _synchre match else only when it is followed by a colon, as in an else statement, drops yield, which cannot be told from a yield expression at the start of a continuation line, and adds with, del, global, nonlocal, pass and finally, which cannot start a continuation line.

  19. terryjreedy
    added 3 commits that reference this issue on Sep 22, 2026

    gh-85560: Fix IDLE paren matching with "else" or "yield" at the start…

    2073473

    [3.14] gh-85560: Fix IDLE paren matching with "else" or "yield" at th…

    28f5294

    [3.13] gh-85560: Fix IDLE paren matching with "else" or "yield" at th…

    8cceb10
  20. terryjreedy
    closed this as completedon Sep 22, 2026
  21. github-project-automation
    moved this to Done in IDLE Issueson Sep 22, 2026
  22. terryjreedy
    added a commit that references this issue on Oct 9, 2026

    [3.15] gh-85560: Fix IDLE paren matching with "else" or "yield" at th…

    439a440
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

topic-IDLEtype-bugAn unexpected behavior, bug, or error

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions