Repository navigation
IDLE: ColorDelegatorTest test_incremental_editing failures #88448
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2021-06-03.03:17:41.798>
created_at = <Date 2021-06-01.20:38:25.128>
labels = ['expert-IDLE', 'type-bug', 'tests', '3.11']
title = 'IDLE: ColorDelegatorTest test_incremental_editing failures'
updated_at = <Date 2021-06-03.03:17:41.798>
user = 'https://github.com/terryjreedy'bugs.python.org fields:
activity = <Date 2021-06-03.03:17:41.798>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2021-06-03.03:17:41.798>
closer = 'terry.reedy'
components = ['IDLE', 'Tests']
creation = <Date 2021-06-01.20:38:25.128>
creator = 'terry.reedy'
dependencies = []
files = ['50084']
hgrepos = []
issue_num = 44282
keywords = ['patch']
message_count = 14.0
messages = ['394870', '394914', '394918', '394919', '394921', '394925', '394936', '394937', '394939', '394941', '394942', '394959', '394962', '394973']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'vstinner', 'taleinat', 'miss-islington']
pr_nums = ['26404', '26491', '26499']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44282'
versions = ['Python 3.11']
Activity
terryjreedy commentedon Jun 1, 2021
idlelib.idle_test.test_colordelagator.ColorDelegatorTest.test_incremental_editing has failed on at least on the following machines.
Pip32 CI Pipelines Win32
Gen3x https://buildbot.python.org/all/#/builders/464 x86 Gentoo Installed with X 3.x
GenPR https://buildbot.python.org/all/#/builders/465 x86 Gentoo Installed with X PR
Failures seen. Since tests are sequential without subtesting, failure means successors are not tested.
line keyword start want got
569 ('BUILTIN', '1.0'), ('1.0', '1.3') () Gen3x, x3; GenPR
573 ('BUILTIN', '1.0'), ('1.0', '1.3') () Gen3x, x5
586 ('BUILTIN', '1.0'), ('1.0', '1.3') () GenPR
597 ('KEYWORD', '1.0'), () ('1.0', '1.1') Pip32, x2
As far as I can tell, looking at Windows buildbots, the 1 failure on Win32 has not been repeated. If there have been any on CI machines, I have not been informed.
A few years ago, I believe Gentoo was the only *nix buildbot running tkinter and IDLE tests. It has a problem with a builtin at the beginning of the line about 10-20% of test runs. This are the majority of its failures in the last week. Perhaps there is a timing issue we can fix.
vstinner commentedon Jun 2, 2021
Buildbots are very unstable these days. I propose to revert the change to give more time to IDLE developers to investigate the issue, and to be able to identify other regressions on buildbots.
https://pythondev.readthedocs.io/ci.html#revert-on-fail
I created #26487 to revert
taleinat commentedon Jun 2, 2021
Please note that the referenced PR is not the correct one.
I'll add a link to the right one shortly.
taleinat commentedon Jun 2, 2021
The latest PR where changes were made to IDLE's colorizer and its tests, including specifically adding the problematic test, is #70038.
taleinat commentedon Jun 2, 2021
I'm investigating this now. If nothing obvious comes up, I'll revert that PR.
taleinat commentedon Jun 2, 2021
On Ubuntu 20.04, I've managed to occasionally get this issue to reproduce locally by running the test many times in parallel, repeatedly:
parallel -n8 -N0 './python -m test -ugui test_idle -m test_colorizer' ::: $(seq 20)
This appears to be an issue with our new method of running tests with a Tk mainloop in the background, using the @run_in_tk_mainloop decorator.
Investigating further.
terryjreedy commentedon Jun 2, 2021
The confusion is my fault: PR-26404 is where I first saw this failure, for the previously added tests, not where the test was added. I unlinked the erroneous message.
Running 12 duplicate tests in parallel on Windows on a 6 core (12 CPU) machine with
f:\dev\3x>python -m test -j12 -ugui -m test_incremental_editing test_idle test_idle test_idle test_idle test_idle test_idle test_idle test_idle test_idle test_idle test_idle test_idle
resulted in nearly all failing. Half or less failed with -j6, usually one with -j3, none without -j. Are the processess failing because of interaction with each other? Or from be switched with other processes?
Back with -j12: increasing the after delay in run_in_tk_mainloop from 1 to 5 cut the failures down to 0-4 in 12. Increasing further to 8 resulting in no failures. Note that human keystrokes are at least, say, 50 milliseconds (1/20 second) apart.
I am reluctant to add a delay that is mostly not needed, so i tried the uploaded 'adaptive' code to only add a long delay if needed. Uploaded. But when a tested failed once, it failed again even with the delay, and contrary to the results above.
terryjreedy commentedon Jun 2, 2021
The other thing I thought is parameterizing the decorator to pass in a delay only for this test, but that require another layer of wrapping, which I cannot do right now.
So I think we should just add skiptest for now. In fact, we only need to run it when editing colordelagator.
taleinat commentedon Jun 2, 2021
Yes, running test_idle in parallel with itself usually causes some failures due to UI focus being stolen from one instance by another. That's why I just ran this test rather than the whole test suite.
The delay needs to be placed carefully. Changing root.after(1, ...) to use 50 rather than 1 (these are milliseconds) seems to resolve the issue entirely on my machine, at the cost of this single test taking 0.4 seconds to run rather than about 0.045 seconds.
I can do that. Though I fear this may happen for other such tests as well... Worth a shot I guess.
taleinat commentedon Jun 2, 2021
On my machine it running the test many times with parallelization no longer causes any failures with this change. Also, running other tests which us the @run_in_tk_mainloop decorator, with the same 1ms delay as before, doesn't cause any such failures. PR forthcoming.
taleinat commentedon Jun 2, 2021
See PR #70678.
vstinner commentedon Jun 2, 2021
New changeset adef445 by Tal Einat in branch 'main':
bpo-44282: Fix occasional test_incremental_editing failures on buildbots (GH-26491)
adef445
vstinner commentedon Jun 3, 2021
New changeset 9c89d62 by Miss Islington (bot) in branch '3.10':
bpo-44282: Fix occasional test_incremental_editing failures on buildbots (GH-26491) (GH-26499)
9c89d62
vstinner commentedon Jun 3, 2021
The 12 latest build on https://buildbot.python.org/all/#/builders/464 are green (success). Can this issue be closed now? We can reopen it later if the issue strikes back.