Repository navigation
test_default_timeout() of test_threading.BarrierTests failure: BrokenBarrierError #56080
Description
| BPO | 11871 |
|---|---|
| Nosy | @db3l, @vstinner, @skrah, @native-api |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-04-18.21:59:03.506>
labels = ['3.8', '3.7', 'tests', '3.9']
title = 'test_default_timeout() of test_threading.BarrierTests failure: BrokenBarrierError'
updated_at = <Date 2022-02-04.10:23:01.964>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2022-02-04.10:23:01.964>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2011-04-18.21:59:03.506>
creator = 'vstinner'
dependencies = []
files = ['22761']
hgrepos = []
issue_num = 11871
keywords = ['patch', 'needs review']
message_count = 15.0
messages = ['133995', '134583', '141124', '141241', '141263', '141270', '141274', '200564', '340889', '340949', '344118', '347533', '361753', '394791', '412502']
nosy_count = 6.0
nosy_names = ['db3l', 'vstinner', 'skrah', 'neologix', 'python-dev', 'Ivan.Pozdeev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue11871'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']
Activity
vstinner commentedon Apr 18, 2011
While trying to reproduce issue bpo-11870 using "gdb -args ./python Lib/test/regrtest.py -F -v --timeout=600 test_threading", I had the following error on Linux:
----------------------
ERROR: test_default_timeout (test.test_threading.BarrierTests)
The error occured on:
neologix commentedon Apr 27, 2011
The most obvious explanation for that failure is that the barrier's timeout is too low.
If the last thread waits on the barrier more than 0.1s after the first thread, then you'll get a BrokenBarrierError.
A 0.1s delay is not that much, 100ms was the default quantum with Linux O(1) scheduler...
neologix commentedon Jul 25, 2011
The attached patch bumps the barrier's default timeout to 300ms: it should be more than enough (unless you got a really crappy scheduler, or a really heavily loaded machine), especially since this problem doesn't seem to occur often (AFAICT).
neologix commentedon Jul 27, 2011
Victor, can I commit it?
vstinner commentedon Jul 27, 2011
YES YOU CAN
python-dev commentedon Jul 27, 2011
New changeset aa9c0fdf2143 by Charles-François Natali in branch '3.2':
Issue bpo-11871: In test_threading.BarrierTests, bump the default barrier timeout
http://hg.python.org/cpython/rev/aa9c0fdf2143
New changeset e8da570d29a8 by Charles-François Natali in branch 'default':
Issue bpo-11871: In test_threading.BarrierTests, bump the default barrier timeout
http://hg.python.org/cpython/rev/e8da570d29a8
neologix commentedon Jul 27, 2011
:-)
skrah commentedon Oct 20, 2013
It looks like it happened again:
http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%203.x/builds/5223/steps/test/logs/stdio
======================================================================
ERROR: test_default_timeout (test.test_threading.BarrierTests)
----------------------------------------------------------------------
Ran 138 tests in 22.880s
vstinner commentedon Apr 26, 2019
The test still fails randomly:
https://buildbot.python.org/all/#/builders/3/builds/2469
db3l commentedon Apr 26, 2019
I should mention that a high level of test parallelism on the part of my worker might have be a contributing factor in this most recent case.
The worker was recently upgraded to a faster 4-core VM, but with limited I/O. In a test run the test processes invariably end up stuck on I/O heavy tests, idling the CPUs.
So I've been running the tests under -j8, as I found it the most effective combination of supporting tests stuck on I/O while keeping the CPUs busy, but it does mean that in some cases there's a lot pending on the CPUs, and depending on the exact test ordering in a run presumably some more sensitive tests could be impacted.
I have in fact seen an increase in random tests generating warnings (fail, then pass) than the worker had previously. I suspect the benefits of the extra parallelism on total test time (-j8 is about 20% faster than -j4) probably isn't valuable enough and will most likely be reducing it a bit.
native-api commentedon May 31, 2019
Got this issue today in AppVeyor's PR check: https://ci.appveyor.com/project/python/cpython/builds/24945165, so it's not local to David's worker.
(At rerun, the test succeeeded, so the check status was not affected.)
vstinner commentedon Jul 9, 2019
test_threading failed on AMD64 Windows8.1 Refleaks 3.x, the unittest errors are missing, but unraisable exceptions look similar:
https://buildbot.python.org/all/#/builders/80/builds/646
0:21:11 load avg: 7.30 [156/419/3] test_threading failed (2 min 56 sec) -- running: test_concurrent_futures (7 min 40 sec), test_email (1 min 34 sec)
beginning 6 repetitions
123456
.....Warning -- Unraisable exceptionWarning -- Unraisable exception
Exception ignored in thread started byException ignored in thread started by: : <function Bunch.__init__.<locals>.task at 0x000000C7CAC57AF0><function Bunch.__init__.<locals>.task at 0x000000C7CAC57AF0>
File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\test\lock_tests.py", line 939, in f
i = barrier.wait()
i = barrier.wait() File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\threading.py", line 620, in wait
File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\threading.py", line 620, in wait
self._wait(timeout)
File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\threading.py", line 660, in _wait
self._wait(timeout)
File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\threading.py", line 660, in _wait
raise BrokenBarrierError
threading. BrokenBarrierErrorraise BrokenBarrierError:
vstinner commentedon Feb 10, 2020
I didn't see this failure recently, I close the issue.
vstinner commentedon May 31, 2021
10 years ago, the issue is not solved. Recent failure on AMD64 Windows8.1 Refleaks PR buildbot:
https://buildbot.python.org/all/#/builders/470/builds/45
The issue started to make the buildbot fail since I added a threading.excepthook in libregrtest.
vstinner commentedon Feb 4, 2022
The race condition still exists in tests. Recent failure on AMD64 Windows8.1 Refleaks 3.x:
https://buildbot.python.org/all/#/builders/511/builds/249
File "D:\buildarea\3.x.ware-win81-release.refleak\build\Lib\test\lock_tests.py", line 1021, in f
File "D:\buildarea\3.x.ware-win81-release.refleak\build\Lib\test\lock_tests.py", line 1021, in f
^^^
File "D:\buildarea\3.x.ware-win81-release.refleak\build\Lib\test\lock_tests.py", line 1021, in f
i = barrier.wait()i = barrier.wait()
File "D:\buildarea\3.x.ware-win81-release.refleak\build\Lib\threading.py", line 683, in wait
^
File "D:\buildarea\3.x.ware-win81-release.refleak\build\Lib\threading.py", line 683, in wait
self._wait(timeout)
^^^^ ^self._wait(timeout)^
^ self._wait(timeout)
^^ ^ ^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
^^ File "D:\buildarea\3.x.ware-win81-release.refleak\build\Lib\threading.py", line 723, in _wait
^^^^^^^^^^
File "D:\buildarea\3.x.ware-win81-release.refleak\build\Lib\threading.py", line 723, in _wait
^^^^^ ^raise BrokenBarrierError^
vstinner commentedon Nov 3, 2022
Sadly, I don't have time to investigate/fix this old issue, I just close it.