Repository navigation
Get to the bottom of intermittent failures on Travis #6586
Description
At this stage these are mainly my own notes on errors we have been seeing on Travis to see if I can find a pattern that can give a clue what the issue is:
From #6585 python2.7
======================================================================
FAIL: matplotlib.tests.test_backend_pdf.test_hatching_legend.test
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/matplotlib-2.0.0b1+40.g2855857-py2.7-linux-x86_64.egg/matplotlib/testing/decorators.py", line 55, in failer
result = f(*args, **kwargs)
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/matplotlib-2.0.0b1+40.g2855857-py2.7-linux-x86_64.egg/matplotlib/testing/decorators.py", line 251, in do_test
self._tol, in_decorator=True)
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/matplotlib-2.0.0b1+40.g2855857-py2.7-linux-x86_64.egg/matplotlib/testing/compare.py", line 337, in compare_images
rms = calculate_rms(expectedImage, actualImage)
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/matplotlib-2.0.0b1+40.g2855857-py2.7-linux-x86_64.egg/matplotlib/testing/compare.py", line 248, in calculate_rms
"actual size {1}".format(expectedImage.shape, actualImage.shape))
ImageComparisonFailure: image sizes do not match expected size: (144, 72, 3) actual size (346, 461, 3)
======================================================================
FAIL: matplotlib.tests.test_mathtext.test_mathfont_dejavuserif_12.test
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/matplotlib-2.0.0b1+40.g2855857-py2.7-linux-x86_64.egg/matplotlib/testing/decorators.py", line 55, in failer
result = f(*args, **kwargs)
File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.7/site-packages/matplotlib-2.0.0b1+40.g2855857-py2.7-linux-x86_64.egg/matplotlib/testing/decorators.py", line 261, in do_test
'(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: /home/travis/build/matplotlib/matplotlib/result_images/test_mathtext/mathfont_dejavuserif_12.png vs. /home/travis/build/matplotlib/matplotlib/result_images/test_mathtext/mathfont_dejavuserif_12-expected.png (RMS 48.746)
----------------------------------------------------------------------
Metadata
Metadata
Assignees
Labels
No labels
Activity
jenshnielsen commentedon Jun 15, 2016
2.x on python 2.7
tacaswell commentedon Jun 19, 2016
From #6611 on py3.5
tacaswell commentedon Jun 20, 2016
LPy on v2.x branch
tacaswell commentedon Jun 21, 2016
OSX
jenshnielsen commentedon Jul 10, 2016
The backend pdf errors of the form
image sizes do not match expected size: (144, 72, 3) actual size (346, 461, 3)are hopefully fixed by #6713jenshnielsen commentedon Jul 11, 2016
Following the merge of #6713 locally:
jenshnielsen commentedon Jul 11, 2016
2.x python 3.4
jenshnielsen commentedon Jul 11, 2016
Travis python 3.5 without multiprocessing
jenshnielsen commentedon Jul 11, 2016
Locally python 3.5 4 processes
jenshnielsen commentedon Jul 11, 2016
In the last example
test_given_colors_levels_and_extendsShould be:
but looks like:
This seems to be the content from
test_contour_datetime_axiswhich is leaking when that test fails:jenshnielsen commentedon Jul 12, 2016
On travis 2.x
python 3.4
jenshnielsen commentedon Jul 12, 2016
On Travis OSX master. This one is known to be timing sensitive
jenshnielsen commentedon Jul 12, 2016
Travis python3.5 on 2.x branch so run with multiprocessing
jenshnielsen commentedon Jul 15, 2016
Master 2.7 after the merge of #6741
jenshnielsen commentedon Jul 24, 2016
On 3.5 so no multiprocessing
Kojoley commentedon Jul 27, 2016
OSX. Pytest. No xdist (multiprocessing).
Same commit but different outcomes:
I cannot suggest anything about these fails on
test_mathtexttests, but maybe it is worth adding some sort ofrcParamsdump toimage_comparisondecorator before raisingImageComparisonFailure?Kojoley commentedon Jul 29, 2016
Python: nightly. https://travis-ci.org/Kojoley/matplotlib/jobs/148118137
Kojoley commentedon Aug 8, 2016
https://travis-ci.org/Kojoley/matplotlib/jobs/150594362
QuLogic commentedon Jan 20, 2017
After several tries in mock, I was able to get
matplotlib.tests.test_mathtext.test_mathfont_stixsans_11to fail:Still not sure how to debug that though.
jkseppan commentedon Jan 22, 2017
At least some of the image-size mismatches were because of tests assuming that
plt.subplotcreates a new figure, while another test had already created a figure of the wrong size. See #7726.dopplershift commentedon Feb 2, 2017
https://travis-ci.org/matplotlib/matplotlib/jobs/196800319
jkseppan commentedon Feb 2, 2017
Unfortunately the artifact uploading doesn't seem to be working (#7739) so we can't see what exactly happens with that. How about printing a base64 encoding of the diff image if it isn't too large? PNG compression should be pretty good on typical difference images.
QuLogic commentedon Feb 3, 2017
With pytest, we might want to try the pytest-rerunfailures plugin. It won't fix it, but it would make for less waiting for a full rebuild.
QuLogic commentedon Mar 19, 2020
Our tests generally do not intermittently fail any more, so I don't think we need to leave this open.