Skip to content

Hypothesis health checks don't pass reliably if "CI" is not set in the environment. #132

Description

@mgorny

With current version of hypothesis:

$ tox -e py36
GLOB sdist-make: /tmp/hyperlink/setup.py
py36 create: /tmp/hyperlink/.tox/py36
py36 installdeps: idna==2.9, pytest==5.2.4, coverage==4.5.4, pytest-cov==2.10.0, hypothesis==5.8.6
py36 inst: /tmp/hyperlink/.tox/.tmp/package/1/hyperlink-20.0.1.dev0.zip
py36 installed: atomicwrites==1.4.0,attrs==19.3.0,coverage==4.5.4,hyperlink @ file:///tmp/hyperlink/.tox/.tmp/package/1/hyperlink-20.0.1.dev0.zip,hypothesis==5.8.6,idna==2.9,importlib-metadata==1.7.0,more-itertools==8.4.0,packaging==20.4,pluggy==0.13.1,py==1.9.0,pyparsing==2.4.7,pytest==5.2.4,pytest-cov==2.10.0,six==1.15.0,sortedcontainers==2.2.2,wcwidth==0.2.5,zipp==3.1.0
py36 run-test-pre: PYTHONHASHSEED='2292208485'
py36 run-test: commands[0] | pytest --cov=hyperlink --cov-report=term-missing:skip-covered --doctest-modules src/hyperlink
=============================================================== test session starts ===============================================================
platform linux -- Python 3.6.11, pytest-5.2.4, py-1.9.0, pluggy-0.13.1
cachedir: .tox/py36/.pytest_cache
rootdir: /tmp/hyperlink, inifile: pytest.ini
plugins: hypothesis-5.8.6, cov-2.10.0
collected 130 items                                                                                                                               

src/hyperlink/_url.py ....................                                                                                                  [ 15%]
src/hyperlink/test/test_common.py ......                                                                                                    [ 20%]
src/hyperlink/test/test_decoded_url.py ..........                                                                                           [ 27%]
src/hyperlink/test/test_hypothesis.py ......F.......                                                                                        [ 38%]
src/hyperlink/test/test_parse.py .                                                                                                          [ 39%]
src/hyperlink/test/test_scheme_registration.py ......                                                                                       [ 43%]
src/hyperlink/test/test_url.py .........................................................................                                    [100%]

==================================================================== FAILURES =====================================================================
______________________________________________ TestHypothesisStrategies.test_hostnames_ascii_nolead _______________________________________________

self = <hyperlink.test.test_hypothesis.TestHypothesisStrategies testMethod=test_hostnames_ascii_nolead>

    @given(hostnames(allow_leading_digit=False, allow_idn=False))
>   def test_hostnames_ascii_nolead(self, hostname):
        # type: (Text) -> None
        """
        hostnames(allow_leading_digit=False, allow_idn=False) generates
        ASCII host names without leading digits.
E       hypothesis.errors.FailedHealthCheck: Data generation is extremely slow: Only produced 1 valid examples in 1.01 seconds (40 invalid ones and 0 exceeded maximum size). Try decreasing size of the data you're generating (with e.g.max_size or max_leaves parameters).
E       See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.too_slow to the suppress_health_check settings for this test.

src/hyperlink/test/test_hypothesis.py:177: FailedHealthCheck
------------------------------------------------------------------- Hypothesis --------------------------------------------------------------------
You can add @seed(2426485286323854916506411185325111964) to this test or run pytest with --hypothesis-seed=2426485286323854916506411185325111964 to reproduce this failure.

---------- coverage: platform linux, python 3.6.11-final-0 -----------
Name                                    Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------------------------
src/hyperlink/_socket.py                   22     20      2      0     8%   3-53
src/hyperlink/_url.py                     690     21    349     12    96%   28-29, 52-53, 115, 122, 126, 253, 280, 378, 391, 571, 886-887, 1201, 1822-1823, 1845-1847, 2370-2372, 113->115, 195->197, 250->253, 277->280, 375->378, 388->391, 570->571, 884->886, 1200->1201, 1519->1524, 1821->1822, 2274->exit
src/hyperlink/hypothesis.py               111      5     37      2    95%   11-14, 101, 244, 100->101, 243->244
src/hyperlink/test/__init__.py             12      6      2      1    50%   14-28, 13->14
src/hyperlink/test/test_hypothesis.py      72      4      6      0    95%   10-11, 18-19
src/hyperlink/test/test_socket.py          29     26      4      1    12%   5-6, 9-45, 8->9
src/hyperlink/test/test_url.py            579      3     16      1    99%   940, 1472-1473, 1471->1472
-----------------------------------------------------------------------------------
TOTAL                                    1766     85    422     17    95%

6 files skipped due to complete coverage.

========================================================= 1 failed, 129 passed in 59.10s ==========================================================
ERROR: InvocationError for command /tmp/hyperlink/.tox/py36/bin/pytest --cov=hyperlink --cov-report=term-missing:skip-covered --doctest-modules src/hyperlink (exited with code 1)
_____________________________________________________________________ summary _____________________________________________________________________
ERROR:   py36: commands failed

(this is with 20.0.0 release)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions