Repository navigation
Hook up Codecov for coverage reporting on C code #2
Description
Activity
I have been running lcov regularly. gcov / lcov coverage of C code is a bit tricky for some modules. Some tests require a real terminal (ncurses), X server (tkinter, idle), root privileges (spwd) or special system configuration (yp for nis). The
make coverage-lcovtarget ignores coverage of 3rd party code, too.@ # remove 3rd party modules, system headers and internal files with @ # debug, test or dummy functions. @lcov --remove $(COVERAGE_INFO) \ '*/Modules/_blake2/impl/*' \ '*/Modules/_ctypes/libffi*/*' \ '*/Modules/_decimal/libmpdec/*' \ '*/Modules/_sha3/kcp/*' \ '*/Modules/expat/*' \ '*/Modules/zlib/*' \ '*/Include/*' \ '*/Modules/xx*.c' \ '*/Parser/listnode.c' \ '*/Python/pyfpe.c' \ '*/Python/pystrcmp.c' \ '/usr/include/*' \ '/usr/local/include/*' \ '/usr/lib/gcc/*' \ --output-file $(COVERAGE_INFO)Would we run lcov as a separate coverage step from coverage.py? Or somehow run them simultaneously? I don't know what kind of slowdown we're looking at here if we had both on and if we might run afoul of Travis' free tier limitations.
On my machine it takes almost 10 minutes to run Python's test suite with a gcov instrumented build of Python. My laptop is pretty fast (8 cores
i7-4900MQ CPU @ 2.80GHz, 16 GB RAM). I'm able to utilze all cores with 10 parallel test process. It's going to be hard enough to get lcov finish on Travis without coverage.py.$ make coverage $ time make test ... real 9m40.226s user 56m49.298s sys 1m17.350sOK, then we can look at running it as a separate test. If you want to try and get this working now then send a PR at https://github.com/brettcannon/cpython-ci-test as that's where I plan to pull the initial config files for Travis and Codecov.
I have to look into it. I'd like to use a service that can use lcov database so we can re-use out-of-tree build and lcov excludes from
make coverage-lcov.It looks like lcov is supported by Codecov: https://github.com/codecov/codecov-bash/blob/master/codecov#L822
Did you want me to leave this open for you, @tiran to look into lcov?
Please take a look at https://github.com/abalkin/cpython/tree/ci or specifically at https://codecov.io/gh/abalkin/cpython/branch/ci. @brettcannon is that the "appropriate merging" that you are looking for?
@abalkin unfortunately no as there's no coverage for the Lib/ directory. If you include coverage runs for both lcov and coverage.py do the results show up in Codecov as if they were run together?
Right. I thought I've seen both, but clearly my focus was on C coverage.
We are evaluating codecov's commercial offering in my firm at the moment, so I should have a working setup "soon".
Reacted by Brett Cannon- changed the title
[-]Hook up Codecov for coverage reporting[/-][+]Hook up Codecov for coverage reporting on C code[/+]on Apr 10, 2017 Tweaked the title in hopes that someone is able to look into getting C code coverage working.
Looked into this. Got it working on my fork here: https://codecov.io/gh/ammaraskar/cpython
Codecov makes this incredibly simple, they support auto merging of multiple reports, so both the python and C parts show up in the coverage report.
Some disadvantages, the coverage instrumented build takes longer to test on travis than without coverage. My build ran for ~40 minutes compared to ~20 minutes currently. This isn't too big of a concern since the coverage build is optional and does not block the travis status but its something to consider.
and here it is as part of the pull request: https://codecov.io/gh/python/cpython/tree/f2148586600ea3a2c4a04550105f32d561f100aa
Great work! Hopefully one of us will be able to review python/cpython#7773 shortly.
This is now merged!
See https://github.com/brettcannon/cpython-ci-test/blob/master/codecov.yml for an initial configuration (only differs from the default thanks to the
newconfig for comments).