Repository navigation
find by dichotomy the failing test #66797
Description
| BPO | 22607 |
|---|---|
| Nosy | @birkenfeld, @terryjreedy, @vstinner, @bitdancer, @voidspace, @xdegaye, @serhiy-storchaka |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2017-07-11.08:19:37.067>
created_at = <Date 2014-10-11.08:31:29.604>
labels = ['type-feature', 'tests']
title = 'find by dichotomy the failing test'
updated_at = <Date 2017-07-11.08:19:37.066>
user = 'https://github.com/xdegaye'bugs.python.org fields:
activity = <Date 2017-07-11.08:19:37.066>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-07-11.08:19:37.067>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2014-10-11.08:31:29.604>
creator = 'xdegaye'
dependencies = []
files = ['36875', '36987', '36994']
hgrepos = []
issue_num = 22607
keywords = ['patch']
message_count = 15.0
messages = ['229067', '229068', '229069', '229070', '229072', '229073', '229095', '229096', '229097', '229099', '229627', '229761', '229838', '298137', '298144']
nosy_count = 7.0
nosy_names = ['georg.brandl', 'terry.reedy', 'vstinner', 'r.david.murray', 'michael.foord', 'xdegaye', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '29512'
type = 'enhancement'
url = 'https://bugs.python.org/issue22607'
versions = ['Python 3.5']
Activity
xdegaye commentedon Oct 11, 2014
This issue stems from bpo-22588.
See message 228968 for the rationale:
Automatize the dichotomy process used to to identify memory leaks, crash, reference leak, resource leak, etc. in a failing test.
xdegaye commentedon Oct 11, 2014
See msg 228968 for the rationale.
xdegaye commentedon Oct 11, 2014
With the attached patch (the patch does reintroduce the bug in 'test_incref_decref_API' of bpo-22588 for testing purposes), it is possible to find the failing subtest rapidly:
After identifying the failing test, print the list of subtests in this test and their number (35 subsets):
$ export SUBTEST_RANGE="[]"
$ ./python -m test -m test__testcapi test_capi
Then run:
$ ./python -m test -m test__testcapi -R 23:23 test_capi
after modifying, each time, the range of subtests to execute, with:
$ export SUBTEST_RANGE="range(1,18)" # tests 1-17 result: fail
$ export SUBTEST_RANGE="range(1,9)" # tests 1-8 result: pass
$ export SUBTEST_RANGE="range(9,13)" # tests 9-12 result: fail
$ export SUBTEST_RANGE="range(9,11)" # tests 9-10 result: fail
$ export SUBTEST_RANGE="[9]" # so it must be test #9, check it now
The strong limitation with this solution is that the subTest context manager must now be enclosed in a 'try except unittest.SkipTest' clause and that the context manager is used more than 100 times in the test suite.
birkenfeld commentedon Oct 11, 2014
I don't think this feature is generally useful enough to be included.
Since you need to modify the test code anyway (adding the try-except), it is probably just as much work to do the selection there.
Why only add selection of subtests, and not of all tests?
In addition, the patch would have to be reworked: eval()ing an environment variable is not acceptable.
BTW, I think the commonly used term for this is "bisection".
vstinner commentedon Oct 11, 2014
I requested the feature because I regulary need bisect (once a month, or
maybe two months).
birkenfeld commentedon Oct 11, 2014
Always within subtests?
xdegaye commentedon Oct 11, 2014
Sorry for not being more explicit and for being lazy doing a copy paste from msg 229022:
xdegaye commentedon Oct 11, 2014
@ Georg Brandl
You seem to be confusing the feature itself with the implementation.
This feature is generally useful.
The fact that there is an acceptable implementation is another matter (and subtest_in_range.diff is not an implementation).
birkenfeld commentedon Oct 11, 2014
You yourself were calling it a "solution".
A feature proposal should either have a description of the new feature, or a patch. Since there is no feature description, I reviewed the patch as being an implementation of the desired feature.
As described in the issue, I disagree. If the intended feature is more comprehensive, please supply an adequate description or a patch that implements it.
birkenfeld commentedon Oct 11, 2014
I see, this was split off another issue (which was already closed). I agree that a bit more verbosity in the initial description would have prevented confusion :)
terryjreedy commentedon Oct 18, 2014
I believe the goal, and a better title, is "Automate leak discovery within a group of tests". Bisection or dichotomy is a means, not a goal, and should not be part of the title.
"Leak discovery" means 'find a test within the group that has a leak (which we know or suspect might be occurring)'.
'Test' can mean multiple modules, one module, one TestCase class, one test_xxx method, one subtest, or one assert. What might it mean in 'group of tests'? Pinning a leak on a particular statement likely has to be done manually; eliminate that. On the other hand, it is feasible (see below) to automatically blame a particular test method and perhaps a particular subtest. So interprete 'tests' as 'test methods or subtests'.
Unittest can run all test methods in a module, all test methods in a test class, or an individual test method. Idle already has a module browser (Class Browser), based on stdlib's pyclbr and Idle's TreeWidget, that could be adapted (perhaps using ttk.Treeview instead) to produce a module-test_class-test tree. Add buttons to select a particular leak test and others to select manual or auto nagivation of the tree and we might have something worthwhile. It could go in Tools/Scripts or PyPI.
I have not used subtests yet since they are not available on 2.7, but I can imagine semi-automated editing, saving to a temp file, and running. I would need experience with real subtest bisection to comment on the patch.
xdegaye commentedon Oct 21, 2014
The attached patch adds the '-X' and '-Y' options to the regrtest tool, allowing to select a range of tests and a range of their subtests. The patch is missing the test cases for the moment.
Limitation:
Does not work very well with nested subtest (nested subtests are currently only used by the unittest test suite itself): subtest numbers are flattened across all the nested subtests.
Here is the sequence of commands that would have allowed to find the subtest responsible for the crash in test_capi at bpo-22588, without modifying any code:
./python -m test -X "[]" test_capi # Get the test count: 18 tests.
./python -m test -X "range(1,10)" -R 23:23 test_capi # pass
./python -m test -X "range(10,15)" -R 23:23 test_capi # pass
./python -m test -X "range(15,17)" -R 23:23 test_capi # pass
./python -m test -X "[17]" -R 23:23 test_capi # pass
./python -m test -X "[18]" -R 23:23 test_capi # fail
./python -m test -X "[18]" -Y "[]" test_capi # Test 18 has 35 subtests.
./python -m test -X "[18]" -Y "range(1,19)" -R 23:23 test_capi # fail
./python -m test -X "[18]" -Y "range(1,10)" -R 23:23 test_capi # fail
./python -m test -X "[18]" -Y "range(1,6)" -R 23:23 test_capi # pass
./python -m test -X "[18]" -Y "range(6,8)" -R 23:23 test_capi # pass
./python -m test -X "[18]" -Y "[8]" -R 23:23 test_capi # pass
./python -m test -X "[18]" -Y "[9]" -R 23:23 test_capi # fail
Output of the last command:
Test# 18: test__testcapi
subTest# 9: internal, {'name': 'test_incref_decref_API'}
xdegaye commentedon Oct 22, 2014
This new version of the patch uses a specific exception to skip tests and fixes a bug when invoking the overriden and wrapped subTest method.
serhiy-storchaka commentedon Jul 11, 2017
Wasn't the similar feature implemented in bpo-29512?
vstinner commentedon Jul 11, 2017
http://bugs.python.org/issue22588#msg228968
Ok, this use case is now well supported by the new test.bisect tool. Yes, this issue is a duplicate of the issue bpo-29512. I forgot this old issue!