regrtest refleak: implement bisection feature

@vstinner

BPO	29512
Nosy	@vstinner, @PierreQuentel, @serhiy-storchaka, @1st1
PRs	bpo-29512: Add test.bisect, bisect failing tests #2452 [3.6] bpo-29512, bpo-30776: Backport regrtest enhancements from master to 3.6 #2513 [3.5] bpo-29512, bpo-30764, bpo-30776: Backport regrtest enhancements from 3.6 to 3.5 #2540 [2.7] bpo-29512, bpo-30764: Backport regrtest enhancements from 3.5 to 2.7 #2541 [2.7] bpo-29512: Fix Lib/test/bisect.py shebang #4522 [2.7] bpo-29512: Rename Lib/test/bisect.py to bisect_cmd.py #7229 bpo-29512: Change bisect to bisect_cmd in docstring #13040 [3.7] bpo-29512: Change bisect to bisect_cmd in docstring (GH-13040) #13051 [2.7] bpo-29512: Change bisect to bisect_cmd in docstring (GH-13040) #13050
Files	bisect_test.py bisect_test.py bisect_test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-30.15:25:05.761>
created_at = <Date 2017-02-09.11:46:03.741>
labels = ['3.7', 'type-feature', 'tests']
title = 'regrtest refleak: implement bisection feature'
updated_at = <Date 2019-05-02.02:52:21.824>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-05-02.02:52:21.824>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-30.15:25:05.761>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-02-09.11:46:03.741>
creator = 'vstinner'
dependencies = []
files = ['46918', '46933', '46934']
hgrepos = []
issue_num = 29512
keywords = ['patch']
message_count = 21.0
messages = ['287405', '294827', '294962', '294964', '294965', '295449', '295450', '297081', '297093', '297421', '297566', '297576', '297586', '299043', '299045', '299047', '306825', '306876', '318134', '318191', '318192']
nosy_count = 4.0
nosy_names = ['vstinner', 'quentel', 'serhiy.storchaka', 'yselivanov']
pr_nums = ['2452', '2513', '2540', '2541', '4522', '7229', '13040', '13051', '13050']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29512'
versions = ['Python 3.7']

It would be great to be able to list which methods cause a reference leak.

Example right now on the 3.6 branch:
----

$ ./python -m test -R 3:3 test_os 
Run tests sequentially
0:00:00 [1/1] test_os
beginning 6 repetitions
123456
......
test_os leaked [124, 124, 124] references, sum=372
test_os leaked [114, 114, 114] memory blocks, sum=342
test_os failed in 32 sec

1 test failed:
test_os

Total duration: 32 sec
Tests result: FAILURE
----

Which methods caused the refleak?

Is it possible to list test methods of a test file using unittest?

First step: I created the issue bpo-30523, "unittest: add --list-tests option to only display the list of test names, don't run tests".

I don't understand how bpo-30523 and bpo-30540 can help to resolve this issue.

Serhiy: "I don't understand how bpo-30523 and bpo-30540 can help to resolve this issue."

bpo-30523 produces a list of all test methods
bpo-30540 allows to pass a file to filter tests
attached bisect_test.py takes an input file produced by --list-tests and uses --matchfile to run tests

Example of usage:

./python bisect_test.py test_tools tests bisect -n 5

bisect -R 3:3 on test_tools
tests contains the list of test identifiers to test_tools
write result into the file "bisect"
stop bisection when there is 5 tests or less

The next step is to add a command to regrtest implementing all steps. Or maybe this script is enough? At least, it's enough for my needs ;-)

Updated script: support Windows, fix cmdline help.

Oops, I uploaded the wrong version. It should now work.

New changeset 84d9d14 by Victor Stinner in branch 'master':
bpo-29512: Add test.bisect, bisect failing tests (bpo-2452)
84d9d14

Ok, I pushed a first implementation. It's problaby buggy and incomplete, but it's better than nothing, and can be enhanced later.

I prefer to play with it longer before starting to backport it to other branches, to not have to backport changes multiple times.

Thanks Yury for the review ;-)

New changeset a3ca94d by Victor Stinner in branch '3.6':
[3.6] bpo-29512, bpo-30776: Backport regrtest enhancements from master to 3.6 (bpo-2513)
a3ca94d

New changeset 0b12107 by Victor Stinner in branch '3.5':
[3.5] bpo-29512, bpo-30764, bpo-30776: Backport regrtest enhancements from 3.6 to 3.5 (bpo-2540)
0b12107

New changeset d7955b8 by Victor Stinner in branch '2.7':
[2.7] bpo-29512, bpo-30764: Backport regrtest enhancements from 3.5 to 2.7 (bpo-2541)
d7955b8

Ok! "python -m test.bisect" is now available in 2.7, 3.5, 3.6 and master branches ;-)

Maybe it's me who is doing things wrong, but when I run Lib/test/test_httpservers.py I get this strange error :

Traceback (most recent call last):
  File "C:\cpython\Lib\test\test_httpservers.py", line 7, in <module>
    from http.server import BaseHTTPRequestHandler, HTTPServer, \
  File "c:\python36\lib\http\server.py", line 92, in <module>
    import email.utils
  File "c:\python36\lib\email\utils.py", line 28, in <module>
    import random
  File "c:\python36\lib\random.py", line 48, in <module>
    import bisect as _bisect
  File "C:\cpython\Lib\test\bisect.py", line 27, in <module>
    import tempfile
  File "c:\python36\lib\tempfile.py", line 45, in <module>
    from random import Random as _Random
ImportError: cannot import name 'Random'

I see that the bisect module is loaded from location /test/bisect.py because of the script introduced in the resolution of this issue. Is it intentional to give it the same name as a module of the standard library ?

There are many ways of running tests, and some of them are broken now.

$ ./python Lib/test/test_bisect.py
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

======================================================================
ERROR: test_backcompatibility (main.TestBisectC)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "Lib/test/test_bisect.py", line 34, in setUp
    (self.module.bisect_right, [], 1, 0),
AttributeError: module 'bisect' has no attribute 'bisect_right'

...

Workarounds:

./python -m test test_httpservers
./python -I Lib/test/test_httpservers.py

Maybe it's me who is doing things wrong, but when I run Lib/test/test_httpservers.py I get this strange error :

...
File "c:\python36\lib\random.py", line 48, in <module>
import bisect as _bisect
File "C:\cpython\Lib\test\bisect.py", line 27, in <module>
...

Oh. Lib/test/ should not be in sys.path. "import bisect" loaded Lib/test/bisect.py instead of Lib/bisect.py. See my commit 3050987 in Python 2.7:

* regrtest now removes '' and [Lib/test/](https://github.com/python/cpython/blob/main/Lib/test) from sys.path
* Use absolute import in test_bisect

(regrtest in master already does something similar.)

But the sys.path is only fixed when regrtest is used...

There are many ways of running tests, and some of them are broken now.
$ ./python Lib/test/test_bisect.py

... if you run Lib/test/test_xxx.py directly, Lib/test/ is added to sys.path, except if you use -I:

$ ./python -I Lib/test/test_bisect.py 
....................................

Ran 36 tests in 0.094s

OK

I didn't know that "./python Lib/test/test_bisect.py" was broken :-(

Maybe Lib/test/support/init.py can remove Lib/test/ from sys.path?

Another option is to move or rename Lib/test/bisect.py. For example, replace "./python -m test.bisect ..." with "./python -m test bisect ..." (or maybe "./python -m test --bisect ..."). The problem is that bisect has extra options, but we can add a bisect prefix to these options.

I reopen this issue.

New changeset 448c673 by Victor Stinner in branch '2.7':
bpo-29512: Fix Lib/test/bisect.py shebang (bpo-4522)
448c673

Python 2.7.14 was released with test.bisect.

While there are corner cases, do we really have to do something?

The simplest fix is just to rename Lib/test/bisect.py to Lib/test/bisect_cmd.py in Python 2.7. It's a tool written for CPython developer, it's CLI is not stable nor documented.

It just mean replacing "./python -m test.bisect ..." with "./python -m test.bisect_cmd ...", not a big deal.

What do you think of renaming the script?

I created the PR 7229 to rename Lib/test/bisect.py to Lib/test/bisect_cmd.py.

New changeset 823c295 by Victor Stinner in branch '2.7':
bpo-29512: Rename Lib/test/bisect.py to bisect_cmd.py (bpo-7229)
823c295

One year later, all issues have now been fixed ;-)

Navigation Menu

Repository navigation

regrtest refleak: implement bisection feature #73698

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

regrtest refleak: implement bisection feature #73698

Description

Activity

vstinner commented on Feb 9, 2017

vstinner commented on May 31, 2017

serhiy-storchaka commented on Jun 1, 2017

vstinner commented on Jun 1, 2017

vstinner commented on Jun 1, 2017

vstinner commented on Jun 8, 2017

vstinner commented on Jun 8, 2017

vstinner commented on Jun 28, 2017

vstinner commented on Jun 28, 2017

vstinner commented on Jun 30, 2017

vstinner commented on Jul 3, 2017

vstinner commented on Jul 3, 2017

vstinner commented on Jul 3, 2017

PierreQuentel commented on Jul 25, 2017

serhiy-storchaka commented on Jul 25, 2017

vstinner commented on Jul 25, 2017

vstinner commented on Nov 23, 2017

vstinner commented on Nov 24, 2017

vstinner commented on May 29, 2018

vstinner commented on May 30, 2018

vstinner commented on May 30, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions