test.support has way too many imports

@terryjreedy

BPO	40275
Nosy	@terryjreedy, @vstinner, @pablogsal, @miss-islington, @shihai1991
PRs	bpo-40275: Add asyncioutils in test.support #19592 [WIP] bpo-40275: Add loggingutils in test.support #19599 bpo-40275: Avoid importing asyncio in test.support #19600 bpo-40275: Avoid importing logging in test.support #19601 bpo-40275: Avoid importing socket in test.support #19603 bpo-40275: Move transient_internet from test.support to socket_helper #19711 bpo-40275: Move requires_hashdigest from test.support to hashlib_helper #19716 bpo-40275: Remove duplicate import of locale module #19761 bpo-40275: Fix name error in support.socket_helper #19825 bpo-40275: import locale lazily in gettext #19905 bpo-40275: lazy import modules in test.support #20128 bpo-40275: lazy import module in test.supports' decorators #20131 bpo-40275: reduce importing module nums by lazy import in libregrtest #20207 bpo-40275: Adding a threading_helper in test.support #20263 [WIP] bpo-40275: add filesystem_helper in test.support #20459 bpo-40275: test.support.check_impl_detail() uses sys.implementation #20468 bpo-40275: test.support imports subprocess lazily #20471 bpo-40275: Fix a calling bug in threading_helper #20479 bpo-40275: remove TESTFN_ENCODING in test.support #20482 bpo-40275: Fix test.support.threading_helper #20488 bpo-40275: Adding filesystem_helper submodule in test.support #20625 bpo-40275: Adding filesystem_helper submodule in test.support #20689 bpo-40275: Add os_helper submodule in test.support #20732 bpo-40275: Add os_helper submodule in test.support #20765 bpo-40275: Add import_helper submodule in test.support #20794 bpo-40275: Add warnings_helper submodule in test.support #20797 bpo-40275: test.supports imports lazily fnmatch, glob, struct #20810 bpo-40275: Move TransientResource to test_urllib2net #20812 bpo-40275: Use new test.support helper submodules in tests #20824 bpo-40275: Use test helpers to replace test.support [part 2] #20849 bpo-40275: Remove redundant import in libregrtest #20957 bpo-40275: Use new test.support helper submodules in tests #21151 bpo-40275: Remove all new modules imported by tests #21167 bpo-40275: Use new test.support helper submodules in tests #21169 bpo-40275: Use new test.support helper submodules in tests #21219 bpo-40275: Use new test.support helper submodules in tests #21314 bpo-40275: Use new test.support helper submodules in tests #21315 bpo-40275: Use new test.support helper submodules in tests #21317 bpo-40275: Use new test.support helper submodules in tests #21412 bpo-40275: Use new test.support helper submodules in tests #21448 bpo-40275: Use new test.support helper submodules in tests #21449 bpo-40275: Use new test.support helper submodules in tests #21450 bpo-40275: Use new test.support helper submodules in tests #21451 bpo-40275: Use new test.support helper submodules in tests #21452 bpo-40275: Use new test.support helper submodules in tests #21727 bpo-40275: Use new test.support helper submodules in tests #21743 bpo-40275: Use new test.support helper submodules in tests #21764 bpo-40275: Use new test.support helper submodules in tests #21772 bpo-40275: Remove test helpers import in test.support #21771 bpo-40275: Use new test.support helper submodules in tests #21785 bpo-40275: Fix failed test cases by using test helpers #21811
Files	count_imports.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-09-04.15:37:27.746>
created_at = <Date 2020-04-13.22:24:51.262>
labels = ['tests', '3.10']
title = 'test.support has way too many imports'
updated_at = <Date 2020-09-04.15:37:27.744>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-09-04.15:37:27.744>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-09-04.15:37:27.746>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-04-13.22:24:51.262>
creator = 'vstinner'
dependencies = []
files = ['49197']
hgrepos = []
issue_num = 40275
keywords = ['patch']
message_count = 80.0
messages = ['366337', '366765', '366766', '366767', '366768', '366776', '366780', '366786', '366787', '366789', '366810', '366815', '366823', '367261', '367262', '367265', '367603', '367629', '367818', '368073', '368814', '368816', '368840', '369029', '369030', '369032', '369062', '369214', '369296', '370142', '370146', '370150', '370152', '370154', '370217', '370221', '371188', '371253', '371295', '371296', '371297', '371298', '371299', '371303', '371751', '372339', '372361', '372381', '372690', '372691', '373081', '373082', '373109', '373403', '374738', '374739', '374740', '374828', '374829', '374931', '375002', '375021', '375039', '375042', '375043', '375051', '375059', '375060', '375067', '375068', '375069', '375122', '375124', '375150', '375151', '375269', '375315', '375367', '375368', '376375']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'vstinner', 'pablogsal', 'miss-islington', 'shihai1991']
pr_nums = ['19592', '19599', '19600', '19601', '19603', '19711', '19716', '19761', '19825', '19905', '20128', '20131', '20207', '20263', '20459', '20468', '20471', '20479', '20482', '20488', '20625', '20689', '20732', '20765', '20794', '20797', '20810', '20812', '20824', '20849', '20957', '21151', '21167', '21169', '21219', '21314', '21315', '21317', '21412', '21448', '21449', '21450', '21451', '21452', '21727', '21743', '21764', '21772', '21771', '21785', '21811']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40275'
versions = ['Python 3.10']

test_threading.ThreadJoinOnShutdown.test_reinit_tls_after_fork() does crash on AIX: bpo-40068. One of the issue is that logging does crash at fork: bpo-40091. The strange thing is that test_threading doesn't use logging. I'm quite sure that logging is imported through test.support.

"import test.support" imports not less than 171... That's quite "heavy". It includes "heavy" modules like concurrent.futures, asyncio or multiprocessing.

It's maybe time to slice again test.support until submodules to reduce the number of imports to the bare minimum. For example, "import test.support" imports asyncio, whereas it's only used by very few tests. Maybe we should add "test.support.asyncioutils".

$ ./python
Python 3.9.0a5+ (heads/pycore_interp:a1ff2c5cf3, Apr 13 2020, 12:27:13) 
>>> import sys
>>> import test

>> before=set(sys.modules)
>> import test.support
>> after=set(sys.modules)

>>> len(after - before)
171

>>> import pprint
>>> pprint.pprint(sorted(after - before))
['__mp_main__',
 '_asyncio',
 '_bisect',
 '_blake2',
 '_bz2',
 '_collections',
 '_compat_pickle',
 '_compression',
 '_contextvars',
 '_datetime',
 '_elementtree',
 '_functools',
 '_hashlib',
 '_heapq',
 '_lzma',
 '_opcode',
 '_operator',
 '_pickle',
 '_posixsubprocess',
 '_queue',
 '_random',
 '_sha3',
 '_sha512',
 '_socket',
 '_sre',
 '_ssl',
 '_string',
 '_struct',
 '_sysconfigdata_d_linux_x86_64-linux-gnu',
 '_weakrefset',
 'argparse',
 'array',
 'asyncio',
 'asyncio.base_events',
 'asyncio.base_futures',
 'asyncio.base_subprocess',
 'asyncio.base_tasks',
 'asyncio.constants',
 'asyncio.coroutines',
 'asyncio.events',
 'asyncio.exceptions',
 'asyncio.format_helpers',
 'asyncio.futures',
 'asyncio.locks',
 'asyncio.log',
 'asyncio.protocols',
 'asyncio.queues',
 'asyncio.runners',
 'asyncio.selector_events',
 'asyncio.sslproto',
 'asyncio.staggered',
 'asyncio.streams',
 'asyncio.subprocess',
 'asyncio.tasks',
 'asyncio.transports',
 'asyncio.trsock',
 'asyncio.unix_events',
 'base64',
 'binascii',
 'bisect',
 'bz2',
 'collections',
 'collections.abc',
 'concurrent',
 'concurrent.futures',
 'concurrent.futures._base',
 'contextlib',
 'contextvars',
 'copy',
 'copyreg',
 'datetime',
 'difflib',
 'dis',
 'email',
 'email.base64mime',
 'email.charset',
 'email.encoders',
 'email.errors',
 'email.header',
 'email.quoprimime',
 'enum',
 'errno',
 'faulthandler',
 'fnmatch',
 'functools',
 'gc',
 'gettext',
 'glob',
 'grp',
 'gzip',
 'hashlib',
 'heapq',
 'importlib',
 'importlib._bootstrap',
 'importlib._bootstrap_external',
 'importlib.abc',
 'importlib.machinery',
 'importlib.util',
 'inspect',
 'itertools',
 'keyword',
 'linecache',
 'locale',
 'logging',
 'logging.handlers',
 'lzma',
 'math',
 'multiprocessing',
 'multiprocessing.context',
 'multiprocessing.process',
 'multiprocessing.reduction',
 'nntplib',
 'opcode',
 'operator',
 'pickle',
 'platform',
 'pprint',
 'pwd',
 'pyexpat',
 'pyexpat.errors',
 'pyexpat.model',
 'queue',
 'quopri',
 'random',
 're',
 'reprlib',
 'resource',
 'select',
 'selectors',
 'shutil',
 'signal',
 'socket',
 'sre_compile',
 'sre_constants',
 'sre_parse',
 'ssl',
 'string',
 'struct',
 'subprocess',
 'sysconfig',
 'tempfile',
 'test.support',
 'test.support.testresult',
 'threading',
 'token',
 'tokenize',
 'traceback',
 'types',
 'typing',
 'typing.io',
 'typing.re',
 'unittest',
 'unittest.async_case',
 'unittest.case',
 'unittest.loader',
 'unittest.main',
 'unittest.result',
 'unittest.runner',
 'unittest.signals',
 'unittest.suite',
 'unittest.util',
 'urllib',
 'urllib.error',
 'urllib.response',
 'warnings',
 'weakref',
 'xml',
 'xml.etree',
 'xml.etree.ElementPath',
 'xml.etree.ElementTree',
 'zlib']

"import test.support" imports not less than 171... That's quite "heavy".

If we split some "heavy" modules to xxxutils, what benefits will we make in fact(more exact module importing behavior)?

+1 for splitting test.support on several submodules!

Some imports can also be lazy.

asyncio is now imported in unittest. Removing direct import from test.support does not help.

asyncio is now imported in unittest. Removing direct import from test.support does not help.

Oh, thanks, you are right. Looks like we need check which submodules should be splitted.

logging is also imported in unittest and indirectly in asyncio.

These three PRs combined reduce the number of imported modules from 179 to 105 (not including 24 modules imported by the bare interpreter) and reduce the hot import time from 160 ms to 80 ms.

These three PRs combined reduce the number of imported modules from 179 to 105 (not including 24 modules imported by the bare interpreter) and reduce the hot import time from 160 ms to 80 ms.

Good job, serhiy. I tested your patches in my vm, the number of importing module have been reduced. Could you paste your performance benchmark test? I got a no siginificant change :( when I run:
$ ./python -m pyperf timeit --compare-to python3.9 "from test import support" -p 100
python3.9: ..................................................................................................... 820 ns +- 206 ns

python: ..................................................................................................... 809 ns +- 201 ns

Mean +- std dev: [python3.9] 820 ns +- 206 ns -> [python] 809 ns +- 201 ns: 1.01x faster (-1%)
Not significant!

I used -X importtime.

Your benchmark measures the performance of look up in sys.modules.

I used -X importtime.
copy that, thanks.

In master branch:
import time: 5011 | 133562 | test.support

After apply serhiy's patches:
import time: 4863 | 66940 | test.support

What is the practical impact on the time taken for test runs? How much does the total time for a test run reduce as a result of refactoring test.support to minimise imports?

test__osx_support and test_selectors are broken, examples:

https://github.com/python/cpython/pull/21806/checks?check_run_id=966438645

2020-08-10T12:04:20.7613070Z ======================================================================
2020-08-10T12:04:20.7613710Z ERROR: test__remove_unsupported_archs (test.test__osx_support.Test_OSXSupport)
2020-08-10T12:04:20.7616560Z ----------------------------------------------------------------------

2020-08-10T12:04:20.7617240Z Traceback (most recent call last):
2020-08-10T12:04:20.7617860Z   File "/Users/runner/work/cpython/cpython/Lib/test/test__osx_support.py", line 23, in setUp
2020-08-10T12:04:20.7618560Z     self.env = test.support.EnvironmentVarGuard()
2020-08-10T12:04:20.7621650Z AttributeError: module 'test.support' has no attribute 'EnvironmentVarGuard'

and

2020-08-10T12:01:40.0736200Z ======================================================================
2020-08-10T12:01:40.0738440Z ERROR: test_register_bad_fd (test.test_selectors.KqueueSelectorTestCase)
2020-08-10T12:01:40.0742700Z ----------------------------------------------------------------------

2020-08-10T12:01:40.0744640Z Traceback (most recent call last):
2020-08-10T12:01:40.0747460Z   File "/Users/runner/work/cpython/cpython/Lib/test/test_selectors.py", line 539, in test_register_bad_fd
2020-08-10T12:01:40.0753630Z     bad_f = support.make_bad_fd()
2020-08-10T12:01:40.0757650Z AttributeError: module 'test.support' has no attribute 'make_bad_fd'

Hai Shi: Would you mind to investigate this issue?

Hai Shi: Would you mind to investigate this issue?
Oh, sure, I will check the test cases again.

New changeset 490c542 by Hai Shi in branch 'master':
bpo-40275: Fix failed test cases by using test helpers (GH-21811)
490c542

Update.

"import test.support" now imports 37 modules, instead of 171 previously. It's way better!

23:26:20 vstinner@apu$ ./python
Python 3.10.0a0 (heads/master:598a951844, Aug  7 2020, 17:24:10) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import test
>>> before=set(sys.modules)
>>> import test.support
>>> after=set(sys.modules)
>>> len(after - before)
37
>>> import pprint; pprint.pprint(sorted(after - before))
['_datetime',
 '_elementtree',
 '_heapq',
 '_sysconfigdata_d_linux_x86_64-linux-gnu',
 '_testcapi',
 '_weakrefset',
 'argparse',
 'copy',
 'datetime',
 'difflib',
 'fnmatch',
 'gettext',
 'heapq',
 'math',
 'pprint',
 'pyexpat',
 'pyexpat.errors',
 'pyexpat.model',
 'signal',
 'sysconfig',
 'test.support',
 'test.support.testresult',
 'traceback',
 'unittest',
 'unittest.case',
 'unittest.loader',
 'unittest.main',
 'unittest.result',
 'unittest.runner',
 'unittest.signals',
 'unittest.suite',
 'unittest.util',
 'weakref',
 'xml',
 'xml.etree',
 'xml.etree.ElementPath',
 'xml.etree.ElementTree']

Shorter list if imports made by unittest are ignored:

23:27:42 vstinner@apu$ ./python
Python 3.10.0a0 (heads/master:598a951844, Aug  7 2020, 17:24:10) 
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, unittest, test
>>> before=set(sys.modules)
>>> before=set(sys.modules); import test.support; after=set(sys.modules)
>>> len(after) - len(before)
17
>>> import pprint; pprint.pprint(sorted(after - before))
['_datetime',
 '_elementtree',
 '_sysconfigdata_d_linux_x86_64-linux-gnu',
 '_testcapi',
 'copy',
 'datetime',
 'math',
 'pyexpat',
 'pyexpat.errors',
 'pyexpat.model',
 'sysconfig',
 'test.support',
 'test.support.testresult',
 'xml',
 'xml.etree',
 'xml.etree.ElementPath',
 'xml.etree.ElementTree']

Hi, victor:
About https://bugs.python.org/issue40275#msg369214, shall we continue to improve the libregretest?

Running test cases sequential: you have use a mechanism to unload the modules which load in the running test cases stage: https://github.com/python/cpython/blob/master/Lib/test/libregrtest/main.py#L437
Running test cases concurrently: It have use subprocess to isolate the module resources, no?
I am not sure about it.

About https://bugs.python.org/issue40275#msg369214, shall we continue to improve the libregretest?

If someone wants to continue the effort in libregrtest, I suggest to open a new issue. This one already has a long list of commits and messages about test.support.

See test.libregrtest.runtest_mp module for the "multiprocessing" implementation which uses subprocess. It's used when you run tests using -jN option. Almost all buildbots use -jN: see make buildbottest.

If someone wants to continue the effort in libregrtest, I suggest to open a new issue.
+1. And if there have any other test cases blocked by test.support, I suggest this bpo.

oh, sorry, missing a word: I suggest this bpo.->I suggest close this bpo.

Sadly, the work done on this issue is useless until test.regrtest imports less modules as well.

So I created bpo-41718 follow-up issue: "test.regrtest has way too many imports".

I consider that the work is done on test.support, so I close this issue.

Navigation Menu

Repository navigation

test.support has way too many imports #84456

61 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test.support has way too many imports #84456

Description

Activity

vstinner commented on Apr 13, 2020

shihai1991 commented on Apr 19, 2020

serhiy-storchaka commented on Apr 19, 2020

serhiy-storchaka commented on Apr 19, 2020

shihai1991 commented on Apr 19, 2020

serhiy-storchaka commented on Apr 19, 2020

serhiy-storchaka commented on Apr 19, 2020

shihai1991 commented on Apr 19, 2020

serhiy-storchaka commented on Apr 19, 2020

shihai1991 commented on Apr 19, 2020

vsajip commented on Apr 20, 2020

61 remaining items

vstinner commented on Aug 10, 2020

shihai1991 commented on Aug 10, 2020

vstinner commented on Aug 10, 2020

vstinner commented on Aug 10, 2020

shihai1991 commented on Aug 12, 2020

vstinner commented on Aug 13, 2020

shihai1991 commented on Aug 14, 2020

shihai1991 commented on Aug 14, 2020

vstinner commented on Sep 4, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions