Improve import time of various stdlib modules

Feature or enhancement

Proposal:

As noted in https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173, typing isn't the slowest stdlib module in terms of import time, but neither is it one of the quickest. We should speed it up, if possible.

Links to previous discussion of this feature:

https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173

Linked PRs

gh-109653: `typing.py`: improve import time by creating soft-deprecat…

Retitling this issue to reflect a change of scope: part of the reason for the slow import time of typing is the slow import time of some of the modules it imports. There may be low-hanging fruit in some other stdlib modules as well.

pythongh-109653: Improve `enum` import time by avoiding import of `fu…

An idea: review the output of a linter for "unused" imports.

Some may be false positives due to needing an import's side effects, but I expect there will be some that are no longer needed.

Some may be false positives due to needing an import's side effects ...

How do you feel about making an intentional effort to get rid of such reliance on side effects in the stdlib?
Explicit is better than implicit :)

An idea: review the output of a linter for "unused" imports.

FWIW, I can only find one in the stdlib that I'm 100% confident with, using ruff/pycln:

cpython/Lib/asyncio/tasks.py

Line 20 in e8be0c9

import warnings

I think this is because Victor went through them all fairly recently, in #105411. There are a fair few in Lib/test/, though. I'll file a PR in a bit.

gh-109653: Improve `enum` import time by avoiding import of `functool…

This is also interesting:

dataclasses copies recursive_repr to avoid a dependency:

cpython/Lib/dataclasses.py

Lines 248 to 266 in 51863b7

    
           # This function's logic is copied from "recursive_repr" function in 
        
           # reprlib module to avoid dependency. 
        
           def _recursive_repr(user_function): 
        
               # Decorator to make a repr function return "..." for a recursive 
        
               # call. 
        
               repr_running = set() 
        
               @functools.wraps(user_function) 
        
               def wrapper(self): 
        
                   key = id(self), _thread.get_ident() 
        
                   if key in repr_running: 
        
                       return '...' 
        
                   repr_running.add(key) 
        
                   try: 
        
                       result = user_function(self) 
        
                   finally: 
        
                       repr_running.discard(key) 
        
                   return result 
        
               return wrapper

But, it imports functools:

cpython/Lib/dataclasses.py

Line 7 in 51863b7

import functools
And it imports reprlib.recursive_repr:

cpython/Lib/functools.py

Line 20 in 51863b7

from reprlib import recursive_repr

One more interesting case:

functools says that import types is deferred to singledispatch():

cpython/Lib/functools.py

Line 19 in 51863b7

# import types, weakref # Deferred to single_dispatch()
But several lines below types module is imported:

cpython/Lib/functools.py

Line 22 in 51863b7

from types import GenericAlias

pythongh-109653: Avoid a top-level import of `types` in `functools`

pythongh-109653: What's new: Note improved import times for several s…

gh-109653: What's new: Note improved import times for several stdlib …

I believe there are futher improvements to be made here - how would you like to handle these? For example, importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless.

I believe there are futher improvements to be made here - how would you like to handle these? For example, importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless.

Sure -- I felt like the low-hanging fruit had been tackled here, and this issue hadn't seen much activity in quite a while, so I felt like it was a good time to close. But more improvements are obviously always welcome. Maybe open a new issue if you have some more targeted improvements you want to propose?

Please link the new issue here so we all can start following that as well!

#118761

pythongh-109653: What's new: Note improved import times for several s…

pythongh-109653: `typing.py`: improve import time by creating soft-de…

pythongh-109653: Improve `enum` import time by avoiding import of `fu…

pythongh-109653: Remove unused imports in the `Lib/` directory (pytho…

pythongh-109653: Avoid a top-level import of `types` in `functools` (p…

pythongh-109653: Reduce the import time of `random` by 60% (python#11…

pythongh-109653: Defer importing `warnings` in several modules (pytho…

pythongh-109653: Improve the import time of `email.utils` (python#109824

Navigation Menu

Repository navigation

Improve import time of various stdlib modules #109653

Feature or enhancement

Proposal:

Links to previous discussion of this feature:

Linked PRs

43 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# This function's logic is copied from "recursive_repr" function in
	# reprlib module to avoid dependency.
	def _recursive_repr(user_function):
	# Decorator to make a repr function return "..." for a recursive
	# call.
	repr_running = set()

	@functools.wraps(user_function)
	def wrapper(self):
	key = id(self), _thread.get_ident()
	if key in repr_running:
	return '...'
	repr_running.add(key)
	try:
	result = user_function(self)
	finally:
	repr_running.discard(key)
	return result
	return wrapper

Improve import time of various stdlib modules #109653

Description

Feature or enhancement

Proposal:

Links to previous discussion of this feature:

Linked PRs

Activity

AlexWaygood commented on Sep 23, 2023

hugovk commented on Sep 23, 2023

itamaro commented on Sep 23, 2023

AlexWaygood commented on Sep 23, 2023

sobolevn commented on Sep 24, 2023

sobolevn commented on Sep 24, 2023

43 remaining items

layday commented on May 8, 2024

AlexWaygood commented on May 8, 2024

ofek commented on May 8, 2024

layday commented on May 8, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions