Repository navigation
PEP 726: Support defining __setattr__() and __delattr__() in a module #106016
Description
The Python stdlib sys module implements multiple API with simple attributes like sys.path, sys.modules, etc. Converting these APIs to getter and setter functions would be an annoying and expensive migration, a lot of code expect these attributes and access them directly.
It's not possible to execute code before or after a module attribute is set to validate inputs or to update internal states.
Example:
$ python3.12
>>> import sys
>>> sys.modules = 123
>>> import asyncio
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 1260, in _find_and_load
AttributeError: 'int' object has no attribute 'get'
If sys.modules is set to an invalid type, import stops working and produces an error which makes no sense.
Another problem is that the internals of Python have a copy of sys attributes and so don't respect sys changes. Recently, I discovered that PyImport_AddModule() uses a "copy" of sys.modules (a reference to the dict). Overriding sys.modules has no effect: it still changes the original sys.modules dict. See #105998 (comment) for technical details.
Supporting __setattr__() would allow to update the internal reference to sys.modules (PyInterpreterState.imports.modules).
Adding __delattr__() would help prevent to delete critical sys attributes which makes the code more complicated. For example, code using sys.stderr has to check if the attribute exists and if it's not None. Currently, it's possible to remove any sys attribute, including functions:
$ python3.12
>>> import sys
>>> del sys.excepthook
>>> 1+
sys.excepthook is missing
File "<stdin>", line 1
1+
^
SyntaxError: invalid syntax
Notice the sys.excepthook is missing error mesage.
In Python 3.7, support for __getattr__() and dir() was added to modules by PEP 562 – Module __getattr__ and __dir__.
This change alone doesn't allow to validate changes of mutable objects like sys.path list. For example, it was proposed to deprecate or disallow adding bytes strings to sys.path. For this, sys.path should use a different type (such as a list subclass): it's not directly related to this issue.
Linked PRs
- gh-106016: Support customizing of module attributes access with __setattr__/__delattr__ (PEP 726) #108261
- gh-106016: Add Lib/test/test_module/ directory #108293
- [3.11] gh-106016: Add Lib/test/test_module/ directory (GH-108293) #108304
- [3.12] gh-106016: Add Lib/test/test_module/ directory (#108293) #108306
Activity
vstinner commentedon Jun 23, 2023
Currently, it's possible to implement this feature by overriding the module in
sys.moduleswith a different class which implements__setattr__(). Proof-of-conceptmymodule.py:Example:
But this approach doesn't work for C extensions (which is what I care about here) ;-)
I found the code (that I updated for Python 3) in an old comp.lang.python discussion (2010) about overriding
__setattr__()of a module: https://groups.google.com/g/comp.lang.python/c/2H53WSCXhoMThis feature was also requested recently: https://discuss.python.org/t/extend-pep-562-with-setattr-for-modules/25506
erlend-aasland commentedon Jun 23, 2023
Feature or bug? ;)
vstinner commentedon Jun 23, 2023
Feature request :-)
erlend-aasland commentedon Jun 23, 2023
Since, module
__getattr__and__dict__required a (small) PEP (PEP-562), I assume this feature should also get a (small) PEP :)skirpichev commentedon Jun 23, 2023
I did a draft PEP, with
__setattr__support only. Despite for some support from core devs, there are no sponsors yet.Let me know if it does make sense to extend this effort.
pythongh-106016: Support customizing of module attributes access with…
pythongh-106016: Add Lib/test/test_module/ directory
gh-106016: Add Lib/test/test_module/ directory (#108293)
pythongh-106016: Add Lib/test/test_module/ directory (pythonGH-108293)
pythongh-106016: Add Lib/test/test_module/ directory (python#108293)
pythongh-106016: Support customizing of module attributes access with…
[3.11] gh-106016: Add Lib/test/test_module/ directory (GH-108293) (#1…
[3.12] gh-106016: Add Lib/test/test_module/ directory (#108293) (#108306
[-]Support defining __setattr__() and __delattr__() in a module[/-][+]PEP 726: Support defining __setattr__() and __delattr__() in a module[/+]joaoe commentedon Nov 11, 2023
Hi.
I see there is a PEP for this
https://peps.python.org/pep-0726/
There is a use case I've encountered recently which would require a proper efficient support for setattr and that is the warnings module.
As you well know, it is not thread safe, and that module has a lot of monkeypatch going one replacing the showwarning function. Exactly what is described in that PEP.
Fixing that would require changing how catch_warnings() and filters work, so that no monkeypatching happens. However, that is very non-backwards compatible if 3rd parties are trying to do the same. A way to fix that would be to detect attributes being set in the module and issue an deprecation warning. Later fail with an error. And that requires a module.setattr. However, adding a class for such an important module could perhaps represent other performance or compatibility problems.
#91505
skirpichev commentedon Nov 12, 2023
@joaoe, I think this scenario at least partially mentioned in the discussion thread: https://discuss.python.org/t/32640/19. And also in the PEP (the last paragraph of the "Discussion" section), let me know if you feel this should be rephrased somehow. I'm going to submit a SC issue to ask for final decision.
Practical benefits for package or module maintainers are very important (see https://discuss.python.org/t/32640/17) to the fate of PEP. So, I'm suggesting to share your thoughts in the discussion thread.
vstinner commentedon Dec 20, 2023
The decision is now in the hands of the Steering Council: python/steering-council#216 For now, I prefer to close this issue. Another issue can be opened once the PEP will be approved.