sys.modules doc entry should reflect restrictions #56842

Activity

1. ericsnowcurrently commented on Jul 25, 2011

   

   ericsnowcurrently

   on Jul 25, 2011

   MemberAuthor

   More actions

   The sys.modules dict is a special object. It is the only variable of the CPython interpreter object that is exposed in the sys module[1]. Everything else in sys lives in the module. However, the modules dict lives in the interpreter object and is bound to the sys module separately. No other variable of the interpreter object gets this treatment.

   This situation sets up an unexpected behavior for sys.modules. There are many places, mostly in Python/import.c, where the modules dict gets used and not by pulling from sys.modules. These places use interp->modules directly[2]. So if sys.modules is re-bound, the imp module is using/reporting an out of sync modules dict.

   One could argue that re-binding a module global is risky and should be avoided. I agree. Here is the use case that prompted me to march ahead anyway:

   class BaseTest(TestCase):
       @classmethod
       def setUpClass(cls):
           cls.sysmodules = sys.modules
           sys.modules = sys.modules.copy()
       @classmethod
       def tearDownClass(cls):
           sys.modules = cls.sysmodules

   I was writing some import related tests and wanted sys.modules to be returned to its initial state after each test. I realise that Lib/test/support.py provides CleanImport and others address this, but you have to provide the module names to clean up. This is an unfortunate hassle sometimes when several layers of imports happen during the import of the module you care about.

   So the result was an exception when I tried importing an extension module, like "_sqlite3". This is because in importdl.h the new module is added to the dict returned by PyImport_GetModuleDict(), not to the one at sys.modules.

   For now I am doing the following to get the same effect:

   class BaseTest(TestCase):
       @classmethod
       def setUpClass(cls):
           cls.sysmodules = sys.modules.copy()
       @classmethod
       def tearDownClass(cls):
           for name in sys.modules:
               del sys.modules[name]
           for name in cls.sysmodules:
               sys.modules[name] = cls.sysmodules[name]

   However, this is less efficient, sort of. I expect that the current direct use of interp->modules in the CPython code is [much?] more efficient than PySys_GetObject("modules") calls.

   Proposal

   In light of all this I recommend that either use of interp->modules be replaced by PySys_GetObject("modules") calls, or the sys module documentation[3] be updated to make clear that sys.modules should not be re-bound (in a CPython implementation detail note). I'm guessing that the first option is right out. The documentation addition would be just right.

   [1] variables of the interpreter object found by grepping "interp->" in the CPython source:
   modules
   modules_by_index
   next
   codec_search_path
   codec_search_cache
   codec_error_registry
   codecs_initialized
   fscodec_initialized
   modules_reloading
   builtins
   sysdict
   tstate_head
   tscdump
   dlopenflags
   [2] see PyImport_GetModuleDict() in Python/import.c
   [3] http://docs.python.org/dev/library/sys.html#sys.modules
2. ericsnowcurrently

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jul 25, 2011
3. ncoghlan commented on Jul 25, 2011

   

   ncoghlan

   on Jul 25, 2011

   Contributor

   More actions

   +1 for making this limitation explicit. See the caveat on locals() [1] for an example of how to note this kind of restriction.

   [1] http://docs.python.org/dev/library/functions.html#locals
4. ericsnowcurrently commented on Jul 25, 2011

   

   ericsnowcurrently

   on Jul 25, 2011

   MemberAuthor

   More actions

   Would an implementation detail note be inappropriate here? I only ask because it looks like the imp module's use of interp->modules is implementation specific.

   Here's a patch for Doc/library/sys.rst that adds the note.
5. ericsnowcurrently

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   on Jul 25, 2011
6. ericsnowcurrently

   changed the title [-]sys.modules gets special treatment[/-] [+]sys.modules doc entry should reflect restrictions[/+] on Jul 25, 2011
7. ericsnowcurrently

   assigned

   docspython

   on Jul 25, 2011
8. merwok commented on Jul 29, 2011

   

   merwok

   on Jul 29, 2011

   Member

   More actions

   The note’s spirit is good, but I think something more concise would do.

   Side note: Please don’t mix up unrelated cosmetic changes in your diffs.
9. ericsnowcurrently commented on Apr 18, 2012

   

   ericsnowcurrently

   on Apr 18, 2012

   MemberAuthor

   More actions

   The original motivator: http://mail.python.org/pipermail/python-dev/2011-July/112497.html
10. ericsnowcurrently commented on Apr 18, 2012

    

    ericsnowcurrently

    on Apr 18, 2012

    MemberAuthor

    More actions

    also, bpo-14615 is related to making sys.modules authoritative.
11. ericsnowcurrently commented on Feb 17, 2013

    

    ericsnowcurrently

    on Feb 17, 2013

    MemberAuthor

    More actions

    One proposal would lead to the sys module growing descriptors:

    http://mail.python.org/pipermail/python-ideas/2013-January/019075.html

    In that case, sys.modules could update the underlying interp->modules.
12. ericsnowcurrently commented on Jun 25, 2013

    

    ericsnowcurrently

    on Jun 25, 2013

    MemberAuthor

    More actions

    bpo-17953 addressed part of this.
13. ericsnowcurrently commented on Sep 4, 2017

    

    ericsnowcurrently

    on Sep 4, 2017

    MemberAuthor

    More actions

    We're dropping PyInterpreterState.modules (bpo-28411).
14. ericsnowcurrently

    closed this as completedon Sep 4, 2017
15. ezio-melotti

    transferred this issue fromon Apr 10, 2022