PEP: 395

Title: Qualified Names for Modules

Version: $Revision$

Last-Modified: $Date$

Author: Nick Coghlan <ncoghlan@gmail.com>

Status: Withdrawn

Type: Standards Track

Topic: Packaging

Content-Type: text/x-rst

Created: 04-Mar-2011

Python-Version: 3.4

Post-History: 05-Mar-2011, 19-Nov-2011

PEP Withdrawal

==============

This PEP was withdrawn by the author in December 2013, as other significant

changes in the time since it was written have rendered several aspects

obsolete. Most notably :pep:`420` namespace packages rendered some of the

proposals related to package detection unworkable and :pep:`451` module

specifications resolved the multiprocessing issues and provide a possible

means to tackle the pickle compatibility issues.

A future PEP to resolve the remaining issues would still be appropriate,

but it's worth starting any such effort as a fresh PEP restating the

remaining problems in an updated context rather than trying to build on

this one directly.

Abstract

========

This PEP proposes new mechanisms that eliminate some longstanding traps for

the unwary when dealing with Python's import system, as well as serialisation

and introspection of functions and classes.

It builds on the "Qualified Name" concept defined in :pep:`3155`.

Relationship with Other PEPs

----------------------------

Most significantly, this PEP is currently deferred as it requires

significant changes in order to be made compatible with the removal

of mandatory __init__.py files in :pep:`420` (which has been implemented

and released in Python 3.3).

This PEP builds on the "qualified name" concept introduced by :pep:`3155`, and

also shares in that PEP's aim of fixing some ugly corner cases when dealing

with serialisation of arbitrary functions and classes.

It also builds on :pep:`366`, which took initial tentative steps towards making

explicit relative imports from the main module work correctly in at least

*some* circumstances.

Finally, :pep:`328` eliminated implicit relative imports from imported modules.

This PEP proposes that the de facto implicit relative imports from main

modules that are provided by the current initialisation behaviour for

``sys.path[0]`` also be eliminated.

What's in a ``__name__``?

=========================

Over time, a module's ``__name__`` attribute has come to be used to handle a

number of different tasks.

The key use cases identified for this module attribute are:

1. Flagging the main module in a program, using the ``if __name__ ==

"__main__":`` convention.

2. As the starting point for relative imports

3. To identify the location of function and class definitions within the

running application

4. To identify the location of classes for serialisation into pickle objects

which may be shared with other interpreter instances

Traps for the Unwary

====================

The overloading of the semantics of ``__name__``, along with some historically

associated behaviour in the initialisation of ``sys.path[0]``, has resulted in

several traps for the unwary. These traps can be quite annoying in practice,

as they are highly unobvious (especially to beginners) and can cause quite

confusing behaviour.

Why are my imports broken?

--------------------------

There's a general principle that applies when modifying ``sys.path``: *never*

put a package directory directly on ``sys.path``. The reason this is

problematic is that every module in that directory is now potentially

accessible under two different names: as a top level module (since the

package directory is on ``sys.path``) and as a submodule of the package (if

the higher level directory containing the package itself is also on

``sys.path``).

As an example, Django (up to and including version 1.3) is guilty of setting

up exactly this situation for site-specific applications - the application

ends up being accessible as both ``app`` and ``site.app`` in the module

namespace, and these are actually two *different* copies of the module. This

is a recipe for confusion if there is any meaningful mutable module level

state, so this behaviour is being eliminated from the default site set up in

version 1.4 (site-specific apps will always be fully qualified with the site

name).

However, it's hard to blame Django for this, when the same part of Python

responsible for setting ``__name__ = "__main__"`` in the main module commits

the exact same error when determining the value for ``sys.path[0]``.

The impact of this can be seen relatively frequently if you follow the

"python" and "import" tags on Stack Overflow. When I had the time to follow

it myself, I regularly encountered people struggling to understand the

behaviour of straightforward package layouts like the following (I actually

use package layouts along these lines in my own projects)::

project/

setup.py

example/

__init__.py

foo.py

tests/

__init__.py

test_foo.py

While I would often see it without the ``__init__.py`` files first, that's a

trivial fix to explain. What's hard to explain is that all of the following

ways to invoke ``test_foo.py`` *probably won't work* due to broken imports

(either failing to find ``example`` for absolute imports, complaining

about relative imports in a non-package or beyond the toplevel package for

explicit relative imports, or issuing even more obscure errors if some other

submodule happens to shadow the name of a top-level module, such as an

``example.json`` module that handled serialisation or an

``example.tests.unittest`` test runner)::

# These commands will most likely *FAIL*, even if the code is correct

# working directory: project/example/tests

./test_foo.py

python test_foo.py

python -m package.tests.test_foo

python -c "from package.tests.test_foo import main; main()"

# working directory: project/package

tests/test_foo.py

python tests/test_foo.py

python -m package.tests.test_foo

python -c "from package.tests.test_foo import main; main()"

# working directory: project

example/tests/test_foo.py

python example/tests/test_foo.py

# working directory: project/..

project/example/tests/test_foo.py

python project/example/tests/test_foo.py

# The -m and -c approaches don't work from here either, but the failure

# to find 'package' correctly is easier to explain in this case

That's right, that long list is of all the methods of invocation that will

almost certainly *break* if you try them, and the error messages won't make

any sense if you're not already intimately familiar not only with the way

Python's import system works, but also with how it gets initialised.

For a long time, the only way to get ``sys.path`` right with that kind of

setup was to either set it manually in ``test_foo.py`` itself (hardly

something a novice, or even many veteran, Python programmers are going to

know how to do) or else to make sure to import the module instead of

executing it directly::

# working directory: project

python -c "from package.tests.test_foo import main; main()"

Since the implementation of :pep:`366` (which defined a mechanism that allows

relative imports to work correctly when a module inside a package is executed

via the ``-m`` switch), the following also works properly::

# working directory: project

python -m package.tests.test_foo

The fact that most methods of invoking Python code from the command line

break when that code is inside a package, and the two that do work are highly

sensitive to the current working directory is all thoroughly confusing for a

beginner. I personally believe it is one of the key factors leading

to the perception that Python packages are complicated and hard to get right.

This problem isn't even limited to the command line - if ``test_foo.py`` is

open in Idle and you attempt to run it by pressing F5, or if you try to run

it by clicking on it in a graphical filebrowser, then it will fail in just

the same way it would if run directly from the command line.

There's a reason the general "no package directories on ``sys.path``"

guideline exists, and the fact that the interpreter itself doesn't follow

it when determining ``sys.path[0]`` is the root cause of all sorts of grief.

In the past, this couldn't be fixed due to backwards compatibility concerns.

However, scripts potentially affected by this problem will *already* require

fixes when porting to the Python 3.x (due to the elimination of implicit

relative imports when importing modules normally). This provides a convenient

opportunity to implement a corresponding change in the initialisation

semantics for ``sys.path[0]``.

Importing the main module twice

-------------------------------

Another venerable trap is the issue of importing ``__main__`` twice. This

occurs when the main module is also imported under its real name, effectively

creating two instances of the same module under different names.

If the state stored in ``__main__`` is significant to the correct operation

of the program, or if there is top-level code in the main module that has

non-idempotent side effects, then this duplication can cause obscure and

surprising errors.

In a bit of a pickle

--------------------

Something many users may not realise is that the ``pickle`` module sometimes

relies on the ``__module__`` attribute when serialising instances of arbitrary

classes. So instances of classes defined in ``__main__`` are pickled that way,

and won't be unpickled correctly by another python instance that only imported

that module instead of running it directly. This behaviour is the underlying

reason for the advice from many Python veterans to do as little as possible

in the ``__main__`` module in any application that involves any form of

object serialisation and persistence.

Similarly, when creating a pseudo-module (see next paragraph), pickles rely

on the name of the module where a class is actually defined, rather than the

officially documented location for that class in the module hierarchy.

For the purposes of this PEP, a "pseudo-module" is a package designed like

the Python 3.2 ``unittest`` and ``concurrent.futures`` packages. These

packages are documented as if they were single modules, but are in fact

internally implemented as a package. This is *supposed* to be an

implementation detail that users and other implementations don't need to

worry about, but, thanks to ``pickle`` (and serialisation in general),

the details are often exposed and can effectively become part of the public

API.

While this PEP focuses specifically on ``pickle`` as the principal

serialisation scheme in the standard library, this issue may also affect

other mechanisms that support serialisation of arbitrary class instances

and rely on ``__module__`` attributes to determine how to handle

deserialisation.

Where's the source?

-------------------

Some sophisticated users of the pseudo-module technique described

above recognise the problem with implementation details leaking out via the

``pickle`` module, and choose to address it by altering ``__name__`` to refer

to the public location for the module before defining any functions or classes

(or else by modifying the ``__module__`` attributes of those objects after

they have been defined).

This approach is effective at eliminating the leakage of information via

pickling, but comes at the cost of breaking introspection for functions and

classes (as their ``__module__`` attribute now points to the wrong place).

Forkless Windows

----------------

To get around the lack of ``os.fork`` on Windows, the ``multiprocessing``

module attempts to re-execute Python with the same main module, but skipping

over any code guarded by ``if __name__ == "__main__":`` checks. It does the

best it can with the information it has, but is forced to make assumptions

that simply aren't valid whenever the main module isn't an ordinary directly

executed script or top-level module. Packages and non-top-level modules

executed via the ``-m`` switch, as well as directly executed zipfiles or

directories, are likely to make multiprocessing on Windows do the wrong thing

(either quietly or noisily, depending on application details) when spawning a

new process.

While this issue currently only affects Windows directly, it also impacts

any proposals to provide Windows-style "clean process" invocation via the

multiprocessing module on other platforms.

Qualified Names for Modules

===========================

To make it feasible to fix these problems once and for all, it is proposed

to add a new module level attribute: ``__qualname__``. This abbreviation of

"qualified name" is taken from :pep:`3155`, where it is used to store the naming

path to a nested class or function definition relative to the top level

module.

For modules, ``__qualname__`` will normally be the same as ``__name__``, just

as it is for top-level functions and classes in :pep:`3155`. However, it will

differ in some situations so that the above problems can be addressed.

Specifically, whenever ``__name__`` is modified for some other purpose (such

as to denote the main module), then ``__qualname__`` will remain unchanged,

allowing code that needs it to access the original unmodified value.

If a module loader does not initialise ``__qualname__`` itself, then the

import system will add it automatically (setting it to the same value as

``__name__``).

Alternative Names

-----------------

Two alternative names were also considered for the new attribute: "full name"

(``__fullname__``) and "implementation name" (``__implname__``).

Either of those would actually be valid for the use case in this PEP.

However, as a meta-issue, :pep:`3155` is *also* adding a new attribute (for

functions and classes) that is "like ``__name__``, but different in some cases

where ``__name__`` is missing necessary information" and those terms aren't

accurate for the :pep:`3155` function and class use case.

:pep:`3155` deliberately omits the module information, so the term "full name"

is simply untrue, and "implementation name" implies that it may specify an

object other than that specified by ``__name__``, and that is never the

case for :pep:`3155` (in that PEP, ``__name__`` and ``__qualname__`` always

refer to the same function or class, it's just that ``__name__`` is

insufficient to accurately identify nested functions and classes).

Since it seems needlessly inconsistent to add *two* new terms for attributes

that only exist because backwards compatibility concerns keep us from

changing the behaviour of ``__name__`` itself, this PEP instead chose to

adopt the :pep:`3155` terminology.

If the relative inscrutability of "qualified name" and ``__qualname__``

encourages interested developers to look them up at least once rather than

assuming they know what they mean just from the name and guessing wrong,

that's not necessarily a bad outcome.

Besides, 99% of Python developers should never need to even care these extra

attributes exist - they're really an implementation detail to let us fix a

few problematic behaviours exhibited by imports, pickling and introspection,

not something people are going to be dealing with on a regular basis.

Eliminating the Traps

=====================

The following changes are interrelated and make the most sense when

considered together. They collectively either completely eliminate the traps

for the unwary noted above, or else provide straightforward mechanisms for

dealing with them.

A rough draft of some of the concepts presented here was first posted on the

python-ideas list ([1]_), but they have evolved considerably since first being

discussed in that thread. Further discussion has subsequently taken place on

the import-sig mailing list ([2]_. [3]_).

Fixing main module imports inside packages

------------------------------------------

To eliminate this trap, it is proposed that an additional filesystem check be

performed when determining a suitable value for ``sys.path[0]``. This check

will look for Python's explicit package directory markers and use them to find

the appropriate directory to add to ``sys.path``.

The current algorithm for setting ``sys.path[0]`` in relevant cases is roughly

as follows::

# Interactive prompt, -m switch, -c switch

sys.path.insert(0, '')

::

# Valid sys.path entry execution (i.e. directory and zip execution)

sys.path.insert(0, sys.argv[0])

::

# Direct script execution

sys.path.insert(0, os.path.dirname(sys.argv[0]))

It is proposed that this initialisation process be modified to take

package details stored on the filesystem into account::

# Interactive prompt, -m switch, -c switch

in_package, path_entry, _ignored = split_path_module(os.getcwd(), '')

if in_package:

sys.path.insert(0, path_entry)

else:

sys.path.insert(0, '')

# Start interactive prompt or run -c command as usual

# __main__.__qualname__ is set to "__main__"

# The -m switches uses the same sys.path[0] calculation, but:

# modname is the argument to the -m switch

# modname is passed to ``runpy._run_module_as_main()`` as usual

# __main__.__qualname__ is set to modname

::

# Valid sys.path entry execution (i.e. directory and zip execution)

modname = "__main__"

path_entry, modname = split_path_module(sys.argv[0], modname)

sys.path.insert(0, path_entry)

# modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``

# __main__.__qualname__ is set to modname

::

# Direct script execution

in_package, path_entry, modname = split_path_module(sys.argv[0])

sys.path.insert(0, path_entry)

if in_package:

# Pass modname to ``runpy._run_module_as_main()``

else:

# Run script directly

# __main__.__qualname__ is set to modname

The ``split_path_module()`` supporting function used in the above pseudo-code

would have the following semantics::

def _splitmodname(fspath):

path_entry, fname = os.path.split(fspath)

modname = os.path.splitext(fname)[0]

return path_entry, modname

def _is_package_dir(fspath):

return any(os.exists("__init__" + info[0]) for info

in imp.get_suffixes())

def split_path_module(fspath, modname=None):

"""Given a filesystem path and a relative module name, determine an

appropriate sys.path entry and a fully qualified module name.

Returns a 3-tuple of (package_depth, fspath, modname). A reported

package depth of 0 indicates that this would be a top level import.

If no relative module name is given, it is derived from the final

component in the supplied path with the extension stripped.

"""

if modname is None:

fspath, modname = _splitmodname(fspath)

package_depth = 0

while _is_package_dir(fspath):

fspath, pkg = _splitmodname(fspath)

modname = pkg + '.' + modname

return package_depth, fspath, modname

This PEP also proposes that the ``split_path_module()`` functionality be

exposed directly to Python users via the ``runpy`` module.

With this fix in place, and the same simple package layout described earlier,

*all* of the following commands would invoke the test suite correctly::

# working directory: project/example/tests

./test_foo.py

python test_foo.py

python -m package.tests.test_foo

python -c "from .test_foo import main; main()"

python -c "from ..tests.test_foo import main; main()"

python -c "from package.tests.test_foo import main; main()"

# working directory: project/package

tests/test_foo.py

python tests/test_foo.py

python -m package.tests.test_foo

python -c "from .tests.test_foo import main; main()"

python -c "from package.tests.test_foo import main; main()"

# working directory: project

example/tests/test_foo.py

python example/tests/test_foo.py

python -m package.tests.test_foo

python -c "from package.tests.test_foo import main; main()"

# working directory: project/..

project/example/tests/test_foo.py

python project/example/tests/test_foo.py

# The -m and -c approaches still don't work from here, but the failure

# to find 'package' correctly is pretty easy to explain in this case

With these changes, clicking Python modules in a graphical file browser

should always execute them correctly, even if they live inside a package.

Depending on the details of how it invokes the script, Idle would likely also

be able to run ``test_foo.py`` correctly with F5, without needing any Idle

specific fixes.

Optional addition: command line relative imports

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With the above changes in place, it would be a fairly minor addition to allow

explicit relative imports as arguments to the ``-m`` switch::

# working directory: project/example/tests

python -m .test_foo

python -m ..tests.test_foo

# working directory: project/example/

python -m .tests.test_foo

With this addition, system initialisation for the ``-m`` switch would change

as follows::

# -m switch (permitting explicit relative imports)

in_package, path_entry, pkg_name = split_path_module(os.getcwd(), '')

qualname= <<arguments to -m switch>>

if qualname.startswith('.'):

modname = qualname

while modname.startswith('.'):

modname = modname[1:]

pkg_name, sep, _ignored = pkg_name.rpartition('.')

if not sep:

raise ImportError("Attempted relative import beyond top level package")

qualname = pkg_name + '.' modname

if in_package:

sys.path.insert(0, path_entry)

else:

sys.path.insert(0, '')

# qualname is passed to ``runpy._run_module_as_main()``

# _main__.__qualname__ is set to qualname

Compatibility with PEP 382

~~~~~~~~~~~~~~~~~~~~~~~~~~

Making this proposal compatible with the :pep:`382` namespace packaging PEP is

trivial. The semantics of ``_is_package_dir()`` are merely changed to be::

def _is_package_dir(fspath):

return (fspath.endswith(".pyp") or

any(os.exists("__init__" + info[0]) for info

in imp.get_suffixes()))

Incompatibility with PEP 402

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:pep:`402` proposes the elimination of explicit markers in the file system for

Python packages. This fundamentally breaks the proposed concept of being able

to take a filesystem path and a Python module name and work out an unambiguous

mapping to the Python module namespace. Instead, the appropriate mapping

would depend on the current values in ``sys.path``, rendering it impossible

to ever fix the problems described above with the calculation of

``sys.path[0]`` when the interpreter is initialised.

While some aspects of this PEP could probably be salvaged if :pep:`402` were

adopted, the core concept of making import semantics from main and other

modules more consistent would no longer be feasible.

This incompatibility is discussed in more detail in the relevant import-sig

threads ([2]_, [3]_).

Potential incompatibilities with scripts stored in packages

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The proposed change to ``sys.path[0]`` initialisation *may* break some

existing code. Specifically, it will break scripts stored in package

directories that rely on the implicit relative imports from ``__main__`` in

order to run correctly under Python 3.

While such scripts could be imported in Python 2 (due to implicit relative

imports) it is already the case that they cannot be imported in Python 3,

as implicit relative imports are no longer permitted when a module is

imported.

By disallowing implicit relatives imports from the main module as well,

such modules won't even work as scripts with this PEP. Switching them

over to explicit relative imports will then get them working again as

both executable scripts *and* as importable modules.

To support earlier versions of Python, a script could be written to use

different forms of import based on the Python version::

if __name__ == "__main__" and sys.version_info < (3, 3):

import peer # Implicit relative import

else:

from . import peer # explicit relative import

Fixing dual imports of the main module

--------------------------------------

Given the above proposal to get ``__qualname__`` consistently set correctly

in the main module, one simple change is proposed to eliminate the problem

of dual imports of the main module: the addition of a ``sys.metapath`` hook

that detects attempts to import ``__main__`` under its real name and returns

the original main module instead::

class AliasImporter:

def __init__(self, module, alias):

self.module = module

self.alias = alias

def __repr__(self):

fmt = "{0.__class__.__name__}({0.module.__name__}, {0.alias})"

return fmt.format(self)

def find_module(self, fullname, path=None):

if path is None and fullname == self.alias:

return self

return None

def load_module(self, fullname):

if fullname != self.alias:

raise ImportError("{!r} cannot load {!r}".format(self, fullname))

return self.main_module

This metapath hook would be added automatically during import system

initialisation based on the following logic::

main = sys.modules["__main__"]

if main.__name__ != main.__qualname__:

sys.metapath.append(AliasImporter(main, main.__qualname__))

This is probably the least important proposal in the PEP - it just

closes off the last mechanism that is likely to lead to module duplication

after the configuration of ``sys.path[0]`` at interpreter startup is

addressed.

Fixing pickling without breaking introspection

----------------------------------------------

To fix this problem, it is proposed to make use of the new module level

``__qualname__`` attributes to determine the real module location when

``__name__`` has been modified for any reason.

In the main module, ``__qualname__`` will automatically be set to the main

module's "real" name (as described above) by the interpreter.

Pseudo-modules that adjust ``__name__`` to point to the public namespace will

leave ``__qualname__`` untouched, so the implementation location remains readily

accessible for introspection.

If ``__name__`` is adjusted at the top of a module, then this will

automatically adjust the ``__module__`` attribute for all functions and

classes subsequently defined in that module.

Since multiple submodules may be set to use the same "public" namespace,

functions and classes will be given a new ``__qualmodule__`` attribute

that refers to the ``__qualname__`` of their module.

This isn't strictly necessary for functions (you could find out their

module's qualified name by looking in their globals dictionary), but it is

needed for classes, since they don't hold a reference to the globals of

their defining module. Once a new attribute is added to classes, it is

more convenient to keep the API consistent and add a new attribute to

functions as well.

These changes mean that adjusting ``__name__`` (and, either directly or

indirectly, the corresponding function and class ``__module__`` attributes)

becomes the officially sanctioned way to implement a namespace as a package,

while exposing the API as if it were still a single module.

All serialisation code that currently uses ``__name__`` and ``__module__``

attributes will then avoid exposing implementation details by default.

To correctly handle serialisation of items from the main module, the class

and function definition logic will be updated to also use ``__qualname__``

for the ``__module__`` attribute in the case where ``__name__ == "__main__"``.

With ``__name__`` and ``__module__`` being officially blessed as being used

for the *public* names of things, the introspection tools in the standard

library will be updated to use ``__qualname__`` and ``__qualmodule__``

where appropriate. For example:

- ``pydoc`` will report both public and qualified names for modules

- ``inspect.getsource()`` (and similar tools) will use the qualified names

that point to the implementation of the code

- additional ``pydoc`` and/or ``inspect`` APIs may be provided that report

all modules with a given public ``__name__``.

Fixing multiprocessing on Windows

---------------------------------

With ``__qualname__`` now available to tell ``multiprocessing`` the real

name of the main module, it will be able to simply include it in the

serialised information passed to the child process, eliminating the

need for the current dubious introspection of the ``__file__`` attribute.

For older Python versions, ``multiprocessing`` could be improved by applying

the ``split_path_module()`` algorithm described above when attempting to

work out how to execute the main module based on its ``__file__`` attribute.

Explicit relative imports

=========================

This PEP proposes that ``__package__`` be unconditionally defined in the

main module as ``__qualname__.rpartition('.')[0]``. Aside from that, it

proposes that the behaviour of explicit relative imports be left alone.

In particular, if ``__package__`` is not set in a module when an explicit

relative import occurs, the automatically cached value will continue to be

derived from ``__name__`` rather than ``__qualname__``. This minimises any

backwards incompatibilities with existing code that deliberately manipulates

relative imports by adjusting ``__name__`` rather than setting ``__package__``

directly.

This PEP does *not* propose that ``__package__`` be deprecated. While it is

technically redundant following the introduction of ``__qualname__``, it just

isn't worth the hassle of deprecating it within the lifetime of Python 3.x.

Reference Implementation

========================

None as yet.

References

==========

.. [1] Module aliases and/or "real names"

(https://mail.python.org/pipermail/python-ideas/2011-January/008983.html)

.. [2] PEP 395 (Module aliasing) and the namespace PEPs

(https://mail.python.org/pipermail/import-sig/2011-November/000382.html)

.. [3] Updated PEP 395 (aka "Implicit Relative Imports Must Die!")

(https://mail.python.org/pipermail/import-sig/2011-November/000397.html)

.. [4] Elaboration of compatibility problems between this PEP and PEP 402

(https://mail.python.org/pipermail/import-sig/2011-November/000403.html)

Copyright

=========

This document has been placed in the public domain.

..

Local Variables:

mode: indented-text

indent-tabs-mode: nil

sentence-end-double-space: t

fill-column: 70

End: