Repository navigation
attribute error due to circular import #40584
Description
| BPO | 992389 |
|---|---|
| Nosy | @tim-one, @loewis, @brettcannon, @pjeby, @ncoghlan, @larryhastings, @merwok, @florentx, @ericsnowcurrently |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2014-11-14.01:55:49.659>
created_at = <Date 2004-07-16.15:09:33.000>
labels = ['interpreter-core', 'type-feature']
title = 'attribute error due to circular import'
updated_at = <Date 2014-11-14.01:56:17.085>
user = 'https://bugs.python.org/dcjim'bugs.python.org fields:
activity = <Date 2014-11-14.01:56:17.085>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2014-11-14.01:55:49.659>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2004-07-16.15:09:33.000>
creator = 'dcjim'
dependencies = []
files = ['1344', '33177']
hgrepos = []
issue_num = 992389
keywords = ['patch']
message_count = 25.0
messages = ['21663', '21664', '21665', '21666', '21667', '65198', '65337', '65436', '65439', '84767', '84799', '84803', '84844', '84922', '84966', '84993', '92114', '92115', '92116', '145614', '175592', '186889', '206433', '206475', '231147']
nosy_count = 14.0
nosy_names = ['tim.peters', 'loewis', 'jhylton', 'brett.cannon', 'dcjim', 'pje', 'ncoghlan', 'Rhamphoryncus', 'bronger', 'larry', 'eric.araujo', 'flox', 'sbt', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'needs patch'
status = 'closed'
superseder = '17636'
type = 'enhancement'
url = 'https://bugs.python.org/issue992389'
versions = ['Python 3.4']
Activity
dcjim commentedon Jul 16, 2004
This bug applied to 2.3 and 2.4. It probably applies to
earlier versions, but who cares? :)
Under some circumstances, code like:
fails with an attribute error for "foo" if foo is still
being imported.
I've attached a zip file of a demo package "eek" that
demonstrates the problem. If you unzip the package and:
you'll get the attribute error described above.
I think the problem is that eek's foo attribute isn't
set until the import of foo is finished. This is too late.
jhylton commentedon Nov 7, 2004
Logged In: YES
user_id=31392
Are the semantics of import clear enough to confirm that
this is a bug? Specifically, this seems to revolve around
circular imports and code in __init__. I agree that this
behavior is confusing, but I don't know if that has more to
do with the nature of circular imports than any bug.
ncoghlan commentedon Nov 15, 2004
Logged In: YES
user_id=1038590
If the semantics aren't clear, then isn't that a bug in
itself? If the behaviour of Jim's code is officially
undefined, then the docs need to say so.
A simple rule would seem to be "a package should not try to
import itself or any subpackages with an absolute import in
its __init__.py file".
Given the nature of __all__ and __path__ for packages, I
can't see a way to safely set eek's foo attribute before
foo's __init__.py has been processed.
tim-one commentedon Nov 16, 2004
Logged In: YES
user_id=31435
Nick, the semantics of circular imports aren't clear even if no
packages are involved. Note that the Reference manual is
silent about such cases. In effect, it's defined by the
implementation, and some people think they know how that
works -- although nobody has credibly claimed to fully
understand the implementation consequences since Gordon
McMillan vanished <0.5 wink>.
While I expect this bug report to sit here for years (it's hard
to imagine anyone caring enough to devote the time needed
to untangle this subsystem), I'll note in passing that this
case "works" if bar.py uses relative import instead; i.e., if it
begins with
instead of with
Then it stops referencing attributes that don't exist before
the whole chain of imports completes.
loewis commentedon Jul 3, 2006
Logged In: YES
user_id=21627
Lowering the priority, as this apparently is not a
high-priority issue.
ncoghlan commentedon Apr 8, 2008
I think the lowered priority got lost somewhere along the line.
bronger commentedon Apr 11, 2008
I have a very similar issue (maybe the same?) at the moment.
Assume the follwing package structure:
main.py
package/
__init__.py [empty]
moduleX.py
moduleY.py
main.py says:
moduleX.py says:
and moduleY.py says:
However, this doesn't work:
If I turn the relative imports to absolutes ones, it works. But I'd
prefer the relative notation for intra-package imports. That's their
purpose after all.
If you split a large module into chunks, cyclic imports are hardly
avoidable (and there's nothing bad about it; it worked fine before PEP-328).
Note that "import absolute.path.to.module as short" doesn't work either.
So currently, in presence of cyclic imports in a package, the only
remedy is to use the full absolute paths everywhere in the source code,
which is really awkward in my opinion.
ncoghlan commentedon Apr 13, 2008
This is actually a pretty tough problem - fixing it would involve some
fairly subtle changes to the way imports from packages are handled.
Given that I'm of the opinion that permitting circular imports in a code
base is an extraordinarily bad coding practice (if I'm ever tempted to
create a circular import, I consider it a sign that I need to separate
out some of the common code into a utility module), I'm not personally
going to be putting any effort into solving it (although I wouldn't
necessarily oppose anyone else trying to fix it).
However, I'll give a detailed description of the problem and a possible
solution in case anyone else wants to tackle it (since I believe there's
already a similar trick done for the sys.modules cache).
At the moment, when resolving an import chain __import__ only sets the
parent package's attribute for the submodule after the submodule import
is complete. This is what Jim describes in his original post, and is the
cause of the failure to resolve the name. Deferring the lookups solves
the problem because it means the package attributes are checked only
after the whole import chain is complete, instead of trying to get
access to a half-executed module during the import itself.
The most likely solution to the problem would be to change the attribute
on the parent package to be handled in a fashion closer to the way the
sys.modules cache is handled: set the attribute on the parent package
*before* executing the module's code, and delete that attribute if a
problem is encountered with the import.
The consequence of this would be that circular imports would be
permitted, although the module's imported in this fashion would be seen
in a half constructed state. So instead of the import failing with an
exception (which is what happens now), you would instead get a module
which you can't actually use (since most its attributes won't actually
be filled in yet, as the circular imports will normally occur near the
top of the file). Attempts to use methods, attributes and functions from
the module may or may not work depending on the order in which the
original module does things.
A clean failure indicating "You have a circular import, get rid of it"
seems better to me than possible hard to diagnose bugs due to being able
to retrieve things from a half-constructed module, but opinions
obviously differ on that. However, I really don't see this changing
without a PEP.
16 remaining items
ncoghlan commentedon Oct 16, 2011
Changed the issue title to state clearly that the core issue is with circular imports that attempt to reference module contents at import time, regardless of the syntactic form used. All of the following module level code can fail due to this problem:
[-]attribute error after non-from import[/-][+]attribute error due to circular import[/+][-]attribute error after non-from import[/-][+]attribute error due to circular import[/+]sbt commentedon Nov 14, 2012
In Torsten's example
can be replaced with
or
If that is not pretty enough then perhaps the new syntax
could be introduced and made equivalent to (*).
ncoghlan commentedon Apr 14, 2013
The implementation of issue bpo-17636 (making IMPORT_FROM fall back to sys.modules when appropriate) will make "import x.y" and "from x import y" equivalent for resolution purposes during import.
That covers off the subset of circular references that we want to allow, so I'm closing this one in favour of the more precisely defined proposal.
ncoghlan commentedon Dec 17, 2013
I'm reopening this, since PEP-451 opens up new options for dealing with it (at least for loaders that export the PEP-451 APIs rather than only the legacy loader API, which now includes all the standard loaders other than the ones for builtins and extension modules)
The attached patch doesn't have an automated test and is quite rough around the edges (as the additional check for the parent being in sys.modules confuses a couple of the importlib tests), but it proves the concept by making the following work:
[ncoghlan@lancre py3k]$ cd ../play [ncoghlan@lancre play]$ mkdir issue992389 [ncoghlan@lancre play]$ cat > issue992389/mod.py from . import mod print("Success!") [ncoghlan@lancre play]$ python3 -c "import issue992389.mod" Traceback (most recent call last): File "<string>", line 1, in <module> File "./issue992389/mod.py", line 1, in <module> from . import mod ImportError: cannot import name mod [ncoghlan@lancre play]$ ../py3k/python -c "import issue992389.mod" Success!pjeby commentedon Dec 17, 2013
The new patch will have weird results in the case of a parent module that defines an attribute that's later replaced by an import, e.g. if foo/init.py defines a variable 'bar' that's a proxy for the foo.bar module. This is especially problematic if this proxy is used during the process of importing foo.bar
At the very least, this code should NOT be deleting the original foo.bar attribute, but rather restoring its previous value.
All in all, I don't think this is a productive route to take. It was discussed on Python-dev previously and IIRC I outlined all the other reasons why back then. The approach in bpo-17636 is the only one that doesn't change the semantics of any existing, not-currently-broken code.
In contrast, the proposed change here introduces new side-effects and *more* volatile state and temporal coupling. I don't think this should go in, since the other approach *only* affects execution paths that would currently raise ImportError.
ncoghlan commentedon Nov 14, 2014
Belatedly agreeing with PJE on this one.
If concerns around circular imports continue to be raised in a post Python 3.5 world (with the bpo-17636 change), then we can look at revisiting this again, but in the meantime, lets just go with the sys.modules fallback.