Fixing circular imports PREMIUM

Let's talk about fixing circular imports in Python.

What does a circular import look like?

We have a Django project (called sparkle) with an application in it called users (Django apps are just special Python packages):

sparkle/
├── db.sqlite3
├── manage.py
├── requirements.txt
├── sparkle
│   ├── asgi.py
│   ├── celery.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── users
    ├── admin.py
    ├── apps.py
    ├── __init__.py
    ├── migrations
    │   ├── 0001_initial.py
    │   ├── __init__.py
    ├── models.py
    ├── tasks.py
    ├── tests.py
    └── views.py

That users app has a tasks module within it that contains Celery tasks.

Celery tasks are typically run in the background. For example, in this task (in users/tasks.py) we're sending a welcome email to a user:

from celery import shared_task
from django.core.mail import send_mail

from .models import User


@shared_task
def send_welcome_email(user_id):
    user = User.objects.get(id=user_id)
    send_mail(
        "Welcome to our site",
        "Hello there!\nWelcome to our site. We hope you enjoy using it.",
        "[email protected]",
        [user.email],
        fail_silently=False,
    )

We trigger this Celery task in our app's models module (users/models.py).

from django.contrib.auth.models import AbstractBaseUser, BaseUserManager
from django.db import models
from django.dispatch import receiver
from django.db.models.signals import post_save

from .tasks import send_welcome_email


class UserManager(BaseUserManager):
    def create_user(self, email, password=None, **extra_fields):
        if not email:
            raise ValueError("Users must have an email address")
        user = self.model(email=self.normalize_email(email), **extra_fields)
        user.set_password(password)
        user.save(using=self._db)
        return user

    def create_superuser(self, email, password, **extra_fields):
        user = self.create_user(email, password=password, **extra_fields)
        user.is_staff = True
        user.is_superuser = True
        user.save(using=self._db)
        return user


class User(AbstractBaseUser):
    email = models.EmailField(max_length=255, unique=True)
    is_active = models.BooleanField(default=True)
    is_staff = models.BooleanField(default=False)
    is_superuser = models.BooleanField(default=False)

    objects = UserManager()

    USERNAME_FIELD = "email"

    def __str__(self):
        return self.email


@receiver(post_save, sender=User)
def send_welcome_email_handler(sender, instance, created, **kwargs):
    if created:
        send_welcome_email.delay(instance.id)

We import the task (send_welcome_email) and then whenever a new user is created, which we're using Django's signals system to check for, we end up executing this task in the background (which is what that .delay call does):

@receiver(post_save, sender=User)
def send_welcome_email_handler(sender, instance, created, **kwargs):
    if created:
        send_welcome_email.delay(instance.id)

This code suffers from circular imports.

If we try to run Django's createsuperuser command, we'll see an error:

(venv) ~/sparkle $ python manage.py createsuperuser
Traceback (most recent call last):
  File "/home/trey/sparkle/manage.py", line 22, in <module>
    main()
  ...
  File "/home/trey/sparkle/users/models.py", line 6, in <module>
    from .tasks import send_welcome_email
  File "/home/trey/sparkle/users/tasks.py", line 4, in <module>
    from .models import User
ImportError: cannot import name 'User' from partially initialized module 'users.models' (most likely due to a circular import) (/home/trey/sparkle/users/models.py)

The error says we can't import User from a partially initialized module. In fact, Python even adds that this is most likely due to a circular import.

When do circular imports happen?

When a module is imported, all the code in that module is run, including the import statements:

from django.contrib.auth.models import AbstractBaseUser, BaseUserManager
from django.db import models
from django.dispatch import receiver
from django.db.models.signals import post_save

from .tasks import send_welcome_email

Those import statements are within our models module (users/models.py). When Python hits the line from .tasks import send_welcome_email, it'll then have to run the code in our tasks module (users/tasks.py) because importing modules runs code.

But the tasks module imports from the models module:

from celery import shared_task
from django.core.mail import send_mail

from .models import User

But if we try to import from our models module now, we'll have a problem because we haven't fully initialized the models module.

So we have a problem. These two modules depend on each other.

This is called a circular import or a dependency loop.

Now, circular imports usually involve just two modules, but it is possible for them to involve more.

For example if we have a requests package like this:

• requests/__init__.py imports from requests/api.py
• requests/api.py imports from requests/utils.py
• requests/utils.py imports from requests (implicitly importing from __init__.py)

We'd end up with a circular import because requests imports from requests.api which imports from requests.utils which imports from requests.

Fix: combining two modules into one

While it is good that Python is usually pretty smart at warning us about circular imports, how do we fix these things?

Circular imports between two modules typically indicate that the first module needs an object from the second, and the second needs an object from the first. We could resolve this problem by combining those modules into one.

Here's our new, combined users.models module:

from celery import shared_task
from django.contrib.auth.models import AbstractBaseUser, BaseUserManager
from django.core.mail import send_mail
from django.db import models
from django.dispatch import receiver
from django.db.models.signals import post_save


...


@receiver(post_save, sender=User)
def send_welcome_email_handler(sender, instance, created, **kwargs):
    if created:
        send_welcome_email.delay(instance.id)


@shared_task()
def send_welcome_email(user_id):
    user = User.objects.get(id=user_id)
    send_mail(
        "Welcome to our site",
        "Hello there!\nWelcome to our site. We hope you enjoy using it.",
        "[email protected]",
        [user.email],
        fail_silently=False,
    )

In our users.models module, we aren't importing from users.tasks anymore because we don't even have a tasks module! Instead, we've just stuck everything we need at the bottom of our models module.

While this does work, it's not the most elegant solution in our case. Combining two modules into one often makes for code that's a bit messier than it needs to be.

Fix: splitting one module into two

Often, a circular import can be fixed by moving some of your code into a new module and then importing from that new module instead.

Here, we've made a users.receivers module that has that Django signal receiver we were using before (at the bottom of users/models.py) to trigger that Celery task:

from django.db.models.signals import post_save
from django.dispatch import receiver

from .models import User
from .tasks import send_welcome_email


@receiver(post_save, sender=User)
def send_welcome_email_handler(sender, instance, created, **kwargs):
    if created:
        send_welcome_email.delay(instance.id)

Our tasks module is the same as it was originally now. It still imports from our users.models module:

from celery import shared_task
from django.core.mail import send_mail

from .models import User

But our models module no longer imports from users.tasks:

from django.contrib.auth.models import AbstractBaseUser, BaseUserManager
from django.db import models

...

class User(AbstractBaseUser):
    email = models.EmailField(max_length=255, unique=True)
    is_active = models.BooleanField(default=True)
    is_staff = models.BooleanField(default=False)
    is_superuser = models.BooleanField(default=False)

    objects = UserManager()

    USERNAME_FIELD = "email"

    def __str__(self):
        return self.email


import users.receivers

Although, it does import from users.receivers at the very bottom of our module. It does this only because Django needs to actually know about this code. This code does need to run somewhere, and so we've had to import it, but we stuck that import after everything else in our module.

It's very strange to see this, but putting an import at the bottom of a module can avoid circular dependencies. If that import was at the top of our module, we'd end up having a three-module circular dependency.

Another way to ensure our users.receivers module is imported by Django is to use the AppConfig class for our users app (in users.apps.py). Instead of importing users.receivers at the bottom of a module, we could add a ready method to our AppConfig object that has that import statement:

from django.apps import AppConfig


class UsersConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'users'

    def ready(self):
        import users.receivers

Regardless of how we handle this issue, splitting your modules up into a third module can sometimes fix the problem.

Fix: moving problematic code between modules

Another fix for circular dependencies could be to move our code.

We could move some of our code that's problematic into the other module to avoid a circular import.

Instead of putting our Django signal handler in our users.models module, we could put it in our users.tasks module:

from celery import shared_task
from django.core.mail import send_mail
from django.db.models.signals import post_save
from django.dispatch import receiver

from .models import User


@shared_task
def send_welcome_email(user_id):
    user = User.objects.get(id=user_id)
    send_mail(
        "Welcome to our site",
        "Hello there!\nWelcome to our site. We hope you enjoy using it.",
        "[email protected]",
        [user.email],
        fail_silently=False,
    )


@receiver(post_save, sender=User)
def send_welcome_email_handler(sender, instance, created, **kwargs):
    if created:
        send_welcome_email.delay(instance.id)

Again though, this feels a little weird.

It doesn't really seem appropriate that a Django signal receiver that checks whether a model object has been created is actually living in our users.tasks module, which is all about Celery tasks.

Moving chunks of code to a different module can sometimes fix the problem, and sometimes it is a pretty good solution, in our case, this doesn't seem like the most appropriate solution.

Fix: using "lazy imports"

If you only need to import a module in specific cases, one of the better solutions for circular imports is something called a lazy import.

A lazy import typically means putting an import statement inside a function.

Here, we've modified our users.models module to use a lazy import:

from django.contrib.auth.models import AbstractBaseUser, BaseUserManager
from django.db import models
from django.dispatch import receiver
from django.db.models.signals import post_save


...

@receiver(post_save, sender=User)
def send_welcome_email_handler(sender, instance, created, **kwargs):
    from .tasks import send_welcome_email
    if created:
        send_welcome_email.delay(instance.id)

We've left our tasks module unchanged, but we no longer import from tasks in our models module, at least not at the top of our module. Instead, we're importing from tasks inside our send_welcome_email_handler function.

This looks very weird, but it works! There's nothing stopping us from sticking an import statement inside a function in Python.

This is called a lazy import because we're only importing the module just before it's needed.

Lazy imports might seem a little bit sloppy, and they can get out of hand sometimes, but they can be a pretty reasonable approach to working around the issue of circular dependencies.

Lazy imports are often an easy fix

Sometimes circular imports are a sign of a design problem. When you hit a circular import situation, ask yourself "why do these two modules depend on each other?"

It might be that these two modules are so tightly coupled that they should really be just one module.

But it's more often the case that one or more of your modules is trying to do too much at once and moving some of your code into a third or fourth module might make sense. Dependency loops might be resolved if even one module in the loop stops importing from the other one and instead imports from a third module.

But if you're not quite ready to refactor how your modules are set up, a quick fix might be to use a lazy import instead.