Separating Celery application and worker in Docker containers

3 min readAug 23, 2020

Each node submits new tasks to a remote server where a postman service acts as a receiver. The tasks are added to a task queue to be consumed by worker services.

31 Jan 2022 Update: Fixes to support Celery 5

In this use case, there are multiple nodes physically located at different places. These nodes could be PCs, tablets, or mobile phones. Each node independently submits new tasks to the remote backend at irregular intervals. For example, one node may submit tasks at 20 minutes intervals while another node may submit tasks between 10 to 30 seconds intervals.

Sounds like a good fit for Celery.

Independent services

Ideally, the postman and the worker should be independent services.

Roles of postman service:

Centralized receiver
Aggregate tasks from various nodes
Distribute tasks to appropriate task queues

Roles of worker services:

Implement task processing logic
Scale out to different tasks
Scale up to handle high volume of tasks

After looking around, most of the articles found were variations of the Get Started guide that used the delay() method. Those examples typically had the task dispatcher and worker in the same process. Eventually, what caught my attention was the send_task() method. However, I could not find a good example implementation for it.

So, I decided to write one.

Decoupled Celery example

Source code: https://github.com/tanchinhiong/decoupled-celery-example

This example uses Docker Compose to create a network of services to illustrate the interactions between separated Celery application and worker services.

A minimal working example consists of:

Celery application (i.e. postman service)
Celery worker (i.e. worker service)
Broker (i.e. RabbitMQ)

A visualization tool is not strictly necessary, but very handy to have around, so I’ve included Flower as well for good measure. These 4 services are defined in a minimal docker-compose.yml.

To get started with this example, simply run make recompose.

Flower web UI (left) and terminal running docker-compose (right).

Under the hood

The worker service implements a simple add function that returns the sum of 2 numbers. The key here is the @app.task(name='addTask') decorator that informs Celery of a named task. The task name does not need to match the function name, so free feel to choose a representative name.

The application service dispatches tasks to the worker service by providing the task name and the task arguments in send_task('addTask', (i, 3)). In this example, 100 tasks are created for the numbers 1 to 100, and the worker service would add 3 to each of those numbers.

Ready to scale

You may have also noticed in the terminal image above that there are ForkPoolWorkers. This was because the worker service was initiated using the option --autoscale 10 in the Dockerfile which allowed Celery to scale up to 10 processes within the worker service. In the future, the same worker service can be scaled up by creating more service replicas. Furthermore, different worker services can be implemented to create a logical separation for other types of tasks (i.e. subtract, multiply, divide).

Conclusion

Recap:

Separate Celery application and worker as independent services
Use named tasks in worker services
Use send_task() in application service dispatch a named task
Use --autoscale option to get more worker processes for the same task
Create new worker services for new tasks

Hope this becomes a useful guide for separating application and worker services in your development.