Designing a Scalable Notification System: From Basics to Fan-Out Architectures

Notifications are everywhere — whether it’s Slack pinging you about a new message, Instagram reminding you of a post, or your bank alerting you about a transaction. While they seem simple on the surface, designing a scalable and reliable notification system is one of the most complex backend engineering challenges.

This blog will break down the essentials of notification system design — starting from a simple welcome email, moving towards asynchronous processing, event-driven architectures, fan-out design patterns, ingestion mechanisms, and finally exploring how Slack decides what notifications to send.

🌱 The Basics: A Simple Notification Flow

Let’s start with the simplest case.

• A user signs up on your app.
• The app stores their details in the database.
• Immediately, an email API (like Gmail or SES) is called to send a “Welcome” email.

Flow:

User Signup → App Server → Email API → User Inbox

Sounds straightforward, right? But here’s the catch — this naive design doesn’t scale.

⚠️ Challenges in Scaling

1. Latency — External APIs like Gmail can take seconds to respond. A signup shouldn’t wait that long.
2. Failures — If Gmail is down, your signup flow breaks.
3. Rate Limits — Email providers often limit requests (e.g., 100 requests/second). If 10,000 users sign up in a spike, most won’t receive their emails.

These issues make synchronous notification handling infeasible for real-world applications.

🔄 Asynchronous Processing with Queues

To solve this, notifications must move out of the critical path of user actions.

Instead of sending emails synchronously, we push requests into a queue.

User Signup → App Server → Queue → Worker → Email API → User Inbox

• When we move to an asynchronous notification system, a few important components come into play:

1. App Server → Enqueues the Event

• Instead of sending an email directly (which is slow), the app server just places the request into a queue.
• Think of it like dropping a task into a “to-do box” and moving on — this makes signup super fast.

2. Worker → Processes Requests in the Background

• A worker is a dedicated service or microservice whose only job is to pick events from the queue and process them (e.g., send emails, SMS, or push notifications).
• Since it runs separately, it doesn’t block the main application.

3. Retries → Automatic Reprocessing on Failures

• If sending an email fails (due to a temporary network issue or API downtime), the worker can retry the request after some delay.
• This improves reliability without bothering the user.

4. Dead Letter Queue (DLQ) → Final Safety Net

• If a request fails repeatedly (say, after 3 retries), it gets moved into a Dead Letter Queue.
• The DLQ acts like a “quarantine zone” for problematic messages. Developers can inspect them later to find out what went wrong (e.g., invalid email address, broken integration).

✅ With this setup:

• Users don’t wait for notifications during signup.
• Failures don’t cause system crashes.
• Developers have visibility into persistent errors via the DLQ.

Async processing only decouples execution but still ties the app to specific queues, so we move to event-driven architecture for flexibility and multiple independent consumers.

📡 Event-Driven Architecture

Next step: What if you want to send multiple notifications (Email, SMS, Push)? Hardcoding each channel into your backend is messy.

Instead, we use an Event-driven approach:

• App publishes an event: UserSignedUp.
• A Message Broker (e.g., Kafka, AWS SNS, RabbitMQ) delivers this event.
• When a user signs up, the app just says “Hey, something happened!” and passes that message to a message broker.
• Different helpers (Email, SMS, Push) each pick up the message and do their job — this way, the app doesn’t have to talk to everyone directly.
• Different consumers (Email Service, SMS Service, Push Service) process it independently.

Event Flow:

User Signup → Event Publisher → Message Broker → [Email Worker, SMS Worker, Push Worker]

Now, adding a new channel is as simple as adding a new subscriber. No changes to the main app code!

🌐 Fan-Out Architecture

The Fan-out pattern takes event-driven design a step further.

In fan-out, a single event is broadcast to multiple queues/services for parallel processing so one event is copied and sent to many queues at once. This lets multiple services (email, SMS, push) process the same event in parallel without depending on each other.

For example, using AWS SNS + SQS + Lambda:

• SNS Topic publishes the event.
  — Think of SNS as a loudspeaker that shouts, “A new signup happened!” so everyone nearby can hear.
• Multiple SQS Queues subscribe (one for Email, one for SMS, one for Push).
  — Each queue is like a mailbox — one for email, one for SMS, one for push — each getting a copy of the loudspeaker’s message.
• Lambdas consume messages and process notifications independently.
  — Lambdas are like little workers that open their mailbox, read the message, and do their job (send email, SMS, or push).
• Diagram:

User Signup Event  
        ↓  
       SNS (Topic)  
      ↙   ↓   ↘  
   SQS1  SQS2  SQS3  
   Email  SMS   Push  
    ↓      ↓     ↓  
 Worker  Worker Worker

This ensures:

• Independent scaling of each channel.
• Failures in one channel don’t affect others.
• Easier extensibility for future channels (WhatsApp, Slack bots, etc.).

🛠️ Ingestion Mechanism — Smarter Notification Routing

Not every event should trigger a notification. Sending too many alerts can annoy users.

This is where ingestion mechanisms come in. They act as a filtering & routing layer before pushing notifications downstream.

Ingestion checks:

1. User Preferences — Has the user opted out of emails?
2. Activity Signals — If the user is actively online, don’t send a “You have a new message” email.
3. Rate Controls — Avoid spamming multiple notifications in a short time.

This ensures only meaningful, context-aware notifications are sent.

💬 How Slack Handles Notifications

Slack provides a masterclass in notification design. Their decision flow ensures users are notified without being spammed:

1. Message received → Event generated.
2. Ingestion Layer checks:

• Is the user active on desktop? → No notification.
• Is the user idle? → Send push/mobile notification.
• Do preferences allow email notifications? → Maybe send email.

3. Fan-out execution — Slack pushes events to different notification channels (desktop, mobile push, email).

4. Consistency — If you read the message on one device, notifications are cleared across others.

ByteByteGo’s Slack Flowchart explains this beautifully.

✅ Best Practices for Building Notification Systems

• Asynchronous first → Don’t block user actions.
• Queues + DLQs → Handle retries & failures gracefully.
• Fan-out with Event Brokers → Scalability and flexibility.
• Ingestion Mechanisms → Respect user preferences and context.
• Observability → Track success/failure rates, latency, and dropped messages.

🏗️ Final Architecture Overview

Here’s how a modern notification system ties everything together:

User Action (e.g., Signup)  
        ↓  
   Event Publisher  
        ↓  
   Message Broker (SNS/Kafka/RabbitMQ)  
        ↓  
   Fan-Out → [Email Queue, SMS Queue, Push Queue]  
        ↓  
 Ingestion Layer (User Preferences, Rate Limit, Context)  
        ↓  
  Workers / Lambdas  
        ↓  
   Notification Delivery (Email API, Twilio, FCM, etc.)

This architecture ensures scalability, flexibility, and a great user experience.

🎯 Conclusion

Building a notification system may seem trivial at first, but designing it to scale across millions of users requires careful engineering. By combining event-driven design, fan-out architecture, ingestion mechanisms, and real-world strategies like Slack’s intelligent routing, you can build a robust system that not only works — but delights users.

👉 Next time you receive a Slack ping or a welcome email, you’ll know the engineering magic happening behind the scenes.