Data Center Architecture

Apache Kafka has revolutionized how modern enterprises handle real-time data streams. Let's explore 8 powerful use cases that demonstrate why Kafka has become the de-facto standard for distributed event streaming: 1. Streaming Data Processing - Ingest high-velocity data from social platforms (Twitter, Instagram, Facebook, TikTok) - Direct streaming to Apache Spark for real-time processing - Enable ML model training on live data streams - Maintain high throughput with topic-based partitioning 2. Log Aggregation - Centralize logs from distributed systems - Enable real-time log processing through Kafka Connect - Stream to analytics engines (Spark) for insights - Support multiple downstream consumers (monitoring, analytics, archival) 3. Message Queuing - Decouple producers from consumers - Guarantee message ordering within partitions - Enable message replay capabilities - Support multiple consumer groups with different processing speeds - Ensure reliable message delivery with configurable persistence 4. Data Replication - Synchronize data across multiple database instances (DB1→DB6) - Enable cross-datacenter replication - Maintain consistency with exactly-once delivery semantics - Support real-time data mirroring for disaster recovery 5. Monitoring & Alerting - Collect metrics from microservices in real-time - Process streams via Apache Flink for complex event processing - Enable real-time dashboards for system health - Trigger immediate alerts on anomaly detection - Support predictive monitoring use cases 6. Change Data Capture (CDC) - Track database changes in real-time - Support various sink connectors:   • Elasticsearch for search   • Redis for caching   • Custom databases for replication - Maintain data consistency across polyglot persistence architectures 7. System Migration - Enable zero-downtime migrations - Implement dual-write patterns - Verify data consistency with pre-migration reconciliation - Support gradual cutover strategies - Minimize risk with rollback capabilities 8. Real-Time Analytics - Process events through partitioned topics - Scale consumers independently - Support multiple consumer groups for different analytics needs - Enable real-time aggregations and windowed computations - Power live dashboards and business metrics Key Advantages Across Use Cases: • Horizontal scalability with partitioned topics • Fault tolerance with replication • Message persistence with configurable retention • High throughput (millions of messages/second) • Low latency (sub-10ms) • Schema evolution support • Strong ordering guarantees Implementation Considerations: - Proper partition strategy design - Topic configuration optimization - Consumer group design - Monitoring and alerting setup - Disaster recovery planning - Schema management - Resource capacity planning Have I overlooked anything? Please share your thoughts—your insights are priceless to me.

Building Data Pipelines has levels to it: - level 0 Understand the basic flow: Extract → Transform → Load (ETL) or ELT This is the foundation. - Extract: Pull data from sources (APIs, DBs, files) - Transform: Clean, filter, join, or enrich the data - Load: Store into a warehouse or lake for analysis You’re not a data engineer until you’ve scheduled a job to pull CSVs off an SFTP server at 3AM! level 1 Master the tools: - Airflow for orchestration - dbt for transformations - Spark or PySpark for big data - Snowflake, BigQuery, Redshift for warehouses - Kafka or Kinesis for streaming Understand when to batch vs stream. Most companies think they need real-time data. They usually don’t. level 2 Handle complexity with modular design: - DAGs should be atomic, idempotent, and parameterized - Use task dependencies and sensors wisely - Break transformations into layers (staging → clean → marts) - Design for failure recovery. If a step fails, how do you re-run it? From scratch or just that part? Learn how to backfill without breaking the world. level 3 Data quality and observability: - Add tests for nulls, duplicates, and business logic - Use tools like Great Expectations, Monte Carlo, or built-in dbt tests - Track lineage so you know what downstream will break if upstream changes Know the difference between: - a late-arriving dimension - a broken SCD2 - and a pipeline silently dropping rows At this level, you understand that reliability > cleverness. level 4 Build for scale and maintainability: - Version control your pipeline configs - Use feature flags to toggle behavior in prod - Push vs pull architecture - Decouple compute and storage (e.g. Iceberg and Delta Lake) - Data mesh, data contracts, streaming joins, and CDC are words you throw around because you know how and when to use them. What else belongs in the journey to mastering data pipelines?

Most of the production Kubernetes Cluster I have set up is running in AWS EKS! (stopped counting after around 100 or so 😇 ) but every single time before making it accessible from outside! I have to make a choice! You guessed it right! I am talking about ingress! What is Ingress (if you are new to it in Kubernetes) Ingress in Kubernetes is a resource that manages external access to services within a cluster, typically via HTTP or HTTPS. It provides routing rules to direct incoming traffic to the correct service based on URL paths, domain names, or other criteria. There are a few choices available before making the selection: 𝐀𝐖𝐒 𝐀𝐋𝐁 𝐈𝐧𝐠𝐫𝐞𝐬𝐬 𝐂𝐨𝐧𝐭𝐫𝐨𝐥𝐥𝐞𝐫: It integrates AWS Application Load Balancers (ALB) with Kubernetes, managing ALB resources to route traffic to Kubernetes services based on Ingress rules. It watches for Kubernetes Ingress resources, automatically provisioning and configuring an ALB to route external traffic based on defined routing rules. It manages target groups, listener rules, and security groups to direct traffic to the appropriate backend services within the cluster. (Sample ingress in the comment section) 𝐍𝐆𝐈𝐍𝐗 𝐈𝐧𝐠𝐫𝐞𝐬𝐬 𝐂𝐨𝐧𝐭𝐫𝐨𝐥𝐥𝐞𝐫:   It is a Kubernetes component that uses NGINX as a reverse proxy and load balancer to manage Ingress resources. It provides routing, load balancing, SSL termination, and advanced traffic control features for applications within a Kubernetes cluster. In AWS EKS, the NGINX Ingress Controller can be configured with a Network Load Balancer (NLB) to handle external traffic, enabling high-performance, low-latency routing. The NLB forwards traffic to the NGINX pods within the cluster, which then direct requests to the appropriate services based on Ingress rules. (sample ingress in the comment section) The above two always have been primary choices, other than that Gloo and Contour were also put in the discussions, based on requirements we usually went it either the ALB Ingress controller or NGINX ingress controller! Now, you might ask when to use what? Primarily I can say, that ALB Ingress supports basic HTTP and HTTPS routing, focusing on layer 7 load balancing with AWS-managed scaling and security but limited to ALB’s routing capabilities, then comes NGINX Ingress, it offers advanced traffic control (e.g., custom rewrites, rate limiting, authentication plugins) and is more flexible if you need detailed routing rules or custom configurations (but a lot of tweaks and optimizing required) Repost if you find this post useful ♺ Cheers, Sandip Das

The Netherlands is exploring innovative ways to make data centers more energy-efficient by developing floating data centers that use canal water for cooling. Data centers require enormous amounts of electricity, not only to power servers but also to cool the equipment and prevent overheating. Traditional data centers rely heavily on air-conditioning systems, which consume significant energy and increase operational costs. To reduce this energy demand, engineers in the Netherlands have proposed floating server facilities that use nearby water sources such as canals, lakes, or ports for natural cooling. The concept works by circulating water from the canal through specialized heat exchangers. The water absorbs heat generated by the servers and carries it away, reducing the need for energy-intensive cooling equipment. This method can significantly lower energy consumption and reduce the environmental footprint of large-scale computing infrastructure. Floating data centers also offer additional benefits such as modular construction, flexible deployment, and efficient land use in densely populated cities. The Netherlands, known for its extensive canal networks and expertise in water engineering, provides an ideal environment for testing this approach. As global demand for cloud computing, artificial intelligence, and digital services continues to rise, innovative cooling solutions like floating data centers could play a major role in making the world’s digital infrastructure more sustainable and energy-efficient. #DataCenterInnovation #GreenTechnology #SustainableComputing #TechInfrastructure #FutureEngineering

A Senior Data Engineer candidate was asked to design an incremental ingestion pipeline during his interview at Google. Another candidate in a different loop at Facebook got the same prompt. CDC pipelines look simple until you add one layer of reality: – Add late arriving updates? Now you need watermarks, reprocessing windows, and correctness guarantees. – Add duplicates and retries? Now idempotency becomes the whole game. – Add schema changes? Now your pipeline breaks at 2 AM unless you plan compatibility. – Add backfills? Now you are doing surgery on live tables without double counting. – Add merge cost? Now your “incremental” job is slower than a full reload. Here’s my checklist of 15 things you must get right when building incremental ingestion with CDC: 1. Start with the business contract → Define what “correct” means: latest state per entity, full history, or both. This single decision changes your table design, merges, and backfills. 2. Choose the right ingestion model: snapshot + CDC vs pure CDC → Snapshot + CDC is safest for bootstrapping and recovery. Pure CDC is leaner but brittle if you miss events. 3. Pick a stable primary key strategy → If your upstream keys are messy, create a durable surrogate key. Your entire dedupe and merge logic depends on this. 4. Capture an ordering signal you can trust → Use a reliable change version: log sequence number, commit timestamp, or monotonically increasing version. Avoid “updated_at” unless you fully trust the source. 5. Design for idempotency from day one  → Assume every event can arrive twice. Your writes must be safe to re-run without changing results. 6. Handle deletes explicitly → CDC isn’t just inserts and updates. Support tombstones or delete flags and define how downstream tables interpret them. 7. Preserve raw events before you transform → Land the raw change feed in a bronze layer. If downstream logic is wrong, raw becomes your rewind button. 8. Build a dedupe rule that survives retries and replays → Dedupe by (primary_key + change_version) or (primary_key + event_id). If event_id is missing, generate one deterministically from the payload plus version. 9. Use watermarks, but never trust them blindly → Watermark = “I have processed up to here.” Still keep a safety lookback window because late data is guaranteed in production. 10. Implement a reprocessing window for late arrivals  → Recompute the last N hours or days on every run based on observed lateness. This is the simplest way to get correctness without constant firefighting. 11. Plan schema evolution with compatibility rules → Decide: backward compatible only, or allow breaking changes with a controlled rollout. Use versioned schemas and block unsafe changes automatically. (Continued in comments.)

The Hyperscale Difference: Why AWS, Azure, and Google Cable Differently It's not just clean wiring. The way these hyperscalers design their Optical Fiber Containment is a direct reflection of their core network strategy: AWS ☁️ (Resilience Scheme): Design mandates focus on physical separation and redundancy across Availability Zones (AZs). Their containment ensures multiple, distinct pathways to guarantee maximum uptime. Show extensive, robust wire mesh baskets and traditional ladder trays carrying dense fiber runs, reflecting their scale and foundational infrastructure. Azure 🟦 (Compliance Scheme): Design prioritizes standardization and zoning for enterprise integration. Their containment facilitates auditable pathways and clear boundaries for hybrid cloud and high-security zones. Focus on structured, rigid fiber runner systems (raceways), possibly with specific mounting brackets, to highlight their modularity and enterprise compliance Google 🟡 (Performance Scheme): Design is built around extreme performance and custom hardware. Their containment uses dense, bespoke raceways to safeguard signal integrity and optimize routing for their high-core-count fiber. Combine various elements: baskets, trays, and ladder racks for diverse cable types, with a prominent Subzero Frame - like contained rack to emphasize high-density, performance-focused infrastructure. The Takeaway: Their fiber containment systems are a physical map of their unique cloud missions. AWS prioritizes uptime, Azure prioritizes integration, and Google prioritizes speed. #DataCenter #CloudComputing #AWS #Azure #GCP #OpticalFiber #Networking

Look, this is going to shock some of you. But someone’s got to say it. Your ‘cloud’ isn’t floating somewhere out there. It’s anchored in giant rooms you may never see. Ever wondered where your photos, messages, and video calls live? Not in the sky, that’s for sure! They live in data centres. Huge, secure buildings that power everything online. Every click, every stream, every late-night email? It all runs through these digital ‘backbones’. → Massive servers work day and night, always on → Cooling systems stop everything from overheating → Backup power keeps things on, even if the lights go out → Security guards and lots of cameras protect your data 24/7 Hyperscalers like Google, AWS, and Microsoft build these rooms on a scale most of us can’t imagine. (Think football stadiums full of blinking lights!) Why does this matter? → Data centres make sure your apps and websites are always on → They keep your information safe and ready, for whenever you need it → They help businesses grow, learn, and connect across the world But there’s more. As our digital world grows, so does the need for energy. That’s why the industry is changing fast: → Smarter cooling to use less power → Greener energy to lower carbon footprints → New tech to make storage faster and safer I’ve spent years inside these whirring rooms. Planning, building, leading teams to deliver for some of the world’s biggest names. (Trust me, it never gets old seeing a new data hall go live!) Next time you share a photo or join a video call, remember … your cloud is anchored in a real place, with real people working behind the scenes. What surprises you most about data centres? Anything you wish you could see behind those doors?

A Complete Overview of Telecom Infrastructure – From Tower to Core 1. Base Transceiver Station (BTS) – The Foundation The BTS site is the first point of contact for mobile users and includes three essential subsystems: A. Power System Ensures 24/7 operation through: • Grid Power (primary source, stepped down via transformers) • Diesel Generator (backup for outages) • Backup Batteries (DC power during failures) • ATS (Automatic Transfer Switch) (automates switching between power sources) • Power Supply Control Cabinet (converts AC to DC) • DCDU (DC Distribution Unit – powers BBUs, RRUs, etc.) B. Radio Access Network (RAN) Enables wireless access and signal processing: • RF Antennas (4G/5G communication interface) • AISG (remotely adjusts antenna tilt and alignment) • Jumper Cables (connect RRUs to antennas) • RRU (Remote Radio Unit) – manages RF signal processing • BBU (Baseband Unit) – handles digital signal processing and traffic control C. Transmission System Links BTS to the core network: • Microwave Antennas (wireless backhaul) • ODU/IDU (Outdoor & Indoor Units – convert and process microwave signals) • IF Cable (connects ODU to IDU) • Router (routes and manages data traffic) 2. Transmission & Transport Network Transports data between access points and core: • Access Network: Connects mobile devices and IoT via radio towers and fiber • Transport Network: Aggregates and transports traffic using: • Microwave Links • Optical Fiber • DWDM (Dense Wavelength Division Multiplexing) for high-bandwidth transmission 3. Core Network – The Brain of the System Responsible for data switching, routing, and service control: • Mobile Core (EPC/5GC): Handles mobility, authentication, and session management • IMS (IP Multimedia Subsystem): Supports VoIP, video calls, and messaging • PCRF/PCF: Policy and charging control • HSS/UDM: Subscriber database and identity management • Gateways (SGW, PGW/UPF): Connect mobile users to external networks 4. Service & Application Layer Where services are hosted and managed: • Data Centers: Host platforms for: • Billing & Charging • Content Delivery (VoD, streaming) • Security & Firewalls • Network Slicing & Cloud Platforms • Edge Computing: Brings processing closer to users for low latency 5. Network Operations & Management Ensures performance, reliability, and optimization: • NOC (Network Operations Center): Central monitoring and fault resolution • OSS/BSS Systems: Support operations and business functions • EMS/NMS: Element and network-level management tools • AI/ML: Used for predictive maintenance, anomaly detection, and optimization Common Physical Components Throughout the Network • Fiber Optics / Patch Cords • CPRI/eCPRI Links (for fronthaul between RRU & BBU) • Ethernet Switches • Racks & Cabinets • GPS/Clock Synchronization Equipment This ecosystem enables seamless voice, data, and video services across billions of connected devices globally.

This is how you measure your AI system as an AI Engineer 👇 For regular software you would track metrics like uptime, error rate, p95 latency. However, they say little about whether the system is fast where users feel it, affordable at scale or correct. Here are the metrics we track when building LLM systems. It is useful to group them by the question they answer: 𝟭. 𝗜𝘀 𝗶𝘁 𝗳𝗮𝘀𝘁? (𝗟𝗮𝘁𝗲𝗻𝗰𝘆) ➡️ Time to first token (TTFT): how long the user is exposed to a blank screen, the number that defines perceived latency. ➡️ Inter-token latency (ITL): how smoothly tokens stream after the first one. ➡️ End-to-end latency at p50 / p95 / p99, dominated by output length, track it per use case rather than globally. 𝟮. 𝗖𝗮𝗻 𝗶𝘁 𝘀𝗰𝗮𝗹𝗲? (𝗧𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗮𝗻𝗱 𝗰𝗼𝘀𝘁) ➡️ Tokens per second per user vs total system throughput, the two trade off against each other on the same hardware. ➡️ Input and output tokens per request to measure your unit economics. ➡️ Cache hit rate - prompt caching is often the technique that reduces cost the most. ➡️ Cost per successful task, not cost per request, a cheap request that fails is a waste. 𝟯. 𝗜𝘀 𝗶𝘁 𝗰𝗼𝗿𝗿𝗲𝗰𝘁? (𝗤𝘂𝗮𝗹𝗶𝘁𝘆) ➡️ Task success rate on a labeled eval set, re-run on every prompt or model change. ➡️ Groundedness for RAG - is the answer supported by the retrieved context. ➡️ Retrieval precision@k and recall@k - generation cannot fix what retrieval never surfaced. ➡️ LLM-as-judge scores over time, calibrated against human labels. ➡️ User feedback signals: thumbs, edits to generated output, free form feedback. 𝟰. 𝗗𝗼𝗲𝘀 𝗶𝘁 𝗵𝗼𝗹𝗱 𝘂𝗽? (𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆) ➡️ Error, timeout and rate-limit rates per provider. ➡️ Retry and fallback rate - how often you silently switch to a backup model. ➡️ Guardrail trigger and refusal rates. 𝟱. 𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝘆𝗼𝘂𝗿 𝗮𝗴𝗲𝗻𝘁 𝗯𝗲𝗵𝗮𝘃𝗲? (𝗔𝗴𝗲𝗻𝘁 𝗺𝗲𝘁𝗿𝗶𝗰𝘀) ➡️ Tool-call error rate. ➡️ Steps and tokens per completed task - drift here means cost is rising while accuracy remains the same ➡️ Context window utilization - the early warning for compaction and truncation issues. Read more about this in my newsletter: https://lnkd.in/dhiscYbm ❗️ Latency and reliability show up on day one because standard infra emits them. Quality, cost per task, and agent behavior need deliberate instrumentation, and they are where AI systems fail in production. Which metric caught a real problem for you that the standard dashboards missed? 👇

You Only Need This to Build a Comprehensive Data Center — Fast . . What if deploying a data center didn’t require years of construction, complex coordination, and endless delays? Modular Data Centers are reshaping the equation. Pre-engineered, factory-built, and rapidly deployable, they integrate power, cooling, racks, fire suppression, and monitoring into a single ready-to-run system. Instead of building everything on-site, you assemble intelligent building blocks designed for speed, scalability, and predictability. The real advantage isn’t just faster deployment it’s operational efficiency. Reduced construction risk, standardized performance, easier expansion, and optimized energy design. Need capacity? Add modules. Need redundancy? Scale by design. Modular infrastructure transforms data center growth from a major project into a controlled upgrade path, exactly what modern digital demand requires. 💬 Have you ever worked on a modular deployment? What surprised you the most?