serving

TensorFlow Serving Basics

Simulates LLM serving admission control strategies (None, MaxConcurrency, TokenBudget, SLOAware, Predictive) across 135 configurations. Key findings: tight token budget (4096) achieves 10x better goodput than loose budget; predictive control is the only strategy maintaining SLO compliance at 100ms; SLOAware is fundamentally unstable under sustained

Load generator and performance analyzer for LLM inference serving. Measures TTFT, TPOT, and throughput against vLLM/TGI/Ollama or a built-in mock, and flags serving anti-patterns with tuning advice.

A flexible, high-performance serving system for machine learning models

Production-ready LLM inference server optimized for AMD ROCm/MI300X. OpenAI-compatible API, PagedAttention, continuous batching, tensor parallelism, INT4/INT8 quantization.

TIDAL: Toolkit for Inference, Deployment, Adaptation, and Learning for efficient AI systems

Iteration-level continuous batching scheduler comparing FCFS, SJF, Fair, and Preemptive policies on real GPU inference. Validated across 5 seeds: SJF reduces short-job TTFT by 81.5% at the cost of 34% lower Jain fairness.

Single-node inference control plane. OpenAI-compatible API over vLLM, SGLang, and TensorRT-LLM.

Event-driven benchmark of KV-cache migration policies for LLM serving scale-down, measuring session drop rate, infrastructure linger, and user-visible pause across workload types and concurrent session counts.

Serve oversized MoE models on Apple Silicon with MLX, Unified Memory, and SSD streaming

Professional Python project: deploying and serving machine learning models.

Profiling-driven SGLang optimization for quantized agentic LLM serving

An end-to-end Data Engineering project implementing a Medallion Architecture for real-time ride-sharing analytics.

Async FastAPI logging proxy that sits in front of a local vLLM instance serving BCCard/gemma-4-31B-it-FP8-Dynamic.

Open research / proof of concept: pinned-expert MoE serving on Apple Silicon (MLX). Concurrency gains measured; >RAM streaming is a slow feasibility demo, not a product. AGPL-3.0.

This is base project of bentoml and machine learning model with poetry env

FastAPI + Docker serving demo for CTR-style model inference

A single-binary HTTP load tester for **model-serving endpoints** (or any HTTP service). No dependencies beyond

Best Free AI Video Enhancement Tools 2026 - Ultra HD Upscale & Repair 🚀