Production-tested Kubernetes deployment recipes for LLM inference using NVIDIA Dynamo.

No recipe for your model and hardware combination? Point an AI coding agent at this repository and ask it to author or adapt one; the repo's agent skills guide the process.

Prerequisites: This guide assumes you have already installed the Dynamo Kubernetes Platform. If not, follow the Kubernetes Deployment Guide first.

These recipes compare Dynamo performance features with benchmark results, each including both baseline and optimized deployment configurations:

These recipes demonstrate aggregated or disaggregated serving:

GAIE Column: Indicates whether the recipe includes integration with the Gateway API Inference Extension (GAIE) — a Kubernetes SIG project that extends the Gateway API for AI inference workloads, providing load balancing, model routing, and request management.

Legend:

• Deployment: ✅ = Complete deploy.yaml manifest available
• Benchmark: ✅ = Includes an AIPerf benchmark manifest

These recipes demonstrate functional deployments with Dynamo features, but have not yet been performance-tuned or paired with benchmark manifests.

These recipes are under active development and may require additional setup steps (e.g., container patching). They are functional but not yet fully validated for production use.

Each complete recipe follows this standard structure:

<model-name>/
├── README.md (optional)           # Model-specific deployment notes
├── model-cache/
│   ├── model-cache.yaml          # PersistentVolumeClaim for model storage
│   └── model-download.yaml       # Job to download model from HuggingFace
└── <framework>/                  # vllm, sglang, or trtllm
    └── <deployment-mode>/        # agg, disagg, disagg-single-node, etc.
        ├── deploy.yaml           # Complete DynamoGraphDeployment manifest
        └── perf.yaml (optional)  # AIPerf benchmark job

In addition, accuracy/ is a shared, model-agnostic accuracy check (deliberately outside the per-model structure above): point it at any deployed recipe to compare the served model's benchmark score against its model card. See accuracy/README.md.

1. Dynamo Platform Installed

The recipes require the Dynamo Kubernetes Platform to be installed. Follow the installation guide:

• Kubernetes Deployment Guide - Quickstart (~10 minutes)
• Detailed Installation Guide - Advanced options

2. GPU Cluster Requirements

Ensure your cluster has:

• GPU nodes matching recipe requirements (see table above)
• GPU operator installed
• Appropriate GPU drivers and container runtime

3. HuggingFace Access

Configure authentication to download models:

export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}

# Create HuggingFace token secret
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token-here" \
  -n ${NAMESPACE}

4. Storage Configuration

Update the storageClassName in <model>/model-cache/model-cache.yaml to match your cluster:

# Find your storage class name
kubectl get storageclass

# Edit the model-cache.yaml file and update:
# spec:
#   storageClassName: "your-actual-storage-class"

Step 1: Download Model

cd recipes
# Update storageClassName in model-cache.yaml first!
kubectl apply -f <model>/model-cache/ -n ${NAMESPACE}

# Wait for download to complete (may take 10-60 minutes depending on model size)
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=6000s

# Monitor progress
kubectl logs -f job/model-download -n ${NAMESPACE}

Step 2: Deploy Service

Update the image in <model>/<framework>/<mode>/deploy.yaml.

kubectl apply -f <model>/<framework>/<mode>/deploy.yaml -n ${NAMESPACE}

# Check deployment status
kubectl get dynamographdeployment -n ${NAMESPACE}

# Check pod status
kubectl get pods -n ${NAMESPACE}

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l nvidia.com/dynamo-graph-deployment-name=<deployment-name> -n ${NAMESPACE} --timeout=600s

Step 3: Test Deployment

# Port forward to access the service locally
kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}

# In another terminal, test the endpoint
curl http://localhost:8000/v1/models

# Send a test request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name>",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'

Step 4: Run Benchmark (Optional)

# Only if perf.yaml exists in the recipe directory
kubectl apply -f <model>/<framework>/<mode>/perf.yaml -n ${NAMESPACE}

# Monitor benchmark progress
kubectl logs -f job/<benchmark-job-name> -n ${NAMESPACE}

# View results after completion
kubectl logs job/<benchmark-job-name> -n ${NAMESPACE} | tail -50

export NAMESPACE=dynamo-demo
kubectl create namespace ${NAMESPACE}

# Create HF token secret
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}

# Deploy
cd recipes
kubectl apply -f qwen3-32b-fp8/model-cache/ -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=6000s
kubectl apply -f qwen3-32b-fp8/trtllm/agg/deploy.yaml -n ${NAMESPACE}

# Test
kubectl port-forward svc/qwen3-32b-fp8-agg-frontend 8000:8000 -n ${NAMESPACE}

For Qwen3-0.6B with vLLM (Aggregated), an example of integration with the Inference Gateway is provided.

First, deploy the Dynamo Graph per instructions above, substituting the Qwen3-0.6B recipe.

Then follow Deploy Inference Gateway Section 2 to install GAIE.

Update the containers.epp.image in the deployment file, i.e. qwen3-0.6b/vllm/agg/gaie/deploy.yaml. It should match the release tag and be in the format nvcr.io/nvidia/ai-dynamo/frontend:<version> e.g. nvcr.io/nvidia/ai-dynamo/frontend:0.9.0 The recipe assumes you are using Kubernetes discovery backend and sets the DYN_DISCOVERY_BACKEND env variable in the epp deployment. If you want to use etcd enable the lines below and remove the DYN_DISCOVERY_BACKEND env var.

- name: ETCD_ENDPOINTS
  value: "dynamo-platform-etcd.$(PLATFORM_NAMESPACE):2379" #  update dynamo-platform to appropriate namespace

export DEPLOY_PATH=qwen3-0.6b/vllm/agg/
# DEPLOY_PATH=<model>/<framework>/<mode>/
kubectl apply -R -f "$DEPLOY_PATH/gaie" -n "$NAMESPACE"

See deepseek-r1/trtllm/disagg/wide_ep/gb200/deploy.yaml for the complete multi-node WideEP configuration.

Each deploy.yaml contains:

• ConfigMap: Engine-specific configuration (embedded in the manifest)
• DynamoGraphDeployment: Kubernetes resource definitions
• Resource limits: GPU count, memory, CPU requests/limits
• Image references: Container images with version tags

Model Configuration:

# In deploy.yaml under worker args:
args:
  - python3 -m dynamo.vllm --model <your-model-path> --served-model-name <name>

GPU Resources:

resources:
  limits:
    gpu: "4"  # Adjust based on your requirements
  requests:
    gpu: "4"

Scaling:

services:
  decode:
    replicas: 2  # Scale to multiple workers

Router Mode:

# In Frontend args:
args:
  - python3 -m dynamo.frontend --router-mode kv --http-port 8000
# Options: round-robin, kv (KV-aware routing)

Container Images:

image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:x.y.z
# Update version tag as needed

Pods stuck in Pending:

• Check GPU availability: kubectl describe node <node-name>
• Verify storage class exists: kubectl get storageclass
• Check resource requests vs. available resources

Model download fails:

• Verify HuggingFace token is correct
• Check network connectivity from cluster
• Review job logs: kubectl logs job/model-download -n ${NAMESPACE}

Workers fail to start:

• Check GPU compatibility (driver version, CUDA version)
• Verify image pull secrets if using private registries
• Review pod logs: kubectl logs <pod-name> -n ${NAMESPACE}

For more troubleshooting:

• Install Dynamo
• Observability Documentation

• Kubernetes Deployment Guide - Platform installation and concepts
• API Reference - DynamoGraphDeployment CRD specification
• vLLM Backend Guide - vLLM-specific features
• SGLang Backend Guide - SGLang-specific features
• TensorRT-LLM Backend Guide - TensorRT-LLM features
• Observability - Monitoring and logging
• Benchmarking Guide - Performance testing

We welcome contributions of new recipes! See CONTRIBUTING.md for:

• Recipe submission guidelines
• Required components checklist
• Testing and validation requirements
• Documentation standards

A production-ready recipe must include:

• ✅ Complete deploy.yaml with DynamoGraphDeployment
• ✅ Model cache PVC and download job
• ✅ Benchmark recipe (perf.yaml) for performance testing
• ✅ Verification on target hardware
• ✅ Documentation of GPU requirements