Cedana Helm Charts

Repository with a collection of helm charts officially maintained to deploy Cedana, on a Kubernetes cluster.

Chart for installing the controller and cedana daemon.

Currently you can install the chart using this repo or you can use the oci repository.

# clone the repo
git clone https://github.com/cedana/cedana-helm-charts --depth 1

# install from local chart files
helm install cedana ./cedana-helm-charts/cedana-helm --create-namespace -n cedana-system \
--set config.url=$CEDANA_URL \
--set config.authToken=$CEDANA_AUTH_TOKEN \
--set config.clusterId=$CLUSTER_ID

# alternatively, you can use the OCI repo
helm install cedana oci://registry-1.docker.io/cedana/cedana-helm --create-namespace -n cedana-system \
--set config.url=$CEDANA_URL \
--set config.authToken=$CEDANA_AUTH_TOKEN \
--set config.clusterId=$CLUSTER_ID

# To install cedana on CPU only nodes
helm install cedana oci://registry-1.docker.io/cedana/cedana-helm --create-namespace -n cedana-system \
--set config.url=$CEDANA_URL \
--set config.authToken=$CEDANA_AUTH_TOKEN \
--set config.clusterId=$CLUSTER_ID
--set config.pluginsGpuVersion=none

config.url, config.authToken, and config.clusterId are required values.

The CRIU nvidia-dev and infiniband plugins are installed on every node by default. They let CRIU checkpoint processes holding open /dev/nvidia* (e.g. NVML users) or /dev/infiniband/* fds, and are inert on nodes without those devices. They follow config.pluginsCriuVersion. To opt out (the plugin is removed from the node):

--set config.pluginsCriuNvidiaDev=false
--set config.pluginsCriuInfiniband=false

By default the Cedana helper runs on every node in the cluster. To limit it to a specific set of nodes, label those nodes and set a matching nodeSelector:

• Label the nodes you want Cedana to run on:

kubectl label nodes <node-name> cedana.ai/enabled=true

• Set the selector at install time (or in a values override):

helm install cedana ./cedana-helm \
  --set daemonHelper.nodeSelector."cedana\.ai/enabled"=true

The helper and health-check DaemonSets use a dedicated cedana-helper ServiceAccount. The chart does not install the EKS Pod Identity Agent, create IAM roles or policies, or create Pod Identity associations.

1. Install the EKS Pod Identity Agent add-on on the cluster.
2. Create an IAM role with a least-privilege policy for the checkpoint bucket. Typical permissions are s3:ListBucket on the bucket and s3:GetObject, s3:PutObject, s3:DeleteObject, s3:AbortMultipartUpload, and multipart-list permissions on the checkpoint prefix.
3. Associate that role with the chart namespace and the cedana-helper ServiceAccount.
4. Install or upgrade with:

config:
  checkpointDir: s3://my-checkpoint-bucket/cedana
  awsCredentialsMode: eksPodIdentity
  awsRegion: us-east-1

daemonHelper:
  serviceAccount:
    create: true
    name: cedana-helper
    automount: false
    annotations: {}

To use an externally managed ServiceAccount, set daemonHelper.serviceAccount.create: false and provide its non-empty name. Use config.awsCredentialsMode: ambient for IRSA, node roles, or other AWS default-chain credential sources. Static mode remains the default and continues to read config.awsAccessKeyId and config.awsSecretAccessKey.

The Cedana controller reads this same nodeSelector to decide which nodes to manage, so it only applies the cedana.ai/not-ready taint to in-scope nodes. Nodes outside the selector are never tainted, and if you narrow the selector later, any existing not-ready taint is removed from the now-out-of-scope nodes automatically. You do not need to configure the taint or its toleration yourself — leave the built-in cedana.ai/not-ready toleration in place; it is required for Cedana to function.

This is the common case ("run Cedana on these nodes"). It does not prevent other workloads from also running on those nodes. To dedicate nodes exclusively to Cedana, see Dedicated Node for Controller below, which additionally taints the node to repel other workloads.

By default the controller manages a cedana.ai/not-ready:NoSchedule taint on each node: it taints the node while Cedana is starting up there and removes the taint once Cedana is ready, gating other workloads off the node until it can checkpoint/restore. If you'd rather manage node readiness yourself (via your own tooling, GitOps, or an admission controller), disable it:

helm install cedana ./cedana-helm \
  --set controllerManager.taintNodes=false

When disabled, the controller is completely hands-off — it will neither add nor remove the cedana.ai/not-ready taint — and you are responsible for keeping workloads off nodes until Cedana is ready. This is enabled (true) by default.

To checkpoint and restore Dynamo/vLLM or SGLang prefill and decode workers, turn on the controller's P/D disagg webhook:

helm install cedana ./cedana-helm \
  --set controllerManager.pdDisaggWebhook.enabled=true

The controller then creates the webhook's cedana-pd-webhook Service, its self-signed serving cert (Secret cedana-pd-webhook-pd-webhook-tls) and the cedana-pd-disagg MutatingWebhookConfiguration itself; there is nothing else to install. It sets up only pods annotated cedana.ai/pd-disagg (see the Cedana docs). Setting it back to false removes the webhook configuration and Service on the controller's next start. The RBAC it needs is part of the chart either way. This is disabled (false) by default.

To run the Cedana Controller exclusively on a specific node:

• Taint the node to prevent general workloads:

kubectl taint node <node-name> dedicated=cedana-manager:NoSchedule

• Label the node to target it for scheduling:

kubectl label node <node-name> dedicated=cedana-manager

Helm values.yaml uses:

• tolerations to allow scheduling on tainted nodes.
• affinity to restrict placement to labeled nodes.

Note: Do not forget to uncomment the tolerations and affinity code blocks before performing the helm install.

For workloads that require large shared memory, you can optionally increase the /dev/shm size on all nodes:

# Enable SHM configuration with default 10G size
helm install cedana ./cedana-helm-charts/cedana-helm --create-namespace -n cedana-system \
--set config.url=$CEDANA_URL \
--set config.authToken=$CEDANA_AUTH_TOKEN \
--set config.clusterId=$CLUSTER_ID \
--set hostConfig.shmConfig.enabled=true

# Customize SHM size (e.g., 20G)
helm install cedana ./cedana-helm-charts/cedana-helm --create-namespace -n cedana-system \
--set config.url=$CEDANA_URL \
--set config.authToken=$CEDANA_AUTH_TOKEN \
--set config.clusterId=$CLUSTER_ID \
--set hostConfig.shmConfig.enabled=true \
--set hostConfig.shmConfig.size="20G"

When cluster metrics are enabled, you must populate the metrics S3 bucket name explicitly. The S3 region and credentials will automatically fall back to the main config settings if omitted.

The monitoring stack includes Prometheus (for metrics collection) and Vector (for shipping logs/metrics to S3). Individual subcharts can be enabled via clusterMetrics.prometheus.enabled and clusterMetrics.vector.enabled respectively.

Note that clusterMetrics.enabled must be set to true to use these subcharts. If clusterMetrics.enabled is false while any sub-component is enabled, the installation will fail validation.

1. Fetch the subchart dependencies:

helm dependency update ./cedana-helm-charts/cedana-helm

2. Install with metrics enabled (e.g. enabling both Prometheus and Vector):

helm install cedana ./cedana-helm-charts/cedana-helm --create-namespace -n cedana-system \
  --set config.url=$CEDANA_URL \
  --set config.authToken=$CEDANA_AUTH_TOKEN \
  --set config.clusterId=$CLUSTER_ID \
  --set config.awsRegion=$AWS_REGION \
  --set config.awsAccessKeyId=$AWS_ACCESS_KEY_ID \
  --set config.awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY \
  --set clusterMetrics.enabled=true \
  --set clusterMetrics.prometheus.enabled=true \
  --set clusterMetrics.vector.enabled=true \
  --set clusterMetrics.s3.bucket=$METRICS_S3_BUCKET

If your cluster has NVIDIA GPUs and you want to monitor GPU metrics (utilization, memory, temperature, etc.), set clusterMetrics.gpu.enabled=true as well. Note that by default, the GPU exporter DaemonSet will only schedule pods on GPU-enabled nodes labeled with nvidia.com/gpu.present=true:

  --set clusterMetrics.gpu.enabled=true

See https://docs.cedana.ai/get-started/using-the-cedana-platform for usage instructions with the Cedana Platform!

You can find samples on the Cedana Platform or check out the cedana-samples for example spec files.

If you want to use our tools, but have specific security requirements reach out and let us know about it.

• Daemonset requires privilege escalation for daemon container, and it updates the host directly with required dependencies and our cedana-daemon application.
• Currently our controller requires access to all pods to be able to list, checkpoint and restore them.
• SHM configuration requires privileged access to modify host filesystem and mount points.

To uninstall completely, simply run:

helm uninstall cedana -n cedana-system