Skip to main content
deploy/k8s/ is a kustomize overlay covering the namespace, config, secrets, RBAC, deployment, service, ingress and the two PVCs.

​
What it creates

Point the image at your registry before applying:
deploy/k8s/kustomization.yaml

​
The storage claim is the part that matters

ReadWriteMany is not optional, and RWO fails quietly. An RWO claim does not error — Kubernetes gives each pod its own volume. A multi-node job then splits its checkpoint shards across pods and the run cannot be resumed, with nothing in the logs to say why.The executor’s health check verifies TUNEPLANE_K8S_STORAGE_SHARED against the claim’s real accessModes. Read it after the first deploy rather than assuming.
The console mounts that claim at exactly the path TUNEPLANE_STORAGE_ROOT names — it stages job directories into runs/<user>/<exp>/<run_id>/work, measures disk pressure, and reclaims. Training pods mount the same claim, so a path the console writes is the path a job opens. Size it for weight cache plus dataset cache plus every run directory. 2Ti is a starting point, not a recommendation; the weight cache alone is tens of gigabytes per base model.
deploy/k8s/configmap.yaml
Four of those are worth understanding rather than copying:
Training pods report back to it. The external ingress address would leave the cluster and come back in; the service DNS name does not. Never 127.0.0.1 — that is the worker’s own loopback.
A mismatch presents as “the cluster came up but workers never registered”, and the only clue is a single version warning easily lost in startup output.
Ray’s object store lives in /dev/shm. At the default, anything but a tiny batch OOMs immediately. It is RAM-backed, so this plus the memory limit must stay under physical memory.
Without it, an unpullable image or a nodeSelector nothing satisfies leaves a job Pending forever while holding its queue slot.

​
Secrets

secret.example.yaml exists to show the shape. Replace it with a real source — kubectl create secret, Sealed Secrets, or the External Secrets Operator — and remove it from kustomization.yaml before production. At minimum: TUNEPLANE_WEB_JWT_SECRET, TUNEPLANE_DB_URL, TUNEPLANE_REDIS_URL, TUNEPLANE_S3_SECRET_KEY, TUNEPLANE_SECRET_ENC_KEY, and the OIDC client secret.

​
Replicas

The manifest ships one replica, and that is the safe default. Before scaling up:
  • TUNEPLANE_DB_URL must point at Postgres. SQLite is single-writer
  • TUNEPLANE_WEB_JWT_SECRET must be fixed, or replicas reject each other’s tokens
  • TUNEPLANE_REDIS_URL must be set, or background roles stop entirely rather than run unsafely across replicas — storage accounting, diagnosis, the watchdog and the daily report all go quiet
  • Run the background roles in their own pod. Set TUNEPLANE_INPROCESS_WORKERS=0 and apply the worker Deployment — worker.yaml in the kustomize path, worker.enabled=true in the chart. The diagnosis and daily-report ticks call an LLM and hold the event loop for seconds at a time; out of process they stop inflating API tail latency, and the console’s replicas become stateless. One replica is correct: every role elects a leader through Redis, so a second one idles

​
Confirm it worked

Then check the executor’s own view — it is what tells you whether the storage claim is really RWX: