Prerequisites
- A Kubernetes cluster (GPU nodes with the device plugin installed);
- KubeRay operator (installed via helm, v1.4+);
- Object storage (required: job packages are distributed into containers via presigned URLs);
- GPU nodes labeled by series:
kubectl label node <n> tuneplane/gpu-series=h200.
Configuration
Key semantics
Job Capsule distribution
Job Capsule distribution
The console assembles user code, the authoritative JobSpec, manifest, and
runner.pex into a content-addressed Capsule and uploads it to object storage. HTTPS archives use Ray’s runtimeEnvYAML.working_dir; for HTTP-only storage, an init container in every Ray Pod verifies the transport SHA-256 and exposes the same file:// path to the runtime-env agent. Training images need no platform bootstrap package.RayJob shape
RayJob shape
head = machine zero of the primary pool; multi-pool jobs get one worker group per pool (nodeSelector by series, pinned via Ray custom resources).
shutdownAfterJobFinishes=true + backoffLimit=0: training never retries blindly. The platform deletes the RayJob immediately after archiving terminal logs; TTL is only a failure fallback.Capacity and scheduling
Capacity and scheduling
Cluster capacity is computed live from node
nvidia.com/gpu capacity plus series labels; unschedulable nodes count as blocked. The per-series capacity shown in the UI and the nodes the scheduler can actually land on come from the same source.Troubleshooting jobs stuck in Pending
Troubleshooting jobs stuck in Pending
The real cause only lives in K8s Events (image pull failure / no node satisfying the nodeSelector / not enough GPUs) — the job detail’s event view surfaces them directly. Jobs exceeding the preRunning deadline are failed automatically, so they never occupy the queue indefinitely.
GPU-less clusters (testing only)
GPU-less clusters (testing only)
TUNEPLANE_K8S_GPU_PASSTHROUGH=0: Pods do not request nvidia.com/gpu, while Ray logical resources work as usual. For running the closed loop on kind/CI; production keeps the default.Playground
Supported: standalone vLLM Pods (not RayJobs), accessed via the K8s API proxy, reclaimed on TTL expiry.Quick verification
CPU-only custom jobs
This Fleet also acceptscustom/custom with one pool, nodes: 1,
gpus_per_node: 0, positive cpus and memory_gb, and no GPU series. It creates
an ordinary Pod with CPU/memory requests equal to limits and restartPolicy: Never. No GPU device plugin, Ray runtime or GPU node label is needed for this
workload. Existing Fleet tolerations allow it to use free CPU capacity on GPU
nodes as well; it does not request any accelerator.
The Capsule archive is delivered through object storage, verified before
extraction, and executed with the image’s Python. The image needs Python and
Bash; the plugin may impose additional Python/SDK requirements. Existing storage,
namespace and Pod RBAC configuration still apply. The platform owns observation,
log archival, stop and terminal cleanup. Scratch and multi-machine CPU requests
are not supported.