tp submit, real assembly and scheduling, a container that actually trains, real ingest of
metrics, logs and artifacts, and assertions on SUCCEEDED. Only the GPU attach is simulated;
nothing else is mocked.
Run it before go-live, after an upgrade, and after any change to a backend.
It needs a checkout of the platform repository, so this is a step for whoever operates the
deployment rather than for somebody using a released console.
Run it
--keep preserves the scene. The harness runs the real tuneplane from this repository’s cli/, creates an independent temporary project with tp init, then submits from there.
The GPU simulation switches
Scheduling, quotas, capacity, and the ledger all execute with real semantics. Production keeps the default (passthrough).
What it asserts
- The state machine reaches SUCCEEDED (passing through PENDING / RUNNING);
- Metrics closed loop:
train/losscan be read back from the API; - Logs closed loop: training log lines are stored and queryable;
- Artifacts closed loop: checkpoint registration is visible.
CI
.github/workflows/e2e.yml: unit tests → local closed loop (sim + real SFT matrix) → kuberay closed loop (kind + operator), with no cross-repository token.
What it does not cover
The slurm backend is not yet in the closed loop (requires containerized slurmrestd + Apptainer): batch script rendering, the REST client, hetjob, and status mapping are covered by unit tests. Details and roadmap in the repo’sdocs/ops/e2e-closed-loop.md.