> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Smoke-test a deployment

> Run submit, schedule, train and ingest end to end, with no GPU.

A real `tp submit`, real assembly and scheduling, a container that actually trains, real ingest of
metrics, logs and artifacts, and assertions on `SUCCEEDED`. Only the GPU attach is simulated;
nothing else is mocked.

Run it before go-live, after an upgrade, and after any change to a backend.

It needs a checkout of the platform repository, so this is a step for whoever operates the
deployment rather than for somebody using a released console.

## Run it

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# local executor: simulated training workload (~10 seconds)
uv run python scripts/e2e/local_loop.py

# local executor: real TRL SFT (actual training of a tiny model on CPU)
uv run python scripts/e2e/local_loop.py --flavor sft

# kuberay executor: kind cluster + operator + MinIO distribution (~40 seconds)
bash scripts/e2e/setup-kind.sh
uv run python scripts/e2e/kuberay_loop.py
```

On failure, container logs / K8s events / platform-side logs are dumped automatically; `--keep` preserves the scene. The harness runs the real `tuneplane` from this repository's `cli/`, creates an independent temporary project with `tp init`, then submits from there.

## The GPU simulation switches

| Setting                             | Semantics                                                                                 |
| ----------------------------------- | ----------------------------------------------------------------------------------------- |
| `TUNEPLANE_LOCAL_GPU_PASSTHROUGH=0` | local: allocation / accounting / labels work as usual, containers launch without `--gpus` |
| `TUNEPLANE_K8S_GPU_PASSTHROUGH=0`   | kuberay: Pods don't request `nvidia.com/gpu`, Ray logical resources work as usual         |

Scheduling, quotas, capacity, and the ledger all execute with real semantics. **Production keeps the default (passthrough).**

## What it asserts

1. The state machine reaches SUCCEEDED (passing through PENDING / RUNNING);
2. Metrics closed loop: `train/loss` can be read back from the API;
3. Logs closed loop: training log lines are stored and queryable;
4. Artifacts closed loop: checkpoint registration is visible.

## CI

`.github/workflows/e2e.yml`: unit tests → local closed loop (sim + real SFT matrix) → kuberay closed loop (kind + operator), with no cross-repository token.

## What it does not cover

The slurm backend is not yet in the closed loop (requires containerized slurmrestd + Apptainer): batch script rendering, the REST client, hetjob, and status mapping are covered by unit tests. Details and roadmap in the repo's `docs/ops/e2e-closed-loop.md`.