Skip to main content

​
Login & Authentication

Add --device-flow to log in with a device code: the CLI shows a short code; open the prompted URL in any device’s browser and enter it. For CI, use tp login --token <service token>.
Run tp login again. Login state lives in ~/.tuneplane/; multiple machines log in independently without affecting each other. A disabled account also shows up as 401 — check your account status first.

​
Rejected Submissions

The experiment lock has drifted from the server-side catalog (the platform published a new recipe). Use tp recipe status <exp> to see the differences, tp recipe upgrade <exp> to upgrade, or add --upgrade-recipe when submitting.
The platform requires submissions to be traceable to an exact commit. Run git commit and retry; if you really need to submit with dirty changes, use --allow-dirty (untracked files are listed as warnings — be careful not to include large or sensitive files).
Fix each reported error: misspelled keys (struct mode does not allow new keys), out-of-range values, batch-size divisibility. After fixing, confirm locally with tp validate <exp> before resubmitting.
The config references a gated model / dataset your HF account is not authorized for, or the dataset id is misspelled (the full org/name is required). Request access on the HF website first, then confirm your account is linked on the console’s HuggingFace page.
A custom --image may use a tag; pin @sha256:… for production. The registry host must be on TUNEPLANE_ALLOWED_IMAGE_REGISTRIES. An empty list rejects custom user images entirely. Custom images.
Catalog custom/custom is external observability. Pass --observability-url on submit. Console charts still need tuneplane.report (install tuneplane in the image). Custom training.
The scaffold script prints that you have not filled in a training command. Replace it. Working directory is TUNEPLANE_WORK_DIR; run "${TUNEPLANE_EXP_DIR}/train.py".
The submission waits in the queue instead of failing; check current usage with tp status. If you’re in a hurry, free capacity by stopping your own older jobs with tp job stop, or ask an admin to adjust your quota.

​
Job Issues

Check in order: quota usage (tp status), idle GPUs on the cluster (console dashboard), whether the schedule window is closed (jobs are held when the remaining window is below the threshold), and whether maintenance mode is on.
Usually a slow image pull / no node satisfying the nodeSelector / not enough GPUs. The Events section of the job details shows the raw K8s Events; the job is automatically marked failed once the preRunning deadline is exceeded.
Check the Logs tab first. Training may still be loading the model. If logs look fine and metrics are missing, TUNEPLANE_INGEST_URL must be reachable from the training container (not 127.0.0.1). Custom jobs also need tuneplane.report in the training code; see Custom training.
This is automatic retry (within budget, with a cooldown). The retry count is visible in the job details; if you don’t want retries, manually stop the job after it fails.
When the job’s failure reason is marked “OOM killer”: lower train_micro_batch_size, enable activation checkpointing, lower vLLM gpu_memory_utilization, or switch to a larger-shape profile. The AI analysis in the Diagnostics tab usually suggests the exact changes.

​
Logs & Observability

On SSE disconnects the CLI automatically reconnects and resumes; for persistent drops check your reverse proxy config (nginx needs buffering disabled for /api). Historical logs can be replayed anytime: tp job logs <ID> -n 0.
Confirm the method has a validation phase (val_period > 0) and a validation step has been reached; GRPO/PPO samples are generated during validation rounds, so being empty early in training is normal.
Still stuck? Open the job details in the console → Diagnostics to run an AI diagnosis, or hand the run id to an admin to check the server-side logs.