Configuration reference

export TUNEPLANE_DEFAULT_FLEET_KIND=kuberay
export TUNEPLANE_STORAGE_ROOT=/mnt/shared/tuneplane
export TUNEPLANE_WEB_JWT_SECRET=$(openssl rand -hex 32)

Every setting is an environment variable prefixed TUNEPLANE_. There is no configuration file — a deployment is described entirely by its environment, which is what makes the same image work under Docker Compose, Kubernetes and systemd without a translation layer. Most of them are also editable in the console, under Settings, and that is the surface to reach for once the deployment is running. The environment is the baseline: the console stores a sparse override, a saved value wins over the variable, and anything the console does not override falls back to the environment as it was when the process started. A save takes effect immediately, without a restart, on every replica. Secrets are stored encrypted and never read back out. What the console does not offer is env-only by necessity rather than by omission: what the process needs before it can read its own settings table, paths the server itself opens (such as TUNEPLANE_RUNTIME_REGISTRY_FILE), and the storage roots every job container is mounted on.

This page is generated from server.core.config.WebSettings by scripts/gen_settings_docs.py, and CI fails if a setting is added without a description. If a knob exists, it is on this page.

The five you cannot skip

Everything else has a working default. These do not.

local | node | kuberay | slurm

default:"local"

The backend of the Fleet seeded when the console has none yet — read once, at first start. After that the backend is a property of the Fleet a workload was placed on, and registering a second Fleet is how a console gains another. There is no silent fallback between backends, so set this explicitly even when you want local. See backends.

path

The one path a deployment configures. It must be an absolute path, and it must resolve identically on the console, on every node, and inside every container. Empty means single-machine development.

string

Generated per process when unset, which logs everyone out on restart and breaks multi-replica deployments outright. Set a fixed strong random value for anything but a laptop.

url

Points at Postgres for a team deployment. Left empty it uses a local SQLite file, which is correct for one machine and wrong for two.

url

The https address users reach this deployment at. The OIDC redirect URI is built from it, so single sign-on cannot work without it.

Reading the tables

Required means the deployment refuses to start without it.
generated means a value is derived per process when you do not supply one.
A dash means the default is empty, and empty usually means the feature is off.

Absolute paths are validated at startup: a relative TUNEPLANE_STORAGE_ROOT is refused rather than resolved against whatever directory the process happened to start in.

Core

Environment variable	Type	Default	Description
`TUNEPLANE_CACHE_TTL`	`float`	`5.0`	How long in-process caches hold a value before re-reading it.
`TUNEPLANE_COOKIE_SECURE`	`bool	None`	—	Whether the session cookie carries the Secure flag. Follows whether public_url is https when unset; set it false to debug over plain http locally.
`TUNEPLANE_DB_PATH`	`Path`	`.tuneplane-data/web.db`	Where SQLite lives, relative to the console’s working directory. Deliberately not under the storage root: that root is usually a shared filesystem, and SQLite over NFS has well-known locking problems. Unused once db_url points at Postgres.
`TUNEPLANE_DB_URL`	`str	None`	—	Database URL. Empty falls back to the SQLite file at db_path. A team deployment points this at Postgres.
`TUNEPLANE_GITHUB_ALLOWED_ORGS`	`str`	—	Comma-separated organization logins whose members may sign in. Empty means any GitHub account can create an account here. Setting it also makes the consent screen ask for `read:org`; a member whose membership is private and unshared is refused.
`TUNEPLANE_GITHUB_CLIENT_ID`	`str`	—	OAuth app client ID from GitHub. Setting it and the secret puts a “Sign in with GitHub” button on the login page, with no restart. The callback URL to register is `<public URL>/api/auth/social/github/callback`.
`TUNEPLANE_GITHUB_CLIENT_SECRET`	`str`	—	OAuth app client secret from GitHub. Stored encrypted and never returned by the API.
`TUNEPLANE_GOOGLE_ALLOWED_DOMAINS`	`str`	—	Comma-separated email domains allowed to sign in with Google. Empty means any Google account can create an account here, which on a console reachable from the internet is open registration. Both the verified email domain and the Workspace `hd` claim are accepted.
`TUNEPLANE_GOOGLE_CLIENT_ID`	`str`	—	OAuth client ID from the Google Cloud console. Setting it and the secret puts a “Sign in with Google” button on the login page, with no restart. The redirect URI to register is `<public URL>/api/auth/social/google/callback`.
`TUNEPLANE_GOOGLE_CLIENT_SECRET`	`str`	—	OAuth client secret from the Google Cloud console. Stored encrypted and never returned by the API.
`TUNEPLANE_GOOGLE_ONE_TAP`	`bool`	`false`	Show Google’s One Tap prompt on the login page, which offers to sign in with an account the browser is already signed into. Off by default because the prompt appears without the visitor asking for it. Requires the Google client ID and secret.
`TUNEPLANE_HOST`	`str`	`127.0.0.1`	Address the console binds to. The CLI’s —host overrides it.
`TUNEPLANE_INGEST_URL`	`str	None`	—	Where training processes POST their logs and metrics. With a remote cluster, do not use 127.0.0.1 — that is the worker’s own loopback. Give the console URL as reachable from inside the cluster network.
`TUNEPLANE_JWT_HOURS`	`int`	`8`	How long an access token stays valid.
`TUNEPLANE_LDAP_ATTR_DISPLAY_NAME`	`str`	`displayName`	Which directory attribute holds the person’s display name.
`TUNEPLANE_LDAP_ATTR_EMAIL`	`str`	`mail`	Which directory attribute holds the email address, usually `mail`.
`TUNEPLANE_LDAP_ATTR_GROUPS`	`str`	—	Which directory attribute holds the user’s groups, usually `memberOf`, read as team membership on every sign-in. Mapping them onto teams is a separate entitlement from signing in through LDAP.
`TUNEPLANE_LDAP_ATTR_USERNAME`	`str`	`uid`	Which directory attribute is the login name, usually `uid` or `sAMAccountName`.
`TUNEPLANE_LDAP_BIND_DN`	`str`	—	A read-only account used to find the user before binding as them. Empty attempts an anonymous search. Not needed when a direct bind template is set.
`TUNEPLANE_LDAP_BIND_PASSWORD`	`str`	—	The password for the service account above.
`TUNEPLANE_LDAP_ENABLED`	`bool`	`false`	Whether the ordinary login form also checks passwords against the directory. Local accounts are always tried first, so the bootstrap administrator can still sign in when the directory is down.
`TUNEPLANE_LDAP_INSECURE_SKIP_VERIFY`	`bool`	`false`	Do not verify the directory’s TLS certificate. For debugging against a self-signed server; never in production, where it removes the only thing distinguishing the directory from anyone impersonating it.
`TUNEPLANE_LDAP_SERVER_URI`	`str`	—	One or more LDAP URLs, comma separated and tried in order so a second domain controller is a failover. Use `ldaps://` or turn on StartTLS: this is the one sign-in method that reads the user’s password, and a plain connection puts it on the wire in the clear.
`TUNEPLANE_LDAP_START_TLS`	`bool`	`false`	Upgrade a plain connection to TLS before binding. Unnecessary on `ldaps://`, which is TLS from the first byte.
`TUNEPLANE_LDAP_TIMEOUT_S`	`float`	`5.0`	How long to wait on the directory. Deliberately short: this runs inside a sign-in request, so a directory that hangs must cost one slow login rather than a stuck worker.
`TUNEPLANE_LDAP_USER_BASE_DN`	`str`	—	The subtree searched for the person signing in, e.g. `ou=people,dc=example,dc=com`.
`TUNEPLANE_LDAP_USER_DN_TEMPLATE`	`str`	—	Bind straight as this DN and skip the search, e.g. `uid={username},ou=people,dc=example,dc=com`. Needs no service account, but reads nothing back — email, display name and groups are available only through the search path.
`TUNEPLANE_LDAP_USER_FILTER`	`str`	`(uid={username})`	The filter that finds one person. `{username}` is substituted and escaped. A filter matching more than one entry is refused rather than guessed at, because guessing would sign somebody in as whichever entry the directory happened to return first.
`TUNEPLANE_NO_AUTH`	`bool`	`false`	Treat every request as an admin and skip authentication entirely. Development only — never set this on anything reachable by other people.
`TUNEPLANE_OIDC_CLIENT_ID`	`str`	—	Client id registered with the identity provider.
`TUNEPLANE_OIDC_CLIENT_SECRET`	`str`	—	Client secret registered with the identity provider.
`TUNEPLANE_OIDC_DEFAULT_ROLE`	`str`	`operator`	Role given to a user on their first single sign-on login. Not admin: the first administrator should be somebody a person deliberately promoted.
`TUNEPLANE_OIDC_DISCOVERY_URL`	`str`	—	Explicit discovery document URL. Empty derives one by appending /.well-known/openid-configuration to the issuer; Casdoor serves a per-application path and needs this set.
`TUNEPLANE_OIDC_GROUP_CLAIM`	`str`	—	The claim holding the user’s groups, read as team membership on every sign-in. Empty (the default) means the platform never touches team membership — which is what a deployment that manages teams by hand needs, since reconciling against an absent claim would empty everybody out. A group the directory names but the platform does not have is skipped, never created.
`TUNEPLANE_OIDC_GROUP_PREFIX`	`str`	—	Stripped from the front of each group before it is read as a team key. Directories hand out paths and DNs where the platform wants a key.
`TUNEPLANE_OIDC_INSECURE_SKIP_VERIFY`	`bool`	`false`	Skip TLS verification against the identity provider. For debugging a self-signed certificate on an internal network; never in production.
`TUNEPLANE_OIDC_ISSUER`	`str`	—	OIDC issuer URL, for example a Casdoor application. Setting it turns on single sign-on.
`TUNEPLANE_OIDC_SCOPE`	`str`	`openid profile email`	Scopes requested at login.
`TUNEPLANE_PORT`	`int`	`8080`	Port the console listens on. The CLI’s —port overrides it.
`TUNEPLANE_PUBLIC_URL`	`str	None`	—	The address users and identity providers reach this deployment at. Used to build the OIDC redirect URI, so a team deployment must set it to the real https domain.
`TUNEPLANE_REDIS_PREFIX`	`str`	`lab`	Key namespace, so several environments can share one Redis without colliding.
`TUNEPLANE_REDIS_URL`	`str	None`	—	Redis for shared cache, distributed locks, rate limiting and immediate token revocation. Empty disables all four and the console behaves as a single instance.
`TUNEPLANE_REFRESH_DAYS`	`int`	`30`	How long a refresh token stays valid. It is delivered only as an httpOnly cookie, so browser JavaScript cannot read it.
`TUNEPLANE_REPO_ROOT`	`Path`	generated	Project root the console reads experiment definitions and local bookkeeping from. Defaults to the working directory.
`TUNEPLANE_SAML_ATTR_DISPLAY_NAME`	`str`	`displayName`	Which assertion attribute holds the person’s display name.
`TUNEPLANE_SAML_ATTR_EMAIL`	`str`	`email`	Which assertion attribute holds the email address. Matched loosely, so the bare name, the SAML URN and the Microsoft claim URI all resolve.
`TUNEPLANE_SAML_ATTR_GROUPS`	`str`	—	Which assertion attribute holds the user’s groups, read as team membership on every sign-in. Mapping them onto teams is a separate entitlement from signing in through SAML, and empty means the platform never touches team membership.
`TUNEPLANE_SAML_ATTR_USERNAME`	`str`	—	Which assertion attribute holds the login name. Empty falls back to the NameID, which is what a minimally configured identity provider sends.
`TUNEPLANE_SAML_ENABLED`	`bool`	`false`	Whether the login page offers SAML. Commercial edition only, and the button appears only once an identity provider is described below — a switch with no IdP behind it would draw a button that goes nowhere.
`TUNEPLANE_SAML_IDP_METADATA_URL`	`str`	—	Where the console fetches the identity provider’s metadata: its entity id, sign-on endpoint and signing certificate. Re-read periodically, so rotating the IdP’s certificate needs no restart.
`TUNEPLANE_SAML_IDP_METADATA_XML`	`str`	—	The identity provider’s metadata document, pasted. For an air-gapped network where the console cannot reach the IdP even though the user’s browser can. Wins over the URL when both are set.
`TUNEPLANE_SAML_SIGN_REQUESTS`	`bool`	`false`	Sign the AuthnRequest sent to the identity provider. Requires the keypair above, and is off by default because most identity providers do not ask for it.
`TUNEPLANE_SAML_SP_CERTIFICATE`	`str`	—	The PEM certificate matching the private key, published in this console’s SP metadata so the identity provider can verify what it signs.
`TUNEPLANE_SAML_SP_ENTITY_ID`	`str`	—	How this console names itself to the identity provider. Empty derives it from the public URL, which is both unique and a document the IdP can fetch.
`TUNEPLANE_SAML_SP_PRIVATE_KEY`	`str`	—	PEM private key used to sign authentication requests and to decrypt encrypted assertions. Needed only if the identity provider asks for either; most do not.
`TUNEPLANE_SAML_WANT_ASSERTIONS_SIGNED`	`bool`	`true`	Refuse an assertion that is not signed. Leave this on: an unsigned assertion is a sign-in anybody who can reach this console can forge. Turn it off only for an identity provider that signs the response envelope instead, and turn that requirement on in its place.
`TUNEPLANE_SAML_WANT_RESPONSE_SIGNED`	`bool`	`false`	Refuse a SAML response whose envelope is not signed. Off by default: signing the assertion is the more common half, and demanding both from an identity provider that does one is a sign-in nobody can complete.
`TUNEPLANE_SERVE`	`bool`	`false`	Bind 0.0.0.0 instead of loopback. Turn it on for a team or container deployment.
`TUNEPLANE_STATIC_DIR`	`Path	None`	—	Directory holding the built console assets (web/dist). The server wheel ships them, so this is normally left alone.
`TUNEPLANE_TRUSTED_PROXY_HOPS`	`int`	`1`	How many reverse proxies sit in front of the console. It decides which entry of `X-Forwarded-For` is believed to be the caller, counted from the right — each proxy appends the peer it saw, so the leftmost entry is whatever the client chose to send. Rate limits and the `ip` column of the audit trail both read it. Set it to 0 when the console is exposed directly, so the header is ignored entirely.
`TUNEPLANE_WEB_JWT_SECRET`	`str`	generated	Signing key for access tokens. Generated per process when unset, which is fine on one machine and wrong for anything else: two replicas would sign with different keys, so every restart logs everyone out.

Proxied submission

Environment variable	Type	Default	Description
`TUNEPLANE_CLUSTER_PROFILE`	`str`	—	Default CLUSTER_PROFILE forwarded to jobs. A profile named by the client is recorded but does not override this.
`TUNEPLANE_NEMO_RL_DIR`	`str`	—	Path to NeMo-RL inside the training container, forwarded to the job as NEMO_RL_DIR. Required to submit NeMo-RL jobs. This is a path in the image, not on the host and not in the CLI project.

Storage

Environment variable	Type	Default	Description
`TUNEPLANE_AUDIT_RETENTION_DAYS`	`int`	`0`	How long an audit row is kept, in days. 0 keeps them forever, which is the default. Pruning does not break the tamper-evident chain: each run records the range it removed and the hash of the last row it deleted, and the verification endpoint accepts a gap that such a record explains — those records are themselves never deleted. Enforcing a retention period is a commercial feature.
`TUNEPLANE_AUDIT_RETENTION_INTERVAL_S`	`float`	`86400.0`	How often the audit retention task runs.
`TUNEPLANE_CLUSTER_SECRETS_FILE`	`str`	—	Path to a secrets file inside the container. When set, the path is forwarded and the secret values never enter the Ray dashboard.
`TUNEPLANE_DATA_SCAN_ALLOW_TERMS`	`str`	—	Comma-separated terms to suppress as known false positives.
`TUNEPLANE_DATA_SCAN_DENY_TERMS`	`str`	—	Comma-separated terms this deployment treats as sensitive, in addition to the built-in rules. Terms, never patterns: an operator pasting a regex into a settings box is how a config change becomes a backtrack that hangs a worker.
`TUNEPLANE_DATA_SCAN_POLICY`	`str`	`report`	What the platform does about sensitive values found in a dataset. `off` does not scan; `report` scans, records and shows the findings while admitting the job anyway; `block` refuses a submission whose data has a high-confidence finding. Blocking keys on severity, never on any finding — the low-confidence rules hit every base64 blob and git SHA in a code corpus.
`TUNEPLANE_DB_MAX_OVERFLOW`	`int`	`10`	Connections the pool may open beyond `db_pool_size` under a burst. Ignored on SQLite, which pools nothing.
`TUNEPLANE_DB_POOL_RECYCLE_S`	`int`	`1800`	Recycle a pooled connection after this many seconds. Set it below any proxy or Postgres idle timeout, so a stale connection is replaced rather than discovered by a failing query. Ignored on SQLite.
`TUNEPLANE_DB_POOL_SIZE`	`int`	`5`	Connections kept open to the database. Sized per process, because an API container wants many short checkouts and a worker container few long-held ones. Ignored on SQLite.
`TUNEPLANE_DB_POOL_TIMEOUT_S`	`float`	`30.0`	How long a request waits for a free connection before failing. Short in the API on purpose, because queueing behind a background scan turns one slow job into a slow console. Ignored on SQLite.
`TUNEPLANE_INGEST_TOKEN_DAYS`	`int`	`30`	How long a job’s ingest token stays valid. It has to outlast the longest training run you expect, or a long job loses the ability to report partway through.
`TUNEPLANE_JOB_RUNNER_MODE`	`str`	`bundled`	How the platform runtime reaches a job. `bundled` injects a content-addressed PEX, so the training image needs nothing. `image-installed` requires tuneplane preinstalled in the image and exists only as a migration rollback — there is never an automatic fallback between the two.
`TUNEPLANE_JOB_RUNNER_PATH`	`str`	—	Where the bundled runner PEX is read from. Empty means <repo_root>/dist/tuneplane-runner.pex.
`TUNEPLANE_JOB_RUNNER_PYTHON`	`str`	—	Deployment-level override for the interpreter the runner uses. Empty uses whatever the runtime artifact declares.
`TUNEPLANE_JOB_RUNNER_REQUIRES_PYTHON`	`str`	`>=3.10,<3.14`	Python versions the bundled runner supports. A training image outside this range is refused rather than failing later.
`TUNEPLANE_LAUNCH_ADOPT_WINDOW_S`	`float`	`900.0`	How long a job row may sit without taking the handle of a workload launched for it, before that workload counts as orphaned and is reclaimed. It bounds how long a reclaim waits, because the dequeue loop relaunches under the same handle within a tick.
`TUNEPLANE_LAUNCH_RECLAIM_GRACE_S`	`float`	`30.0`	How long a launch reclaim waits before its first check. Long enough that an ordinary launch has already promoted and stood the policy down, so the common case never reaches the handler.
`TUNEPLANE_LICENSE_FILE`	`str`	—	A file holding the licence instead. Useful because a licence is a few hundred bytes nobody wants in a Helm values file or a shell history, and mounting a secret as a file is how Kubernetes prefers to do this. `license_key` wins if both are set.
`TUNEPLANE_LICENSE_KEY`	`str`	—	The commercial licence, as the signed string the vendor issued. Empty runs the community edition, which is a supported configuration rather than a degraded one — everything in it works and keeps working. Verified offline: there is no call home, ever. Settable from the admin console, and it takes effect on the next request rather than needing a restart, because entitlement is checked per request rather than at mount time.
`TUNEPLANE_MAX_UPLOAD_MB`	`int`	`1024`	Size ceiling for a submitted job package, compressed. The uncompressed ceiling is fifty times this, which is what stops a compression bomb.
`TUNEPLANE_OUTBOX_BATCH_SIZE`	`int`	`32`	Terminal actions delivered per tick. Bounded, because one of them submits a job and another writes a dataset version.
`TUNEPLANE_OUTBOX_POLL_INTERVAL_S`	`float`	`5.0`	How often the outbox worker delivers what finished runs owe. Short, because the user’s “your run finished” notification is now on the other side of it.
`TUNEPLANE_OUTBOX_RETENTION_DAYS`	`int`	`7`	How long a delivered action is kept. Long enough to answer whether last night’s publication actually ran, short enough that this does not become a second job history.
`TUNEPLANE_PASSTHROUGH_ENV`	`dict`	generated	Non-secret environment variables forwarded into every job, such as an HF endpoint mirror.
`TUNEPLANE_PIPELINE_STALL_GRACE_MINUTES`	`int`	`15`	How long after a submission’s last job finishes the platform may still be starting the next step of it. Inside the window a declared-but-missing step reads as still coming; past it the job list marks the submission stalled. Raise it where the queue routinely makes a chained job wait longer than this.
`TUNEPLANE_QUEUE_POLL_INTERVAL`	`float`	`5.0`	How often the dequeue loop looks for work.
`TUNEPLANE_QUOTA_ENFORCE`	`bool`	`true`	Whether the soft quota actually blocks admission. Turning it off records usage without refusing anything.
`TUNEPLANE_SERVER_SECRETS_FILE`	`str`	—	Server-side KEY=VALUE secrets file, injected into jobs when cluster_secrets_file is not used.
`TUNEPLANE_STOP_CONFIRM_WAIT_S`	`float`	`15.0`	How long a user-initiated pause keeps asking the backend whether the workload has actually terminated. PAUSED does not hold GPU quota, so recording it while a pod is still terminating hands the next admission cards that are not free yet. Backends whose stop is synchronous never wait at all; this is for the ones that answer “accepted”, such as Slurm and Kubernetes.
`TUNEPLANE_STORAGE_ROOT`	`str`	—	The one path a deployment configures. Everything the platform writes derives from it: caches, run directories, job packages and control-plane state. The invariant is that this path is identical on the console, on every node, and inside every container — a shared filesystem or a same-path bind mount satisfies it. Empty means single-machine development, falling back to ./.tuneplane-data/storage.
`TUNEPLANE_SUBMIT_DRY_RUN`	`bool`	`false`	Assemble the runtime environment but never actually submit to Ray. Debugging aid.
`TUNEPLANE_VOLUME_ROOT`	`str`	—	Root of the governed Volumes a job may read, distinct from its training data. Must sit outside the storage root, which is checked at startup - that root is mounted writable into job containers, so a volume beneath it would be reachable by a second, writable path and the read-only mount would be decorative. Mount JuiceFS here, with the S3 settings pointed at its gateway, to give one content both an object API and a filesystem. Empty means this deployment does not support volumes.

Automatic retry

Environment variable	Type	Default	Description
`TUNEPLANE_AUTO_RETRY_DELAY_S`	`float`	`60.0`	How long to wait after a failure before resubmitting, so a transient fault has time to clear.
`TUNEPLANE_AUTO_RETRY_ENABLED`	`bool`	`false`	Whether a failed job is resubmitted automatically, resuming from its last checkpoint under the same run id. Off by default — turn it on deliberately.
`TUNEPLANE_AUTO_RETRY_MAX`	`int`	`1`	How many automatic retries a job gets. A job can override it with its own max_retries.
`TUNEPLANE_AUTO_RETRY_ON`	`str`	`started`	Which failures are retried. `started` retries only jobs that reached RUNNING — a configuration or environment error usually fails before that, and retrying it just fails again. `all` retries every failure.

LLM judge

Environment variable	Type	Default	Description
`TUNEPLANE_JUDGE_CACHE_ENABLED`	`bool`	`true`	Cache deterministic scores (temperature at or below zero) so the same prompt is not paid for twice.
`TUNEPLANE_JUDGE_ENABLED`	`bool`	`false`	Whether training jobs get a judge endpoint and token injected, so a reward function can score with an LLM against a Rubric. The upstream credential and the audit trail both stay on the platform side.
`TUNEPLANE_JUDGE_LLM_API_KEY`	`str`	—	API key for the judge’s upstream. Empty reuses agent_llm_api_key.
`TUNEPLANE_JUDGE_LLM_BASE_URL`	`str`	—	OpenAI-compatible upstream the judge calls. Empty reuses the diagnosis agent’s LLM configuration.
`TUNEPLANE_JUDGE_LLM_MODEL`	`str`	—	Model the judge scores with. Empty reuses agent_llm_model.
`TUNEPLANE_JUDGE_LLM_TIMEOUT`	`float`	`60.0`	Request timeout for one judge call.

Agent sandbox

Environment variable	Type	Default	Description
`TUNEPLANE_ENVIRONMENT_ALLOWED_HOSTS`	`str`	—	Hosts a job may reach for an `openenv-remote` environment, comma separated. Empty means none, deliberately: a training container usually has the run of the internal network, so accepting whatever host a manifest names would turn an environment reference into an outbound channel written by somebody else.
`TUNEPLANE_ENVIRONMENT_SELF_REGISTRATION`	`bool`	`false`	Whether an environment may authorise its own host by proving one answers there. Off by default, because turning it on moves the egress judgement from “an administrator named this host” to “an OpenEnv server answered here when the version was sealed, and we recorded who asked” — a weaker claim about the host and a much stronger one about the audit trail.
`TUNEPLANE_REFLOW_ENABLED`	`bool`	`true`	Whether the Reflow background task runs. It is on by default and does nothing until a deployment sets a capture sample rate above zero — capture is enabled per deployment, never globally.
`TUNEPLANE_REFLOW_FLUSH_INTERVAL_S`	`int`	`60`	How often captured traffic is flushed from memory into the buffer.
`TUNEPLANE_SANDBOX_CPUS`	`str`	`1`	CPU limit for one sandbox container.
`TUNEPLANE_SANDBOX_ENDPOINT`	`str`	—	An external E2B-compatible sandbox service to use instead of the built-in container.
`TUNEPLANE_SANDBOX_IMAGE`	`str`	—	Image for the first-party code sandbox. Off unless configured: running model-generated code is not something to enable by default, and there is no sensible image to guess.
`TUNEPLANE_SANDBOX_MAX_CONCURRENCY`	`int`	`8`	How many sandboxes may run at once.
`TUNEPLANE_SANDBOX_MEMORY`	`str`	`512m`	Memory limit for one sandbox container.
`TUNEPLANE_SANDBOX_TIMEOUT_S`	`float`	`30.0`	Wall-clock limit for one sandbox execution.
`TUNEPLANE_SANDBOX_TOKEN`	`str`	—	Bearer token for that external sandbox service.
`TUNEPLANE_SANDBOX_USER`	`str`	`65534:65534`	User the sandbox process runs as. Non-root by default; change it only if the image needs a different unprivileged account.

Playground

Environment variable	Type	Default	Description
`TUNEPLANE_PLAYGROUND_ENABLED`	`bool`	`false`	Whether users may start Playground sessions to chat with a run’s artifacts. The local backend only.
`TUNEPLANE_PLAYGROUND_IMAGE`	`str`	—	Inference image for Playground sessions. Must be a digest-pinned OCI reference.
`TUNEPLANE_PLAYGROUND_LOG_PULL_INTERVAL_S`	`float`	`5.0`	How often a live session’s container output is copied into the log ledger. A serving container carries no SDK and reports nothing, so without this pull a session shows no logs until it ends — and the minutes a model spends loading are the only ones anybody watches. 0 turns it off.
`TUNEPLANE_PLAYGROUND_MAX_GPUS`	`int`	`4`	GPU ceiling for a single Playground session.
`TUNEPLANE_PLAYGROUND_PORT_MAX`	`int`	`18099`	High end of that port range. It must not overlap the deployment range: where they overlap, a throwaway session can take the port a stable endpoint needs.
`TUNEPLANE_PLAYGROUND_PORT_MIN`	`int`	`18000`	Low end of the host port range Playground sessions claim, one port each.
`TUNEPLANE_PLAYGROUND_TTL_S`	`float`	`3600.0`	How long an idle session lives before it is stopped and its GPUs released.

Model deployments

Environment variable	Type	Default	Description
`TUNEPLANE_ALLOWED_IMAGE_REGISTRIES`	`str`	—	Registry hosts a user may name with `tp submit --image`, comma separated and matched exactly. Empty leaves first-party framework overrides unrestricted but closes custom images entirely; a production deployment should configure it explicitly.
`TUNEPLANE_DEFAULT_FLEET_KIND`	`str`	`local`	The backend of the Fleet seeded when a console has none yet. `local` is one container on the console host, `node` one container on a remote machine running the node daemon, `kuberay` an ephemeral RayCluster per job, `slurm` an exclusive allocation through slurmrestd. Read once, at first start — after that the backend is a property of the Fleet a workload was placed on, so registering a second Fleet is how a console gains a backend, and changing this moves no existing workload.
`TUNEPLANE_DEPLOYMENT_ALLOW_TRUST_REMOTE_CODE`	`bool`	`false`	Whether a deployment may load a model that executes code from its own repository. Off by default, because it means running code the model author wrote.
`TUNEPLANE_DEPLOYMENT_HEALTH_TIMEOUT_S`	`float`	`10.0`	Timeout for one health probe against a serving revision.
`TUNEPLANE_DEPLOYMENT_MAX_CONCURRENT_REQUESTS`	`int`	`0`	How many inference requests one deployment may have in flight through the console at once, or 0 for no cap. Backpressure rather than a quota - a saturated engine serves no faster for having more sockets open, and a refusal the caller can retry beats a queue that slows every request already in it.
`TUNEPLANE_DEPLOYMENT_MAX_GPUS`	`int`	`8`	GPU ceiling for one deployment revision.
`TUNEPLANE_DEPLOYMENT_PORT_MAX`	`int`	`18299`	High end of that port range.
`TUNEPLANE_DEPLOYMENT_PORT_MIN`	`int`	`18100`	Low end of the host port range deployments claim.
`TUNEPLANE_DEPLOYMENT_READINESS_TIMEOUT_S`	`float`	`900.0`	How long a new revision may take to become ready before it is judged failed. Large models loading from cold storage need the headroom.
`TUNEPLANE_DEPLOYMENT_SGLANG_IMAGE`	`str`	—	Serving image for SGLang deployments. Must be digest-pinned.
`TUNEPLANE_DEPLOYMENT_SHARED_ROOTS`	`str`	—	Extra read-only paths a deployment may load model weights from, comma separated.
`TUNEPLANE_DEPLOYMENT_VLLM_IMAGE`	`str`	—	Serving image for vLLM deployments. Must be digest-pinned.
`TUNEPLANE_IMAGE_EVALKIT`	`str`	—	Image for benchmark evaluation jobs. Unlike the training runtimes this one has no public upstream, so every deployment builds it (`./deploy/docker/build-runtimes.sh evalkit --push`). Leave it empty only if the image is registered in TUNEPLANE_RUNTIME_REGISTRY_FILE instead — with neither, `tp bench run` and the automatic evaluation pipeline cannot be submitted.
`TUNEPLANE_IMAGE_NEMO_RL`	`str`	—	Default image for NeMo-RL jobs. A tag is enough. A user’s `--image` takes precedence.
`TUNEPLANE_IMAGE_OPENRLHF`	`str`	—	Default image for OpenRLHF jobs.
`TUNEPLANE_IMAGE_PYTORCH`	`str`	—	Default image for native PyTorch training jobs (the `pytorch/*` methods). Like the other training runtimes it is built by the deployment (`./deploy/docker/build-runtimes.sh pytorch --push`), and its torch version must match the one the recipe pins — the job refuses to start otherwise. Leave it empty only if the image is registered in TUNEPLANE_RUNTIME_REGISTRY_FILE instead.
`TUNEPLANE_IMAGE_TRL`	`str`	—	Default image for TRL jobs.
`TUNEPLANE_IMAGE_VERL`	`str`	—	Default image for verl jobs.
`TUNEPLANE_INPROCESS_WORKERS`	`bool`	`true`	Whether background roles — dequeue, diagnosis, watchdog, daily report — run inside the web process. Turn it off to run them as separate processes (`python -m server.worker --role <name>`): the slow LLM-calling ticks stop occupying the event loop and inflating API tail latency, and the web replicas become stateless and freely scalable. On by default so a single-container deployment keeps working unchanged.
`TUNEPLANE_MODEL_DEPLOYMENTS_ENABLED`	`bool`	`false`	Whether managed model deployments — long-lived serving for application traffic — are available at all.
`TUNEPLANE_NODE_ADDRESSES`	`str`	—	Comma-separated node URLs, for example `http://gpu-1:7070,http://gpu-2:7070`. Requires the storage root to be mounted at the same path on the console and on every node.
`TUNEPLANE_NODE_INDEX_URL`	`str`	—	Package index the generated `install.sh` fetches the tuneplane-node package from. Empty means PyPI; an intranet deployment points this at its own mirror, since the machine running the script may have no route to the public internet.
`TUNEPLANE_NODE_LAUNCH_TIMEOUT_S`	`float`	`120.0`	Timeout for a launch request, covering container start on a node that already has the image. A first-time image pull runs asynchronously and does not count against this.
`TUNEPLANE_NODE_REQUEST_TIMEOUT_S`	`float`	`15.0`	Timeout for an ordinary request to an agent.
`TUNEPLANE_NODE_STATE_DIR`	`str`	`~/.tuneplane-node`	Where a node keeps the identity and credential it received when it joined a fleet. Not under the storage root on purpose - a node that cannot see the shared mount still has to be able to say who it is.
`TUNEPLANE_NODE_TOKEN`	`str`	—	Bearer token shared between the console and every agent. Both sides must have it; a missing token refuses startup rather than running unauthenticated.
`TUNEPLANE_RUNTIME_REGISTRY_FILE`	`str`	—	Deployment-level map from runtime_id to an execution artifact. Required on Slurm, which needs SIF or SQSH. Optional on local and kuberay, where an entry overrides the catalog’s image reference — which is how an internal mirror gets used.
`TUNEPLANE_SLURM_RUNTIME_PROFILE`	`str`	—	Which container runtime Slurm uses. Must be chosen explicitly; there is no auto-detection and no fallback between profiles.

Slurm executor

Environment variable	Type	Default	Description
`TUNEPLANE_SLURM_ACCOUNT`	`str`	—	Accounting account charged for jobs.
`TUNEPLANE_SLURM_CLUSTER_WAIT_TIMEOUT_S`	`int`	`1800`	How long to wait for every node in an allocation to join the Ray cluster before giving up.
`TUNEPLANE_SLURM_CPUS_PER_NODE`	`int`	`1`	CPUs requested per node.
`TUNEPLANE_SLURM_JWT_FILE`	`str`	—	File holding the JWT used for slurmrestd authentication.
`TUNEPLANE_SLURM_PARTITION`	`str`	—	Partition jobs are submitted to.
`TUNEPLANE_SLURM_QOS`	`str`	—	QoS requested for jobs.
`TUNEPLANE_SLURM_RAY_PORT`	`int`	`1200`	Port the Ray head listens on inside an allocation.
`TUNEPLANE_SLURM_REST_API_VERSION`	`str`	`v0.0.43`	slurmrestd API version this deployment talks.
`TUNEPLANE_SLURM_REST_URL`	`str`	—	slurmrestd endpoint. The control plane always goes through REST; there is no sbatch or SSH fallback.
`TUNEPLANE_SLURM_REST_USER`	`str`	—	Username the control plane authenticates to slurmrestd as.
`TUNEPLANE_SLURM_STORAGE_ROOT`	`str`	—	Where the storage root appears on the cluster’s parallel filesystem. Empty means the same absolute path as storage_root, which is the normal case.
`TUNEPLANE_SLURM_TIMEOUT`	`float`	`30.0`	Timeout for one slurmrestd request.
`TUNEPLANE_SLURM_TIME_LIMIT`	`str`	`24:00:00`	Wall-clock limit requested per allocation.
`TUNEPLANE_SLURM_TLS_VERIFY`	`bool`	`true`	Whether to verify slurmrestd’s TLS certificate. Leave it on outside debugging.
`TUNEPLANE_SLURM_WORKER_PORT_MAX`	`int`	`2999`	High end of that range.
`TUNEPLANE_SLURM_WORKER_PORT_MIN`	`int`	`2000`	Low end of the port range Ray workers use.

KubeRay executor

Environment variable	Type	Default	Description
`TUNEPLANE_K8S_API_SERVER`	`str`	—	Kubernetes API server address. Resolution order is explicit configuration, then the in-cluster service account, then unavailable. The console usually runs inside the cluster, so this is normally left empty.
`TUNEPLANE_K8S_CA_PATH`	`str`	—	CA bundle used to verify the Kubernetes API server.
`TUNEPLANE_K8S_CPU_LIMIT`	`str`	—	CPU limit for training pods. Empty means unbounded, decided by node capacity.
`TUNEPLANE_K8S_GPU_NODE_LABEL`	`str`	`tuneplane/gpu-series`	Node label whose value names the GPU series, so the Kubernetes scheduler places a job on the right hardware natively.
`TUNEPLANE_K8S_GPU_PASSTHROUGH`	`bool`	`true`	Whether pods actually request nvidia.com/gpu. False is simulation mode for a cluster with no GPUs — kind, OrbStack, CI — where Ray’s logical num-gpus still works so the platform’s own end-to-end tests can run.
`TUNEPLANE_K8S_IMAGE_PULL_POLICY`	`str`	`IfNotPresent`	Image pull policy for training pods.
`TUNEPLANE_K8S_IMAGE_PULL_SECRET`	`str`	—	Pull credentials for an internal registry.
`TUNEPLANE_K8S_INSECURE_SKIP_VERIFY`	`bool`	`false`	Skip TLS verification against the Kubernetes API. Debugging a self-signed certificate only.
`TUNEPLANE_K8S_MEMORY_LIMIT`	`str`	—	Memory limit for training pods. Empty means unbounded.
`TUNEPLANE_K8S_NAMESPACE`	`str`	—	Namespace training resources are created in. Empty uses the service account’s own namespace.
`TUNEPLANE_K8S_NCCL_SOCKET_IFNAME`	`str`	—	Which network interface NCCL uses. On a multi-NIC machine, leaving it unset lets NCCL pick the wrong one — which presents as “training starts but is absurdly slow” or a hang in all-reduce, and is very hard to trace. Common values are bond0, eth0, ib0.
`TUNEPLANE_K8S_SERVICE_ACCOUNT`	`str`	—	Service account the training pods run as, when they need cluster access of their own.
`TUNEPLANE_K8S_SHM_SIZE`	`str`	`64Gi`	Size of /dev/shm in a training pod. Ray’s object store lives there, and the container default of 64MB makes anything but a tiny batch OOM immediately.
`TUNEPLANE_K8S_STORAGE_MOUNT`	`str`	—	Where that PVC appears inside the pod. Empty means the same absolute path as storage_root, so the console and the container agree by default.
`TUNEPLANE_K8S_STORAGE_PVC`	`str`	—	PVC holding the storage root. Without it every job re-downloads tens of gigabytes of weights, which on an internal link costs more than the training does.
`TUNEPLANE_K8S_STORAGE_SHARED`	`bool`	`true`	Whether the claim is readable by every pod of one job at once. An RWO claim does not fail a multi-node job outright — it gives each pod its own volume, so checkpoint shards split across nodes and the run cannot be resumed. Declared here and verified against the live PVC.
`TUNEPLANE_K8S_TIMEOUT`	`float`	`30.0`	Timeout for one Kubernetes API request.
`TUNEPLANE_K8S_TOKEN`	`str`	—	Bearer token for the Kubernetes API, when not using the in-cluster service account.
`TUNEPLANE_K8S_VOLUME_PVC`	`str`	—	Optional second claim holding the volume root, mounted read-only. Empty means this deployment cannot honour volume references on KubeRay, and a job that requests one is rejected at admission.
`TUNEPLANE_KUBERAY_ACTIVE_DEADLINE_S`	`int`	`0`	Wall-clock ceiling for a running job, so a runaway does not hold cards indefinitely. 0 disables it.
`TUNEPLANE_KUBERAY_HEAD_MODE`	`str`	`colocated`	Whether the Ray head is also the first GPU compute node. `colocated` matches the shape NeMo-RL and KubeRay document. A dedicated GPU-less head changes the allocation shape and is not switched to implicitly.
`TUNEPLANE_KUBERAY_PRERUNNING_DEADLINE_S`	`int`	`1800`	How long a job may sit before running before it is judged failed. Without it, an unpullable image or a nodeSelector nothing satisfies leaves the job Pending forever while holding its queue slot. 0 disables the deadline.
`TUNEPLANE_KUBERAY_RAY_VERSION`	`str`	`2.55.1`	Ray version declared to KubeRay. It must match the Ray inside the training image. A mismatch presents as “the cluster came up but workers never registered”, with one easily-missed warning line as the only clue.
`TUNEPLANE_KUBERAY_SUBMISSION_MODE`	`str`	`K8sJobMode`	How the job is submitted. `K8sJobMode` starts a separate submitter pod; `SidecarMode` injects the submitter into the head pod — one pod and one scheduling round fewer, at the cost of not supporting a submitter pod template.
`TUNEPLANE_KUBERAY_TTL_SECONDS`	`int`	`600`	Kubernetes garbage-collection backstop. Normal terminal states are cleaned up by the platform as soon as logs are archived; this TTL only catches what the control plane failed to remove.

Local executor

Environment variable	Type	Default	Description
`TUNEPLANE_DISK_WATERMARK_PCT`	`int`	`90`	Stop dequeuing for everyone once the storage root passes this usage percentage, to protect the checkpoint writes of jobs already running. 0 turns it off. A different question from the storage quota: a quota asks who is using too much, a watermark asks whether the disk is about to fill.
`TUNEPLANE_LOCAL_CHECK_EXTERNAL_GPUS`	`bool`	`true`	Whether to look for processes occupying GPUs from outside the platform — somebody who logged in and started a notebook. Without a device plugin as a backstop this check earns its keep: otherwise the platform hands out a card that is already busy.
`TUNEPLANE_LOCAL_CHECK_GPU_HEALTH`	`bool`	`true`	Whether to check GPU health with nvidia-smi: pending ECC retirements, uncorrected errors, hardware throttling. A failing card stops being allocated and the capacity view says why. These faults raise no error — they produce bad results or a silent slowdown.
`TUNEPLANE_LOCAL_CLI_TIMEOUT`	`float`	`60.0`	Timeout for one docker or podman CLI call.
`TUNEPLANE_LOCAL_CONTAINER_TTL_S`	`int`	`600`	Cleanup backstop for orphaned containers. A normal terminal state archives logs and removes the workload immediately; this only catches what reconciliation missed.
`TUNEPLANE_LOCAL_CPU_LIMIT`	`str`	—	Passed to `--cpus`. Empty means unbounded.
`TUNEPLANE_LOCAL_GPU_COUNT`	`int`	`0`	Number of physical GPUs. 0 detects them with nvidia-smi, which a containerised console should not rely on: the probe needs the host’s NVIDIA Container Toolkit injection, and a failed injection reads as zero cards.
`TUNEPLANE_LOCAL_GPU_PASSTHROUGH`	`bool`	`true`	Whether containers actually receive GPUs. False is simulation mode: allocation, accounting and labels all behave normally but the container runs without `--gpus`, which is how the platform’s end-to-end tests run on a machine with no NVIDIA runtime. Production must leave this on.
`TUNEPLANE_LOCAL_IMAGE_PULL_TIMEOUT_S`	`float`	`3600.0`	Timeout for the explicit pull before a first launch with a new image. Training images are routinely tens of gigabytes, so this cannot share the sixty-second CLI timeout — that would make “first use of a new image” fail every time.
`TUNEPLANE_LOCAL_MEMORY_LIMIT`	`str`	—	Passed to `--memory`. Empty means unbounded.
`TUNEPLANE_LOCAL_NETWORK`	`str`	`host`	Container network mode. Host networking is the simplest workable choice: a job has to reach back to the console to report status and artifacts, and on a single machine there is nothing to isolate it from.
`TUNEPLANE_LOCAL_RUNTIME`	`str`	`auto`	Container runtime. `auto` probes docker, then podman, then falls back to a bare process. Process mode isolates nothing and can leave GPU memory behind, so it is for a development machine only — that isolation is the entire reason for containerising in the first place.
`TUNEPLANE_LOCAL_SHM_SIZE`	`str`	—	Size of /dev/shm in a training container. Empty follows k8s_shm_size. It is RAM-backed, so this plus the memory limit must stay under physical memory.
`TUNEPLANE_LOCAL_STOP_TIMEOUT`	`int`	`30`	How many seconds a stopped job gets after SIGTERM to save a checkpoint before SIGKILL.
`TUNEPLANE_REGISTRY_RECONCILE_INTERVAL_S`	`float`	`300.0`	How often the dataset and corpus projection is rebuilt from object storage. It is a backstop now that the write paths upsert directly, but it is also the only thing that repairs an entry changed outside the API. 0 turns it off.
`TUNEPLANE_STORAGE_SCAN_INTERVAL_S`	`float`	`600.0`	How often the storage root is walked to produce per-user usage and platform totals. 0 never scans, and storage quotas and the dashboard card then report “not measured”. A full scan takes minutes on a large tree, which is why it is a background role rather than something computed on request.
`TUNEPLANE_TERMINAL_CLEANUP_GRACE_S`	`float`	`2.0`	How long to wait after a lifecycle response before cleaning up, so the training process can finish writing its last stderr.
`TUNEPLANE_TERMINAL_LOG_TAIL_LINES`	`int`	`5000`	How many trailing log lines to pull from the container, pod or Slurm archive when the training side did not manage to report them itself.

Object storage

Environment variable	Type	Default	Description
`TUNEPLANE_ALLOWED_RECIPES`	`str`	—	Which post-training methods this deployment offers, as `<framework>/<method>`, comma separated. Empty enables everything in the catalog. The catalog says what the platform supports; this says what this deployment opens to its users — a cluster without the memory for distillation turns it off here rather than in the catalog.
`TUNEPLANE_HWCONFIG_REFRESH_INTERVAL`	`float`	`15.0`	How often each replica reloads the hardware profile configuration.
`TUNEPLANE_RECIPE_STRICT_PARAMS`	`bool`	`true`	Whether submission rejects hyperparameter keys the recipe never declared. On by default: a misspelled name that is silently ignored leaves the user believing they tuned something.
`TUNEPLANE_S3_ACCESS_KEY`	`str`	—	Access key for that endpoint. Server-side only.
`TUNEPLANE_S3_BUCKET`	`str`	`tuneplane`	Bucket everything is written into.
`TUNEPLANE_S3_ENDPOINT`	`str`	—	S3-compatible endpoint for job packages, artifacts, datasets and archives. Credentials stay on the server: clients and jobs only ever receive presigned URLs, scoped to one method and one key with an expiry. So credentials never reach the Ray dashboard, a container environment, or a job log, and a leaked URL costs one key for one window rather than the whole bucket. Unconfigured means object storage is off and everything falls back to shared-filesystem paths.
`TUNEPLANE_S3_JOB_PRESIGN_TTL`	`int`	`259200`	Validity of a presigned URL handed to a job — its code package, or an `--init-from` artifact. It has to cover queueing as well as fetching: waiting hours in a queue is normal, and a URL that expires while queued fails the job on its first step.
`TUNEPLANE_S3_PRESIGN_TTL`	`int`	`3600`	Validity of an ordinary presigned URL.
`TUNEPLANE_S3_REGION`	`str`	`us-east-1`	Region passed to the S3 client.
`TUNEPLANE_S3_SECRET_KEY`	`str`	—	Secret key for that endpoint. Server-side only.
`TUNEPLANE_S3_SIGNATURE_VERSION`	`str`	`s3v4`	Signature version. Leave it at s3v4 unless the endpoint predates it.
`TUNEPLANE_TIMEZONE`	`str`	`Asia/Shanghai`	The IANA timezone the deployment’s business calendar runs on. Scheduling windows, daily GPU-hour boundaries and the billing month are all computed in it, so it has to be one zone for everybody — a quota that reset at a different instant per person would not be a quota. It is not a display setting. Timestamps in the console are rendered in each reader’s own browser timezone; the few sentences the server composes itself name this zone inline.

Scheduling windows

Environment variable	Type	Default	Description
`TUNEPLANE_ARGILLA_API_KEY`	`str`	—	API key for that Argilla instance.
`TUNEPLANE_ARGILLA_URL`	`str`	—	Argilla instance used for human preference annotation. Its address is a deployment-level setting at the same trust level as the database and object storage — on an internal deployment it is already on a private network, so the private-network guard that webhooks get does not apply here.
`TUNEPLANE_ARGILLA_WORKSPACE`	`str`	`argilla`	Argilla workspace annotation datasets are created in.
`TUNEPLANE_FAIRSHARE_WINDOW_H`	`float`	`24.0`	How far back fair-share looks. Too short and somebody who just finished a large job is immediately at the front again; too long and usage from weeks ago still holds today’s work back.
`TUNEPLANE_SCHEDULE_DISPATCH_MIN_REMAINING_S`	`float`	`1800.0`	Minimum time that must remain in a window for a job to be dispatched into it.
`TUNEPLANE_SCHEDULE_ENFORCE`	`bool`	`false`	Whether scheduling windows actually block work. Off means the configured windows are displayed but nothing is refused.
`TUNEPLANE_SCHEDULE_STOP_GRACE_S`	`float`	`600.0`	Grace period after a window closes before running jobs are stopped.
`TUNEPLANE_SCHED_POLICY`	`str`	`priority-fifo`	Dequeue ordering. `priority-fifo` is priority descending then first-come-first-served. `fair-share` puts whoever has used less recently ahead within one priority band. A single team will not notice the difference; several teams sharing one cluster will, because first-come-first-served lets one person queue twenty jobs inside their quota and fill the queue — which becomes an interpersonal problem immediately, and gets blamed on the platform.
`TUNEPLANE_SCHED_PREEMPTION`	`bool`	`false`	Turns a team quota from a ceiling into a guarantee. A team may run above it while the cluster has room, and that overflow is reclaimed — paused, with automatic resume from the last checkpoint — when a team below its own guarantee is waiting. Off by default, in which case a team quota is a hard cap and nothing is ever taken back. Borrowing and reclaim are one switch: lending without reclaim lets one team hold the cluster indefinitely.
`TUNEPLANE_SCHED_PREEMPT_MIN_RUNTIME_S`	`float`	`600.0`	How long a job is safe from reclaim after it starts. Without a floor a busy queue reclaims one job, admits another, reclaims that one, and the cluster spends its day writing checkpoints.
`TUNEPLANE_SCHED_RESERVE_AFTER_S`	`float`	`900.0`	How long a job may be denied on capacity before the scheduler starts holding cards for it. Without reservation the queue starves large jobs, and the platform itself generates the stream that starves them: lifecycle jobs and Playground sessions take one GPU each, so a denied four-GPU job is skipped and the one-GPU jobs behind it claim the capacity that was just freed, every pass, forever. The console shows it as “waiting for capacity”, indefinitely. Reserving costs idle cards while the reservation fills, which is why it only applies to jobs already starved this long. 0 disables reservation.
`TUNEPLANE_SWEEP_EARLY_STOP_ENABLED`	`bool`	`true`	Deployment-wide kill switch for sweep early stopping. Early stopping is configured per sweep by the client, so this exists purely so that a behaviour which actively kills jobs can be turned off globally without editing any sweep.

Maintenance mode

Environment variable	Type	Default	Description
`TUNEPLANE_MAINTENANCE_MODE`	`bool`	`false`	When on, new submissions queue but are never dispatched, and drained jobs stay PAUSED with auto-resume set. Turning it off resumes automatically: the queue worker resubmits each paused job under its original run id and training continues from its checkpoint.
`TUNEPLANE_MAINTENANCE_NOTE`	`str`	—	Message shown to anyone whose submission is held by maintenance mode.

Runtime watchdog

Environment variable	Type	Default	Description
`TUNEPLANE_MCP_ENABLED`	`bool`	`true`	Whether the MCP server is mounted, exposing job configuration, metrics and logs to AI agents.
`TUNEPLANE_MCP_TOKEN_DAYS`	`int`	`30`	Validity of an MCP OAuth access token.
`TUNEPLANE_ONBOARDING_CARD`	`bool`	`true`	Whether the overview shows the first-run guide that walks a new user through installing the CLI, signing in and submitting a first job. It hides itself once any job exists.
`TUNEPLANE_PREMIUM_PUBLIC`	`bool`	`false`	Whether AI diagnosis, Ask Agent and MCP access are open to every signed-in user rather than administrators only.
`TUNEPLANE_WATCHDOG_CLUSTER_RECONCILE`	`bool`	`true`	Whether to reconcile cluster-wide Ray GPU usage against the ledger. This check does not depend on anything the client reports, which makes it the most reliable of the three.
`TUNEPLANE_WATCHDOG_CLUSTER_TOLERANCE`	`int`	`0`	Tolerance in GPUs for the cluster-level reconciliation.
`TUNEPLANE_WATCHDOG_ENABLED`	`bool`	`true`	Whether the runtime watchdog runs. Bare-metal Ray has no cgroup or GPU hard isolation, so admission is a soft gate applied before the job starts; the watchdog periodically reconciles what the cluster is actually using against what the ledger says, alerts and audits on a mismatch, and can stop a job that reliably exceeds its allocation. It is defence in depth — a hard quota needs infrastructure like Kubernetes or Kueue.
`TUNEPLANE_WATCHDOG_ENFORCE`	`bool`	`false`	Whether the watchdog stops a job it reliably judges over-allocated, rather than only alerting.
`TUNEPLANE_WATCHDOG_FATAL_HANG_S`	`float`	`90.0`	How long a job may stay silent after its logs already show a fatal worker crash before it is marked FAILED and stopped. 0 disables it.
`TUNEPLANE_WATCHDOG_GPU_MIN_MEM_MIB`	`float`	`2048.0`	Memory a card must hold before it counts as in use by this job. It excludes idle cards, which report roughly 0.6 GB of leftover context, so a machine’s idle GPUs are not miscounted as over-allocation.
`TUNEPLANE_WATCHDOG_GPU_TOLERANCE`	`int`	`0`	How many GPUs a job may exceed its accounted allocation by before it counts as over.
`TUNEPLANE_WATCHDOG_INTERVAL`	`float`	`120.0`	How often the watchdog reconciles.
`TUNEPLANE_WATCHDOG_SILENCE_ALERT_S`	`float`	`1800.0`	How long an active job with a non-zero allocation may report no telemetry before an alert fires, which usually means collection was disabled or the job is wedged.
`TUNEPLANE_WATCHDOG_STARTUP_GRACE_S`	`float`	`300.0`	How long after a job reaches RUNNING before the watchdog will judge it, avoiding the noise of startup and warm-up.
`TUNEPLANE_WATCHDOG_TELEMETRY_WINDOW_S`	`float`	`300.0`	How recent telemetry has to be to count as current.

Diagnosis agent

Environment variable	Type	Default	Description
`TUNEPLANE_AGENT_COMPARE_MAX_TURNS`	`int`	`6`	Tool-call rounds the comparison assistant may take.
`TUNEPLANE_AGENT_DIAGNOSE_ON_FAIL`	`bool`	`true`	Whether a failure triggers a diagnosis immediately.
`TUNEPLANE_AGENT_DIAGNOSE_RUNNING`	`bool`	`true`	Whether jobs are diagnosed while still running, not only after they finish.
`TUNEPLANE_AGENT_DIAGNOSIS_COOLDOWN`	`int`	`900`	Minimum gap between two diagnoses of the same job.
`TUNEPLANE_AGENT_DIAGNOSIS_INTERVAL`	`float`	`300.0`	How often running jobs are considered for diagnosis.
`TUNEPLANE_AGENT_DIAGNOSIS_MAX_TURNS`	`int`	`8`	Tool-call rounds the diagnosis agent may take before it must answer.
`TUNEPLANE_AGENT_ENABLED`	`bool`	`true`	Whether the diagnosis agent runs at all.
`TUNEPLANE_AGENT_LLM_API_KEY`	`str`	—	API key for that endpoint.
`TUNEPLANE_AGENT_LLM_BASE_URL`	`str`	`https://api.openai.com/v1`	OpenAI-compatible endpoint the diagnosis agent calls, for example `https://api.openai.com/v1`.
`TUNEPLANE_AGENT_LLM_MODEL`	`str`	`gpt-4o-mini`	Model the diagnosis agent reasons with. The default is a light model to keep costs down, but multi-step diagnosis is sensitive to model capability — production should use a stronger one.
`TUNEPLANE_AGENT_LLM_TEMPERATURE`	`float`	`0.3`	Sampling temperature. A negative value omits the parameter entirely, which some reasoning models require — they reject any explicit temperature with a 400.
`TUNEPLANE_AGENT_LLM_TIMEOUT`	`float`	`60.0`	Timeout for one LLM call.

Secrets and integrations

Environment variable	Type	Default	Description
`TUNEPLANE_HF_ENDPOINT`	`str`	—	Hub API mirror for new job, Playground, and model serving containers, injected as HF_ENDPOINT. Enter the URL only, without export or quotes, for example https://nexus.example/repository/huggingface-proxy. Saving applies to new workloads without restarting the console; existing containers must be recreated. Empty uses the official Hub and disables the mirror-specific environment defaults. See air-gapped deployments.
`TUNEPLANE_HF_HUB_DISABLE_XET`	`bool`	`true`	With a mirror endpoint configured, inject HF_HUB_DISABLE_XET as 1 when enabled or 0 when disabled. Keep enabled for Nexus REST proxies. Applies to newly created job, Playground, and model serving containers; existing containers must be recreated. Explicit workload environment values take precedence.
`TUNEPLANE_HF_HUB_DOWNLOAD_TIMEOUT`	`int`	`300`	With a mirror endpoint configured, inject HF_HUB_DOWNLOAD_TIMEOUT into newly created job, Playground, and model serving containers. Defaults to 300 seconds to allow proxy cache misses. Must be positive. Existing containers must be recreated; explicit workload environment values take precedence.
`TUNEPLANE_HF_LOGIN_ENABLED`	`bool`	`false`	Offer “Sign in with Hugging Face” on the login page. Separate from the Hugging Face account binding below, which lets an already-signed-in user pull gated repositories — same OAuth application, two unrelated decisions. Sign-in asks only for identity scopes.
`TUNEPLANE_HF_OAUTH_BASE_URL`	`str`	`https://huggingface.co`	Hugging Face base URL. Change it only for an enterprise Hub.
`TUNEPLANE_HF_OAUTH_CLIENT_ID`	`str`	—	Client id of the Hugging Face OAuth application.
`TUNEPLANE_HF_OAUTH_CLIENT_SECRET`	`str`	—	Client secret of that application.
`TUNEPLANE_HF_OAUTH_ENABLED`	`bool`	`false`	Whether users can link a Hugging Face account, which is what lets a job read a gated model or push an export.
`TUNEPLANE_HF_OAUTH_SCOPES`	`str`	`openid profile email read-repos gated-repos`	Scopes requested when a user links their account.
`TUNEPLANE_HF_OAUTH_WRITE_ENABLED`	`bool`	`false`	Whether linking may request write scopes. Off by default, so the safe default is that nothing is ever uploaded.
`TUNEPLANE_HF_PREFLIGHT_ENABLED`	`bool`	`true`	Whether submission checks up front that the Hub resources a job names are reachable and authorised. Only takes effect when the Hugging Face integration is on.
`TUNEPLANE_HUB_HUGGINGFACE_ENABLED`	`bool`	`false`	Whether a workload may read from Hugging Face — a job referencing `hf:org/name` in a spec, or a model deployment serving a repository from it. Off by default; both pull the same weights over the same egress, so one switch governs both. Separate from Hugging Face sign-in and account linking, which are about credentials rather than about whether the platform reaches the hub at all.
`TUNEPLANE_HUB_MODELSCOPE_ENABLED`	`bool`	`false`	Whether a workload may read from ModelScope — a job referencing `ms:org/name` in a spec, or a model deployment serving a repository from it. Off by default. Public repositories only; private ModelScope repositories are not supported.
`TUNEPLANE_MODELSCOPE_API_BASE`	`str`	`https://modelscope.cn`	Where the console asks whether a ModelScope reference exists. Distinct from the ModelScope endpoint below, which is the download endpoint injected into job containers — a proxy serving file downloads need not serve the metadata API.
`TUNEPLANE_MODELSCOPE_API_TIMEOUT`	`int`	`60`	Inject MODELSCOPE_API_TIMEOUT when a ModelScope endpoint is configured. Default 60, minimum 1. Uses the current modelscope_hub SDK variable name. Applies to new workloads; explicit workload environment values take precedence.
`TUNEPLANE_MODELSCOPE_DOWNLOAD_MAX_RETRIES`	`int`	`5`	Inject MODELSCOPE_DOWNLOAD_MAX_RETRIES when a ModelScope endpoint is configured. Default 5, minimum 0. Uses the current modelscope_hub SDK variable name. Applies to new workloads; explicit workload environment values take precedence.
`TUNEPLANE_MODELSCOPE_DOWNLOAD_PARALLEL_WORKERS`	`int`	`1`	Inject MODELSCOPE_DOWNLOAD_PARALLEL_WORKERS when a ModelScope endpoint is configured. Default 1, minimum 1. Uses the current modelscope_hub SDK variable name. Applies to new workloads; explicit workload environment values take precedence.
`TUNEPLANE_MODELSCOPE_DOWNLOAD_TIMEOUT`	`int`	`300`	Inject MODELSCOPE_DOWNLOAD_TIMEOUT when a ModelScope endpoint is configured. Default 300, minimum 1. Uses the current modelscope_hub SDK variable name. Applies to new workloads; explicit workload environment values take precedence.
`TUNEPLANE_MODELSCOPE_ENDPOINT`	`str`	—	Inject MODELSCOPE_ENDPOINT into new job, Playground, and model serving containers. Enter a URL without export or quotes. Empty leaves ModelScope SDK defaults unchanged and disables the related download settings. Requires a compatible ModelScope SDK in the image; this does not change the engine’s model provider. Existing containers must be recreated.
`TUNEPLANE_SECRET_ENC_KEY`	`str`	—	Fernet key used to encrypt stored secrets at rest. Required once the Hugging Face integration is enabled — there is no implicit fallback, because a silently unencrypted secret store is worse than a refusal to start.

Webhook channels

Environment variable	Type	Default	Description
`TUNEPLANE_DAILY_REPORT_AGENT_MAX_TURNS`	`int`	`6`	Tool-call rounds the source-analysis agent may take.
`TUNEPLANE_DAILY_REPORT_DEEP_ANALYSIS`	`bool`	`false`	Whether the agent reads job working directories to identify new methods and techniques. Needs an LLM configured, and costs noticeably more.
`TUNEPLANE_DAILY_REPORT_DEEP_ANALYSIS_MAX`	`int`	`3`	How many projects one report may analyse deeply, which is the cost ceiling.
`TUNEPLANE_DAILY_REPORT_ENABLED`	`bool`	`false`	Whether the daily summary of post-training progress is generated and sent.
`TUNEPLANE_DAILY_REPORT_KEEP_DAYS`	`int`	`90`	How long the web version of a report stays readable before its link returns 404.
`TUNEPLANE_DAILY_REPORT_PROMPT`	`str`	—	Custom system prompt for the report. Empty uses the built-in default.
`TUNEPLANE_DAILY_REPORT_SECRET`	`str`	—	Signing secret for that dedicated webhook. Empty falls back to the DingTalk secret.
`TUNEPLANE_DAILY_REPORT_TIME`	`str`	`09:00`	Time of day the report is sent, HH:MM in the service timezone.
`TUNEPLANE_DAILY_REPORT_WEBHOOK_URL`	`str`	—	Dedicated webhook for the daily report. Empty falls back to the DingTalk webhook.
`TUNEPLANE_DAILY_REPORT_WEEKDAYS_ONLY`	`bool`	`true`	Whether the report is sent on weekdays only.
`TUNEPLANE_DINGTALK_AT_ALL`	`bool`	`false`	Whether to @-mention everyone on push.
`TUNEPLANE_DINGTALK_AT_MOBILES`	`str`	—	Phone numbers to @-mention on push, comma separated.
`TUNEPLANE_DINGTALK_ENABLED`	`bool`	`false`	Whether DingTalk push is active. Off silences both mirrored site messages and the daily report.
`TUNEPLANE_DINGTALK_NOTIFY_SITE_MESSAGES`	`bool`	`false`	Whether in-app notifications are mirrored into the DingTalk group.
`TUNEPLANE_DINGTALK_SECRET`	`str`	—	Signing secret, required when the robot’s security setting is set to signing.
`TUNEPLANE_DINGTALK_SITE_MESSAGE_KINDS`	`str`	`error,warning`	Which notification levels are mirrored, comma separated, to keep the group readable.
`TUNEPLANE_DINGTALK_WEBHOOK_URL`	`str`	—	DingTalk group robot webhook, including its access token.
`TUNEPLANE_WEBHOOK_ALLOWED_HOSTS`	`str`	—	Outbound host allowlist, comma separated. Empty forbids every webhook, deliberately: the console can usually reach the internal network, so allowing an arbitrary outbound target turns it into an internal port scanner.
`TUNEPLANE_WEBHOOK_DAILY_REPORT_URL`	`str`	—	Separate target for the daily report. Empty falls back to the primary target.
`TUNEPLANE_WEBHOOK_ENABLED`	`bool`	`false`	Whether the generic webhook channel is active. The console sends one templated HTTP request — extensions that would otherwise run inside the console process cross a process boundary instead of becoming plugins.
`TUNEPLANE_WEBHOOK_NOTIFY_SITE_MESSAGES`	`bool`	`false`	Whether in-app notifications are mirrored to the webhook.
`TUNEPLANE_WEBHOOK_SITE_MESSAGE_KINDS`	`str`	—	Which notification levels are mirrored, comma separated. Empty mirrors all of them.
`TUNEPLANE_WEBHOOK_TEMPLATE`	`str`	—	JSON body template, with {{title}} and {{text}} placeholders. Empty uses a generic shape.
`TUNEPLANE_WEBHOOK_URL`	`str`	—	Primary webhook target, used for mirrored site messages and as the daily-report fallback.

Get a deployment up

Executors

Platform governance

Running it

Configuration reference

The five you cannot skip

Reading the tables

Core

Proxied submission

Storage

Automatic retry

LLM judge

Agent sandbox

Playground

Model deployments

Slurm executor

KubeRay executor

Local executor

Object storage

Scheduling windows

Maintenance mode

Runtime watchdog

Diagnosis agent

Secrets and integrations

Webhook channels

​The five you cannot skip

​Reading the tables

​Core

​Proxied submission

​Storage

​Automatic retry

​LLM judge

​Agent sandbox

​Playground

​Model deployments

​Slurm executor

​KubeRay executor

​Local executor

​Object storage

​Scheduling windows

​Maintenance mode

​Runtime watchdog

​Diagnosis agent

​Secrets and integrations

​Webhook channels

The five you cannot skip

Reading the tables

Core

Proxied submission

Storage

Automatic retry

LLM judge

Agent sandbox

Playground

Model deployments

Slurm executor

KubeRay executor

Local executor

Object storage

Scheduling windows

Maintenance mode

Runtime watchdog

Diagnosis agent

Secrets and integrations

Webhook channels