Skip to main content
For a handful of GPU machines, when you do not want Kubernetes or Slurm. Each machine runs one tuneplane-node daemon; the console places jobs over a narrow HTTP API — launch, observe, stop, logs, reap. The container spec is the same one local builds. GPU selection and docker run happen on the machine, under a single lock, and allocation truth stays in the container labels.
A job must fit on one node. Multi-node gang scheduling is not implemented, and a submission that needs it is rejected at admission rather than half-placed.

​
Prerequisites

  1. Docker + NVIDIA Container Toolkit on every GPU machine
  2. Shared storage mounted at the same path on the console host and every machine (TUNEPLANE_STORAGE_ROOT)
  3. A node Fleet on the console — Fleets → New fleet, or seeded at first start with TUNEPLANE_DEFAULT_FLEET_KIND=node

​
Registering a machine

Registration is an administrator’s job, not a user’s. The Fleet page mints a join token and shows one line to paste on the machine:
It installs the tuneplane-node package, registers the machine, and starts the daemon. Running it again on a machine that already has the node daemon is the upgrade path. On a machine that already has the package — a re-join, or one your configuration management installed — the same registration without the installer:
The join token is the whole authorization: you need root on the machine, not an account on the console. What comes back is this node’s identity and a credential, written under TUNEPLANE_NODE_STATE_DIR; tuneplane-node serve heartbeats with them. Mixed GPU types need nothing: the node reports its own cards at join, and the console maps the card name onto a hardware series. A card the registry does not know is reported at join so an administrator can add it — until then, a job that pins a series is not placed there.

​
Node configuration

Set on the machine, not on the console:
Run tuneplane-node serve as a systemd service. The daemon does not train; it only starts job containers.

​
Console configuration

TUNEPLANE_NODE_ADDRESSES and TUNEPLANE_NODE_TOKEN name a node set in the console’s own configuration. They are the pre-registration path, kept for a deployment that has not registered its machines yet; a Fleet with registered nodes never reads them.

​
Semantics

Least-loaded fit among nodes whose series matches the request and that have enough free GPUs. If nothing fits, launch raises NoCapacity and the job stays QUEUED.
Observe returns no signal for jobs on an unreachable node. Reconciliation skips that round. When the node returns, state converges from the real container. Those GPUs show as blocked in capacity while the node is down.
First pull on a node is async. The job stays PENDING with a pull note. TUNEPLANE_NODE_LAUNCH_TIMEOUT_S covers start when the image is already local.
Draining a node stops new placement and leaves running jobs alone. It is how a machine leaves the fleet without killing work.
Longer notes: docs/ops/executor-backends.md in this repository.