Native logs and the telemetry gateway

The Rust collector reads declared TensorBoard or JSONL sources without importing the training framework. It sends bounded batches to an independent telemetry gRPC service. The currently shipped gateway is a Python process; it has separate RPC capacity and a separate database pool from Node control. It is not an LLM inference proxy.

Deploy the listener

The gateway needs the migrated PostgreSQL database, the same signing configuration as the control plane, and the object-store configuration used for rich records and artifacts. Provide a trusted server certificate whose hostname matches the address reached by collectors.

python -m tuneplane_server.services.telemetry.gateway_main \
  --bind 0.0.0.0:8444 \
  --certificate /tls/tls.crt \
  --private-key /tls/tls.key

Compose supplies the telemetry profile. Helm supplies telemetryGateway.enabled, telemetryGateway.tlsSecret and port 8444. These are separate from nodeGateway and its port 8443. Expose gRPC with a TLS-preserving TCP route and verify reachability from execution hosts. Collectors use dedicated workload tokens bound to run, attempt and member, over server-authenticated TLS. Node control tokens and user login tokens cannot substitute for them. The platform provisions and renews these credentials; do not paste them into training code or experiment configuration. No collector client certificate is required by the normal deployment path.

Install the collector artifact

Build for the execution hosts’ operating system and architecture with the pinned Rust 1.91.0 toolchain and installed target/native toolchain. Linux production artifacts use musl. For example:

uv run --no-sync python scripts/build-rust-collector.py /srv/tuneplane/collectors \
  --target x86_64-unknown-linux-musl

The command prints a content-addressed directory containing manifest.json and tuneplane-collector. Deploy that complete directory where the control plane and launch workers can read it. Configure the managed telemetry profile together:

TUNEPLANE_TELEMETRY_GATEWAY_TARGET=telemetry.tuneplane.corp:8444
TUNEPLANE_TELEMETRY_TRUST_BUNDLE=/srv/tuneplane/certs/telemetry-ca.pem
TUNEPLANE_TELEMETRY_COLLECTOR_ARTIFACT=/srv/tuneplane/collectors/<sha256>
TUNEPLANE_TELEMETRY_SPOOL_ROOT=/var/lib/tuneplane-telemetry
TUNEPLANE_TELEMETRY_SPOOL_BYTES=268435456
TUNEPLANE_TELEMETRY_DRAIN_SECONDS=30

Use the corresponding ARM target for ARM hosts. The launch path verifies the artifact manifest and digest; a binary for the console host is not necessarily suitable for the training host. The official TensorBoard Python library is used for conformance testing, not installed into the training environment by the collector.

Provision local durable storage

The spool root must be private, durable host-local storage on every execution machine, not NFS or another shared network filesystem. Source logs and output artifacts can still use shared storage. Configure backend mounts explicitly; a missing or unsuitable spool grant must not silently fall back to shared output storage. For Local Compose deployments, deploy/docker/compose.telemetry-local.yml mounts the explicitly configured host spool root into the execution services. Kubernetes and Slurm also require their backend-specific local-storage grants. Object delivery does not make a shared filesystem suitable for a SQLite spool. See storage and Node execution.

Select sources and inspect results

The integration artifact declares sources, ownership and metric mappings. Assign one reader to each shared source. Do not assume every framework writes on every rank, discard nonzero-rank sources blindly, or deduplicate equal values from different members. Preserve their source identities; publish an aggregate from its designated owner. TensorBoard scalars and structured JSONL records are distinct inputs. Large TensorBoard images and histograms are not implicitly streamed as scalar data. Rich JSONL channels have their own schemas and budgets. W&B can coexist as an experiment tool; its proprietary local history is not a supported source merely because TensorBoard reading is enabled. Open the Job’s Charts tab. Canonical curves use the frozen mapping; the raw scalar inspector lets you choose attempt and source. Queries bound returned points and series. A reduced plot is not a claim that all raw records were returned. Unknown metric semantics stay explicit rather than being guessed from a familiar tag name.

Failure and recovery checks

Symptom	Check
Node is connected but charts are empty	The telemetry address and CA are independent of the Node control address; inspect source declarations and the collector separately
Data stops during a gateway outage	Confirm the local spool persists and has room; restart the gateway and verify replay resumes
A send succeeds but data is not complete	An ACK follows durable admission; rich-data projection and artifact completion require their own evidence
Duplicate points after retry	Compare stream, delivery sequence, attempt and source identity; equal values alone are not duplicates
A resumed run starts again at step zero	Select the correct attempt; earlier attempt history must remain separate
Preparation is slow	Inspect image pull and verified input staging; asynchronous preparation alone is not a launch failure

A lost ACK replays the same durable batch identity. Disk loss, exhausted source retention and a destroyed spool are different failures from a process restart and do not inherit its recovery guarantee. Verify the actual storage and backend profile before claiming multi-machine durability. The existing HTTP/SDK reporting path still serves supported lifecycle, log and reporting use cases. Native telemetry is not configured merely by setting TUNEPLANE_INGEST_URL.

Get a deployment up

Executors

Platform governance

Running it

Native logs and the telemetry gateway

Deploy the listener

Install the collector artifact

Provision local durable storage

Select sources and inspect results

Failure and recovery checks

​Deploy the listener

​Install the collector artifact

​Provision local durable storage

​Select sources and inspect results

​Failure and recovery checks

Deploy the listener

Install the collector artifact

Provision local durable storage

Select sources and inspect results

Failure and recovery checks