Air-gapped deployments

TUNEPLANE_HF_ENDPOINT=https://mirrors.your-company.com/repository/huggingface

That one setting is what makes weight downloads work behind a mirror. The rest of this page is the other three things that break, and what to do about each.

Model weights

huggingface_hub reads HF_ENDPOINT, and the platform injects TUNEPLANE_HF_ENDPOINT into every training and Playground container as exactly that. Point it at a reverse proxy — Nexus, Artifactory, or anything else that fronts the Hub. Two adjustments travel with it automatically, and both exist because of a failure that is painful to diagnose:

Timeouts are widened

huggingface_hub defaults to a ten-second timeout. A mirror’s first fetch of a file it has never cached goes all the way to the origin, and routinely takes longer than that. The symptom is a download that fails on a cold file and succeeds on a retry, which reads like a flaky network rather than a configuration problem.

Xet is disabled

Xet needs short-lived tokens that a reverse proxy cannot issue. Left on, weight downloads stall on a 400 whose body contains nothing but a URL — no message, no field, nothing to search for.If your image genuinely implements Xet, override it explicitly through TUNEPLANE_PASSTHROUGH_ENV.

A job whose weights cannot be fetched does not fail fast. It starts, logs the download, and sits there until it times out — holding its GPU allocation the whole time. Confirm the mirror works before opening submission to a team.

Container images

Training images come from the recipe catalog, which names public registries such as nvcr.io. On an isolated network you mirror them and redirect:

Setting	What it does
`TUNEPLANE_RUNTIME_REGISTRY_FILE`	Maps a `runtime_id` to the artifact your registry actually holds, overriding the catalog’s reference
`TUNEPLANE_ALLOWED_IMAGE_REGISTRIES`	The hosts a user may name with `--image`. Set it to your internal registry
`TUNEPLANE_K8S_IMAGE_PULL_SECRET`	Pull credentials, on KubeRay

Pin digests rather than tags in production. A tag that moves inside your own mirror is still a tag that moved.

Building the image behind a mirror

The Dockerfile defaults to public upstreams so it builds anywhere. Point it at an internal mirror with build arguments:

docker build \
  --build-arg APT_MIRROR=https://mirrors.your-company.com/repository \
  --build-arg PYPI_INDEX=https://mirrors.your-company.com/repository/pypi-group/simple/ \
  --build-arg NPM_REGISTRY=https://mirrors.your-company.com/repository/npm-group/ \
  --build-arg UV_IMAGE=mirror.your-company.com/astral-sh/uv:python3.13-trixie-slim \
  -t tuneplane-server:0.3.0 .

Leaving APT_MIRROR empty keeps Debian’s own sources, which is what you want on a machine with normal internet access.

The platform runtime

The job side needs tuneplane to report metrics, and the training image will not have it. TUNEPLANE_JOB_RUNNER_MODE=bundled, the default, solves this without network access: the server injects a content-addressed PEX into the job, so the training image needs nothing installed. This is one of the few places where the air-gapped case is the easy one — leave the setting alone.

Datasets

Nothing here reaches the internet. Datasets are pushed with tp dataset push into the deployment’s own object storage, and pulled into the shared cache when a job starts. See datasets.

Benchmark data

Benchmarks fetch their own data, and each harness reaches a different place. Two of the three need something from you. lm-eval (gsm8k, mmlu, humaneval, ifeval, math) downloads from the Hub, so TUNEPLANE_HF_ENDPOINT above already covers it. Nothing extra. evalscope (C-Eval and the other Chinese suites) is ModelScope-native: it downloads evalscope/ceval from www.modelscope.cn. Its override is MODELSCOPE_DOMAIN, and it differs from HF_ENDPOINT in a way that decides your topology: it is a domain, not a URL. The client builds https:// + the value, with no path. A Nexus proxy repository lives at /repository/<name>, so you cannot name it directly — the mirror has to answer at the root of a hostname of its own:

server {
  server_name modelscope.mirrors.your-company.com;
  location / { proxy_pass https://mirrors.your-company.com/repository/modelscope/; }
}

Then hand it to jobs. Nothing in the platform needs changing — the passthrough exists for exactly this:

TUNEPLANE_PASSTHROUGH_ENV='{"MODELSCOPE_DOMAIN": "modelscope.mirrors.your-company.com"}'

Switching evalscope to the Hugging Face hub instead does not work, and the failure is misleading rather than obvious: --dataset-hub huggingface changes where it looks without translating the id, so evalscope/ceval is requested from a mirror that has never had it and the error reads “dataset not found” rather than “no route to ModelScope”. Some suites fetch a fixed URL. lm-eval’s math500, for instance, reads a CSV from openaipublic.blob.core.windows.net. There is no endpoint setting for these: either publish a raw proxy for that host, or accept that the suite is unavailable. This is worth checking per suite before promising a benchmark to anyone.

Confirm it worked

Submit the quickstart job and watch the logs. The sequence you want is:

resolving Qwen/Qwen2.5-1.5B via https://mirrors.your-company.com/repository/huggingface
downloading model-00001-of-00002.safetensors
...
Ray runtime started

A stall between the first and second line is the mirror. A 400 with a bare URL in the body is Xet. A timeout on a large file after roughly ten seconds means the widened timeouts are not being applied — check that the platform, not your own script, is setting HF_ENDPOINT.

Get a deployment up

Executors

Platform governance

Running it

Air-gapped deployments

Model weights

Container images

Building the image behind a mirror

The platform runtime

Datasets

Benchmark data

Confirm it worked

​Model weights

​Container images

​Building the image behind a mirror

​The platform runtime

​Datasets

​Benchmark data

​Confirm it worked

Model weights

Container images

Building the image behind a mirror

The platform runtime

Datasets

Benchmark data

Confirm it worked