Skip to main content
Tasksets live under Assets. Agent evaluations live under Evaluation. A taskset defines the work and its train/eval split. An environment defines where an agent acts. Harbor coordinates the agent, sandbox and verifier for each trial. You can reuse a taskset across models and execution environments.

​
Publish an immutable taskset

Place Harbor task directories beneath a common root. Each directory must contain task.toml; task IDs equal directory names and must be unique.
build records file hashes in taskset.json. To mix training and evaluation tasks, edit individual split fields before publishing. Modified task content requires updated hashes and a new version; existing manifests are never overwritten by build. Symlinks, overlapping task directories and path traversal are rejected. The console displays sealed versions, split counts and the content digest. It exposes no task text or reference-answer download. Private versions are visible only to authorized users. Jobs reference owner/name@version; floating versions are rejected. Authorized local-container jobs receive a read-only taskset mount through TUNEPLANE_JOB_TASKSET_DIR. Other delivery backends are not supported yet. Split labels select the evaluation cohort; they do not hide files from arbitrary code inside an authorized job. Publish held-out tasks as a separate taskset when you need separate access permissions.

​
Prepare a Harbor evaluation

Open Agent evaluations, or follow the link from a taskset version. Choose:
  • A sealed taskset version; only its evaluation split runs.
  • A hosted model endpoint and immutable model revision.
  • Docker or E2B as the sandbox backend available in your runtime.
  • Runtime and harness SHA-256 identities, repetitions, concurrency and trial timeout.
The page validates the request and downloads harbor-evaluation.json. It does not launch a job. Use the harbor-eval experiment in tuneplane-examples, place this file at the project root, and follow that experiment’s README. Its --check-plan mode validates local task content without importing Harbor or creating sandboxes. A platform submission uses tp submit --taskset with a configured local profile and pinned runtime image.

​
Capacity and results

A request supports up to 10,000 planned trials and 256 concurrent trials. Without a selected sandbox pool, this is only a per-job limit. Actual capacity depends on the provider and runtime. Docker access or E2B credentials must be provisioned in the runtime; selecting a backend in the console does not provision it. Infrastructure failures stop new trials. Cooperative cancellation waits for started trials to clean up and saves partial results. Force-kill cleanup requires provider TTLs and reconciliation. Every planned outcome contributes to coverage; missing and unscored results do not become successful evaluations. Text capture is disabled by default. Taskset content, verifier content, runtime, harness and actual execution settings accompany results for comparison. Runtime and harness hashes are declarations, not certification of the installed backend. Shared pool capacity and leases are available through the Sandbox pools page. A console launch/history workflow and native provider certification remain separate work.