tp bench

tp bench ls                                       # what can be evaluated here
tp bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu

<EXP> is an evaluation experiment, created once with tp new my-bench --method evalkit/benchmark and reused. The workflow around these commands is in Run a benchmark.

What `tp bench ls` prints

Built-in benchmarks ship with the platform; published ones arrive through tp recipe sync. Each row says which runner it uses, its primary metric, and its category.

math
  gsm8k      built-in  lm-eval    exact match (strict)  GSM8K · grade-school word problems
  math       built-in  lm-eval    exact match           MATH · competition mathematics
code
  humaneval  built-in  lm-eval    pass@1                HumanEval · code generation

The runner is usually not yours to pick

Which runner, how many few-shot examples, and the sample cap are declared by the benchmark itself — so you do not have to remember that GSM8K wants 5-shot or that HumanEval needs --confirm_run_unsafe_code. --runner is only for a raw suite name the platform has no pack for.If the benchmarks you select span two runners — say gsm8k on lm-eval and ceval on evalscope — the submission is refused and tells you to split it in two. One evaluation job is one process, and quietly splitting it would be worse than saying so.

Examples

tp new my-bench --method evalkit/benchmark              # once per project
tp bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu
tp bench run my-bench -m Qwen/Qwen2.5-7B --suites ceval
tp bench run my-bench -m run:run-4f2a91 --suites gsm8k --limit 20   # smoke test only

`tp bench`

Run standard benchmarks

tp bench <subcommand>

Command	Description
`tp bench external`	Report scores from an external harness
`tp bench ls`	List the available benchmarks
`tp bench new`	Scaffold a safety benchmark pack
`tp bench run`	Submit a benchmark evaluation job

`tp bench external`

Report scores from an external harness

tp bench external <subcommand>

Command	Description
`tp bench external create`	Open an external evaluation
`tp bench external submit`	Report a score file

`tp bench external create`

Open an external evaluation

tp bench external create [options]

Option	Description
`--model`, `-m`	Reference of the model under evaluation (deployment id / model version / HF id) (required)
`--harness`	Which harness is used, e.g. cvdp
`--train-run`	Which training run’s artifact is evaluated; only with this do the gate and the model version accept these scores
`--note`	Note

`tp bench external submit`

Report a score file

tp bench external submit <RUN_ID> [options]

Argument	Description
`RUN_ID`	The run_id that external create returned (required)

Option	Description
`--file`, `-f`	Score JSON file (required)
`--model`, `-m`	Model reference (used to group the variance)
`--harness`	Harness name

`tp bench ls`

List the available benchmarks

tp bench ls [options]

Option	Description
`--category`	One category only: math\|code\|knowledge\|chinese\|instruction\|hardware\|safety

`tp bench new`

Scaffold a safety benchmark pack

tp bench new <NAME> [options]

Argument	Description
`NAME`	Pack name, e.g. refusal-policy (required)

Option	Description
`--owner`	Namespace segment for the local catalog; defaults to your OS user. Not an identity claim — user packs are synced down from the platform, not pushed up from here
`--rubric`	<owner>/<name> of the rubric that decides what counts as a refusal
`--title`	Display title; defaults to the name

`tp bench run`

Submit a benchmark evaluation job

tp bench run <EXP> [options]

Argument	Description
`EXP`	The evaluation experiment (create it with tp new <name> —method evalkit/benchmark) (required)

Option	Description
`--model`, `-m`	HF model id / absolute path on shared storage / run:<run_id> (takes that run’s hf_export) (required)
`--suites`	Comma-separated benchmarks (e.g. gsm8k,mmlu). Run `tp bench ls` for the options (required)
`--runner`	Evaluation backend: lm-eval \| evalscope. Declared by the benchmark itself by default, rarely needed
`--limit` `<int>`	Sample cap per benchmark (for smoke runs)
`--extra-args`	Extra arguments passed straight to the runner
`--profile`	Target hardware and resources in one argument: h200 (the registry default shape), h200:4 (4 GPUs), h200:16 (2 full nodes). Repeat it to split pools per role (the heterogeneous extension point): —profile train=h200:8 —profile rollout=h100:2. Comma-separate roles that share a pool: —profile actor,rollout=h200:8 —profile teacher=h200:8 (repeatable)
`--allow-dirty`	Allow uncommitted changes in the working tree
`--repeat` `<int range>`	Repeat the evaluation this many times to measure the evaluation’s own variance (the gate needs it to call a regression) (default `1`)

CLI reference

Set up

Experiments

Train

Artifacts and evaluation

Resources

Administration

What `tp bench ls` prints

The runner is usually not yours to pick

Examples

`tp bench`

`tp bench external`

`tp bench external create`

`tp bench external submit`

`tp bench ls`

`tp bench new`

`tp bench run`

​What tp bench ls prints

​The runner is usually not yours to pick

​Examples

​tp bench

​tp bench external

​tp bench external create

​tp bench external submit

​tp bench ls

​tp bench new

​tp bench run

What `tp bench ls` prints

The runner is usually not yours to pick

Examples

`tp bench`

`tp bench external`

`tp bench external create`

`tp bench external submit`

`tp bench ls`

`tp bench new`

`tp bench run`