Data shapes and design-lineage splits

data:
  quality:
    group_key: design            # what a row belongs to
    max_group_leak_ratio: 0.0    # how much of the held-out set may share one

A dataset here is a file of JSON objects and nothing more: the platform stores it, profiles it, scans it and hands it to a framework, and what is in a row is the framework’s business. That is the right default, and it stops being right at exactly one place — the point where somebody picks a recipe for a dataset and finds out at minute forty of a training run that the rows were preference pairs and the recipe wanted conversations. This page covers the two things the platform learned to say about the inside of a corpus: what shape its rows claim to be, and what a row belongs to so that a split can hold a whole design out rather than a random sample of one.

The nine shapes

A shape is declared, not inferred, and the checker only tests the claim. Inference was the alternative and it is worse in the way that matters: a heuristic that reads three rows and guesses is right almost always, and the almost is a silent mis-train rather than a refusal. There is no generic and no custom. A format that means nothing cannot be checked and would only be a field to fill in; a corpus whose shape is not here simply declares none, and everything else works as it always did. messages — chat SFT. A non-empty list, each entry carrying a non-empty role and content.

{"messages": [{"role": "user", "content": "Explain clock domain crossing"}, {"role": "assistant", "content": "A CDC is ..."}]}

prompt_completion — plain supervised text. Both keys non-empty strings.

{"prompt": "Write a 4-bit counter", "completion": "module counter(...);\n..."}

preference — DPO and friends.

{"prompt": "Review this always block", "chosen": "The sensitivity list ...", "rejected": "Looks fine."}

fim — fill-in-the-middle, which is how a code model learns to complete inside a file rather than after it. Only middle has to be non-empty: prefix and suffix are context and either may legitimately be empty at the edges of a file.

{"prefix": "always @(posedge clk) begin\n", "middle": "  q <= d;\n", "suffix": "end\n"}

repo_context — a completion with the rest of the repository in front of it. files is a non-empty list of {path, content} with no repeated path, plus a non-empty target. This is the shape RTL-Repo and every repository-level code benchmark use, and the one that cannot be flattened into a prompt string without losing which file is which.

{"files": [{"path": "rtl/uart.v", "content": "module uart(...);"}, {"path": "rtl/pkg.vh", "content": "`define W 8"}], "target": "  assign tx = ...;"}

spec_rtl — a natural-language specification and the module that implements it.

{"spec": "An 8-bit synchronous FIFO with full and empty flags", "rtl": "module fifo(...);\n..."}

testbench — a design and the bench that exercises it. The direction matters and is not symmetric with spec_rtl: here the model writes the bench, which is a verification engineer’s job.

{"dut": "module alu(...);\n...", "testbench": "module tb_alu;\n  initial begin ..."}

log_root_cause — a tool or regression log and what went wrong. The manufacturing-side shape as much as the design-side one.

{"log": "ERROR: [Synth 8-3352] multi-driven net ...", "root_cause": "The reset is driven from two always blocks"}

script — an EDA scripting task: Tcl, a Makefile, a constraint file. Kept apart from prompt_completion because what makes it correct is that the script runs, not that the text reads well.

{"request": "Constrain the 100 MHz input clock", "script": "create_clock -period 10.000 [get_ports clk]"}

A repeated path inside repo_context is refused rather than tolerated: whichever copy the framework keeps decides what the model sees, so the two entries would disagree about the same file in the same example.

What the checker may look at

Shape, and nothing else. Which keys are present, that they hold strings rather than nested structures, that a pair has both halves. Nothing about content, quality, language or length — those belong to the quality report, and a format check that started judging content would be a second opinion about data you own.

Conformance is a ratio, not a gate

from tuneplane.data.formats import DataFormat, conformance, parse

report = conformance(rows, DataFormat.SPEC_RTL)
print(report.rows, report.conforming, report.ratio)
for row_number, problem in report.examples:
    print(row_number, problem)

conformance takes any iterable of rows, so the caller decides how much to read. It reports how many rows it saw, how many conformed, the ratio, and at most ten complaints with their row numbers — a corpus with a million broken rows has one problem, not a million. Two deliberate refusals:

It never becomes a verdict. A gate that fires on any finding is a gate somebody turns off in a week, and a real corpus has malformed rows in it that nobody wants to argue about. What you are owed is the number and the first few examples.
An empty file has no ratio. ratio is None over zero rows, never 1.0. Nothing was checked, and “perfect” is the wrong way to say that.

parse returns None for an absent format — the ordinary case, and not an error — and raises with the sorted list of known names for one it does not recognise.

Splitting an engineering corpus at random leaks

Every contamination check the platform already has compares text. tp dataset check matches normalised 13-grams; the fingerprint comparison intersects sampled n-gram hashes. Both answer the question they were built for, and neither can see the failure that costs an engineering team its evaluation. Engineering corpora are full of near-variants of one artefact: eight revisions of a UART, a module and the wrapper that instantiates it, the same testbench parameterised four ways. Two revisions of one module genuinely share almost no n-grams while being, for the purpose of measuring what a model learned, the same problem. Split those at random and the evaluation set holds siblings of the training set. Every row is distinct. No duplicate ratio fires. The shingle overlap reads clean. The held-out score then measures recall of something the model has already seen, the number goes up, and the model is no better. So the check compares identity instead of text: whatever group_key names — a design, a repository, a case number — and the answer is the share of evaluation groups that also appear in training.

from tuneplane.data.contamination import group_leak

leak = group_leak(train_groups=["uart", "spi"], eval_groups=["uart", "pcie"])
leak.ratio     # 0.5 -- half the held-out designs were trained on
leak.clean     # False
leak.examples  # ("uart",)

Three properties worth knowing:

It is counted over distinct groups, not rows. One design appearing in a thousand evaluation rows is one leak. Counting it a thousand times would let a corpus’s row distribution decide how serious its leak looks.
ratio is None when nothing is held out, never 0.0, for the same reason an empty conformance report has no ratio.
It needs no text at all. Two sets and an intersection, so it can run on metadata before either side is read.

Zero is the only value worth aiming at for a held-out set. Anything above it means part of the score is recall.

Declaring the group key

spec.data.quality is the JobSpec block that carries the ceilings on the training data — the dual of the evaluation gates, which judge whether the model that came out is good enough. Two of its fields are about lineage:

Field	What it means
`group_key`	A metadata field naming the thing a row belongs to. Empty means row-level, which is the previous behaviour and the right default for a corpus of independent examples
`max_group_leak_ratio`	Ceiling on the share of groups appearing on both sides of a split, `0`–`1`

data:
  quality:
    max_duplicate_ratio: 0.1
    group_key: design
    max_group_leak_ratio: 0.0

The contract refuses two inconsistent specs outright:

A ratio outside 0–1.
max_group_leak_ratio without group_key — without it every row is its own group and the ceiling could never be exceeded, which is a line that never gets judged. The same rule already binds max_overlap_ratio to overlap_with.

All four solutions that ship declare a split field: design for the two RTL ones, repo for the script assistant, and case for log triage, where the same failure reported twice is one fact. tp solution show prints it as split by; see solutions.

What is wired up, and what is not

Being exact about this matters more than the feature does.

Piece	Status
The nine shapes and their validators	In the SDK, at `tuneplane.data.formats`, with tests
`group_leak`	In the SDK, at `tuneplane.data.contamination`, with tests
`group_key`, `max_group_leak_ratio`	Parsed and validated by the JobSpec contract
A solution’s `data_format`	Checked against the nine at manifest load
`max_duplicate_ratio`, `max_overlap_ratio`	Enforced at submission against the stored quality report and fingerprints
A group-leak check at submission	Not implemented. Nothing computes the ratio, so the declared ceiling refuses nothing
A format declared on a dataset version	Not implemented. `tp dataset push` has no format flag, and no dataset field carries one
A `data.quality` block in an experiment config	Not carried by `tp submit`. The CLI builds the JobSpec’s data section from the dataset and path arguments and leaves `quality` at its defaults, so the block reaches the platform only from a JobSpec constructed against the API

So today the nine names are a vocabulary a Solution declares and a library check you can run over your own rows — in a preprocessing script, or in CI before a push. group_key is the same: declaring it records the intent and refuses an inconsistent spec; computing the leak is group_leak’s job and calling it is yours. That is a smaller claim than “the platform stops a leaking split”, and it is the true one.

Datasets · Solutions · RTL benchmarks

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Data shapes and design-lineage splits

The nine shapes

What the checker may look at

Conformance is a ratio, not a gate

Splitting an engineering corpus at random leaks

Declaring the group key

What is wired up, and what is not

Next

​The nine shapes

​What the checker may look at

​Conformance is a ratio, not a gate

​Splitting an engineering corpus at random leaks

​Declaring the group key

​What is wired up, and what is not

​Next

The nine shapes

What the checker may look at

Conformance is a ratio, not a gate

Splitting an engineering corpus at random leaks

Declaring the group key

What is wired up, and what is not

Next