trace-to-training-data

name: trace-to-training-data description: Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.

Trace To Training Data

This skill assumes eval-harness-first already graded the traces being converted here — goldens, graders, and runs/<run-id>/results.json all exist before conversion starts. This is the flywheel edge that skill names in its own flow: "the same labeled traces become the training set." Conversion happens here; grading already happened upstream.

Input: graded traces — eval/goldens.jsonl plus runs/<run-id>/results.json, each row carrying a task_id, a verdict from the grader, and a reward when the task supports a scalar score (judge score, execution partial-credit, or an RLVR verifier):

{"task_id": "t-042", "trace_id": "t-042-a3",
 "messages": [{"role": "user", "content": "..."}],
 "verdict": "pass", "reward": 0.91,
 "grader": "exact_match"}

Output format: rows shaped exactly like dataset-curation's Format Selection table — SFT messages rows or DPO prompt/chosen/rejected pairs — so this skill's output is that skill's input with no reshaping step in between.

The Principle

The eval harness already did the labeling work: every trace in results.json carries a verdict, and often a reward, before this skill ever touches it. Converting a graded trace into a training row is mechanical — pick a shape from dataset-curation's table, map fields, write JSONL. Curation is the work that remains — which traces clear a quality bar, which pairs are informative, and which rows must never enter the training set at all.

Treat any conversion step that requires re-judging a trace as a sign the harness is missing a grader, not a gap this skill should paper over. A trace with no verdict or reward isn't convertible yet — route it back to eval-harness-first first, don't hand-label it here to unblock conversion.

SFT From Traces

• Keep the top-reward fraction of successful trajectories, not every passing one. Rank passing traces by reward and take a fraction (the Agent-lightning pattern) rather than every trace that merely cleared the pass bar — a trace that barely passed is a weaker SFT signal than one that scored well above threshold.
• Expert-corrected failures become gold SFT examples directly (the Langfuse pattern) — when a human edits a failing trace's output into a correct one, that correction needs no reward threshold; a human already validated it. Route corrections straight into the SFT set.
• Step-level masking beats whole-trajectory discard for multi-step traces. When only some steps in a multi-step trajectory are bad, mask the loss on the bad steps and keep the good ones, rather than discarding the whole trajectory. SRFT reports 32.2% vs. 30.9% on SWE-bench for step-level critic masking over trajectory discard — a real, if modest, gap from the finer-grained cut.

Preference Pairs From Traces

• Build pairs from passing-vs-failing trajectories on the SAME task, never from unrelated best- and worst-scoring traces pulled across different tasks — cross-task pairs teach the model to prefer one task over another, not one response over another.
• Select the rejected member at μ−2σ of the reward distribution for that task, never the absolute minimum. preference-optimization's Pair Construction section owns the full selection formula; this skill supplies the graded trajectories it consumes.
• Judge-scored delta selection cuts pair volume without cutting signal. Score each candidate pair by chosen-minus-rejected judge delta and keep only the highest-delta subset — the top 5k of a 16.5k candidate pool matched the full pool's downstream result. Build the full candidate set first, then filter by delta; don't cap generation at 5k up front.

Hygiene

• Scan for secrets and PII before any row ships, and redact what's found. Traces sourced from production logs can carry credentials, API keys, tokens, or customer data — run a secret/PII scan over every SFT and DPO row and redact matches; conversion fails closed (the row is dropped, not shipped with the raw content) if sensitive fields remain after redaction. Never commit secrets.
• Eval goldens must never leak into training data. Hold every eval/goldens.jsonl ID out of every converted SFT and DPO set — a trace that also appears as a golden trains on the exact item the checkpoint gets graded against later, silently inflating every subsequent eval run.
• Dedup against the training set, not just within the newly converted rows — exact-match or embedding-similarity, matching dataset-curation's dedup method field, run against whatever training data already exists before this batch merges in.
• Provenance goes into the dataset card. Every converted row must trace back to its source run_id and trace_id — dataset-curation's Provenance field checks for exactly this link back to trace-to-training-data output; a row with no traceable source isn't ready to merge.

Related Skills

• eval-harness-first — produces the graded traces this skill converts; a trace with no verdict or reward isn't convertible yet, route it back there before conversion.
• dataset-curation — owns the target formats and the dataset card this skill's provenance data feeds; converted rows must match its Format Selection table field names exactly, not an approximation of them.
• preference-optimization — consumes the DPO pairs this skill builds and owns the full μ−2σ rejection-selection formula referenced above.

Worked JSONL-to-JSONL conversions — graded trace to SFT row, trace pair to DPO pair, correction to SFT row, the rejection-sampling loop, and the goldens-holdout check — live in references/conversion-recipes.md.