An SFT dataset from agent traces for a small model, measured instead of estimated
To teach Qwen3-0.6B to operate the Pi coding agent, I built the data side first: a pipeline that converts public agent traces into Pi's exact wire format, a harness contract with a fixed tool vocabulary, probes and an evaluation suite. Context limits were set from exact tokenizer counts, not character estimates, and traces that don't fit the tool contract are rejected by deterministic gates with a ledger of why.
Most small-model agent fine-tunes fail before training starts: the traces don't match the harness's tool format, or the context cap silently cuts off the part worth learning. This project built the data side carefully first.
What's in it
- A data pipeline with canonical records, deduplication, redaction, manifests and fixed splits, and a converter into Pi's exact wire format (OpenAI-style function calls and tool results).
- A harness contract: a fixed tool vocabulary, a schema'd harness card, serialisation rules and rollout code, with tests.
- Probes for the base model's raw behaviour and reasoning, before any training.
- An evaluation suite with a spec, schemas and tools.
- Decision reports written from measurements, such as the training context length.
Measured, not estimated
- Context length came from the real tokenizer and chat template, not characters divided by four. The measurement showed that a 4,096-token cap would have thrown away most of the supervised tokens, and that a cap above the model's native context added nothing.
- Bad traces are rejected deterministically. Only 8 of 15 normalised sessions survived conversion in the first pass, and none of the 7 losses was about length: all failed structural gates such as an unsupported tool or legacy edit arguments. Every rejection is logged with its reason.
- Conversions are reproducible. Re-running the converter in memory reproduces the on-disk records byte for byte, along with the reject ledger.
Facts
| Item | Detail |
|---|---|
| Base model | Qwen3-0.6B-Base |
| Target harness | Pi (TypeScript coding agent), OpenAI-style function-calling format |
| Training plan | LoRA SFT on Modal, then RL |
| Status | Data, harness, probes and evaluation built; no results claimed here |
What you can buy
Agent data and environment work built to your spec: trace conversion into your harness's exact format, structural gates with reject ledgers, measured context budgets, and evaluation suites that check tool use rather than final text.
FAQ
Why measure context with the real tokenizer?
Because character-based estimates are wrong in both directions for code and tool calls. A cap set from estimates can silently drop most of what you meant to train on.
Why reject traces instead of repairing them?
A trace that uses a tool your harness doesn't have teaches the model to call something that doesn't exist. Rejecting it with a logged reason keeps the training set honest, and the ledger shows what a repair would have to fix.
Has the model been trained?
This page covers the data and harness work. Training results are not claimed here.
License it, or have me build an environment in the same style.