OpenAI Parameter Golf: what an agent-run research lab tried
Parameter Golf is OpenAI's open challenge to train the best small language model, scored by validation bits per byte, inside a hard budget of under 16,000,000 bytes and a 10-minute run on 8 H100s. I worked it with coding agents as the research staff: I set ideas and priorities and the agent proposed experiments, patched code, ran jobs on Modal and read results. These are notes on how the lab was organised and what it tried.
Parameter Golf is an open optimisation problem with hard constraints. It is not a hyperparameter sweep: architecture, optimiser, schedule, quantisation, compression-aware training and the export format are all fair game, as long as the submission stays under the byte budget and reproduces on the official track.
How the lab was organised
- I stayed at the idea layer. I set priorities and reviewed results. A coding agent acted as the senior engineer: it read code, proposed experiments, patched files, launched jobs and interpreted the numbers.
- Modal was the lab bench. Cheap proxy runs validated plumbing and ranked ideas; only promising ones were promoted to runs that matched the official 8×H100, 10-minute track.
- Operational competence before automation. An autonomous research loop was scaffolded, but deliberately not made the main path until the experiment workflow was understood.
The labs
| Lab | What it explored |
|---|---|
| Main lab | Training experiments on Modal against the official codebase |
| Tensor-intervention lab | Novel weight-level interventions on an accepted modern base (int6 MLP, SmearGate, BigramHash, orthogonal init, Muon with weight decay, SWA) |
| RL lab and MoE-RL lab | Reinforcement-learning and mixture-of-experts ideas applied to pretraining |
| Top-1 lab | Reproducing and recombining the strongest public recipes |
| Autoresearch | A provider-backed research loop, in the style of Karpathy's autoresearch |
What it taught
- Budgets change what "better" means. Under a byte limit, the export format and quantisation are as much a part of the model as its architecture.
- Proxy runs are only useful if they rank ideas the same way the real track does. That is worth checking before trusting them.
- Agents are good research staff when the human owns the questions. The useful split was ideas and review with me, and implementation, runs and first-pass analysis with the agent.
What you can buy
Research environments in the same shape: a fixed budget, a hard metric, a reproducible track, and tasks that ask an agent to improve the metric without breaking the rules. Graders come from the metric and the constraints themselves.
FAQ
What is Parameter Golf?
An open challenge from OpenAI to get the lowest validation bits per byte from a language model whose whole submission is under 16,000,000 bytes, reproducible in a 10-minute run on 8 H100 GPUs.
What is an agent-run research lab?
A setup where a person sets the research questions and reviews results, while a coding agent implements experiments, runs them on rented GPUs and reports back.
What scores did the lab reach?
I'll add final scores here once they're confirmed.
License it, or have me build an environment in the same style.