Running someone else's benchmark well is its own skill: environments have to build, networks have to be locked down without starving the agent, and a failed sandbox must not be scored as a wrong answer. This project stood DeepSWE up for a model it wasn't set up for.
The benchmark
- 113 tasks drawn from active open-source repositories, in TypeScript, Go, Python, JavaScript and Rust.
- Harbor task format: metadata, the instruction the agent sees, a Dockerfile for the environment, a verifier (a test script plus test additions applied at grading time) and a held-out reference solution.
- Pier, a Harbor-compatible runner, adds per-agent network allowlists, so agents reach only their model API while the task stays isolated.
What I added
- Harness configs for MiniMax M3, so the same tasks can run under several agent harnesses.
- A CLI harness adapter for driving command-line agents inside Pier.
- Run scripts for a baseline run and a full run, plus a status tool for long benchmark jobs.
Facts
| Item | Detail |
|---|---|
| Benchmark | DeepSWE by Datacurve, 113 tasks |
| Runner | Pier (Harbor-compatible), sandboxes on Modal |
| Model | MiniMax M3 |
| Results | Not published here yet |
What you can buy
SWE-style environments built and run to your spec: tasks in the Harbor format or as HUD tasksets, verifiers that accept any correct behaviour, network allowlists per agent, and runs across harnesses so you can separate the model from the harness.
FAQ
What is the DeepSWE benchmark?
DeepSWE is Datacurve's benchmark of 113 long-horizon software engineering tasks drawn from active open-source repositories, in TypeScript, Go, Python, JavaScript and Rust. Each task has its own environment and a program-based verifier, and runs through Pier, a Harbor-compatible runner.
What is the Harbor task format?
A task directory with metadata, the agent-facing instruction, an environment definition, a verifier and a held-out reference solution. The verifier checks observable behaviour, so any correct solution passes.
Why run one model under several harnesses?
Because the harness changes results. Comparing harnesses on the same tasks shows how much of a score belongs to the model and how much to the tooling around it.